Lately, there has been an increased interest towards leveraging the potential of artificial intelligence (AI) within academic writing. Out of other AI tools currently available, recent studies have observed that ChatGPT is extensively used to produce scientific abstracts [1]. ChatGPT has also been used for writing literature reviews, especially systematic reviews.

Evaluating ChatGPT’s Role in Scientific Abstract Writing: Strengths, Challenges, and Future Implications

Dr.Olivia | Research Writing and Formatting Consultant

27 Mar, 2025

Dr.Olivia | Research Writing and Formatting Consultant

27 Mar, 2025

Introduction

The purpose of this research is to investigate the effectiveness of ChatGPT in producing scientific abstract related to biomedical research. This research will also evaluate the AI outputs with human-written abstract, including readability, quality, and adherence to regular guidelines [2].

Though AI has the potential to effectively summarize papers with impressive readability, there are concerns regarding precision, adherence to scientific benchmarks, and the scope for biases [3]. Therefore, this research systematically analyses the findings while showcasing the main insights for academic professionals, authors, and researchers as well.

Performance of ChatGPT in Scientific Abstract Generation

Comparative Quality Assessment

The Consolidated Standards of Reporting Trials for Abstracts (CONSORT-A) checklist was used to assess abstracts produced using ChatGPT and was compared with manually written abstracts from high-impact journals through quality assessment [4].

Adherence to CONSORT-A checklist

Adherence with CONSORT-A checklist revealed that both 3.5 and 4.0 versions of ChatGPT underperformed in 10 and 14 out of 18 checklist items, respectively [5].

Readability and Clarity

Though ChatGPT abstracts scored low in terms of quality, abstracts generated by ChatGPT 3.5 were found to be most readable in 62.22% of instances [3].
Abstracts by ChatGPT conveyed the research findings in a simple manner, which increased accessibility to a wider audience [6].

Minimal Errors and Hallucinations

A low hallucination rate of 0.03 errors in each ChatGPT abstract was noticed, which was substantially lower than rates as recorded through other studies.
Precise extraction of objectives and conclusions was specifically effective using ChatGPT 3.5 [7].

Figure.1 Abstract generated from ChatGPT 4.0 (Source: Li, D., et al. (2023)

Factors Influencing ChatGPT’s Performance

The limitations noticed in ChatGPT while generating scientific abstracts were based on diverse factors.

Comparison with High-Impact Journal Abstracts

Abstracts used for comparing with ChatGPT abstracts were sourced from journals having a high impact factor. Such journals are notorious for a thorough process of peer review and observance of reporting standards [8].
On average, human abstracts had an OQS of 66.06%, which was much greater than scores usually recorded in wider journal collections (32.6%-54.1%) [3].

Adherence to Guidelines

Outputs from ChatGPT were not consistent in adhering to CONSORT-A benchmarks, probably owing to its training on varied datasets, where stringent adherence is not necessary [9].

Prompt Engineering Limitations

A uniform prompt was used in the research while producing an abstract to sustain consistency, instead of optimizing the design prompts for enhanced outputs
Research in the future needs to examine the impact of structured prompts on abstracts that are AI-generated [3].

Comparative Model Performance

It was also observed that version 3.5 of ChatGPT outperformed version 4 while producing abstract, realizing a high score of 15.06% with less hallucination.
This finding is in tandem with studies that hint that ChatGPT 3.5 is superior in terms of its ability to resolve challenges (reasoning, chain of thoughts) in specific tasks [10].

Begin Your Editing and formatting Journey With Us!

Strengths of ChatGPT in Academic Writing

Though there were certain restrictions in duplicating abstracts authored by humans, several advantages were also observed:

Concise Summarization

Abstracts produced using ChatGPT were structured and concise, presenting the findings in an effective manner [2].

Enhanced Readability

The potential of ChatGPT to convert intricate scientific jargon into accessible material could allow the research findings to be widely distributed [6].

Potential for Literature Reviews

Research objectives and conclusions were precisely recognized and summed up by AI, rendering it as an impactful tool in literature synthesis and systematic reviews [3].

Writing Assistance for Non-Native Authors

Further, ChatGPT could also act as a powerful tool that supports language. This could assist non-native English speakers to refine scientific abstracts while ascribing to formatting and word limits as recommended by journals [11].

Limitations and Risks of AI-Generated Abstracts

Though the application of AI is promising, the research underlined many crucial risks related to AI-generated abstracts:

Lack of Detail and Scientific Rigor

Though ChatGPT-generated abstracts are readable, they frequently lack crucial methodological aspects, which affect their adherence to reporting guidelines [12].

Potential Bias in AI Training Data

AI models are trained on datasets primarily extracted from well-funded institutions and English language materials, which could introduce biases in content generation [13].

Ethical Considerations in AI-Assisted Writing

The spurt in utilising AI within academic circles also raises ethical concerns pertaining to originality, attribution, and the need to disclose the use of AI in research manuscripts [14].

Connect with us to explore how we can support you in maintaining academic integrity and enhancing the visibility of your research across the world!

We offer the expertise, knowledge, and comprehensive support your Clinical research and publication needs.

Implications for Future Research and Academic Writing

In view of such findings, it is recommended that researchers and academic professionals:

Best Practices for Using AI in Scientific Writing

Limit to drafting: The use of ChatGPT should be restricted to drafting. However, it must not be used as a substitute for human oversight in scientific writing.

Manual verification of AI outputs: Content produced using ChatGPT must be critically assessed for accuracy, validity of references, and guideline adherence
Transparency and ethical disclosure: Journals and academic institutions should mandatorily ask authors to disclose the use of AI while developing manuscripts [15].

Future Research Directions

Evaluate AI’s performance with optimized prompts: Studies need to be carried out to examine whether ChatGPT’s accuracy and adherence to standards can be augmented with refining prompts.
Assess AI’s utility in systematic reviews: Considering its expertise in summing up conclusions and objectives, additional examination of AI should be conducted to help with systematic reviews and meta-analysis [16].
Developing AI detection and verification tools: Developing AI detection tools by the research community will help differentiate AI and human content, maintaining integrity in academics [13].

Figure. 2 Quality, Compliance, and Readability of AI vs. Human-Written Abstracts (Source: (Hwang, Taesoon, et al, 2024)).

Conclusion:

As a supportive tool, ChatGPT holds immense promise within scientific writing. This is particularly true when it comes to summing up abstracts, improving readability, and synthesizing literature reviews. Nonetheless, the adherence to reporting benchmarks, accuracy, and scientific rigor warrant human supervision.

Though certain facets of research writing can be streamlined with AI, it can never be a substitute for the critical thinking, expertise, or ethical responsibility needed in academics.

In the future, researchers need to embrace an approach that maintains equilibrium in approaches, harnessing the power of AI but at the same time ensuring rigor in validating content produced by AI for maintaining integrity in scientific research [15].

At Pubrica, our experts assist you in crafting simplified, clear, and effective scientific abstracts. We ensure that your abstract is written with highest standards of accuracy and integrity.