Skip to content
1887

Access Microbiology open research platform logo Access Microbiology

Volume 6, Issue 6

An evaluation of ChatGPT and Bard (Gemini) in the context of biological knowledge retrieval Open Access

Abstract

ChatGPT and Bard (now called Gemini), two conversational AI models developed by OpenAI and Google AI, respectively, have garnered considerable attention for their ability to engage in natural language conversations and perform various language-related tasks. While the versatility of these chatbots in generating text and simulating human-like conversations is undeniable, we wanted to evaluate their effectiveness in retrieving biological knowledge for curation and research purposes. To do so we asked each chatbot a series of questions and scored their answers based on their quality. Out of a maximal score of 24, ChatGPT scored 5 and Bard scored 13. The encountered issues included missing information, incorrect answers, and instances where responses combine accurate and inaccurate details. Notably, both tools tend to fabricate references to scientific papers, undermining their usability. In light of these findings, we recommend that biologists continue to rely on traditional sources while periodically assessing the reliability of ChatGPT and Bard. As ChatGPT aptly suggested, for specific and up-to-date scientific information, established scientific journals, databases, and subject-matter experts remain the preferred avenues for trustworthy data.

  • Received:
  • Accepted:
  • Published Online:
Keyword(s): biocuration and large language model
Funding
This study was supported by the:
  • SRI International
    • Principle Award Recipient: RonCaspi
© 2024 The Authors
  • This is an open-access article distributed under the terms of the Creative Commons Attribution License.
Loading

Article metrics loading...

/content/journal/acmi/10.1099/acmi.0.000790.v3
2024-06-12
2025-07-19
Loading full text...

Full text loading...

/deliver/fulltext/acmi/6/6/acmi000790.v3.html?itemId=/content/journal/acmi/10.1099/acmi.0.000790.v3&mimeType=html&fmt=ahah

References

  1. Temsah M-H, Altamimi I, Jamal A, Alhasan K, Al-Eyadhy A. ChatGPT surpasses 1000 publications on PubMed: envisioning the road ahead. Cureus 2023; 15:e44769 [View Article] [PubMed]
     [Google Scholar]
  2. Apornvirat S, Namboonlue C, Laohawetwanit T. Comparative analysis of ChatGPT and Bard in answering pathology examination questions requiring image interpretation. Am J Clin Pathol 2024aqae036 [View Article] [PubMed]
     [Google Scholar]
  3. Kirshteyn G, Golan R, Chaet M. Performance of ChatGPT vs. HuggingChat on OB-GYN topics. Cureus 2024; 16:e56187 [View Article] [PubMed]
     [Google Scholar]
  4. Gilson A, Safranek CW, Huang T, Socrates V, Chi L et al. How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? The implications of large language models for medical education and knowledge assessment. JMIR Med Educ 2023; 9:e45312 [View Article] [PubMed]
     [Google Scholar]
  5. Tiwari K, Matthews L, May B, Shamovsky V, Orlic-Milacic M et al. ChatGPT usage in the reactome curation process. bioRxiv 20232023.11.08.566195 [View Article] [PubMed]
     [Google Scholar]
  6. Wong C. These scientists aren’t using ChatGPT - here’s why. Nature 2023 [View Article] [PubMed]
     [Google Scholar]
  7. Rawashdeh B, Kim J, AlRyalat SA, Prasad R, Cooper M. ChatGPT and artificial intelligence in transplantation research: is it always correct?. Cureus 2023; 15:e42150 [View Article] [PubMed]
     [Google Scholar]
  8. Naddaf M. ChatGPT generates fake data set to support scientific hypothesis. Nature 2023; 623:895–896 [View Article]
     [Google Scholar]
  9. Lechien JR, Briganti G, Vaira LA. Accuracy of ChatGPT-3.5 and -4 in providing scientific references in otolaryngology–head and neck surgery. Eur Arch Otorhinolaryngol 2024; 281:2159–2165 [View Article]
     [Google Scholar]
  10. Kabri O. Google executive: ‘Bard is not intended to provide factual answers, just a companion for creative tasks. ctech 2023 https://www.calcalistech.com/ctechnews/article/b1ensqkih accessed 15 April 2024
     [Google Scholar]
  11. Caspi R, Billington R, Keseler IM, Kothari A, Krummenacker M et al. The MetaCyc database of metabolic pathways and enzymes - a 2019 update. Nucleic Acids Res 2020; 48:D445–D453 [View Article] [PubMed]
     [Google Scholar]
  12. Taguchi G, Ubukata T, Nozue H, Kobayashi Y, Takahi M et al. Malonylation is a key reaction in the metabolism of xenobiotic phenolic glucosides in Arabidopsis and tobacco. Plant J 2010; 63:1031–1041 [View Article] [PubMed]
     [Google Scholar]
  13. Setlow B, Cabrera-Martinez R-M, Setlow P. Mechanism of the hydrolysis of 4-methylumbelliferyl-beta-D-glucoside by germinating and outgrowing spores of Bacillus species. J Appl Microbiol 2004; 96:1245–1255 [View Article] [PubMed]
     [Google Scholar]
  14. Heyworth R, Dahlqvist A. Pig intestinal beta-glucosidase activities. II. Evidence for the hydrolysis of 4-methylumbelliferyl beta-D-glucoside and beta-D-galactoside at the same enzyme site. Biochim Biophys Acta 1962; 64:182–184 [View Article] [PubMed]
     [Google Scholar]
  15. Collins MD, Jones D. Distribution of isoprenoid quinone structural types in bacteria and their taxonomic implication. Microbiol Rev 1981; 45:316–354 [View Article] [PubMed]
     [Google Scholar]
  16. Zuk D, Belk JP, Jacobson A. Temperature-sensitive mutations in the Saccharomyces cerevisiae MRT4, GRC5, SLA2 and THS1 genes result in defects in mRNA turnover. Genetics 1999; 153:35–47 [View Article] [PubMed]
     [Google Scholar]
  17. Harnpicharnchai P, Jakovljevic J, Horsey E, Miles T, Roman J et al. Composition and functional characterization of yeast 66S ribosome assembly intermediates. Mol Cell 2001; 8:505–515 [View Article] [PubMed]
     [Google Scholar]
  18. Rodríguez-Mateos M, Abia D, García-Gómez JJ, Morreale A, de la Cruz J et al. The amino terminal domain from Mrt4 protein can functionally replace the RNA binding domain of the ribosomal P0 protein. Nucleic Acids Res 2009; 37:3514–3521 [View Article] [PubMed]
     [Google Scholar]
  19. Lo K-Y, Li Z, Wang F, Marcotte EM, Johnson AW. Ribosome stalk assembly requires the dual-specificity phosphatase Yvh1 for the exchange of Mrt4 with P0. J Cell Biol 2009; 186:849–862 [View Article] [PubMed]
     [Google Scholar]
  20. Woolford JL, Baserga SJ. Ribosome biogenesis in the yeast Saccharomyces cerevisiae. Genetics 2013; 195:643–681 [View Article] [PubMed]
     [Google Scholar]
  21. Konikkat S, Woolford JL. Principles of 60S ribosomal subunit assembly emerging from recent studies in yeast. Biochem J 2017; 474:195–214 [View Article] [PubMed]
     [Google Scholar]
  22. Zheng L, White RH, Cash VL, Jack RF, Dean DR. Cysteine desulfurase activity indicates a role for NIFS in metallocluster biosynthesis. Proc Natl Acad Sci U S A 1993; 90:2754–2758 [View Article] [PubMed]
     [Google Scholar]
  23. Patzer SI, Hantke K. SufS is a NifS-like protein, and SufD is necessary for stability of the [2Fe-2S] FhuF protein in Escherichia coli. J Bacteriol 1999; 181:3307–3309 [View Article] [PubMed]
     [Google Scholar]
  24. Schirmer T, Jenal U. Structural and mechanistic determinants of c-di-GMP signalling. Nat Rev Microbiol 2009; 7:724–735 [View Article] [PubMed]
     [Google Scholar]
  25. Olsén A, Jonsson A, Normark S. Fibronectin binding mediated by a novel class of surface organelles on Escherichia coli. Nature 1989; 338:652–655 [View Article] [PubMed]
     [Google Scholar]
  26. Onizuka T, Endo S, Akiyama H, Kanai S, Hirano M et al. The rbcX gene product promotes the production and assembly of ribulose-1,5-bisphosphate carboxylase/oxygenase of Synechococcus sp. PCC7002 in Escherichia coli. Plant Cell Physiol 2004; 45:1390–1395 [View Article] [PubMed]
     [Google Scholar]
  27. Hillier M. Why does ChatGPT generate fake references?; 2023 https://teche.mq.edu.au/2023/02/why-does-chatgpt-generate-fake-references/ accessed 14 November 2023
  28. Welborn A. ChatGPT and fake citations. Duke University Libraries Blogs; 2023 https://blogs.library.duke.edu/blog/2023/03/09/chatgpt-and-fake-citations/ accessed 14 November 2023
  29. Levins H, Barnhart M. Beware of ChatGPT’s Scientific Journal Citations. Penn LDI 2023 https://ldi.upenn.edu/our-work/research-updates/beware-of-chatgpts-scientific-journal-citations/ accessed 14 November 2023
     [Google Scholar]
  30. Taylor R, Kardas M, Cucurull G, Scialom T, Hartshorn A et al. Galactica: a large language model for science. arXiv preprint 2022 [View Article]
     [Google Scholar]
/content/journal/acmi/10.1099/acmi.0.000790.v3
Loading
An evaluation of ChatGPT and Bard (Gemini) in the context of biological knowledge retrieval
Access Microbiology 6, 000790.v3 (2024); https://doi.org/10.1099/acmi.0.000790.v3
/content/journal/acmi/10.1099/acmi.0.000790.v3
/content/journal/acmi/10.1099/acmi.0.000790.v3
Loading

Data & Media loading...

Most cited Most Cited RSS feed

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error