Computational algorithms have become an essential component of research, with great efforts by the scientific community to raise standards on development and distribution of code. Despite these efforts, sustainability and reproducibility are major issues since continued validation through software testing is still not a widely adopted practice. Here, we report seven recommendations that help researchers implement software testing in microbial bioinformatics. We have developed these recommendations based on our experience from a collaborative hackathon organised prior to the American Society for Microbiology Next Generation Sequencing (ASM NGS) 2020 conference. We also present a repository hosting examples and guidelines for testing, available from https://github.com/microbinfie-hackathon2020/CSIS.

This study was supported by the:
  • Fundação para a Ciência e a Tecnologia (Award 2020.08493.BD)
    • Principle Award Recipient: RafaelMamede
  • ZJLab
    • Principle Award Recipient: LuisPedro Coelho
  • Shanghai Municipal Science and Technology Major Project (Award 2018SHZDZX01)
    • Principle Award Recipient: LuisPedro Coelho
  • Fundação para a Ciência e a Tecnologia (Award SFRH/BD/129483/2017)
    • Principle Award Recipient: CInes Mendes
  • This is an open-access article distributed under the terms of the Creative Commons Attribution License.

Article metrics loading...

Loading full text...

Full text loading...



  1. Stodden V, Seiler J, Ma Z. An empirical analysis of journal policy effectiveness for computational reproducibility. Proc Natl Acad Sci U S A 2018; 115:2584–2589 [View Article] [PubMed]
    [Google Scholar]
  2. Krafczyk M, Shi A, Bhaskar A, Marinov D, Stodden V. Scientific tests and continuous integration strategies to enhance reproducibility in the scientific software context. In Proceedings of the 2nd International Workshop on Practical Reproducible Evaluation of Computer Systems Phoenix, AZ, USA: Association for Computing Machinery; 2019 pp 23–28
    [Google Scholar]
  3. Chang G, Roth CB, Reyes CL, Pornillos O, Chen Y-J et al. Retraction. Science 2006; 314:1875 [View Article] [PubMed]
    [Google Scholar]
  4. Hall BG, Salipante SJ. Retraction: Measures of clade confidence do not correlate with accuracy of phylogenetic trees. PLoS Comput Biol 2007; 3:e158 [View Article] [PubMed]
    [Google Scholar]
  5. Busby B, Lesko M. August 2015 and January 2016 Hackathon participants Federer L. Closing gaps between open software and public data in a hackathon setting: User-centered software prototyping. F1000Res 2016; 5:672 [View Article] [PubMed]
    [Google Scholar]
  6. Gruening B, Sallou O, Moreno P, da Veiga Leprevost F, Ménager H et al. Recommendations for the packaging and containerizing of bioinformatics software. F1000Res 2018; 7:742 [View Article] [PubMed]
    [Google Scholar]
  7. Trockman A, Zhou S, Kästner C, Vasilescu B. Adding sparkle to social coding: an empirical study of repository badges in the npm ecosystem. In Proceedings of the 40th International Conference on Software Engineering Gothenburg, Sweden: Association for Computing Machinery; 2018 pp 511–522
    [Google Scholar]
  8. Karimzadeh M, Hoffman MM. Top considerations for creating bioinformatics software documentation. Brief Bioinform 2018; 19:693–699 [View Article] [PubMed]
    [Google Scholar]
  9. O’Toole Á, Scher E, Underwood A, Jackson B, Hill V et al. Assignment of epidemiological lineages in an emerging pandemic using the pangolin tool. Virus Evol 2021; 7:veab064 [View Article] [PubMed]
    [Google Scholar]
  10. Silva M, Machado MP, Silva DN, Rossi M, Moran-Gilad J et al. chewBBACA: A complete suite for gene-by-gene schema creation and strain identification. Microb Genom 2018; 4: [View Article] [PubMed]
    [Google Scholar]

Data & Media loading...


Supplementary material 1


Most cited this month Most Cited RSS feed

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error