Vladimir A. Makarov, Terry Stouch, Brandon Allgood, Chris D. Willis, Nick Lynch, "Best practices for artificial intelligence in life sciences research", Drug Discovery Today, Volume 26, Issue 5, 2021, Pages 1107-1110, ISSN 1359-6446, https://doi.org/10.1016/j.drudis.2021.01.017. Free pre-print available at https://osf.io/eqm9j/
Abstract: We describe 11 best practices for the successful use of artificial intelligence and machine learning in pharmaceutical and biotechnology research at the data, technology and organizational management levels.
Summary of Our Paper in Drug Discovery Today ("The 11 Commandments")
Walsh, I., Fishman, D., Garcia-Gasulla, D. et al. "DOME: recommendations for supervised machine learning validation in biology", Nature Methods 18, 1122–1127 (2021). https://doi.org/10.1038/s41592-021-01205-4
Abstract: DOME is a set of community-wide recommendations for reporting supervised machine learning–based analyses applied to biological studies. Broad adoption of these recommendations will help improve machine learning assessment and reproducibility.
How do the Pistoia Alliance recommendations for Good Machine Learning Practices align with those by the leading regulatory authorities?
There is a very considerable overlap between our recommendations (conceived in 2019 and first made public in the April of 2020 as a preprint) and the recommendations by the US FDA and the UK MHRA (published in October 2021). Three out of 10 FDA recommendations follow ours exactly, and six more overlap in content with our recommendations. It is not surprising because both sets of recommendations are based on industry experience and common sense. The remaining difference is due to the difference in the intended audiences: we addressed the ML practitioners whereas the FDA wrote its recommendations for the medical community.
Compare to the US FDA and UK MHRA Good Machine Learning Practice for Medical Device Development: Guiding Principles:
Our business analysis is based on enumeration of business personas and the challenges that these personas may face in their work with AI/ML technologies. The proposed Good Machine Learning Practices (GMLPs) answer these challenges.
Persona | Challenges | GMLPs that address these challenges | Author(s) | Reviewer(s) |
Data Scientist | DS1: How do I pick a suitable data modeling technique? | 1. Make recommendations for methods suitable for specific problem classes; or for auto-ML systems 2. DS should learn the application domain and/or work with domain experts | ||
DS2: How do I store and version a model? | 1. Recommend model versioning systems 2. Refer to best practices in ML Model Registration, Deployment, Monitoring 3. Best practices and regulatory standards in requirements management and model provenance | Christophe Chabbert | ||
| Samuel Ramirez | |||
1. Refer to the DOME recommendations 2. Quickly fail models that are bad. A deceptive model is worse than a bad one | Fotis Psomopoulos | |||
Elina Koletou | ||||
1. Recommend best practices from the FAIR Toolkit 2. Evaluate data for “fit for purpose”, in particular, for the metadata quality 3. Refer to best practices in Exploration, Cleaning, Normalizing, Feature Engineering, Scaling | Natalja Kurbatova, Christophe Chabbert, Berenice Wulbrecht | |||
Christophe Chabbert | ||||
DS8: How do I publish AI models? | Publish model code, and testing and training data, sufficient for reproduction of research work, along with model results. Make recommendations for an appropriate minimal information model | Fotis Psomopoulos | ||
DS9: How to protect IP? | ||||
DS10: How should the applicability domain of the model be expressed? | Use Applicability Domain Analysis | Samuel Ramirez | ||
Model User | MS1: How should I manage the data? Includes data protection, versioning, labeling | 1. Recommend best practices from the FAIR Toolkit 2. Evaluate data for “fit for purpose”, in particular, for the metadata quality 3. Refer to best practices in Exploration, Cleaning, Normalizing, Feature Engineering, Scaling | Natalja Kurbatova, Christophe Chabbert, Berenice Wulbrecht | |
MS2: How do I avoid bias? | ||||
MS3: How do I make sure the model produces sensible answers? | 1. Set-up a “human-in-the-loop” system. Recommend tools for this, if they exist 2. Set-up business feedback mechanism for flagging model results that do not align with expectations | Chas Nelson | ||
Architect | Christophe Chabbert | |||
See above for Model User | Christophe Chabbert | |||
1. Refer to best practices in DevOps 2. Automated model packaging for ease of production delivery | Elina Koletou | |||
Elina Koletou | ||||
Executive | E1: How do I learn about the costs and benefits of AI/ML technologies and the limits of possible? | Make recommendations for conferences, training materials, education products, review papers, and books. These must be updated on a frequent cadence | ||
E2: How do I value and justify investments in AI? | 1. Make recommendations for technology valuation methodologies 2. Provide business questions, goals, KPIs 3. Continuously review model performance against the business needs (goals, KPIs) | Prashant Natarajan | ||
E3: How do I enable technology adoption? | 1. Bring expertise together to assess compatibility of new technology with business architecture 2. Ensure sufficient knowledge transfer so users are comfortable with adoption and use the new approach 3. Have a sustainability plan so resources are available when needed | Prashant Natarajan |
Common challenges across ALL personas:
Common responsibility across multiple personas:
Glossary notes: In the context of this document these terms have these meanings:
Fotis | Psomopoulos | CERTH |
Brandon | Allgood | Valo Health |
Christophe | Chabbert | Roche |
Adrian | Schreyer | Exscientia |
Elina | Koletou | Roche |
Frederik | van der Broek | Elsevier |
David | Wöhlert | Elsevier |
John | Overington | Exscientia |
Loganathan | Kumarasamy | Zifo R&D |
Neal | Dunkinson | Scibite |
Simon | Thornber | GSK |
Irene | Pak | BMS |
Berenice | Wullbrecht | Ontoforce |
Prashant | Natarajan | H2O.ai |
Valerie | Morel | Ontoforce |
Yvonna | Li | Roche |
Silvio | Tosatto | Unipd.it |
Natalja | Kurbatova | Zifo R&D |
Mufis | Thalath | Zifo R&D |
Niels | Van Beuningen | Vivenics |
Ton | Van Daelen | BIOVIA (3ds) |
Lucille | Valentine | Gliff.ai |
Adrian | Fowkes | Lhasa |
Chas | Nelson | Gliff.ai |
Paolo | Simeone | Ontoforce |
Christoph | Berns | Bayer |
Mark | Earll | Syngenta |
Vladimir | Makarov | Pistoia Alliance |
EU Artificial Intelligence Act: https://www.consilium.europa.eu/en/press/press-releases/2022/12/06/artificial-intelligence-act-council-calls-for-promoting-safe-ai-that-respects-fundamental-rights/
US FDA SaMD Action Plan (September 2022): https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-software-medical-device; also see announcement at https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-software-medical-device or download PDF at https://www.fda.gov/media/145022/download
Related topic at the Pistoia Alliance, a FAIR Guide for Clinical Data: FAIR4Clin