Good Machine Learning Practices (GMLP)

Persona

Challenges

GMLPs that address these challenges

Author(s)

Data Scientist

DS1: How do I pick a suitable data modeling technique?

1. Make recommendations for methods suitable for specific problem classes; or for auto-ML systems

2. DS should learn the application domain and/or work with domain experts

DS2: How do I store and version a model?

1. Recommend model versioning systems

2. Refer to best practices in ML Model Registration, Deployment, Monitoring

3. Best practices and regulatory standards in requirements management and model provenance

Christophe Chabbert

DS3: How do I tune the hyperparameter values?

1. Hyperparameters in AI/ML models are used to drive the learning process, and as such, cannot be inferred from the training data. Examples include topology and size of the neural network, or the learning rate and batch size.

2. The number of possible combinations of hyperparameter value choices can be large. Grid search may not work. Use Bayesian, evolutionary, or early stopping algorithms.

3. Evaluate the model by using the n-fold cross-validation, aiming to preserve the statistical characteristics such as class distribution or value ranges in both training and test partitions.

4. Evaluate the robustness of the model with the standard deviation or the confidence interval of the performance metric calculated for multiple values of the test data. For a quality model, the standard deviation of the performance metric should be low. Note the ranges of input values for which the prediction quality is high or low.

5. It is highly recommended to assess the generalization performance of an AI/ML model on a data set completely independent from the data sets used in hyperparameter tuning.

DS4: How do I validate a model?

1. Refer to the DOME recommendations

2. Quickly fail models that are bad. A deceptive model is worse than a bad one

Fotis Psomopoulos

DS5: How to assure scalability of AI systems?

Elina Koletou

DS6: How should I acquire and manage data?

1. Recommend best practices from the FAIR Toolkit

2. Evaluate data for “fit for purpose”, in particular, for the metadata quality

3. Refer to best practices in Exploration, Cleaning, Normalizing, Feature Engineering, Scaling 

Natalja Kurbatova, Christophe Chabbert, Berenice Wulbrecht

DS7: Where do I store predictions?

Christophe Chabbert

DS8: How do I publish AI models?

Publish model code, and testing and training data, sufficient for reproduction of research work, along with model results. Make recommendations for an appropriate minimal information model

Fotis Psomopoulos

DS9: How to protect IP?

DS10: How should the applicability domain of the model be expressed?

Model User

MS1: How should I manage the data? Includes data protection, versioning, labeling

1. Recommend best practices from the FAIR Toolkit

2. Evaluate data for “fit for purpose”, in particular, for the metadata quality

3. Refer to best practices in Exploration, Cleaning, Normalizing, Feature Engineering, Scaling

Natalja Kurbatova, Christophe Chabbert, Berenice Wulbrecht

MS2: How do I avoid bias?

MS3: How do I make sure the model produces sensible answers?

1. Set-up a “human-in-the-loop” system. Recommend tools for this, if they exist

2. Set-up business feedback mechanism for flagging model results that do not align with expectations

Chas Nelson

Architect

A1: How to assure scalability of AI systems? 

Christophe Chabbert

A2: How should I manage the data?

See above for Model User

Christophe Chabbert

A3: Continuous Integration/Delivery

1. Refer to best practices in DevOps

2. Automated model packaging for ease of production delivery

Elina Koletou

A4: How to assure performance (execution speed)?

Elina Koletou

Executive

E1: How do I learn about the costs and benefits of AI/ML technologies and the limits of possible?

Make recommendations for conferences, training materials, education products, review papers, and books. These must be updated on a frequent cadence

E2: How do I value and justify investments in AI?

1. Make recommendations for technology valuation methodologies

2. Provide business questions, goals, KPIs

3. Continuously review model performance against the business needs (goals, KPIs)

E3: How do I enable technology adoption?

1. Bring expertise together to assess compatibility of new technology with business architecture

2. Ensure sufficient knowledge transfer so users are comfortable with adoption and use the new approach

3. Have a sustainability plan so resources are available when needed