Page Comparison

Persona

Challenges

GMLPs that address these challenges

Author Responsible for Solution(s)

Data Scientist

DS1: How do I pick a suitable data modeling technique?

1. Make recommendations for methods suitable for specific problem classes; or for auto-ML systems

2. DS should learn the application domain and/or work with domain experts

DS2: How do I store and version a model?

1. Recommend model versioning systems

2. Refer to best practices in ML Model Registration, Deployment, Monitoring

3. Best practices and regulatory standards in requirements management and model provenance

Christophe Chabbert

DS3: How do I tune the hyperparameter values?

DS4: How do I validate a model?

1. Refer to the DOME recommendations

2. Quickly fail models that are bad. A deceptive model is worse than a bad one

Fotis Psomopoulos link to doc

DS5: How to assure scalability of AI systems?

Elina Koletou link to doc

DS6: How should I acquire and manage data?

1. Recommend best practices from the FAIR Toolkit

2. Evaluate data for “fit for purpose”, in particular, for the metadata quality

3. Refer to best practices in Exploration, Cleaning, Normalizing, Feature Engineering, Scaling 

Natalja Kurbatova link to doc , Christophe Chabbert, Berenice Wulbrecht

DS7: Where do I store predictions?

Christophe Chabbert  link to doc

DS8: How do I publish AI models?

Publish model code, and testing and training data, sufficient for reproduction of research work, along with model results. Make recommendations for an appropriate minimal information model

Fotis Psomopoulos

DS9: How to protect IP?

DS10: How should the applicability domain of the model be expressed?

Model User

MS1: How should I manage the data? Includes data protection, versioning, labeling

1. Recommend best practices from the FAIR Toolkit

2. Evaluate data for “fit for purpose”, in particular, for the metadata quality

3. Refer to best practices in Exploration, Cleaning, Normalizing, Feature Engineering, Scaling

Natalja Kurbatova, Christophe Chabbert, Berenice Wulbrecht   link to doc

MS2: How do I avoid bias?

MS3: How do I make sure the model produces sensible answers?

1. Set-up a “human-in-the-loop” system. Recommend tools for this, if they exist

2. Set-up business feedback mechanism for flagging model results that do not align with expectations

Chas Nelson MS3 Chas Nelson 21Jul2022

Architect

A1: How to assure scalability of AI systems? 

Christophe Chabbert link to doc

A2: How should I manage the data?

See above for Model User

Christophe Chabbert, Natalja Kurbatova link to doc

A3: Continuous Integration/Delivery

1. Refer to best practices in DevOps

2. Automated model packaging for ease of production delivery

Elina Koletou link to doc

A4: How to assure performance (execution speed)?

Elina Koletou link to doc

Executive

E1: How do I learn about the costs and benefits of AI/ML technologies and the limits of possible?

Make recommendations for conferences, training materials, education products, review papers, and books. These must be updated on a frequent cadence

E2: How do I value and justify investments in AI?

1. Make recommendations for technology valuation methodologies

2. Provide business questions, goals, KPIs

3. Continuously review model performance against the business needs (goals, KPIs)

E3: How do I enable technology adoption?

1. Bring expertise together to assess compatibility of new technology with business architecture

2. Ensure sufficient knowledge transfer so users are comfortable with adoption and use the new approach

3. Have a sustainability plan so resources are available when needed