Points Made by the Pistoia Alliance Panel in the Discussion on April 5th 2023

Owned by Vladimir Makarov
Last updated: Apr 05, 2023

  • ML/AI models require continuous management in production, which adds to the total cost of ownership of such models and affects ROI.

  • “Experimentalis”: it is important to keep an experimental mindset and to allow oneself to fail quickly in less feasible projects. On the other hand, keeping the ML/AI models in the experimental state forever is a bad practice. Prashant referred to it by the word “experimentalis”, and made an allusion to a disease. Models stuck in the experimental state and not productized never achieve the ROI expected of the new technology.

  • One must involve SMEs in the entire lifecycle of the AI/ML models, from development, to maintenance and eventual retirement, and also in the executive presentation of AI/ML models, for the maximum value.

  • There is substantial value in being able to recognize appropriate workplace roles and personas, as they relate to AI/ML, and that are described in the Pistoia Alliance business analysis.

  • On the topic of regulated / trustworthy / explainable AI: we should define areas of knowledge where the “black box” approach is ok, and where it should not be allowed.

  • Having the ML results you have to be able to trace back to the version of the model and to the dataset version that have been used to generate these results.

  • There is need for high-quality production environments for AI/ML, that include metadata maintenance, feature stores, model versioning, etc. Setting up a proper production environment for AI is not a trivial task, as is evident from the multiplicity of MLOps guides. This means that there may be commercial potential for the software.

Q&A during the talk

Q1A: How should a team be formed and roles assigned, to address the challenges discussed here (experimentation, production, usability, feedback)?

Q1B: What is the relationship between data governance groups and AI/ML project teams?

A1 (Prashant): we touch upon roles and personas in our GMLP effort here at Pistoia. The teams set up for such efforts range across a few different models - COE, hub and spoke, fully integrated, centralized, etc. In my experience, even within a single organization, there can be varied team structures based on varying maturities, team needs, investments, and outcomes

A2 (Bikalpa):
Technical folks (AI,ML, NLP, SWE)
Business Managers (Product Owners, Product Managers, UX)
Domain Experts (fields relevant)