Versions Compared

Old Version 21

changes.mady.by.user Vladimir Makarov

Saved on

compared with

New Version 22

changes.mady.by.user Vladimir Makarov

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Description

Mitigation

Responsible Party

Failure to identify business questions, or picking too many or too few

Draft appropriate business questions - DONE; but not all business questions can be answered with specific technologies, so must take this factor into account

Lee Harland, John Wise, Bruce Press, Peter Revill

Validate that Open Targets either has a ready to use Knowledge Graph implementation, or can be converted into a KG with reasonable cost

The Hyve
Jordan Ramsdell
Robert Gill
Brian Evarts

Open Targets/EBI:

  • Sebastian Lobentanzer

  • Ellen McDonagh

Failure to identify a suitable LLM

Technology research, feature and cost analysis, and selection

Jon Stevens, Etzard Stolte, Helena Deus; Brian Evarts; Wouter Franke, Matthijs van der Zee

Does Open Targets use an ontology?

Perhaps The Hyve team has a ready answer

Failure to download a large volume of data (all of the PubMed as a maximum) for the prompt-tuning of the LLM

TBD

Failure to perform local KG comparison with calculation of a score

  1. Technology research

  2. If no ready-to-use technology exists, estimate bespoke development

  3. If estimates indicate infeasibility, this may become a gap

Failure to generate a proper query for a KG database system by an LLM

Technology research.

  • See refs 7, 8, 13, 14 below

  • BioCypher by EBI may have this capability already - needs evaluation

The Hyve
Jordan Ramsdell
Robert Gill
Brian Evarts

Open Targets/EBI:

Sebastian Lobentanzer

Ellen McDonagh

Failure to build a prototypical target discovery pipeline on the limited budget in case of mounting technical difficulties

Schedule the project in phases. Aim to answer known unknowns and to establish risk mitigation strategies early in this phase (“project elaboration”)

Some proprietary LLMs may be censored, thus introducing uncontrollable bias in the answers that they produce

  • DONE: Strong preference for an open-source, uncensored LLMs

Identified and resolved in the LLM sub-team

...

  1. https://www.sciencedirect.com/science/article/pii/S1359644613001542

  2. https://www.nature.com/articles/s41573-020-0087-3

  3. https://www.epam.com/about/newsroom/press-releases/2023/epam-launches-dial-a-unified-generative-ai-orchestration-platform

  4. https://epam-rail.com/open-source

  5. Open LLM Leaderboard: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard

  6. Chatbot Arena: https://chat.lmsys.org/?arena

  7. Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning

    https://arxiv.org/abs/2310.01061

  8. Knowledge-Consistent Dialogue Generation with Language Models and Knowledge Graphs

    https://openreview.net/forum?id=WhWlYzUTJfP&source=post_page-----97a4cf96eb69--------------------------------

  9. BioChatter Benchmark Results: https://biochatter.org/benchmark-results/#biochatter-query-generation

  10. MBET Benchmark (embeddings) https://huggingface.co/spaces/mteb/leaderboard

  11. Lora-Land and Lorax: https://predibase.com/lora-land

  12. A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databases. Summary: queries over a KG with GPT 4 are much more accurate than queries over a SQL database with GPT 4. https://arxiv.org/abs/2311.07509

  13. https://towardsdatascience.com/evaluating-llms-in-cypher-statement-generation-c570884089b3

  14. https://medium.com/neo4j/enhancing-the-accuracy-of-rag-applications-with-knowledge-graphs-ad5e2ffab663

  15. linkedlifedata.com