...
Link to the spreadsheet with questions: https://docs.google.com/spreadsheets/d/16arTCfdguNGdl916ZdbRkaS7SxDa9x3I/edit?usp=sharing&ouid=111803761008578493760&rtpof=true&sd=true
Link to the document with “ground truth” examples of business questions with answers, derived from the Open Targets database, developed courtesy of the ZS Consulting: https://docs.google.com/document/d/1_WvkgveIxUYb8rS_5wBq4VLL48CTNphq/edit?usp=drive_link&ouid=111803761008578493760&rtpof=true&sd=true We can use this small collection of questions for testing of the future target discovery LLM system.
2024.02.12 sub-team call:
Recording: https://pistoiaalliance-org.zoom.us/rec/share/HMVULVgYhhm0afZyU1LJEcm7XmOPwYahu9-xCmeIgCemdwQ7DwRkU6WoubuX4yZd.BFvjGBf6-0qvu1a2?startTime=1707750476000
Passcode: =8T7nf=eThis sub-team is done.
Questions are defined. Some can be readily answered with the Open Targets KG. Good enough questions are ok. Not everything should be answerable right away.
Let us now take one or more of easy to address (“green”) questions and feed it to KG. Get POC for technical and procedural feasibility.
No need to invest in harder questions now, no need to expand the Open Targets KG
Success criterion #1: compare RAG answer to the opinion from a human scientist who is an expert on the topic of the question
Success criterion #2: compare RAG answer (with a plain language question asked of LLM) to the KG-derived answer produced by an expert data scientist
Strategy: Ask at the next large team call: validate an approach to encase Open Targets KG questions as multiple modules with tuned prompts for a RAG system, one for each of the questions. But say we are successful in the POC, - then what? Need a better vision of success, exciting, complex.
Creating many open-source APIs to public data sources is not exciting. Perhaps define a standard for such APIs?
Need for the API standard is one lesson learned
This is valuable for KG software vendors: ease of “wiring in” additional data, including proprietary data sources
What is the volatility of the data sets that we want to eventually use? For rapidly developing data sources continuous updates may be needed to accommodate ongoing changes in data source structure. So need a data standard for rapidly evolving data sources, with data increments (easier use case) or completely new data dimensions and variables (harder use case). Can a data source present itself to a RAG query system to automate data updates?
Possible new risk: will LLMs be confused by the similar data types from multitude of sources?
Provider / vendor / expert ecosystem needed - not just Open AI ChatGPT with Medline in it
Notes by Lee:
Sorry, I wrote this summary during the meeting and never sent it out, in case its still helpful Vladimir:
The core principle we were working off is that the initial aim of the Pistoia project is to create a suite of “modules” that can be plugged into LLM Chatbots to answer specific questions from public APIs/Knowledge graphs. The benefits of this are (i) modules readily available for all companies to simply plug into their LLMs and (ii) emerging standards/approaches/learning around doing this
!! We should validate this is indeed the case with the steering group!
The subteam was tasked with generating Target ID/Validation questions that could be mapped to data/queries solvable by the Open Targets knowledge graph
So we specifically limited to fairly generic high level questions, not therapy area specific and not detailed AND limited to the kinds of data available in Open Targets
The Hyve have mapped which questions can be answered by the OTKG and how easily that would be
Our recommendation is to just pick one or two that already have all data needed and demonstrate generally the approach works [i.e. create LLM modules]
“Validation” should be that “a bench scientist, asking a natural language question through a Chat LLM should obtain the same data as a data-scientist querying the knowledge graph correctly”
If this works (which it should) we should then look at next steps to focus on (and resourcing) of either/and/or
Building more modules from existing questions
Expanding questions beyond the generic to something deeper
Generating more/new data to support 1&2
Of course, if this is not what the potential funders think is worthwhile, we should stop until we establish what is the next step