Team: The Hyve; Jordan Ramsdell, Robert Gill, Brian Evans + Open Targets/EBI team: Sebastian Lobentanzer, Ellen McDonagh
Link to the working doc: https://docs.google.com/document/d/16ZQkkdoadf0jXg9gb-qCh7eiCajOh9smmh86KnBHeT0/edit?usp=sharing
2024.04.12 Sub-team meeting
Recording: https://pistoiaalliance-org.zoom.us/rec/share/HAVWHW7_RTJW8jWk-acRxqvp3Z0sLNgrk7XUNazYz2I1sCYY78VYBeL_5LfxUmng.2bNwqJbp8nChbndC?startTime=1712930915000 Passcode: K2UD1@Fu
The Hyve team presented a working instance of BioCypher / BioChatter and ran a few queries live. This prototype can only handle queries on a few topics (not a full OT KG inside) and displays answers in tabular or RDF form, not as plain text.
The issue of unreliable Cypher query generation is seen as the biggest problem in the project
VM to inform the sponsors of the above and confirm with them [DONE: sponsors agree]
2024.04.19 Sub-team meeting
Agenda:
Main objective: need to set-up a testing environment to systematically evaluate and improve LLM ability to generate Cypher queries
LLM team members, please remind me of what open-source LLM we picked (in addition to GPT4)
VM will look up in recording
How is Cypher query generation testing done by the BioCypher team?
Results: https://biochatter.org/benchmark/
Process: https://biochatter.org/benchmarking/ and https://biochatter.org/benchmark-developer/
The focus on the repeat_instruction contents?
For the RFP, think on how to implement Abbvie/Jon’s method in this environment
Make sure that the English question, generated query, and the result are displayed for human evaluation
Propose a set of English questions that we will use (limited by the current OT contents in BioCypher) - Confirm with The Hyve that the questions in our list currently flagged as feasible are indeed such
Do we need to write “ideal” Cypher queries for these questions? - yes, make sure we understand the questions asked -this is a line item in the RFP
VM Fwd questions in column M to experts
Comment from Etzard: in his system elastic search is used across documents to by-pass the failing SPARQL queries generated by LLM, is this conceptually similar to the method proposed by Abbvie? VM shared the recording of the Jon’s talk; ask for the slides
In general, it is good for us to collect these and similar hacks that force LLMs to produce better queries
Action item for the team (all members): please think about any additional requirements that we need to include into the RFP
Note, if payments (beyond minimum cloud expenses reimbursement) are needed we must do a competitive RFP/RFQ