...
We will initially test GPT4 and Mistral (picks by the LLM selection team)
How is Cypher query generation testing done by the BioCypher team?
Results: https://biochatter.org/benchmark/
Process: https://biochatter.org/benchmarking/ and https://biochatter.org/benchmark-developer/
The focus on the repeat_instruction contents?
For the RFP, think on how to implement Abbvie/Jon’s method in this environment
Make sure that the English question, generated query, and the result are displayed for human evaluation
Propose a set of English questions that we will use (limited by the current OT contents in BioCypher) - Confirm with The Hyve that the questions in our list currently flagged as feasible are indeed such
Do we need to write “ideal” Cypher queries for these questions? - yes, make sure we understand the questions asked -this is a line item in the RFP
VM Fwd questions in column M to experts
Comment from Etzard: in his system elastic search is used across documents to by-pass the failing SPARQL queries generated by LLM, is this conceptually similar to the method proposed by Abbvie? VM shared the recording of the Jon’s talk; ask for the slides.
In general, it is good for us to collect these and similar hacks that force LLMs to produce better queries
Action item for the team (all members): please think about any additional requirements that we need to include into the RFP
Note, if payments (beyond minimum cloud expenses reimbursement) are needed we must do a competitive RFP/RFQ
...
A testing environment to systematically evaluate and improve LLM ability to generate Cypher queries requires vendor support, and hiring a vendor requires an RFP. Vladimir informed the team about the upcoming RFP and the process for it
We will need to create "correct" answers to the scientific competency questions that we plan to use in the POC. ZS colleagues volunteered to perform this service. Details will be decided next week in a call between Vladimir and Bruce Press. This is FYI only, no action needed.
Participants observed that not all questions contained in our scientific competency question list can be answered based only on the information in Open Targets. For example, any questions that refer to clinical trials may not have complete answers based on the Open Targets contents alone. These issues will not have an effect on the POC project, however. In the future we may have to ask the project funders whether they would like to invest in data improvements or not.
We will have to brainstorm techniques for improvement of LLM performance in writing Cypher queries. (Or any structured query language).
A registry for proposed techniques that would contain high-level or pseudocode algorithmic descriptions of them: https://docs.google.com/document/d/18vPi23prPnrBOX3xAQInilqC6C4ssy2neOl05DDSH1Y/edit?usp=sharing
Peter Dorr shared DOIs to papers that describe other techniques in this field - already captured in the brainstorming document
Peter Dorr agreed to organize a talk by his organization in one of our main team (Wednesday) meetings, where the methods developed by his company can be shared. Exact date TBD
2024.05.10 Sub-team meeting
...
2024.05.24 Sub-team meeting
Passcode: ZA!z3S5t
RFP was announced two days ago, are there any questions?
Q&A captured in recording
Clarified work scope (asked by Etzard)
Clarified the performance time period: initially, 3 months, but with an option to extend (asked by Etzard)
Clarified position on the expected cost proposal: it does exist, but is not published (asked by Rob)
We agreed to focus on making edits and additions to the list of methods in https://docs.google.com/document/d/18vPi23prPnrBOX3xAQInilqC6C4ssy2neOl05DDSH1Y/edit?usp=sharing - updates:
Research: Brian Evarts: see https://github.com/neo4j-labs/text2cypher as an additional possible method
Also, langsmith library for Langchain - Brian Evarts to look up more details
If we could add iterative learning to the process this may also help
NB: Similar ideas are explored by Dataiku: https://content.dataiku.com/dataiku-answers-chatbots
Next meeting will be dedicated to RFP Q&A. We may cancel it in case there are no questions that merit live discussion.
2024.06.21 Sub-team meeting
Agenda:
Last chance for any Q&A about the RFP
Look at the specific business questions that we will use for testing, and the “true” answers for them: https://docs.google.com/document/d/1_WvkgveIxUYb8rS_5wBq4VLL48CTNphq/edit?usp=drive_link&ouid=111803761008578493760&rtpof=true&sd=true (this work was done by the ZS team - many thanks!)
Recording:
https://pistoiaalliance-org.zoom.us/rec/share/j1s8mE9Ytq4aEkMHoYroEKfcx0ThzeArtvbkcmMrB9yqCoJBHMltBgP-hsjLpgWz.-pEWfzVBloiYxqbA?startTime=1718978573000
Passcode: P0@LM7E!