...
A testing environment to systematically evaluate and improve LLM ability to generate Cypher queries requires vendor support, and hiring a vendor requires an RFP. Vladimir informed the team about the upcoming RFP and the process for it
We will need to create "correct" answers to the scientific competency questions that we plan to use in the POC. ZS colleagues volunteered to perform this service. Details will be decided next week in a call between Vladimir and Bruce Press. This is FYI only, no action needed.
Participants observed that not all questions contained in our scientific competency question list can be answered based only on the information in Open Targets. For example, any questions that refer to clinical trials may not have complete answers based on the Open Targets contents alone. These issues will not have an effect on the POC project, however. In the future we may have to ask the project funders whether they would like to invest in data improvements or not.
We will have to brainstorm techniques for improvement of LLM performance in writing Cypher queries. (Or any structured query language).
A registry for proposed techniques that would contain high-level or pseudocode algorithmic descriptions of them: https://docs.google.com/document/d/18vPi23prPnrBOX3xAQInilqC6C4ssy2neOl05DDSH1Y/edit?usp=sharing
Peter Dorr will share shared DOIs to papers that describe other techniques in this field - already captured in the brainstorming document
Peter Dorr agreed to organize a talk by his organization in one of our main team (Wednesday) meetings, where the methods developed by his company can be shared. Exact date TBD