Phase

Milestones

Deliverables

Est Date

Initiation

Project charter

1. A list of candidate pre-competitive projects

2. One or more projects selected by vote

3. Project charter is drafted for the winning idea

4. Raise minimal funds for the Elaboration phase

12/11/23

(Complete)

Elaboration

1. Development plan

2. Cost estimates

1. Risks analysis – see Risk Registry below

2. Technology analysis to address the identified risks

3. Work Breakdown Structure (WBS)

4. Cost estimates

5. Time estimates

6. Gantt Chart for Construction with additional iterations as needed and a work schedule

7. Make feasibility decisions before committing to build

Q1-Q3 2024

(Complete)

Construction

(POC)

1. Learn best ways to use LLMs for NL queries

2. Lessons learned published

1. Results of experiments

2. Lessons learned recorded and published

Q4 2024

(Almost complete in December 2024)

Construction (PROD)

1. Target discovery pipeline

1. Target discovery pipeline – detailed deliverables are not yet known

2. Lessons learned recorded and published

1. to be specified by research project sponsors

TBD

Transition

Sustainability achieved

1. Place the prototype into maintenance mode or outsource for continuous development by another organization (e.g. non-profit)

2. Plan extension work, if any

TBD

Description

Mitigation

Responsible Party

Failure to identify business questions, or picking too many or too few

Draft appropriate business questions - DONE; but not all business questions can be answered with specific technologies, so must take this factor into account

Lee Harland, John Wise, Bruce Press, Peter Revill

Validate that Open Targets either has a ready to use Knowledge Graph implementation, or can be converted into a KG with reasonable cost

• Two competing, not fully compatible platforms: Neo4j/Cypher and GraphDB/SPARQL

  • BioCypher (Neo4j stack)

  • Metaphacts (SPARQL stack)

• Sponsors prefer the Neo4j

• OT partly imported into BioCypher and will be imported into Metaphacts soon

• Requirements and technology mapping to these reqs

• Architecture diagram

• linkedlifedata.com

The Hyve
Jordan Ramsdell
Robert Gill
Brian Evarts

Open Targets/EBI:

• Sebastian Lobentanzer

• Ellen McDonagh

Failure to identify a suitable LLM

• See this comparison

• Recommend to focus on the Cypher query generation ability as the key risk (below)

• Start with one open-source and one closed-source LLMs (say Mistral and GPT 4) and agree to explore others later, and meanwhile close this risk

Jon Stevens, Etzard Stolte, Helena Deus; Brian Evarts; Wouter Franke, Matthijs van der Zee

Failure to generate a proper query for a KG database system by an LLM

Technology research.

• See refs 7, 8, 13, 14 below

• BioCypher by EBI may have this capability already - needs evaluation

The Hyve
Jordan Ramsdell
Robert Gill
Brian Evarts

Open Targets/EBI:

Sebastian Lobentanzer

Ellen McDonagh

Does Open Targets use an ontology?

Yes in general

The Hyve

Failure to download a large volume of data (all of the PubMed as a maximum) for the prompt-tuning of the LLM

CLOSED RISK

This may be unnecessary, TBD

Failure to perform local KG comparison with calculation of a score

CLOSED RISK

Failure to build a prototypical target discovery pipeline on the limited budget in case of mounting technical difficulties

CLOSED: Schedule the project in phases. Aim to answer known unknowns and to establish risk mitigation strategies early

.

It is not yet known whether a product will be built. For now the scope is focused on the technology analysis for the POC

Some proprietary LLMs may be censored, thus introducing uncontrollable bias in the answers that they produce

• DONE: Censorship may already be included in the performance scores, so this is taken care of in the comparison of the LLMs. However, there is team preference for open-source and uncensored LLMs

Identified and resolved in the LLM sub-team