RFI process to evaluate Ontologies Mapping Tools

Document Information

Contents

1.       Summary

2.       Team and Governance

3.       Request for Information

3.1     Existing Functionality

3.2     Missing Functionality

3.3     Supporting Materials

3.4     RFI Process

3.4.1  RFI Schedule

3.4.2  RFI Related Questions / Clarifications

3.4.3  Response Format

3.4.4  RFI terms & conditions

4.       Appendices

4.1     Glossary

1. Summary

The Ontologies Mapping project has been set up to create better mapping tools and services, and to establish best practices for ontology management in the Life Sciences. For our purpose, ontologies can include hierarchical relationships, taxonomies, classifications and/or vocabularies which are becoming increasingly important for support of research and development. This project will inform Pistoia Alliance members about "state of the art" tools and methodologies which map and visualise ontologies, to understand ontology structure, potential overlaps and equivalence of meaning between ontologies in the same data domain.

We have defined the detailed requirements for an Ontologies Mapping tool and used this to conduct an initial survey of existing mapping tools. We concluded from this exercise that some of these are likely to substantially meet our requirements. Therefore, the process for a formal request for information (RFI) has been devised by the project team to allow us to evaluate existing mapping tools against our requirements.

It is expected that the participating tool providers will benefit from meaningful engagement with the Life Science industry. This year the Pistoia Alliance (http://www.pistoiaalliance.org) reached more than 100 member organisations which span all sizes of company such as large biopharmaceutical companies, product and service vendors including publishers, start-up small businesses and major academic institutions. This will give participants highly valuable visibility and an understanding of the ontology mapping requirements of Pistoia member organisations. It will also give the tool providers a unique opportunity to demonstrate how well their product meets our requirements and it will generate valuable feedback on aspects which would benefit from further development.

The Pistoia Alliance will not publicly recommend or rank or otherwise confer some kind of status or stratification on any third party solutions. The RFI results from participating tool providers will remain confidential to each provider and Pistoia Alliance members and they are in no way an endorsement or recommendation of anything.  Further details are described in the RFI terms and conditions.

2. Team and Governance

3. Request for Information

3.1 Existing capability

The tool provider must show how the existing capability of your tool meets our functional and non-functional requirements for ontologies mapping, as described in the accompanying document:-

Link to Tool requirements wiki page

We expect sufficient detail about all aspects of our requirements to enable us to evaluate how well they are satisfied. We have provided a template sheet which lists our requirements for the tool provider to summarise existing capabilities and evidence examples:- 

Link to Capabilities Template sheet wiki page

Although the emphasis in our requirements is mapping between ontologies, please describe any additional functionality. In particular, how this impacts on the tool interface design, performance and technical requirements to run.

3.2 Missing functionality

The tool provider should describe which of our requirements have missing functionality in your tool. It would be helpful to indicate what resource would be required to remedy the missing function, giving specific information about the development work likely to be required. For example, is the missing functionality addressed already in your plans or road map for further product development?

3.3 Supporting materials and evaluation

Source disease and phenotype ontologies for mapping

Two mapping tasks will comprise of pairwise alignment of source ontologies (via BioPortal in OWL format) in the disease and phenotype data domain:-

1) Human Phenotype Ontology (HPO) to Mammalian Phenotype Ontology (MP) and

2) Human Disease Ontology (DOID) to Orphanet and Rare Diseases Ontology (ORDO)

These are well matched tasks that involve ontologies with similar content and number of classes which can used by the tool providers to demonstrate mapping scale and quality.

The tool provider is required to provide an outline of how the mapping results were produced to accompany the mapping outputs in one of the formats described in the requirements.

Any additional data sources used by participating tools to assist with these mapping tasks should be described by the tool provider. Where appropriate, the tool provider will demonstrate mapping performance with and without additional data sources.

Our process for evaluating the quality of the mappings produced by the tool provider is described in the next section.

Evaluation of mapping quality

The quality of mappings will be evaluated by the project team against available reference mappings and though manual inspection. To facilitate our evaluation of mappings the tool provider must include three specific areas of disease or phenotype:-

A) carbohydrate and glucose metabolism

B) obesity and weight abnormality

C) breast cancer and mammary carcinoma

Evaluation of mappings will make use of reference mappings (e.g. NCBO, UMLS, MeSH) and it will include the following aspects:-

• manual assessment to compare with reference mappings
• inspection of perfect matches. Are they all found? Does it miss any out correctly? What is missed?
• evaluation of similar matches based on synonyms
• evaluation based on axiom similarity (same parents, came class descriptions)

The mapping tool should generate measures of quality which will include 1) confidence for the match and 2) level of similarity for the match and any additional measures. Similarity of match will be expected to range from equivalent to close similarity to broadly similar. These measures of quality should be recorded in the mapping output which should be in at least one of the standard formats defined in the requirements document. 

If additional data has been used to make the match, please provide access through links to such data. Performance of the tool with and without additional data sources should be shown if practical.

3.4 RFI Process

3.4.1 RFI Schedule

All times are GMT.

3.4.2 RFI Related Questions / Clarifications

All questions should be sent in writing to ian.harrow@pistoiaalliance.org and martin.romacker@pistoiaalliance.org. Answers to questions will be circulated to all confirmed respondents.

Tool provider respondents may request a Q&A teleconference; if a teleconference is held, all respondents will be invited.

3.4.3 Response Format

Respondents may use their own descriptive format for detailing their submission to this RFI. This will include example mappings including the two tasks described in section 3.3. The mappings must be in one of the format standards listed in the requirements document. 

Submissions should be emailed to ian.harrow@pistoiaalliance.org with the subject title “Ontologies Mapping Tool RFI response by <respondent organisation>”

3.4.4 RFI terms & conditions

This is a Request for Information and not an Request for Proposals (RFP). The project reserves the right to alter the requirements ahead of any future RFP if such an RFP were to be issued.

• Any estimate submitted does not constitute a proposal for the work and will not form part of any future RFP process. The project will not expect any respondent to be constrained by the estimate they submitted in any future bid against all or part of these requirements.
• This RFI is the intellectual property of the Pistoia Alliance and will be openly available to other parties to view.
• The Pistoia Alliance will not be liable for any costs incurred in preparation or submission of a response to this RFI.
• The Pistoia Alliance will accept no liability for any loss caused by response to this RFI.
• The Pistoia Alliance reserves the right to extend timelines if circumstances require.
• The Pistoia Alliance will not publicly recommend or rank or otherwise confer some kind of status or stratification on any third party solutions.

Submissions to the RFI by each participating tool provider will be evaluated in relation to our requirements and these results will remain strictly confidential to the project team and shared only with the original tool provider because feedback should remain confidential. This means that none of the response data or summaries will enter the public domain. It permits the project team to make use of the information to determine whether to run a full RFP to develop alternative solutions, or whether they can then make their own decisions as to which existing tools to use themselves in their own organisations but without making any endorsement or comparison externally.

In the case of an existing solution being found which substantially satisfies our requirements, the public output would be the publication of the tool requirements and an indication that existing tools can be found that meet most of them, but without naming those tools. This could involve a third-party specialist website that is likely to be read by the majority of companies active in ontology mapping. Project team and Pistoia members could share internally a summary grid of tool features against our requirements, but without making any recommendations beyond what readers can infer themselves from looking at the grid. 

4. Appendices

4.1 Glossary

*The scope of this RFI is ontologies mapping in the same or closely related data domains