OBO-Edit:Reasoner Benchmarks

From GO Wiki
Revision as of 12:56, 16 November 2008 by Cjm (talk | contribs) (→‎Ontologies Used)
Jump to navigation Jump to search

PRELIMINARY benchmarks for various reasoners vs various ontologies.

There is an unashamed GO bias here. We are evaluating with an eye to the project growth of GO particularly w.r.t both internal and external. We also care about database queries as well as ontology maintenance.

OBO-Edit Native Reasoners

oboedit_fcr

OBO-Edit Forward Chaining Reasoner. Fast, but only uses the following rules:

  • Transitivity
  • Propagation over/under is_a
  • Intersection_of (aka xps)

John wrote this first in OE1, and then abandoned it. Can still be used for so-xp, and is suitable for certain purposes such as finding trivial redundancies by transitivity, but not recommended in general

oboedit_lpr

LinkPileReasoner (LPR). Successor to FCR.

Rules:

BUGS: there are certain edge cases involving sub-relations and transitivity that do not work. This bug was discovered after John left. The code is difficult to maintain so this reasoner will eventually be abandoned in favour of the RBR (Developer notes: this appears to be due to the fact that sub-relations are computed on the fly rather than cached)

oboedit_rbr

RuleBasedReasoner. Successor to LPR.

The internal design of the RBR is highly simplified compared to the LPR. It was written entirely by cjm. Note the profile is the same as for the LPR, however, the LPR bug does not arise due to the design of the RBR.

Note that the RBR is currently slow for incremental reasoning. This is because it checks all links after any links are added or deleted. This can easily be fixed in future versions. When it is, the LPR will probably be retired altogether.

OWL/DL reasoners

OWL Reasoners are superior to OE reasoners in that they can make use of more expressive constructs, such as complementOf (negation) and universal restrictions (all-only links). However, these constructs are not currently permitted in the OBO 1.2 language.

Some OWL reasoners only recently included transitive_over and arbitrary Relation_composition. They do not allow relation intersections.

Note that OWL reasoners will not allow you to ask questions of the form "what is the heart ventricle part_of?". To allow this we perform an extra step of materializing all transitive relation x class pairs (e.g. "heart part"). This leads to worse performance.

The two main open source OWL reasoners are FaCT++ and Pellet

owlapi_pellet

Pellet executed via the OWLAPI (1.2)

owlapi_pellet_mr

As above, with all class x relation pairs materialized.

owlapi_factpp

FaCT++ executed via the OWLAPI (1.2)

owlapi_factpp_mr

As above, with all class x relation pairs materialized.

pellet

Pellet 2.0 executed directly

oboedit_pwr

OBO-Edit pellet-wrapped reasoner. Pellet, executed with OE using a java object level bridge. Performs the class x relation materialization.

Needs updated to the latest Pellet API. May be discontinued in favour of an OWLAPI bridge

Database Reasoners

These reasoners use essentially the same procedure as the oboedit RBR. However, inferred links are backed by a relational database rather than in-memory. As such they tend to be slower, but are not-memory bound. These are also a good choice when the inferred graph needs to be stored in a database, to allow for advanced queries on annotations.

obdsql

Runs off of the OBD SQL schema. See OBD

Uses: Postgres 8.3

  • Transitivity
  • Propagation over/under is_a
  • Subrelations
  • Intersection_of

godb

Uses: MySQL 5

Relatively limited set of rules:

Will be extended in future

blipkit

Backed by a prolog database


Results

Availability

See reasoner-benchmarks for the raw data.

Ontologies Used

The following ontologies were used:

Preliminary Benchmarks

ONT oboedit_fcr owlapi_pellet_mr owlapi_factpp oboedit_lpr oboedit_rbr obdsql oboedit_pwr owlapi_factpp_mr owlapi_pellet pellet godbsql blipkit
DC_CL.obo 0.096 3.26185 0.321088 0.269457 0.125779 0.344787 3.406521 0 2 0.04
bp_xp_cc.obo 13.476 113.021307 54.366712 1540 80.96
bp_xp_cl.obo 13.589 84.450377 54.120796 1612 89.41
bp_xp_u.obo 14.096 102.442893 59.710887 1545 86.89
caro.obo 0.018 0.234596 0.013313 0.031378 0.026605 1.449529 0.035096 0.159686 0 0 0.01
cc_xp_i.obo 13.36 65.882047 283.275332 1407
cc_xp_p.obo 0.92 161.977906 19.044575 1.905131 148 1 68 3.59
cell.obo 0.391 3.015834 0.17246 2.024176 1.025127 45.124648 7.315394 2.951417 0 47 1.14
fly.obo 4.122 43.970665 1.678357 23.575053 11.259419 685.638543 42.515054 30 687 18.81
fly_xp.obo 10.223 5.514895 0.393729 31.695978 24.125094 522 0.390108 5.480391 0 370 24.02
fma2.obo 33.16 156.072801 10.881829 157.908895 132.435731 61
go.obo 10.507 187.491808 51.812351 40.883224 25.03078 1643.285332 134.506767 9 1143 27.42
go_xp_i.obo 101.995 1205.722121 1384.34991 2395 1117
go_xp_r.obo 56.467 1020.276001 706.097357 32 388.68
hp_xp.obo 47.689 986.551557
mp_xp.obo
nbirn.obo 10.917 984 0 315 109.04
nif.obo 7.274 925 0 287 58.33
sao.obo 0.17 3.125008 0.127607 0.257155 0.166285 20 3.157377 3.214984 1 6 0.52
so-xp.obo 0.351 6.547285 0.320349 1.423436 0.927108 47 3914.710214 57.310919 5.856606 3 25 0.97
tax.obo 0
uber.obo 0.889 11.993522 10.440892 3.818597 2.674329 89 3674.025166 11.939288 0 51 2.28
xenopus_anatomy.obo 0.214 3.45821 0.125743 1.356273 0.677572 24 45.437443 3.069618 0 13 0.71
zfs.obo 1.012 7.729311 0.868834 4.59457 1.945593 81 442.176979 8.255652 1 98 3.28

Methods

The reasoner-benchmarks.pl script was used