OBO-Edit:Reasoner Benchmarks: Difference between revisions
No edit summary |
No edit summary |
||
(29 intermediate revisions by 2 users not shown) | |||
Line 1: | Line 1: | ||
PRELIMINARY benchmarks for various reasoners vs various ontologies. | |||
There is an unashamed GO bias here. We are evaluating with an eye to the project growth of GO particularly w.r.t [[Category:Cross Products|Cross Products]] both internal and external. We also care about database queries as well as ontology maintenance. | |||
== OBO-Edit Native Reasoners == | == OBO-Edit Native Reasoners == | ||
=== oboedit_fcr === | === oboedit_fcr === | ||
OBO-Edit Forward Chaining Reasoner. Fast, but only uses the following rules: | [[OBO-Edit:Forward Chaining Reasoner]] (LPR). Fast, but only uses the following rules: | ||
* Transitivity | * Transitivity | ||
* Propagation over/under [[is_a]] | * Propagation over/under [[is_a]] | ||
* Intersection_of | * Intersection_of (aka xps) | ||
John wrote this first in OE1, and then abandoned it. Can still be used for so-xp, and is suitable for certain purposes such as finding trivial redundancies by transitivity, but not recommended in general | John wrote this first in OE1, and then abandoned it. Can still be used for so-xp, and is suitable for certain purposes such as finding trivial redundancies by transitivity, but not recommended in general | ||
Line 13: | Line 17: | ||
=== oboedit_lpr === | === oboedit_lpr === | ||
[[OBO-Edit:Link Pile Reasoner]] (LPR). Successor to FCR. | |||
Rules: | Rules: | ||
Line 28: | Line 32: | ||
=== oboedit_rbr === | === oboedit_rbr === | ||
[[OBO-Edit:Rule Based Reasoner]] (RBR). Successor to LPR. | |||
* Transitivity | * Transitivity | ||
Line 49: | Line 53: | ||
Note that OWL reasoners will not allow you to ask questions of the form "what is the heart ventricle part_of?". To allow this we perform an extra step of materializing all transitive relation x class pairs (e.g. "heart part"). This leads to worse performance. | Note that OWL reasoners will not allow you to ask questions of the form "what is the heart ventricle part_of?". To allow this we perform an extra step of materializing all transitive relation x class pairs (e.g. "heart part"). This leads to worse performance. | ||
The two main open source OWL reasoners are FaCT++ and Pellet | The two main open source OWL reasoners are FaCT++ and Pellet. | ||
Note: even though the table below gives the suffix ".obo", each reasoner was executed on an owl file. These were generated by go2owl whenever the native form of the ontology is obo. If the source is owl then blipkit is used for the reverse translations. See [http://www.berkeleybop.org/obo-conv.cgi obo-conv]. NOTE THAT THIS GIVES AN ADVANTAGE TO OBO REASONERS AS SOME AXIOMS WILL BE DROPPED. | |||
=== owlapi_pellet === | === owlapi_pellet === | ||
Line 70: | Line 76: | ||
Pellet 2.0 executed directly | Pellet 2.0 executed directly | ||
=== hermit === | |||
Hermit 0.9.2 executed directly. | |||
Note that hermit does not support object property chains -i.e. [[transitive_over]] or arbitrary [[Relation_composition]]. It will fail on any ontology that includes these. This means a lot of entries are blank. | |||
=== oboedit_pwr === | === oboedit_pwr === | ||
Line 93: | Line 105: | ||
=== godb === | === godb === | ||
This is used when loading ontologies into the GO Database. See [[Transitive_closure]] | |||
Uses: MySQL 5 | Uses: MySQL 5 | ||
Line 107: | Line 121: | ||
=== blipkit === | === blipkit === | ||
Backed by a prolog database | Backed by a prolog database. This is the same reasoner used in [http://www.berkeleybop.org/obol obol].See [http://www.blipkit.org blipkit home page] | ||
* Transitivity | * Transitivity | ||
Line 117: | Line 131: | ||
* Arbitrary rules | * Arbitrary rules | ||
== Results == | |||
=== Availability === | |||
See [http://oboedit.org/reasoner-benchmarks reasoner-benchmarks] for the raw data. | |||
=== Ontologies Used === | |||
TODO: show expressivity of each | |||
The following ontologies were used: | |||
* DC_CL.obo | |||
* bp_xp_cc.obo -- [[XP:biological_process_xp_cellular_component]] | |||
* bp_xp_cl.obo -- [[XP:biological_process_xp_cell]] | |||
* bp_xp_u.obo -- [[XP:biological_process_xp_uber_anatomy]] | |||
* caro.obo | |||
* cc_xp_i.obo -- [[XP:cellular_component_xp_self]] and [[XP:cellular_component_xp_go]] | |||
* cc_xp_p.obo -- [[XP:cellular_component_xp_self]] ("X part" terms only) | |||
* cell.obo | |||
* fly.obo | |||
* fly_xp.obo | |||
* fma2.obo | |||
* go.obo | |||
* go_xp_i.obo -- all GO internal xps. See [[:Category:Internal_Cross_Products]] | |||
* go_xp_r.obo -- [[XP:biological_prcess_xp_regulation]] | |||
* hp_xp.obo -- [http://www.berkeleybop.org/ontologies/#human_phenotype_xp] | |||
* mp_xp.obo -- [[XP:mammalian_phenotype_xp]] | |||
* nbirn.obo | |||
* nif.obo | |||
* sao.obo | |||
* so-xp.obo | |||
* tax.obo | |||
* uber.obo | |||
* xenopus_anatomy.obo | |||
* zfs.obo | |||
=== Preliminary Benchmarks === | |||
'''THESE RESULTS ARE PRELIMINARY''' | |||
Blank = program did not complete in allotted time (1-2hrs, varies) OR ran out of memory/heap OR possibly for other reasons, including bugs in my code. At this time, these results should be taken with a pinch of salt. | |||
Also make sure you understand everything above before interpreting. OBO reasoners may have some unfair advantages here. | |||
{|class="wikitable" style="text-align:center" border="1" | |||
|- | |||
! ONT | |||
! hermit | |||
! oboedit_fcr | |||
! owlapi_pellet_mr | |||
! owlapi_factpp | |||
! oboedit_lpr | |||
! oboedit_rbr | |||
! obdsql | |||
! oboedit_pwr | |||
! owlapi_factpp_mr | |||
! owlapi_pellet | |||
! pellet | |||
! godbsql | |||
! blipkit | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/DC_CL.obo DC_CL] | |||
| 0.154 | |||
| 0.096 | |||
| 3.26185 | |||
| 0.321088 | |||
| 0.269457 | |||
| 0.101434 | |||
| 21 | |||
| | |||
| 0.344787 | |||
| 3.406521 | |||
| 0 | |||
| 2 | |||
| 0.04 | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/bp_xp_cc.obo bp_xp_cc] | |||
| | |||
| 13.476 | |||
| | |||
| | |||
| 113.021307 | |||
| 46.33769 | |||
| 1668 | |||
| | |||
| | |||
| | |||
| | |||
| 1540 | |||
| 80.96 | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/bp_xp_cl.obo bp_xp_cl] | |||
| | |||
| 13.589 | |||
| | |||
| | |||
| 84.450377 | |||
| 49.550078 | |||
| 1774 | |||
| | |||
| | |||
| | |||
| | |||
| 1612 | |||
| 89.41 | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/bp_xp_u.obo bp_xp_u] | |||
| | |||
| 14.096 | |||
| | |||
| | |||
| 102.442893 | |||
| 52.879202 | |||
| 2043 | |||
| | |||
| | |||
| | |||
| | |||
| 1545 | |||
| 86.89 | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/caro.obo caro] | |||
| 0.038 | |||
| 0.018 | |||
| 0.234596 | |||
| 0.013313 | |||
| 0.031378 | |||
| 0.027466 | |||
| 0 | |||
| 1.449529 | |||
| 0.035096 | |||
| 0.159686 | |||
| 0 | |||
| 0 | |||
| 0.01 | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/cc_xp_i.obo cc_xp_i] | |||
| | |||
| 13.36 | |||
| | |||
| | |||
| 65.882047 | |||
| 289.564573 | |||
| 1573 | |||
| | |||
| | |||
| | |||
| | |||
| 1407 | |||
| 67.05 | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/cc_xp_p.obo cc_xp_p] | |||
| | |||
| 0.92 | |||
| | |||
| 161.977906 | |||
| 19.044575 | |||
| 1.914166 | |||
| 148 | |||
| | |||
| | |||
| | |||
| 1 | |||
| 68 | |||
| 3.59 | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/cell.obo cell] | |||
| 0.212 | |||
| 0.391 | |||
| 3.015834 | |||
| 0.17246 | |||
| 2.024176 | |||
| 1.039019 | |||
| 32 | |||
| 45.124648 | |||
| 7.315394 | |||
| 2.951417 | |||
| 0 | |||
| 47 | |||
| 1.14 | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/chebt.obo chebt] | |||
| 9.156 | |||
| | |||
| 105.981208 | |||
| 125.216634 | |||
| | |||
| 180.321342 | |||
| 2881 | |||
| | |||
| | |||
| 100.201565 | |||
| 10 | |||
| 4927 | |||
| | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/fly.obo fly] | |||
| 2.847 | |||
| 4.122 | |||
| 43.970665 | |||
| 1.678357 | |||
| 23.575053 | |||
| 10.96353 | |||
| 330 | |||
| | |||
| 685.638543 | |||
| 42.515054 | |||
| 30 | |||
| 687 | |||
| 18.81 | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/fly_xp.obo fly_xp] | |||
| 5.514 | |||
| 10.223 | |||
| 5.514895 | |||
| 0.393729 | |||
| 31.695978 | |||
| 22.514039 | |||
| 526 | |||
| | |||
| 0.390108 | |||
| 5.480391 | |||
| 0 | |||
| 370 | |||
| 24.02 | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/fma2.obo fma2] | |||
| 11.857 | |||
| 33.16 | |||
| 156.072801 | |||
| 10.881829 | |||
| 157.908895 | |||
| | |||
| 7483 | |||
| | |||
| | |||
| 132.435731 | |||
| 61 | |||
| | |||
| | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/go.obo go] | |||
| | |||
| 10.507 | |||
| 187.491808 | |||
| 51.812351 | |||
| 40.883224 | |||
| 23.630003 | |||
| 1364 | |||
| | |||
| 1643.285332 | |||
| 134.506767 | |||
| 9 | |||
| 1143 | |||
| 27.42 | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/go_xp_i.obo go_xp_i] | |||
| | |||
| 101.995 | |||
| | |||
| | |||
| 1205.722121 | |||
| 408.745009 | |||
| 2395 | |||
| | |||
| | |||
| | |||
| | |||
| 1117 | |||
| 607.24 | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/go_xp_r.obo go_xp_r] | |||
| | |||
| 56.467 | |||
| | |||
| | |||
| 1020.276001 | |||
| 43.711079 | |||
| 1427 | |||
| | |||
| | |||
| | |||
| 32 | |||
| | |||
| 388.68 | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/hp_xp.obo hp_xp] | |||
| | |||
| 47.689 | |||
| | |||
| | |||
| 986.551557 | |||
| | |||
| 10736 | |||
| | |||
| | |||
| | |||
| | |||
| | |||
| | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/mp_xp.obo mp_xp] | |||
| | |||
| | |||
| | |||
| | |||
| | |||
| | |||
| | |||
| | |||
| | |||
| | |||
| | |||
| | |||
| | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/mp_xp_ch.obo mp_xp_ch] | |||
| | |||
| | |||
| | |||
| | |||
| | |||
| 2687.581091 | |||
| | |||
| | |||
| | |||
| | |||
| | |||
| | |||
| | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/mp_xp_cl.obo mp_xp_cl] | |||
| | |||
| | |||
| | |||
| | |||
| | |||
| 10.166375 | |||
| | |||
| | |||
| | |||
| | |||
| 222 | |||
| | |||
| 20.3 | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/mp_xp_go.obo mp_xp_go] | |||
| | |||
| | |||
| | |||
| | |||
| | |||
| 78.470274 | |||
| | |||
| | |||
| | |||
| | |||
| 92 | |||
| | |||
| 57.52 | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/mp_xp_ma.obo mp_xp_ma] | |||
| | |||
| | |||
| | |||
| | |||
| | |||
| | |||
| | |||
| | |||
| | |||
| | |||
| | |||
| | |||
| | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/nbirn.obo nbirn] | |||
| 0.003 | |||
| 10.917 | |||
| | |||
| | |||
| | |||
| 27.418409 | |||
| 984 | |||
| | |||
| | |||
| | |||
| 0 | |||
| 315 | |||
| 109.04 | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/ncithesaurus.obo ncithesaurus] | |||
| 2118.256 | |||
| 91.178 | |||
| 632.050947 | |||
| 58.714178 | |||
| | |||
| 122.090551 | |||
| 55000 | |||
| | |||
| 66.600443 | |||
| 745.562156 | |||
| 255 | |||
| 1592 | |||
| | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/nif.obo nif] | |||
| 0.004 | |||
| 7.274 | |||
| | |||
| | |||
| | |||
| 21.840244 | |||
| 925 | |||
| | |||
| | |||
| | |||
| 0 | |||
| 287 | |||
| 58.33 | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/obi.obo obi] | |||
| 1.998 | |||
| 6.435 | |||
| | |||
| 0.90253 | |||
| | |||
| 3.379759 | |||
| 91 | |||
| | |||
| 0.756836 | |||
| | |||
| 7 | |||
| 28 | |||
| 1.04 | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/sao.obo sao] | |||
| | |||
| 0.17 | |||
| 3.125008 | |||
| 0.127607 | |||
| 0.257155 | |||
| 0.171273 | |||
| 20 | |||
| | |||
| 3.157377 | |||
| 3.214984 | |||
| 1 | |||
| 6 | |||
| 0.52 | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/so-xp.obo so-xp] | |||
| 0.397 | |||
| 0.351 | |||
| 6.547285 | |||
| 0.320349 | |||
| 1.423436 | |||
| 0.959837 | |||
| 47 | |||
| 3914.710214 | |||
| 57.310919 | |||
| 5.856606 | |||
| 3 | |||
| 25 | |||
| 0.97 | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/tax.obo tax] | |||
| 0.004 | |||
| | |||
| | |||
| | |||
| | |||
| | |||
| | |||
| | |||
| | |||
| | |||
| 0 | |||
| | |||
| | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/uber.obo uber] | |||
| 0.853 | |||
| 0.889 | |||
| 11.993522 | |||
| 10.440892 | |||
| 3.818597 | |||
| 2.199386 | |||
| 89 | |||
| | |||
| 3674.025166 | |||
| 11.939288 | |||
| 0 | |||
| 51 | |||
| 2.28 | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/xenopus_anatomy.obo xenopus_anatomy] | |||
| 0.163 | |||
| 0.214 | |||
| 3.45821 | |||
| 0.125743 | |||
| 1.356273 | |||
| 0.6789 | |||
| 24 | |||
| | |||
| 45.437443 | |||
| 3.069618 | |||
| 0 | |||
| 13 | |||
| 0.71 | |||
|- | |||
| [http://oboedit.org/reasoner-benchmarks/zfs.obo zfs] | |||
| 0.609 | |||
| 1.012 | |||
| 7.729311 | |||
| 0.868834 | |||
| 4.59457 | |||
| 1.899504 | |||
| 81 | |||
| | |||
| 442.176979 | |||
| 8.255652 | |||
| 1 | |||
| 98 | |||
| 3.28 | |||
|} | |||
== Methods == | |||
The [http://obo.cvs.sourceforge.net/viewvc/obo/obo/website/utils/reasoner-benchmarks.pl?view=log reasoner-benchmarks.pl] script was used. | |||
The times above represent actual reasoning time (in seconds) as reported by the reasoner itself. It should exclude time for program to load etc. | |||
4gb OS X, 32 bit java for OE, java -Xmx2048m | |||
Standard oboInOwl conversion was used for obo ontologies when running with owl reasoners. For OWLAPI tests a custom simple runner was written, see: [http://geneontology.svn.sourceforge.net/viewvc/geneontology/OWLTools/src/owltools/OWLReasonerRunner.java?view=log OWLReasonerRunner] | |||
==Reasoner Benchmarking on Users' Machines== | |||
[[OE2 Benchmarking by Users]] | |||
[[Category:Reasoning]] | [[Category:Reasoning]] |
Latest revision as of 09:02, 12 January 2009
PRELIMINARY benchmarks for various reasoners vs various ontologies.
There is an unashamed GO bias here. We are evaluating with an eye to the project growth of GO particularly w.r.t both internal and external. We also care about database queries as well as ontology maintenance.
OBO-Edit Native Reasoners
oboedit_fcr
OBO-Edit:Forward Chaining Reasoner (LPR). Fast, but only uses the following rules:
- Transitivity
- Propagation over/under is_a
- Intersection_of (aka xps)
John wrote this first in OE1, and then abandoned it. Can still be used for so-xp, and is suitable for certain purposes such as finding trivial redundancies by transitivity, but not recommended in general
oboedit_lpr
OBO-Edit:Link Pile Reasoner (LPR). Successor to FCR.
Rules:
- Transitivity
- Propagation over/under is_a
- Intersection_of
- Subrelations
- transitive_over and arbitrary Relation_composition
- Relation intersections
BUGS: there are certain edge cases involving sub-relations and transitivity that do not work. This bug was discovered after John left. The code is difficult to maintain so this reasoner will eventually be abandoned in favour of the RBR (Developer notes: this appears to be due to the fact that sub-relations are computed on the fly rather than cached)
oboedit_rbr
OBO-Edit:Rule Based Reasoner (RBR). Successor to LPR.
- Transitivity
- Propagation over/under is_a
- Intersection_of
- Subrelations
- transitive_over and arbitrary Relation_composition
- Relation intersections
The internal design of the RBR is highly simplified compared to the LPR. It was written entirely by cjm. Note the profile is the same as for the LPR, however, the LPR bug does not arise due to the design of the RBR.
Note that the RBR is currently slow for incremental reasoning. This is because it checks all links after any links are added or deleted. This can easily be fixed in future versions. When it is, the LPR will probably be retired altogether.
OWL/DL reasoners
OWL Reasoners are superior to OE reasoners in that they can make use of more expressive constructs, such as complementOf (negation) and universal restrictions (all-only links). However, these constructs are not currently permitted in the OBO 1.2 language.
Some OWL reasoners only recently included transitive_over and arbitrary Relation_composition. They do not allow relation intersections.
Note that OWL reasoners will not allow you to ask questions of the form "what is the heart ventricle part_of?". To allow this we perform an extra step of materializing all transitive relation x class pairs (e.g. "heart part"). This leads to worse performance.
The two main open source OWL reasoners are FaCT++ and Pellet.
Note: even though the table below gives the suffix ".obo", each reasoner was executed on an owl file. These were generated by go2owl whenever the native form of the ontology is obo. If the source is owl then blipkit is used for the reverse translations. See obo-conv. NOTE THAT THIS GIVES AN ADVANTAGE TO OBO REASONERS AS SOME AXIOMS WILL BE DROPPED.
owlapi_pellet
Pellet executed via the OWLAPI (1.2)
owlapi_pellet_mr
As above, with all class x relation pairs materialized.
owlapi_factpp
FaCT++ executed via the OWLAPI (1.2)
owlapi_factpp_mr
As above, with all class x relation pairs materialized.
pellet
Pellet 2.0 executed directly
hermit
Hermit 0.9.2 executed directly.
Note that hermit does not support object property chains -i.e. transitive_over or arbitrary Relation_composition. It will fail on any ontology that includes these. This means a lot of entries are blank.
oboedit_pwr
OBO-Edit pellet-wrapped reasoner. Pellet, executed with OE using a java object level bridge. Performs the class x relation materialization.
Needs updated to the latest Pellet API. May be discontinued in favour of an OWLAPI bridge
Database Reasoners
These reasoners use essentially the same procedure as the oboedit RBR. However, inferred links are backed by a relational database rather than in-memory. As such they tend to be slower, but are not-memory bound. These are also a good choice when the inferred graph needs to be stored in a database, to allow for advanced queries on annotations.
obdsql
Runs off of the OBD SQL schema. See OBD
Uses: Postgres 8.3
- Transitivity
- Propagation over/under is_a
- Subrelations
- Intersection_of
godb
This is used when loading ontologies into the GO Database. See Transitive_closure
Uses: MySQL 5
Relatively limited set of rules:
- Transitivity
- Propagation over/under is_a
- Subrelations
- transitive_over and arbitrary Relation_composition
Will be extended in future
blipkit
Backed by a prolog database. This is the same reasoner used in obol.See blipkit home page
- Transitivity
- Propagation over/under is_a
- Intersection_of
- Subrelations
- transitive_over and arbitrary Relation_composition
- Relation intersections
- Arbitrary rules
Results
Availability
See reasoner-benchmarks for the raw data.
Ontologies Used
TODO: show expressivity of each
The following ontologies were used:
- DC_CL.obo
- bp_xp_cc.obo -- XP:biological_process_xp_cellular_component
- bp_xp_cl.obo -- XP:biological_process_xp_cell
- bp_xp_u.obo -- XP:biological_process_xp_uber_anatomy
- caro.obo
- cc_xp_i.obo -- XP:cellular_component_xp_self and XP:cellular_component_xp_go
- cc_xp_p.obo -- XP:cellular_component_xp_self ("X part" terms only)
- cell.obo
- fly.obo
- fly_xp.obo
- fma2.obo
- go.obo
- go_xp_i.obo -- all GO internal xps. See Category:Internal_Cross_Products
- go_xp_r.obo -- XP:biological_prcess_xp_regulation
- hp_xp.obo -- [1]
- mp_xp.obo -- XP:mammalian_phenotype_xp
- nbirn.obo
- nif.obo
- sao.obo
- so-xp.obo
- tax.obo
- uber.obo
- xenopus_anatomy.obo
- zfs.obo
Preliminary Benchmarks
THESE RESULTS ARE PRELIMINARY
Blank = program did not complete in allotted time (1-2hrs, varies) OR ran out of memory/heap OR possibly for other reasons, including bugs in my code. At this time, these results should be taken with a pinch of salt.
Also make sure you understand everything above before interpreting. OBO reasoners may have some unfair advantages here.
ONT | hermit | oboedit_fcr | owlapi_pellet_mr | owlapi_factpp | oboedit_lpr | oboedit_rbr | obdsql | oboedit_pwr | owlapi_factpp_mr | owlapi_pellet | pellet | godbsql | blipkit |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
DC_CL | 0.154 | 0.096 | 3.26185 | 0.321088 | 0.269457 | 0.101434 | 21 | 0.344787 | 3.406521 | 0 | 2 | 0.04 | |
bp_xp_cc | 13.476 | 113.021307 | 46.33769 | 1668 | 1540 | 80.96 | |||||||
bp_xp_cl | 13.589 | 84.450377 | 49.550078 | 1774 | 1612 | 89.41 | |||||||
bp_xp_u | 14.096 | 102.442893 | 52.879202 | 2043 | 1545 | 86.89 | |||||||
caro | 0.038 | 0.018 | 0.234596 | 0.013313 | 0.031378 | 0.027466 | 0 | 1.449529 | 0.035096 | 0.159686 | 0 | 0 | 0.01 |
cc_xp_i | 13.36 | 65.882047 | 289.564573 | 1573 | 1407 | 67.05 | |||||||
cc_xp_p | 0.92 | 161.977906 | 19.044575 | 1.914166 | 148 | 1 | 68 | 3.59 | |||||
cell | 0.212 | 0.391 | 3.015834 | 0.17246 | 2.024176 | 1.039019 | 32 | 45.124648 | 7.315394 | 2.951417 | 0 | 47 | 1.14 |
chebt | 9.156 | 105.981208 | 125.216634 | 180.321342 | 2881 | 100.201565 | 10 | 4927 | |||||
fly | 2.847 | 4.122 | 43.970665 | 1.678357 | 23.575053 | 10.96353 | 330 | 685.638543 | 42.515054 | 30 | 687 | 18.81 | |
fly_xp | 5.514 | 10.223 | 5.514895 | 0.393729 | 31.695978 | 22.514039 | 526 | 0.390108 | 5.480391 | 0 | 370 | 24.02 | |
fma2 | 11.857 | 33.16 | 156.072801 | 10.881829 | 157.908895 | 7483 | 132.435731 | 61 | |||||
go | 10.507 | 187.491808 | 51.812351 | 40.883224 | 23.630003 | 1364 | 1643.285332 | 134.506767 | 9 | 1143 | 27.42 | ||
go_xp_i | 101.995 | 1205.722121 | 408.745009 | 2395 | 1117 | 607.24 | |||||||
go_xp_r | 56.467 | 1020.276001 | 43.711079 | 1427 | 32 | 388.68 | |||||||
hp_xp | 47.689 | 986.551557 | 10736 | ||||||||||
mp_xp | |||||||||||||
mp_xp_ch | 2687.581091 | ||||||||||||
mp_xp_cl | 10.166375 | 222 | 20.3 | ||||||||||
mp_xp_go | 78.470274 | 92 | 57.52 | ||||||||||
mp_xp_ma | |||||||||||||
nbirn | 0.003 | 10.917 | 27.418409 | 984 | 0 | 315 | 109.04 | ||||||
ncithesaurus | 2118.256 | 91.178 | 632.050947 | 58.714178 | 122.090551 | 55000 | 66.600443 | 745.562156 | 255 | 1592 | |||
nif | 0.004 | 7.274 | 21.840244 | 925 | 0 | 287 | 58.33 | ||||||
obi | 1.998 | 6.435 | 0.90253 | 3.379759 | 91 | 0.756836 | 7 | 28 | 1.04 | ||||
sao | 0.17 | 3.125008 | 0.127607 | 0.257155 | 0.171273 | 20 | 3.157377 | 3.214984 | 1 | 6 | 0.52 | ||
so-xp | 0.397 | 0.351 | 6.547285 | 0.320349 | 1.423436 | 0.959837 | 47 | 3914.710214 | 57.310919 | 5.856606 | 3 | 25 | 0.97 |
tax | 0.004 | 0 | |||||||||||
uber | 0.853 | 0.889 | 11.993522 | 10.440892 | 3.818597 | 2.199386 | 89 | 3674.025166 | 11.939288 | 0 | 51 | 2.28 | |
xenopus_anatomy | 0.163 | 0.214 | 3.45821 | 0.125743 | 1.356273 | 0.6789 | 24 | 45.437443 | 3.069618 | 0 | 13 | 0.71 | |
zfs | 0.609 | 1.012 | 7.729311 | 0.868834 | 4.59457 | 1.899504 | 81 | 442.176979 | 8.255652 | 1 | 98 | 3.28 |
Methods
The reasoner-benchmarks.pl script was used.
The times above represent actual reasoning time (in seconds) as reported by the reasoner itself. It should exclude time for program to load etc.
4gb OS X, 32 bit java for OE, java -Xmx2048m
Standard oboInOwl conversion was used for obo ontologies when running with owl reasoners. For OWLAPI tests a custom simple runner was written, see: OWLReasonerRunner