GOlr: Difference between revisions

From GO Wiki
Jump to navigation Jump to search
mNo edit summary
 
(99 intermediate revisions by 3 users not shown)
Line 1: Line 1:
[[Category:Software]]
[[Category:GOlr]]
=Overview=
=Overview=


This page describes the status of the public Solr index for the GO. This index will replace some of the query functionality for GOOSE as well as become the new backend for AmiGO and other services.
This page describes the status of the public Solr index for the GO. This index will replace some of the query functionality for GOOSE as well as become the new backend for AmiGO 2 and other services. A preview server is now active at http://golr.berkeleybop.org.
 
=Public beta=
 
The preview server at http://golr.berkeleybop.org is currently in beta.


=Implementation Progress=
Some preview documentation is at http://cdn.berkeleybop.org/jsapi/bbop-js/docs/index.html
 
==Use==
 
Feel free to use it as you will and start to write for it. As this is
still a little early, we are only loading the GO, CL, ECO, and taxslim, as well as non-IEA annotations from the GAFs. Please see the [[AmiGO_Labs]] caveats, as well as the [[#Downtime | downtime]] information on this page.
 
==Implementation progress==
 
We are now loading properly from [https://code.google.com/p/owltools/ owltools].


We have looked at [http://nginx.org nginx] as a reverse proxy for speed and to prevent unauthorized access to non-select URLs on the Solr server, and found it to be good.
We have looked at [http://nginx.org nginx] as a reverse proxy for speed and to prevent unauthorized access to non-select URLs on the Solr server, and found it to be good.


We are now working on readying stove.lbl.gov to act as the public GO Solr server (GOlr) and AmiGO 2 client. The URLs will eventually be golr.berkeleybop.org and amigo2.berkeleybop.org.
The GOlr server is now active at http://golr.berkeleybop.org (nginx front on stove).


=Schema Progress=
We now also have http://amigo2.berkeleybop.org working on stove and backed by golr.berkeleybop.org.


We are currently working towards a flexible schema as defined in the [http://geneontology.svn.sourceforge.net/viewvc/geneontology/java/gold/solr/conf/schema.xml?revision=4429&content-type=text%2Fplain owltools] code. As we roll it out for the AmiGO backend and as a replacement for common GOOSE queries, we expect to find holes in the schema, documented below.
We are now on Solr 3.6.


Currently, the index is entirely populated by through an owltools command line program, but in the future additional document types may require other scripts for loading. It looks like:
==Schema progress==


https://code.google.com/p/owltools/source/browse/#svn%2Ftrunk%2FOWLTools-Solr%2Fsrc%2Fmain%2Fresources
We are currently working towards a flexible schema as [http://geneontology.svn.sourceforge.net/viewvc/geneontology/java/gold/solr/conf/schema.xml?revision=4429&content-type=text%2Fplain defined] in the owltools code. As we roll it out for the AmiGO backend and as a replacement for common GOOSE queries, we expect to find holes in the schema, documented below.


owltools --solr-config /home/sjcarbon/local/src/svn/owltools/OWLTools-Solr/src/main/resources/ont-config.yaml /home/sjcarbon/local/src/svn/owltools/OWLTools-Solr/src/main/resources/bio-config.yaml /home/sjcarbon/local/src/svn/owltools/OWLTools-Solr/src/main/resources/ann-config.yaml /home/sjcarbon/local/src/svn/owltools/OWLTools-Solr/src/main/resources/ann_ev_agg-config.yaml --solr-schema-dump
Currently, the index is entirely populated by through an owltools command line program, using [https://code.google.com/p/owltools/source/browse/#svn%2Ftrunk%2FOWLTools-Solr%2Fsrc%2Fmain%2Fresources YAML] files for configuration. For more information, see the ''Installation'' section below.


owltools /srv/tmp/go.owl /srv/tmp/cl.owl /srv/tmp/taxslim.owl --solr-url http://localhost:8080/solr/ --solr-purge --solr-config /home/sjcarbon/local/src/svn/owltools/OWLTools-Solr/src/main/resources/ont-config.yaml --solr-load-ontology --solr-load-gafs /srv/tmp/gene_association.mgi
==Query examples for the GOlr Schema==


==Rewriting old GOOSE query examples for the new GOlr Schema==
Example queries on the [http://wiki.geneontology.org/index.php/Example_Solr_Queries wiki].


* Example queries [http://wiki.geneontology.org/index.php/Example_Solr_Queries]
* Some old queries require the use of Solr facets and some modification of code Seth has written to present simpler results queries (i.e. the facet results are not displayed).
** Some old queries require the use of Solr facets and some modification of code Seth has written to present simpler results queries.
* Note: hierarchical queries that 'combine' facet queries can be handled by the facet.pivot functionality that is available in Solr 4. In the meantime, we'll see if we can devise a work-around.
** Note: hierarchical queries that 'combine' facet queries can be handled by the facet.pivot functionality that is available in Solr 4. In the meantime, we'll see if we can devise a work-around.


==Current Issues and Problems with the GOlr Schema==
== Current holes in the GOlr schema ==


* No PANTHER data
* None listed at this time...
** We'll look at creating a new document type from Suzi's PAINT code.


* A few suggestions to make use of the document categories easier:
== Downtime ==
** bioentity 'source' was not loaded correctly in the version I was testing
** annotation should have 'with' field (I believe Seth said this is coming but thought I'd get it written down here)
** ontology_class should have 'synonyms' and 'closure' (I believe Seth said this is coming)
** annotation_aggregate should have 'go_id' as a separate field (I believe Seth said this is coming as 'alternate_id')


Now that we have entered beta, downtimes are sporactic as we work with the software stack. As we approach production, the automatic update times will be decided and posted.


[[Category:SWUG Projects]]
= Local installation (data and schema development) =
[[Category:Software]]
 
[[Category:AmiGO]]
GOlr installation information and Solr installation help can be found [[GOlr:_Installation | here]].
[[Category:Software Progress]]
 
[[Category:Seth]]
=Collaborations=
[[Category:Mary]]
 
In order to make a truly useful system for the community at large, the Gene Ontology Consortium is working directly with other groups on GOlr tools, schema, and library development.
 
==Virtual Fly Brain (VFB)==
 
We're working with [http://www.virtualflybrain.org VFB] on schema and ontology tools development to help enable rich ontology interaction and search at speed. This will also help push the envelope on the types of use cases that we might see.

Latest revision as of 11:40, 12 August 2019

Overview

This page describes the status of the public Solr index for the GO. This index will replace some of the query functionality for GOOSE as well as become the new backend for AmiGO 2 and other services. A preview server is now active at http://golr.berkeleybop.org.

Public beta

The preview server at http://golr.berkeleybop.org is currently in beta.

Some preview documentation is at http://cdn.berkeleybop.org/jsapi/bbop-js/docs/index.html

Use

Feel free to use it as you will and start to write for it. As this is still a little early, we are only loading the GO, CL, ECO, and taxslim, as well as non-IEA annotations from the GAFs. Please see the AmiGO_Labs caveats, as well as the downtime information on this page.

Implementation progress

We are now loading properly from owltools.

We have looked at nginx as a reverse proxy for speed and to prevent unauthorized access to non-select URLs on the Solr server, and found it to be good.

The GOlr server is now active at http://golr.berkeleybop.org (nginx front on stove).

We now also have http://amigo2.berkeleybop.org working on stove and backed by golr.berkeleybop.org.

We are now on Solr 3.6.

Schema progress

We are currently working towards a flexible schema as defined in the owltools code. As we roll it out for the AmiGO backend and as a replacement for common GOOSE queries, we expect to find holes in the schema, documented below.

Currently, the index is entirely populated by through an owltools command line program, using YAML files for configuration. For more information, see the Installation section below.

Query examples for the GOlr Schema

Example queries on the wiki.

  • Some old queries require the use of Solr facets and some modification of code Seth has written to present simpler results queries (i.e. the facet results are not displayed).
  • Note: hierarchical queries that 'combine' facet queries can be handled by the facet.pivot functionality that is available in Solr 4. In the meantime, we'll see if we can devise a work-around.

Current holes in the GOlr schema

  • None listed at this time...

Downtime

Now that we have entered beta, downtimes are sporactic as we work with the software stack. As we approach production, the automatic update times will be decided and posted.

Local installation (data and schema development)

GOlr installation information and Solr installation help can be found here.

Collaborations

In order to make a truly useful system for the community at large, the Gene Ontology Consortium is working directly with other groups on GOlr tools, schema, and library development.

Virtual Fly Brain (VFB)

We're working with VFB on schema and ontology tools development to help enable rich ontology interaction and search at speed. This will also help push the envelope on the types of use cases that we might see.