AmiGO 2 Manual: Installation (2.4.x): Difference between revisions

From GO Wiki
Jump to navigation Jump to search
m (Sjcarbon moved page AmiGO 2 Manual: Installation (new deployment) to AmiGO 2 Manual: Installation (2.4.x): Need more granularity until everything gets packaged again into the repo.)
 
(16 intermediate revisions by 2 users not shown)
Line 1: Line 1:
[[Category:AmiGO_2]]
[[Category:Software]]
[[Category:AmiGO_2_Manual]]
=Preliminaries=
=Preliminaries=


These instructions are intended to illustrate how to install the AmiGO 2 client software on a Ubuntu 14.04 LTS. With careful reading, these instructions should be generalizable to other cases. The installation of this client software only covers AmiGO 2 and some Apache 2 configuration.
The current released production version is 2.4.x.
 
These instructions are intended to illustrate how to install the AmiGO 2 client software on a Ubuntu 16.04 LTS. With careful reading, these instructions should be generalizable to other cases. The installation of this client software only covers AmiGO 2 and some Apache 2 configuration.


For information on the data server that AmiGO 2 uses (covering the installation of and loading using the server software: GOlr schema for Solr, OWLTools, etc.), please see the ''Installation'' section of the [[GOlr]] documentation.
For information on the data server that AmiGO 2 uses (covering the installation of and loading using the server software: GOlr schema for Solr, OWLTools, etc.), please see the ''Installation'' section of the [[GOlr]] documentation.
Line 9: Line 14:
=Packages=
=Packages=


In general, package managers are the preferred way of getting libraries onto a system in a consistent, reversible, and secure manner. However, often you'll find yourself in a situation where you'll need to use CPAN because your preferred (lesser) distribution does not have a robust set of perl libraries.
==Perl==
 
In general, package managers are the preferred way of getting libraries onto a system in a consistent, reversible, and secure manner. However, often you'll find yourself in a situation where you'll need to use CPAN because your preferred distribution or system does not have a robust set of perl libraries.


For users who need to use CPAN, but don't want to mess too deeply with their system (CPAN can turn into a nightmare when installing globally), we'd suggest using CPAN's local::lib and just putting that into your development and run paths: https://metacpan.org/pod/local::lib#The-bootstrapping-technique .
For users who need to use CPAN, but don't want to mess too deeply with their system (CPAN can turn into a nightmare when installing globally), we'd suggest using CPAN's local::lib and just putting that into your development and run paths: https://metacpan.org/pod/local::lib#The-bootstrapping-technique .
==JavaScript==
Given that, in the time that it has taken you to read this, yet another revolution has occurred in JavaScript, staying on top of packages can be hard and distributions do not reliably track them. For this, assuming that you have a recent version of node we do most of our package management with NPM.
To get a recent version of node, we have been using
deb https://deb.nodesource.com/node_0.4.x xenial main
Which was the end product of following the instructions here: https://nodejs.org/en/download/package-manager/ . As you can see, we're working of of the still supported LTS 4.x.


== Ubuntu/Debian ==
== Ubuntu/Debian ==


Also, in addition to anything that you might need for BBOP JS (in the optional case that you are going to develop with it), the following packages are needed:
The following packages are needed (this might need some revision):


  sudo apt-get -u install git make rhino kwalify yui-compressor naturaldocs libcgi-application-perl libcgi-application-plugin-session-perl libcgi-application-plugin-tt-perl libclone-perl libconfig-yaml-perl libdbi-perl libdbd-sqlite3-perl libdbd-mysql-perl libdata-formvalidator-perl libossp-uuid-perl libfile-type-perl libfile-slurp-perl libfreezethaw-perl libgraph-perl libgraphviz-perl libjson-xs-perl liburi-perl libwww-mechanize-perl libxml-libxml-perl libxml-xpath-perl
  sudo apt-get -u install git make kwalify libcgi-application-perl libcgi-application-server-perl libcgi-application-plugin-session-perl libcgi-application-plugin-tt-perl libclone-perl libconfig-yaml-perl libdbi-perl libdbd-sqlite3-perl libdbd-mysql-perl libdata-formvalidator-perl libossp-uuid-perl libfile-type-perl libfile-slurp-perl libfreezethaw-perl libgraph-perl libgraphviz-perl libjson-xs-perl liburi-perl libwww-mechanize-perl libxml-libxml-perl libxml-xpath-perl


The package libsql-tokenizer-perl is also required (by GOOSE for query checking), but may no longer be available in repositories. If you're interested in using a locally configured GOOSE, you may need to manually install this package along the lines of:
The package libsql-tokenizer-perl is also required (by GOOSE for query checking), but may no longer be available in repositories. If you're interested in using a locally configured GOOSE, you may need to manually install this package along the lines of:
Line 30: Line 47:
  sudo apt-get -u install liblocal-lib-perl
  sudo apt-get -u install liblocal-lib-perl


== CentOS/RedHat ==
== CentOS/RedHat ''example'' ==


A comparative of packages installed between Ubuntu and CentOS.
An ''example'' comparative table of packages installed between Ubuntu and CentOS; in this case, the perl packages were manually installed rather than using CPAN (see CPAN note above).


{| class="wikitable"
{| class="wikitable"
Line 130: Line 147:
| NaturalDocs-1.52.zip  
| NaturalDocs-1.52.zip  
| See Note [1]
| See Note [1]
|-
| rhino
| rhino-1.7-0.7.r2.2.el6.noarch
| yum install
|-
|-
| rsync  
| rsync  
| rsync-3.0.6-9.el6.x86_64  
| rsync-3.0.6-9.el6.x86_64  
| yum install
| yum install
|-
| yui-compressor
| yuicompressor-master
| See Note [2]
|-
|-
| [not defined]  
| [not defined]  
Line 159: Line 168:
/usr/local/naturaldocs), copy the zip file into the directory, then unzip the zip
/usr/local/naturaldocs), copy the zip file into the directory, then unzip the zip
file there.
file there.
Note [2]: yui-compressor is an Ubuntu-specific shell script front-end to the
Java based yuicompressor tool. There is no equivalent script in the
RHEL/CentOS distributions, but the AmiGO 2 tools require its use. We have
written a home-grown script for RHEL/CentOS to substitute for the Ubuntu
version. This script is installed in /usr/bin/yui-compressor. The following is the
script contents:
#!/bin/sh
# Quick and dirty front-end script to the yui-compressor that emulates
# /usr/bin/yuicompressor script from Ubuntu release
#
JAVA_HOME=/usr/java/default
YUI_JAR=/usr/local/yuicompressor/build/yuicompressor-2.4.7.jar
# Run YUI Compressor
/usr/bin/java -jar $YUI_JAR "$@"


=AmiGO 2: Client software installation=
=AmiGO 2: Client software installation=


The AmiGO 2 client software can be used completely separately from the server software as long as a Solr server with a compliant schema is available ''somewhere''.
The AmiGO 2 client software can be used completely separately from the server software as long as a Solr server with a compliant schema is available ''somewhere''.
AmiGO 2 '''no longer depends''' on an additional download of BBOP JS, it now will use the bundled copy by default. '''Most users can skip the next paragraph'''.
For widget developers, since most of the JavaScript in AmiGO 2 relies on this library, the first step in installing AmiGO 2 is to get BBOP JS somehow available. While AmiGO 2 repository (below) comes with a packaged version of bbop.js in the _data/ directory (which is also the default when installing AmiGO 2), if you're interested in hacking around BBOP JS in conjunction with AmiGO 2, you'll have to build it yourself and change the BBOP_JS variable in the Makefile to the appropriate location so you can use the "make refresh" command. Since we don't want to repeat ourselves here, full downloading and building instructions can be found on its homepage at https://github.com/kltm/bbop-js . Scroll down a bit to "Local (building)" to get to the pertinent information.


==AmiGO 2 code==
==AmiGO 2 code==


This software contains the GO-specific things that BBOP JS does not support--the client (web server) code for AmiGO 2. For the sake of convenience (and using a similar directory layout to the developers), we'll get it like:
This software contains the client (web server) code for AmiGO 2. For the sake of convenience (and using a similar directory layout to the developers), we'll get it like:


  mkdir -p ~/local/src/git
  mkdir -p ~/local/src/git
Line 194: Line 184:
== Configuration ==
== Configuration ==


Next, we have to configure the AmiGO 2 installation given what we have with our BBOP JS and Apache 2 installations. The file to edit is:
Next, we have to configure the AmiGO 2 installation given what we have with our Apache 2 installations. The file to edit is:


  conf/amigo.yaml
  conf/amigo.yaml
Line 211: Line 201:
The configuration file for our test setup looks like:
The configuration file for our test setup looks like:


---
[https://github.com/geneontology/amigo/blob/master/conf/examples/amigo.yaml.localhost Example configuration]
AMIGO_1X_PUBLIC_CGI_BASE_URL:
 
  comment: The public URL of an available AmiGO 1.x instance for link references.
After "npm install", the "gulp" command can be found in "./node_modules/.bin/gulp". The following instructions assume that this is in your path, but if not, feel free to use the full path of gulp.
  type: url
  value: http://amigo1.geneontology.org/cgi-bin/amigo
AMIGO_BALANCER:
  comment: 'Mark this AmiGO 2 installation as behind a load balanacer or similar system. The practical effect of this flag is that if AmiGO detects *.error files (usually created by the global-message.pl script), it will return a 503 page signalling any upstream system that the instance is not currently ready.'
  type: boolean
  value: 0
AMIGO_BETA:
  comment: Mark the AmiGO site as BETA (triggers some changes in variables and installed features).
  type: boolean
  value: 1
AMIGO_DYNAMIC_PATH:
  comment: The full path to the location of the perl binaries.
  type: directory
  value: /home/foo/local/src/git/amigo/perl/bin
AMIGO_DYNAMIC_URL:
  comment: The URL that AMIGO_DYNAMIC_PATH maps to.
  type: url
  value: http://localhost:9999
AMIGO_GOOGLE_ANALYTICS_ID:
  comment: Google analytics ID. Optional.
  type: string
  value: ''
AMIGO_LAYOUT_SEARCH:
  comment: A whitespace separated list of class IDs to use for live search tabs on the search page.
  type: string
  value: annotation ontology bioentity
AMIGO_PUBLIC_GOLR_URL:
  comment: The public URL of the GOlr server (including slash).
  type: url
  value: http://golr.berkeleybop.org/
AMIGO_ROOT:
  comment: 'Unless you are doing something strange, the parent directory of this file.'
  type: directory
  value: /home/foo/local/src/git/amigo
AMIGO_SERVER_NAME:
  comment: 'The name by which your AmiGO instance wishes to known--think load balancer or cache situations. If you would rather this information not be reported, leave blank.'
  type: string
  value: localhost
AMIGO_STATIC_PATH:
  comment: The full path to a usable static document directory where static and staging are located.
  type: directory
  value: /home/foo/local/src/git/amigo/static
AMIGO_STATIC_URL:
  comment: The URL for the static content. Can be local or remote.
  type: url
  value: http://localhost:9999/static
AMIGO_TERM_REGEXP:
  comment: The regular expressions to identify internal ontology ids.
  type: string
  value: 'all|GO:[0-9]{7}'
AMIGO_VERBOSE:
  comment: Log verbose AmiGO debugging messages.
  type: boolean
  value: 1
AMIGO_WORKING_PATH:
  comment: Please enter the full path to readable/writable directory that will be used for things like temporary files and logs (if enabled).
  type: directory
  value: /tmp
GOLR_TIMESTAMP_LOCATION:
  comment: 'This is the full path to the timestamp file that is optionally produced by OWLTools during GOlr loading. It contains information about download attempt times for the different loaded files. If not found or unusable, AmiGO 2 will gracefully ignore it.'
  type: file
  value: /home/foo/local/src/git/amigo/golr_timestamp.log
GO_BBOPJS_LOCATION:
  comment: 'If this is not redefined to a new location (e.g. the BBOP-JS staging/ directory, it will use the bbop.js bundle packaged with AmiGO 2 (the one used in the unit tests) in the external/ directory.'
  type: file
  value: /home/foo/local/src/git/amigo/external/bbop.js
GO_YAML_LOCATION:
  comment: 'The full path to the YAML config files directory. In that directory, all files like *config.yaml will be scanned for use.'
  type: directory
  value: /home/foo/local/src/git/amigo/metadata


If everything is correct, installing a dev instance AmiGO 2 should be as simple as:
If everything is correct, installing a dev instance AmiGO 2 should be as simple as:


  make install
  npm install
gulp install


This will create configuration and amigo2 JavasScript packages, as well as run unit tests and create API docs.
This will create configuration and amigo2 JavasScript packages, as well as run unit tests and create API docs.
In addition, if you are interested in viewing the data statistics pages within AmiGO, you would then run:
gulp cache


Further more, there will be variations for whether we are developing on the embedded server (all perl) or onto an Apache 2 CGI setup. Both of these are given different subsections.
Further more, there will be variations for whether we are developing on the embedded server (all perl) or onto an Apache 2 CGI setup. Both of these are given different subsections.
Line 301: Line 226:
Running should now be as simple as a:
Running should now be as simple as a:


  make run
  gulp run-amigo


The dev server should now be running on port :9999 (given the amigo.yaml above).
The dev server should now be running on port :9999 (given the amigo.yaml above).
Line 350: Line 275:
You may want to check out various AmiGO 2 system scripts referred to in the message at the end of the installation process: global-message.pl, clean-filesystem.pl, and blank-kvetch.pl. More information can be found in [[AmiGO_2_Manual:_Administration]].
You may want to check out various AmiGO 2 system scripts referred to in the message at the end of the installation process: global-message.pl, clean-filesystem.pl, and blank-kvetch.pl. More information can be found in [[AmiGO_2_Manual:_Administration]].


==All Glory to the Makefile==
==All Glory to the gulpfile==


For some time now, all day-to-day development, handling, distribution, and loading of [[AmiGO_2]] and [[GOlr]] have been handled with the Makefiles in amigo/Makefile and amigo/golr/Makefile respectively. If it cannot be done by changing your environment variables and rerunning the target that you want, you might want to reconsider what you're doing.
For some time now, all day-to-day development, handling, distribution, and loading of [[AmiGO_2]] and [[GOlr]] have been handled with the gulpfile.js in amigo/. If it cannot be done by changing your environment variables and rerunning the target that you want, you might want to reconsider what you're doing.
 
While the Makefiles are not yet fully documented (will add them to our NaturalDocs real soon now), they may be worth perusing. This is issue: https://github.com/kltm/amigo/issues/26
 
== <strike>Images</strike> ==
 
<strike>
A KVM image of Ubuntu 12.04.2 of the above configuration (created 2013-07-24) for the AmiGO 2 fronted (only) is available at "http://api.berkeleybop.org/images/"; the name of the image is "a2.img". Just paste those together (I don't want to have the direct link here). The md5sum is 6d60b045fc38fdd8d98222367a506c51.
 
* it is an 8GB image, which should have no non-AmiGO 2 services or software
* hostname: amigo
* core username: amigo
* pass: ogima
* in conf/amigo.yaml, the wired the address is 192.168.122.213 for testing--you'll need to change that
* the GOlr server is the current public one--you'll need to create your own backend and change it
* ServerName is not set for Apache.
</strike>
 
[[Category:AmiGO_2_Manual]]
[[Category:AmiGO_2]]
[[Category:Software]]

Latest revision as of 19:34, 8 August 2016

Preliminaries

The current released production version is 2.4.x.

These instructions are intended to illustrate how to install the AmiGO 2 client software on a Ubuntu 16.04 LTS. With careful reading, these instructions should be generalizable to other cases. The installation of this client software only covers AmiGO 2 and some Apache 2 configuration.

For information on the data server that AmiGO 2 uses (covering the installation of and loading using the server software: GOlr schema for Solr, OWLTools, etc.), please see the Installation section of the GOlr documentation.

Handling permissions and the like for the various servers is not explicitly covered in this document, but I'd like to make the observation that AmiGO 2 and client stuff can be handled as any user except for the necessary Apache 2 installation (when not using dev deployment directly) and the Solr/GOlr.

Packages

Perl

In general, package managers are the preferred way of getting libraries onto a system in a consistent, reversible, and secure manner. However, often you'll find yourself in a situation where you'll need to use CPAN because your preferred distribution or system does not have a robust set of perl libraries.

For users who need to use CPAN, but don't want to mess too deeply with their system (CPAN can turn into a nightmare when installing globally), we'd suggest using CPAN's local::lib and just putting that into your development and run paths: https://metacpan.org/pod/local::lib#The-bootstrapping-technique .

JavaScript

Given that, in the time that it has taken you to read this, yet another revolution has occurred in JavaScript, staying on top of packages can be hard and distributions do not reliably track them. For this, assuming that you have a recent version of node we do most of our package management with NPM.

To get a recent version of node, we have been using

deb https://deb.nodesource.com/node_0.4.x xenial main

Which was the end product of following the instructions here: https://nodejs.org/en/download/package-manager/ . As you can see, we're working of of the still supported LTS 4.x.

Ubuntu/Debian

The following packages are needed (this might need some revision):

sudo apt-get -u install git make kwalify libcgi-application-perl libcgi-application-server-perl libcgi-application-plugin-session-perl libcgi-application-plugin-tt-perl libclone-perl libconfig-yaml-perl libdbi-perl libdbd-sqlite3-perl libdbd-mysql-perl libdata-formvalidator-perl libossp-uuid-perl libfile-type-perl libfile-slurp-perl libfreezethaw-perl libgraph-perl libgraphviz-perl libjson-xs-perl liburi-perl libwww-mechanize-perl libxml-libxml-perl libxml-xpath-perl

The package libsql-tokenizer-perl is also required (by GOOSE for query checking), but may no longer be available in repositories. If you're interested in using a locally configured GOOSE, you may need to manually install this package along the lines of:

sudo apt-get -u install dh-make-perl
cd /tmp
dh-make-perl --build --cpan SQL::Tokenizer
sudo dpkg -i libsql-tokenizer-perl_???_all.deb

Depending on what you're doing with CPAN in this case, you may also need to install a helper library first:

sudo apt-get -u install liblocal-lib-perl

CentOS/RedHat example

An example comparative table of packages installed between Ubuntu and CentOS; in this case, the perl packages were manually installed rather than using CPAN (see CPAN note above).

Ubuntu CentOS install type
git git yum install
kwalify kwalify gem install
libcgi-application-perl CGI-Application-4.50 CGI-Application-Standard-Config-1.01 CGI-Application-Plugin-ErrorPage-1.21 CGI-Application-Plugin-Redirect-1.00 rpm -i
libcgi-application-plugin-session-perl CGI-Application-Plugin-Session-1.03 rpm -i
libcgi-application-plugin-tt-perl CGI-Application-Plugin-TT-1.05 rpm -i
libclone-perl Clone-0.34 rpm -i
libconfig-yaml-perl Config-YAML-1.42 rpm -i
libdata-formvalidator-perl Data-FormValidator-4.81 rpm -i
libdbd-mysql-perl DBD-mysql-4.023 rpm -i
libdbd-sqlite3-perl DBD-SQLite-1.39 rpm -i
libdbi-perl DBI-1.628 rpm -i
libfile-slurp-perl File-Slurp-9999.19 rpm -i
libfile-type-perl File-Type-0.22 rpm -i
libfreezethaw-perl FreezeThaw-0.5001 rpm -i
libgraph-perl Graph-0.96 rpm -i
libgraphviz-perl GraphViz-2.14 rpm -i
libjson-xs-perl JSON-XS-2.34 rpm -i
libossp-uuid-perl uuid-1.6.2.tar.gz rpm -i
liburi-perl URI-1.60 rpm -i
libwww-mechanize-perl WWW-Mechanize-1.72 rpm -i
libxml-xpath-perl XML-XPath-1.13 rpm -i
make make-3.81-20.el6.x86_64 yum install
naturaldocs NaturalDocs-1.52.zip See Note [1]
rsync rsync-3.0.6-9.el6.x86_64 yum install
[not defined] jline yum install
[not defined] jpackage-utils yum install

Note [1]: NaturalDocs is a set of perl modules that is not distributed or installed in the standard fashion. It is available from http://www.naturaldocs.org/ and is distributed as a .zip file. The zip file does not create a subdirectory, so it simply extracts the files into the current directory. It's probably best to create a directory first (in our case, we usr /usr/local/naturaldocs), copy the zip file into the directory, then unzip the zip file there.

AmiGO 2: Client software installation

The AmiGO 2 client software can be used completely separately from the server software as long as a Solr server with a compliant schema is available somewhere.

AmiGO 2 code

This software contains the client (web server) code for AmiGO 2. For the sake of convenience (and using a similar directory layout to the developers), we'll get it like:

mkdir -p ~/local/src/git
cd ~/local/src/git
git clone git://github.com/kltm/amigo.git
cd amigo

Configuration

Next, we have to configure the AmiGO 2 installation given what we have with our Apache 2 installations. The file to edit is:

conf/amigo.yaml

If you do not have a conf/amigo.yaml file, you can start from the default template with:

cp conf/.initial_values.yaml conf/amigo.yaml

The most important (and fiddly) part of installing AmiGO 2 is this configuration file. For our case, in a local testing environment, the resulting amigo.yaml might look like the following. Please note that:

  • variables names have occasionally been changed, so this might not be exactly like your output
  • we are using a random local IP address for testing--you will have to change that
  • we are using the username foo
  • for the time being we're using a remote GOlr server--again you'll want to setup your own and change this

The configuration file for our test setup looks like:

Example configuration

After "npm install", the "gulp" command can be found in "./node_modules/.bin/gulp". The following instructions assume that this is in your path, but if not, feel free to use the full path of gulp.

If everything is correct, installing a dev instance AmiGO 2 should be as simple as:

npm install
gulp install

This will create configuration and amigo2 JavasScript packages, as well as run unit tests and create API docs.

In addition, if you are interested in viewing the data statistics pages within AmiGO, you would then run:

gulp cache

Further more, there will be variations for whether we are developing on the embedded server (all perl) or onto an Apache 2 CGI setup. Both of these are given different subsections.

Development server (embedded perl)

To run the local embedded development server, you will have to add the following package:

sudo apt-get -u install libcgi-application-server-perl

Running should now be as simple as a:

gulp run-amigo

The dev server should now be running on port :9999 (given the amigo.yaml above).

Production server (Apache 2)

While we are not officially supporting any particular web server or configuration, we make available here some notes about setting up a server on a new installation.

AmiGO 2 was developed with Apache 2 in mind, but I'd suspect that any CGI-capable web server would be sufficient. Configuring and running a web server is outside the scope of this document.

First, we need to install the Apache 2 web server:

sudo apt-get -u install apache2

If you are going to use the suggested ModMacro, you'll also need:

sudo apt-get -u install libapache2-mod-macro
sudo a2enmod macro

Great, now we need to get Apache setup to aim at our source/deployment directory. We have a template to work with using conf/amigo2-apache.conf.

sudo cp conf/amigo2-apache.conf /etc/apache2/sites-available/

This is easily changed by changing sjcarbon to foo. Now start the server.

sudo a2ensite amigo2-apache
sudo a2enmod rewrite
sudo a2enmod cgi
sudo service apache2 restart

You may be able to check the server now at: http://localhost:9999

You may run into permission issues for newer versions of Apache 2 like

Forbidden

You don't have permission to access /amigo on this server.

In this case, please see these documents:

Afterwards

System scripts

You may want to check out various AmiGO 2 system scripts referred to in the message at the end of the installation process: global-message.pl, clean-filesystem.pl, and blank-kvetch.pl. More information can be found in AmiGO_2_Manual:_Administration.

All Glory to the gulpfile

For some time now, all day-to-day development, handling, distribution, and loading of AmiGO_2 and GOlr have been handled with the gulpfile.js in amigo/. If it cannot be done by changing your environment variables and rerunning the target that you want, you might want to reconsider what you're doing.