Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Note

This documentation is for Release 37.x0 and above. For documentation on previous releases, please select from the options below.

  • Release 1.x version of this page
, see  For the
  •  
  • Release 2.x version of this page
, see 

 

These instructions provide a recipe for building your own all-in-one Avalon system from scratch on CentOS or Red Hat Enterprise Linux.  Please note that while an all-in-one installation as outlined here is certainly suitable for testing and demos,  a single, all-in-one, server may not be suitable for production environments.  For recipes on building each piece of the Avalon system on different hosts, (as would typically be done for a production environment), please see the instructions for (standalone) environments here:  Fedora (standalone)MySQL (standalone)Web (standalone)Solr (standalone)Red5 (standalone), and Matterhorn (standalone).

 

Table of Contents
maxLevel2

Ready the Installation Environment

Install EPEL

Code Block
rpm -ivh http://linux.mirrors.es.net/fedora-epel/6/i386/epel-release-6-8.noarch.rpm

This package has libyaml-devel which is required by ruby and not provided by Redhat.

Make sure a valid hostname is resolvable

The default hostname is “avalon.dev”, so name the machine this and enter it into /etc/hosts

Code Block
# hostname
avalon.dev
# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4 avalon.dev

...

  • Release 3.0-3.1 version of this page: v.111.
  • Release 3.2 version of this page: v.116.
  • Release 4.0 version of this page: v.143.
  • Release 5.x version of this page: v.163.
  • Release 6.0 version of this page: v.177
  • Release 6.3-6.5 version of this page: v.201

These instructions provide a recipe for building your own all-in-one Avalon system from scratch on CentOS or Red Hat Enterprise Linux version 7.x.  Please note that while an all-in-one installation as outlined here is certainly suitable for testing and demos, a single, all-in-one, server may not be suitable for production environments.


Table of Contents
maxLevel2

Ready the Installation Environment


Info

The instructions below require being run as root unless specifically noted otherwise.

Storage requirement

Avalon and components need about 20GB of disk space to install.

Open ports requirement

The Avalon Media System requires several ports to be open to client browsers.

Here are the port settings that will need to be configured:

PortPurposeExternal?
80HTTP (Avalon)Yes
1935RTMP (red5)Yes5080
8983HTTP (
red5
Solr)No
8983
8984HTTP (Fedora
/Solr
)No
8080
8980HTTP (
Matterhorn
Nginx)Yes

...

The preferred method is to create a shell script that will do the work for you. Here is an example script that you should look through and customize as needed: avalon-iptables-config.sh

...

Code Block
chmod +x /etc/sysconfig/avalon-iptables-config.sh
/etc/sysconfig/avalon-iptables-config.sh

 

If you run into connection issues you can disable the iptables, by running "service iptables stop". This will completely drop your firewall. When finished troubleshooting run "service iptables start".

Disable SELinux

Code Block
echo 0 > /selinux/enforce 
vim /etc/selinux/config #change the value of `SELINUX` from `enforcing` to `permissive`

...

Create the NUL repository config file:

Code Block
vim /etc/yum.repos.d/nul-public.repo

Append the following code:

Code Block
[nul_public]
name=NUL Library Public RHEL repository
baseurl=http://yumrepo-public.library.northwestern.edu/x86_64
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-nul
cost=150

Install and place the NUL GPG key in the proper location:

Code Block
curl http://yumrepo-public.library.northwestern.edu/RPM-GPG-KEY-nul -o /etc/pki/rpm-gpg/RPM-GPG-KEY-nul

Install development libraries and packages for building Ruby

...

You may have to disable SELinux completely if there's Passenger installation problem

Info
vim /etc/selinux/config #change the value of `SELINUX` to `disabled`

Reboot to apply change

Code Block
shutdown -r now

Install EPEL

This package has libyaml-devel which is required by ruby and not provided by Redhat.

Code Block
yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

Install development libraries and packages for building Ruby

Code Block
yum groupinstall "Development Tools"
yum install readline-devel zlib-devel libyaml-devel libffi-devel openssl-devel libxml2-devel libxslt-devel cmake

Tomcat

...

Install Java 8

Code Block
yum install tomcatjava-7.0.32
service tomcat start
vim /usr/local/tomcat/conf/server.xml

on line 70, change the Tomcat connector port from 8080 to 8983:

Code Block
<Connector port="8983" protocol="HTTP/1.1"

Add Tomcat manager user

By default, no user has access to the Tomcat Manager App. 

Code Block
vim /usr/local/tomcat/conf/tomcat-users.xml

Define a user in tomcat-users.xml with access to the manager-gui role. Below is a very basic example.

Code Block
<tomcat-users>
  <role rolename="manager-gui"/>
  <user username="admin" password="tomcat" roles="manager-gui"/>
</tomcat-users>

Restart Tomcat

Code Block
service tomcat restart

See if you can log in to the manager app at http://<server host name>:8983/manager/html

...

Solr

Avalon makes use of Solr through the Blacklight gem for faceting and relevance-based searching.

Download the solr tarball and unpack it

Download Solr from http://www.apache.org/dyn/closer.cgi/lucene/solr/4.2.0

Code Block
wget http://archive.apache.org/dist/lucene/solr/4.2.0/solr-4.2.0.tgz
tar xvf solr-4.2.0.tgz
mv solr-4.2.0 /usr/local/solr
cd /usr/local/solr/dist/
cp solr-4.2.0.war /usr/local/tomcat/webapps/solr.war
mv solr-4.2.0.war /usr/local/solr/solr.war

Add XML configurations

Edit /usr/local/tomcat/conf/Catalina/localhost/solr.xml and add

Code Block
<Context docBase="/usr/local/solr/solr.war" debug="0" crossContext="true">
 <Environment name="solr/home" type="java.lang.String" value="/usr/local/solr" override="true"/>
</Context>

Edit /usr/local/solr/solr.xml and insert

Code Block
<?xml version="1.0" encoding="UTF-8" ?>
  <solr persistent="true">
    <cores defaultCoreName="avalon" adminPath="/admin/cores" zkClientTimeout="${zkClientTimeout:15000}" hostPort="8983" hostContext="solr">
    <core instanceDir="avalon/" name="avalon"/>
  </cores>
</solr>

Download and Install Avalon specific Solr files from github

Code Block
mkdir /usr/local/solr/lib
mv /usr/local/solr/dist/* /usr/local/solr/lib/
mv /usr/local/solr/contrib /usr/local/solr/lib/
 
wget https://github.com/avalonmediasystem/puppet-solr/archive/master.zip
unzip master
mv puppet-solr-master/files/avalon /usr/local/solr
chown -R tomcat7:tomcat /usr/local/solr/
chown -R tomcat7:tomcat /usr/local/tomcat
service tomcat restart

If you have successfully installed Solr you should be able to access the dashboard page at http://<server host name>:8983/solr

If you can't access the dashboard, check the tomcat logs in /usr/local/tomcat/logs/. Catalina.out and localhost.<date>.log usually provide the best information.

MySQL

Avalon uses MySQL for storing search queries, user data and roles, and as a back end for asynchronously sending requests to Matterhorn.

Install MySQL server 

Code Block
yum install mysql-server
service mysqld start

Create databases and users

Enter the mysql monitor

Code Block
#mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
...etc...
mysql>

Create a database for the Fedora Commons Repository software and add a user to it 

Code Block
create database fedora3;
create user 'fcrepo'@'localhost' identified by 'fcrepo';
grant all privileges on fedora3.* to 'fcrepo'@'localhost';
flush privileges; 

Create a database for the Avalon web application and add a user to it

Code Block
create database rails;
create user 'rails'@'localhost' identified by 'rails';
grant all privileges on rails.* to 'rails'@'localhost';
flush privileges;

Check your work and exit

Code Block
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| fedora3            |
| mysql              |
| rails              |
| test               |
+--------------------+
5 rows in set (0.00 sec)
mysql> select user, host from mysql.user;
+--------+--------------+
| user   | host         |
+--------+--------------+
| root   | 127.0.0.1    |
|        | 129.79.32.87 |
| root   | 129.79.32.87 |
|        | localhost    |
| fcrepo | localhost    |
| rails  | localhost    |
| root   | localhost    |
+--------+--------------+
7 rows in set (0.00 sec)
 
mysql> exit;
Bye

See documentation for your version of MySQL Server for detailed syntax (http://dev.mysql.com/doc/refman/5.1/en/create-database.html )

Fedora Commons Repository

Download and run the fcrepo installer
Code Block
wget http://sourceforge.net/projects/fedora-commons/files/fedora/3.6.2/fcrepo-installer-3.6.2.jar/download
java -jar fcrepo-installer-3.6.2.jar

Use the following values in place of defaults

SettingValue
Installation type
custom
Fedora home directory
/usr/local/fedora
Fedora administrator password
fedoraPassword 
Fedora server host
localhost
Fedora application server context
fedora
Authentication requirement for API-A
false
SSL availability
true
SSL required for API-A
false
SSL required for API-M
true
Servlet engine
existingTomcat
Tomcat home directory
/usr/local/tomcat
Tomcat HTTP port
8983
Tomcat shutdown port
8005
Tomcat Secure HTTP port
8443
Keystore file
default
Keystore password
changeit
Keystore type
JKS
Database
mysql
MySQL JDBC driver
included
Database username
fcrepo
Database password
fcrepo

Accept remaining defaults then add permissions for Tomcat and restart Tomcat

Code Block
chown -R tomcat7:tomcat /usr/local/fedora
service tomcat restart

Red5 Media Server

Red5 is an open source alternative to Adobe Media Server. If using the Adobe Media Server you can skip to the next step.

Create a red5 user.

Code Block
useradd red5

Download and install Red5.

Code Block
wget http://red5.org/downloads/red5/1_0_1/red5-1.0.1.tar.gz
tar xvf red5-1.0.1.tar.gz
mv red5-server-1.0 /usr/local/red5

Download the init script and add it to the init.d directory.

Code Block
wget https://raw.github.com/avalonmediasystem/avalon-installer/master/modules/red5/templates/red5_init_script.erb -O red5_init_script.sh
mv red5_init_script.sh /etc/rc.d/init.d/red5

Give Red5 permissions to the red5 directory and the init script.

Code Block
chown -R red5:red5 /usr/local/red5
chmod +x /etc/rc.d/init.d/red5

FFmpeg

Installation prerequisites

Install prerequisite packages using yum and the NUL-public repository

Code Block
yum install SDL-devel a52dec-devel bzip2-devel faad2-devel freetype-devel frei0r-plugins-devel \
	gsm-devel imlib2-devel lame-devel libdc1394-devel libraw1394-devel librtmp-devel libtheora-devel \
	libva-devel libfaac-devel libvdpau-devel libstdc++-devel libvorbis-devel libvpx-devel \
	mediainfo opencore-amr-devel opencv-devel openjpeg-devel openssl-devel schroedinger-devel \
	speex-devel texi2html vo-aacenc-devel x264-devel xvidcore-devel yasm zlib-devel

Install rpmdev-setuptree

Code Block
yum install rpmdevtools

Install ffmpeg srpm

The following commands need to run under a user other than root. Change to the red5 user to continue.

Code Block
su - red5

Run the ffmpeg installer

Code Block
#rpmdev-setuptree
#rpm -ivh https://github.com/avalonmediasystem/avalon-installer/raw/master/files/ffmpeg/ffmpeg-1.2-59a.el6.src.rpm
Retrieving https://github.com/avalonmediasystem/avalon-installer/raw/master/files/ffmpeg/ffmpeg-1.2-59a.el6.src.rpm
   1:ffmpeg                 warning: user makerpm does not exist - using root
warning: group makerpm does not exist - using root
########################################### [100%]
warning: user makerpm does not exist - using root
warning: group makerpm does not exist - using root
Build ffmpeg binary as non-root and install as root
#rpmbuild -bb rpmbuild/SPECS/ffmpeg12.spec

Log back in as root and finish the install.

Code Block
su - root
rpm -ivh /home/red5/rpmbuild/RPMS/x86_64/ffmpeg-*.rpm

HTTPD

Install and start the httpd service.

Code Block
yum install httpd
service httpd start

Matterhorn

Install Matterhorn

Create a user for Matterhorn and then install Matterhorn

Code Block
useradd matterhorn
 
wget https://github.com/avalonmediasystem/avalon-felix/archive/release/3.0.0.tar.gz
tar xvf 3.0.0
mv avalon-felix-3.0.0 /usr/local/matterhorn

wget https://raw.github.com/avalonmediasystem/config-files/master/matterhorn/matterhorn_init.sh
mv matterhorn_init.sh /etc/init.d/matterhorn

chmod +x /etc/init.d/matterhorn
chown -R matterhorn:matterhorn /usr/local/matterhorn

Add avalon user and create avalon directory.

Code Block
useradd avalon
mkdir /var/www/avalon
chown -R avalon:avalon /var/www/avalon

Create and configure streaming directories.

Code Block
mkdir -p /usr/local/red5/webapps/avalon/streams
mkdir /var/avalon
mkdir -p /var/www/avalon/public/streams

chown red5:avalon /usr/local/red5/webapps/avalon/streams
chmod 0775 /usr/local/red5/webapps/avalon/streams
ln -s /usr/local/red5/webapps/avalon/streams /var/avalon/rtmp_streams
 
mkdir /var/avalon/hls_streams
chown root:root /var/avalon/hls_streams/
ln -s /var/avalon/hls_streams/ /var/www/avalon/public/streams

chmod 0775 /var/avalon/hls_streams/

Configure Matterhorn

Download Matterhorn config and verify property values.

Code Block
wget https://raw.github.com/avalonmediasystem/config-files/master/matterhorn/config.properties
vim config.properties 
And verify the configuration of the streaming directories
Code Block
org.opencastproject.streaming.directory=/var/avalon/rtmp_streams
org.opencastproject.hls.directory=/var/avalon/hls_streams

Move the config to the appropriate spot

Code Block
mv config.properties /usr/local/matterhorn/etc/

Add matterhorn user to the avalon group.

Code Block
usermod -G avalon matterhorn

Optional, but recommended to avoid problems with batch ingest: Change number of processes available to matterhorn user.

Apache Passenger and Ruby

Change current user to avalon then install RVM and ruby 1.9.3.

Code Block
su - avalon
curl -L https://get.rvm.io | bash -s stable --ruby=1.9.3

Source the RVM shell (as avalon user)

Code Block
source /home/avalon/.rvm/scripts/rvm

Install Passenger via Gem (as avalon user)

Code Block
gem install passenger

Check to make sure passenger installed in the expected location (as avalon user)

Code Block
$ passenger-config --root
/home/avalon/.rvm/gems/ruby-1.9.3-p545/gems/passenger-4.0.40

Install Passenger apache module requirements (as root)

Code Block
su - root
yum install curl-devel httpd httpd-devel apr-devel apr-util-devel

Build passenger for your version of Apache and Ruby (as avalon user)

Code Block
su - avalon
passenger-install-apache2-module #copy the suggested Apache configuration file settings for later

Create an apache configuration file (as root)

Code Block
su - root
vim /etc/httpd/conf.d/passenger.conf

Example contents of /etc/httpd/conf.d/passenger.conf, which may need to be changed based off of the current version of ruby and passenger:

Code Block
LoadModule passenger_module /home/avalon/.rvm/gems/ruby-1.9.3-p545/gems/passenger-4.0.40/buildout/apache2/mod_passenger.so
<IfModule passenger_module>
  PassengerRoot /home/avalon/.rvm/gems/ruby-1.9.3-p545/gems/passenger-4.0.40
  PassengerDefaultRuby /home/avalon/.rvm/wrappers/ruby-1.9.3-p545/ruby
  PassengerMaxPoolSize 30
  PassengerPoolIdleTime 300
  PassengerMaxInstancesPerApp 0
  PassengerMinInstances 3
  PassengerSpawnMethod smart-lv2
</IfModule>

Apache security configuration

Code Block
wget https://raw.github.com/avalonmediasystem/config-files/master/sbin/avalon_auth -O /usr/local/sbin/avalon_auth
chmod +x /usr/local/sbin/avalon_auth
wget https://raw.github.com/avalonmediasystem/config-files/master/apache/10-mod_rewrite.conf -P /etc/httpd/conf.d/

Create a virtual host for avalon in /etc/httpd/conf.d/avalon.conf

Code Block
wget https://raw.github.com/avalonmediasystem/config-files/master/apache/20-avalon.conf -P /etc/httpd/conf.d/
vim /etc/httpd/conf.d/20-avalon.conf

Add this line to 20-avalon under the VirtualHost tag:

Code Block
RailsEnv development

Restart apache.  With apache running, check passenger-status

Code Block
/etc/init.d/httpd restart
su - avalon
which passenger-status
#> ~/.rvm/gems/ruby-1.9.3-p448/bin/passenger-status

Avalon

Grab Avalon code from github

Code Block
cd ~
git clone git://github.com/avalonmediasystem/avalon.git
cd avalon
git checkout master #make sure you are in the master branch (should be by default)
mv public/* /var/www/avalon/public/
mv  * /var/www/avalon/

Configure Avalon 

Warning

If using vim with default settings and pasting the the code below, it will automatically comment out the last line. To prevent that, enable paste using the command :set paste and then use just ctrl+shift+v instead of going into insert mode.

Create /var/www/avalon/config/setup_load_paths.rb and add:

Code Block
if ENV['MY_RUBY_HOME'] && ENV['MY_RUBY_HOME'].include?('rvm')
   begin
     gems_path = ENV['MY_RUBY_HOME'].split(/@/)[0].sub(/rubies/,'gems')
     ENV['GEM_PATH'] = "#{gems_path}:#{gems_path}@global"
     require 'rvm'
     RVM.use_from_path! File.dirname(File.dirname(__FILE__))
   rescue LoadError
     raise "RVM gem is currently unavailable."
   end
 end
 # If you're not using Bundler at all, remove lines bellow
 ENV['BUNDLE_GEMFILE'] = File.expand_path('../Gemfile', File.dirname(__FILE__))
 require 'bundler/setup'
Code Block
cd /var/www/avalon/config
cp authentication.yml.example ./authentication.yml

...

Code Block
vim database.yml

Replace database.yml with the correct values for your development environment

Code Block
development:
  adapter: mysql2
  host: localhost
  database: rails
  username: rails
  password: rails
  pool: 5
  timeout: 5000

Install the mysql2 adapter 

Code Block
yum install mysql-devel
su - avalon
gem install activerecord-mysql2-adapter
gem install mysql2
su - root
vim /var/www/avalon/Gemfile

Add this line to the Gemfile

Code Block
gem 'mysql2', '~>0.3.11'

If you are using mysql instead of sqllite (as is done throughout this document), then comment out the following lines in the Gemfile

Code Block
#    gem 'activerecord-jdbcsqlite3-adapter'
#    gem 'jdbc-sqlite3'
#    gem 'sqlite3'

Run the bundle install

Code Block
su - avalon
cd /var/www/avalon
gem update debugger-ruby_core_source
bundle install

Finish configuring Avalon

Edit /var/www/avalon/config/solr.yml

Code Block
  development:
    url: http://localhost:8983/solr/

Edit /var/www/avalon/config/fedora.yml

Code Block
  development:
    user: fedoraAdmin
    password: fedoraPassword
    url: http://127.0.0.1:8983/fedora

Create  /var/www/avalon/config/matterhorn.yml

Code Block
development:
  url: http://matterhorn_system_account:CHANGE_ME@localhost:8080/

Create  /var/www/avalon/config/avalon.yml and base it off of /var/www/avalon/config/avalon.yml.example. Consult the documentation to customize this file for your installation. 

Code Block
development:
  dropbox:
    path: '/var/avalon/dropbox/'
    upload_uri: 'sftp://localhost/var/avalon/dropbox'
    username: 'test'
    password: 'test'
    notification_email_address: ''

Create the database using rake

Code Block
su - avalon
cd /var/www/avalon
rake db:create

If you get an error message saying that you can't connect to the database, take a look at this post and follow some of the troubleshooting steps.

http://stackoverflow.com/questions/5376427/cant-connect-to-local-mysql-server-through-socket-var-mysql-mysql-sock-38

Run the database migrations

Code Block
rake db:migrate

Set rails environment to development, if it has not defaulted to this. On the first line of /var/www/avalon/config/environment.rb make sure it says 'development'

Code Block
ENV['RAILS_ENV'] ||= 'development'

Visit your new Avalon site!

You should be able to visit the webpage with just the hostname (ie http://localhost)

Click on "Sign in" in the upper right corner of the website main page. Set up a default identity with the following properties.

 

Code Block
archivist1@example.com
password/password

 

This is a known identity with administrative privileges.

Start delayed_job

As avalon run

Code Block
cd /var/www/avalon/
bundle exec rake delayed_job:start

Delayed Job handles background jobs.  DelayedJob logs to log/delayed_job.log in the avalon directory. 

Red5 Avalon Security Webapp

Code Block
cd /usr/local/red5/webapps
wget https://github.com/avalonmediasystem/config-files/raw/master/red5/red5-avalon.tar.gz
tar xvzf red5-avalon.tar.gz
chown -R red5:red5 avalon/

Edit /usr/local/red5/webapps/avalon/WEB-INF/red5-web.properties

Code Block
avalon.serverUrl=http://localhost/

Restart Red5

Code Block
service red5 restart

If Red5 is installed and running you should be able to access http://avalon.dev:5080/

Dropbox

Code Block
groupadd -r dropbox
useradd -r avalondrop
usermod -G dropbox avalon
mkdir -p /var/avalon/dropbox
chown avalondrop:dropbox /var/avalon/dropbox
chmod 2775 /var/avalon/dropbox

Edit /etc/ssh/sshd_config

Code Block
# override default of no subsystems
Subsystem sftp internal-sftp
 
# Example of overriding settings on a per-user basis
#Match User anoncvs
# X11Forwarding no
# AllowTcpForwarding no
# ForceCommand cvs server
Match Group dropbox
ChrootDirectory /var/avalon
X11Forwarding no
AllowTcpForwarding no
ForceCommand internal-sftp

Restart SSH

Code Block
service sshd restart

...

1.8.0-openjdk

Main Components

MariaDB

Info
titleMariaDB

MariaDB is now the default database system for CentOS/RHEL7 and can be used interchangeably with MySql.  MySql or PostgreSQL can be substituted if desired.

Avalon uses MariaDB for storing search queries, user data and roles, and as a back end our encoding dashboard. 

Install MariaDB server 

Code Block
yum install mariadb-server
systemctl start mariadb

Fedora Commons Repository

Tomcat

Fedora runs as a webapp in Tomcat

Install Apache Tomcat 

Code Block
titleRHEL 7
yum install tomcat
vim /etc/tomcat/server.xml #line 71, change the Tomcat connector port from 8080 to 8984

Add Tomcat manager user

By default, no user has access to the Tomcat Manager App. Define a user in /etc/tomcat/tomcat-users.xml with access to the manager-gui role. Below is a very basic example.

Code Block
<tomcat-users>
  <role rolename="manager-gui"/>
  <user username="admin" password="<insert strong password here>" roles="manager-gui"/>
</tomcat-users>

Create Fedora user and database

Enter the mariadb client

Code Block
mysql
mariadb> create database fcrepo CHARACTER SET utf8 COLLATE utf8_general_ci;
mariadb> create user 'fcrepo'@'localhost' identified by '<fcrepo_password>';
mariadb> grant all privileges on fcrepo.* to 'fcrepo'@'localhost';

mariadb> create database rails CHARACTER SET utf8 COLLATE utf8_general_ci;
mariadb> create user 'rails'@'localhost' identified by '<rails_pasword>';
mariadb> grant all privileges on rails.* to 'rails'@'localhost';

mariadb> flush privileges;

Check your work and exit

Code Block
mariadb> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| fcrepo             |
| performance_schema |
| test               |
+--------------------+
5 rows in set (0.00 sec)
mariadb> exit;
Bye

Configure Tomcat for Fedora

Append the following to /etc/tomcat/tomcat.conf

Code Block
JAVA_OPTS="-Dfcrepo.modeshape.configuration=classpath:/config/jdbc-mysql/repository.json -Dfcrepo.mysql.username=fcrepo -Dfcrepo.mysql.password=<fcrepo_password> -Dfcrepo.mysql.host=localhost -Dfcrepo.mysql.port=3306 -Dfcrepo.home=/var/avalon/fedora/"

Download and run the fcrepo installer

Code Block
mkdir -p /var/avalon/fedora
chown tomcat:tomcat /var/avalon/fedora
wget https://github.com/fcrepo4/fcrepo4/releases/download/fcrepo-4.7.5/fcrepo-webapp-4.7.5.war -O /usr/share/tomcat/webapps/fedora4.war

Restart Tomcat

Code Block
systemctl restart tomcat

See if you can access Fedora's REST interface at http://<server host name>:8984/fedora4/rest

Try it out on your local machine and on another machine. If you can't reach the app from another machine, your iptables might need to be changed to allow access. If Fedora is not up, check the tomcat logs in /var/log/tomcat/. Catalina.out and localhost.<date>.log usually provide the best information.

Solr

Avalon makes use of Solr through the Blacklight gem for faceting and relevance-based searching.

Install prerequisites

Code Block
yum install lsof

Download the solr tarball and run the installation script

Download Solr from http://archive.apache.org/dist/lucene/solr/

Code Block
wget http://archive.apache.org/dist/lucene/solr/6.6.6/solr-6.6.6.tgz
tar xzf solr-6.6.6.tgz solr-6.6.6/bin/install_solr_service.sh --strip-components=2
bash ./install_solr_service.sh solr-6.6.6.tgz

By default, the script extracts the distribution archive into /opt, configures Solr to write files into /var/solr, and runs Solr as the solr user. Follow the linked guide if you wish to change these defaults.

Create Avalon core for Solr

Code Block
mkdir -p /tmp/avalon_solr/
wget https://raw.githubusercontent.com/avalonmediasystem/avalon/master/solr/config/solrconfig.xml -O /tmp/avalon_solr/solrconfig.xml
wget https://raw.githubusercontent.com/avalonmediasystem/avalon/master/solr/config/schema.xml -O /tmp/avalon_solr/schema.xml
su solr # Needs to run as solr user
/opt/solr/bin/solr create_core -c avalon -d /tmp/avalon_solr
exit

If you have successfully installed Solr you should be able to access the dashboard page at http://<server host name>:8983/solr

Instructions on how to manually start/stop Solr: https://cwiki.apache.org/confluence/display/solr/Running+Solr

Media Streaming Server

An HLS-enabled server like Wowza, Adobe Media Server (commercial) or Nginx + the HLS module (open-source) can take an mp4 created by Avalon and stream it on the fly.

Nginx instructions

Code Block
languagebash
titleInstall Nginx with vod module
rpm -ihv http://installrepo.kaltura.org/releases/kaltura-release.noarch.rpm
yum install kaltura-nginx

Add /etc/nginx/nginx.conf

Code Block
user nginx;
worker_processes 4;

events {
  worker_connections 1024;
}

http {
  server {
    listen 8980;

    vod_mode local;
    vod_last_modified 'Sun, 19 Nov 2000 08:52:00 GMT';
    vod_last_modified_types *;
    vod_metadata_cache metadata_cache 512m;
    vod_response_cache response_cache 128m;
    gzip on;
    gzip_types application/vnd.apple.mpegurl;
    open_file_cache          max=1000 inactive=5m;
    open_file_cache_valid    2m;
    open_file_cache_min_uses 1;
    open_file_cache_errors   on;

    location ~ ^/avalon/(?<stream>.+)/(?<resource>.+\.(?:m3u8|ts)) {
      alias /var/avalon/derivatives/$stream;
      vod hls;

      set $token "$arg_token";
      add_header X-Stream-Auth-Token "$token";
      
      sub_filter_types application/vnd.apple.mpegurl;
      sub_filter_once off;
      sub_filter '.ts' ".ts?token=$token";

      auth_request /auth;
      add_header Access-Control-Allow-Headers '*';
      add_header Access-Control-Expose-Headers 'Server,range,Content-Length,Content-Range';
      add_header Access-Control-Allow-Methods 'GET, HEAD, OPTIONS';
      add_header Access-Control-Allow-Origin '*';
      expires 100d;
    }

    location = /auth {
      # resolver 127.0.0.1;
      proxy_pass http://127.0.0.1/authorize.txt?token=$token&name=$stream;
      proxy_pass_request_body off;
      proxy_set_header Content-Length "";
      proxy_set_header X-Original-URI $request_uri;
    }
  }
}


Info

listen should use a public open port.

alias should point to where the actual stream files are.

proxy_pass needs changing if installing Nginx on a different server.

Add /lib/systemd/system/nginx.service

Code Block
[Unit]
Description=The NGINX HTTP and reverse proxy server
After=syslog.target network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t
ExecStart=/usr/sbin/nginx
ExecReload=/usr/sbin/nginx -s reload
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Add nginx user and let it own nginx stuff

Code Block
useradd -M -s /bin/nologin nginx
chown -R nginx:nginx /etc/nginx /var/log/nginx 

Make 8980 bindable and start nginx

Code Block
semanage port -a -t http_port_t  -p tcp 8980
systemctl start nginx

Later: Avalon config should be updated to be compatible with Nginx:

Code Block
streaming:
  server: :nginx
  http_base: 'http://localhost:8980/avalon'
  content_path: '/var/avalon/derivatives'


Warning

If you enable SSL on Avalon server, you should also enable SSL on the streaming server to avoid Mixed content warning.

FFmpeg & Mediainfo

Info
titleDownload prebuilt ffmpeg

The following prebuilt binaries are provided by a third party. Proceed with caution.

Download and install ffmpeg (for transcoding & thumbnails)

Code Block
mkdir -p /tmp/ffmpeg && cd /tmp/ffmpeg
curl https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz | tar xJ
cp `find . -type f -executable` /usr/bin/

Install Mediainfo (for technical metadata)

Code Block
yum install mediainfo

HTTPD

Install and start the httpd service.

Code Block
yum install httpd
systemctl start httpd

Apache Passenger and Ruby

Change current user to avalon then install RVM and ruby 2.4.1

Code Block
yum install ruby sqlite-devel # Needed to build Ruby using RVM.
useradd avalon
su - avalon
curl -L https://get.rvm.io | bash -s stable --ruby=2.5.5

Source the RVM shell (as avalon user) or close the terminal and open it back up.

Code Block
source /home/avalon/.rvm/scripts/rvm
rvm use 2.5.5
exit

Install Passenger apache module requirements (as root)

Code Block
yum install pygpgme curl
curl --fail -sSLo /etc/yum.repos.d/passenger.repo https://oss-binaries.phusionpassenger.com/yum/definitions/el-passenger.repo

yum install mod_passenger || yum-config-manager --enable cr && yum install mod_passenger

Create a virtual host for avalon

Code Block
wget --no-check-certificate https://raw.github.com/avalonmediasystem/config-files/master/apache/20-avalon.conf -P /etc/httpd/conf.d/
vim /etc/httpd/conf.d/20-avalon.conf

In 20-avalon.conf add this line inside the VirtualHost tag:

Code Block
RailsEnv production

If using SSL, the following fix should be added to address BEAST, POODLE, RC4 issues (after the SSLEngine on)

Code Block
SSLProtocol all -SSLv2 -SSLv3
SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:!RC4:+HIGH:+MEDIUM:-LOW

Modify /etc/httpd/conf.d/passenger.conf

Code Block
PassengerRuby /home/avalon/.rvm/rubies/ruby-2.5.5/bin/ruby

Validate passenger install and restart apache

Code Block
passenger-config validate-install
systemctl start httpd

Avalon

Grab Avalon code from github

Code Block
git clone https://github.com/avalonmediasystem/avalon.git /var/www/avalon
chown -R avalon:avalon /var/www/avalon

Set rails environment to production, if it has not defaulted to this. On the first line of /var/www/avalon/config/environment.rb make sure it says 'production'

Code Block
ENV['RAILS_ENV'] ||= 'production'

Configure database settings
Anchor
mysql2
mysql2

Code Block
cd /var/www/avalon/config
vim database.yml

Replace database.yml with the correct values for your production environment. Note that the pool setting should be equal or exceed the number of concurrent jobs in Sidekiq.

Code Block
production:
  adapter: mysql2
  host: localhost
  database: rails
  username: rails
  password: rails
  pool: 20
  timeout: 5000

Install the mysql2 adapter 

Code Block
yum install cmake #<--will be required for rugged gem
yum install mariadb-devel

Install gems

Run the bundle install

Code Block
# as root
yum install nodejs # Javascript runtime 

# as avalon
su - avalon
cd /var/www/avalon
gem install bundler
bundle install --with mysql production --without development test
exit

Finish configuring Avalon

Edit /var/www/avalon/config/solr.yml and /var/www/avalon/config/blacklight.yml

Code Block
  production:
    url: http://localhost:8983/solr/avalon

Edit /var/www/avalon/config/fedora.yml

Code Block
production:
    user: fedoraAdmin
    password: fedoraAdmin
    url: http://127.0.0.1:8984/fedora4/rest
    base_path: ""

Create streaming directory

Code Block
# as root
mkdir -p /var/avalon/derivatives
chown avalon:avalon /var/avalon/derivatives


Info
titleAvalon config file

Avalon settings now live in /var/www/avalon/config/settings.yml. The default values should be sufficient to start with.

They can be selectively overwritten by creating a settings/<environment>.yml, or by using environment variables. Consult the config gem doc to understand how it works, or Avalon's documentation to customize this file for your installation.

Let Avalon know where your HLS streams are

Code Block
titleconfig/settings/production.local.yml
streaming:
  server: :nginx
  http_base: 'http://localhost:8980/avalon'
  content_path: '/var/avalon/derivatives'

Anchor
secrets
secrets
Change the secrets.yml file:

Code Block
export RAILS_ENV=production
rake secret

grab the output of rake secret and add it to secrets.yml where instruSTDOUTSTDOUTSTDOUTcted.

More information: Configuration Files#config/secrets.yml

Create controlled_vocabulary.yml

Code Block
cp config/controlled_vocabulary.yml.example config/controlled_vocabulary.yml

If you get an error message saying that you can't connect to the database, take a look at this post and follow some of the troubleshooting steps.

http://stackoverflow.com/questions/5376427/cant-connect-to-local-mysql-server-through-socket-var-mysql-mysql-sock-38

Run the database migrations

Code Block
rake db:migrate

Install yarn and node modules

Code Block
# as root
curl --silent --location https://dl.yarnpkg.com/rpm/yarn.repo | sudo tee /etc/yum.repos.d/yarn.repo
yum install yarn

# as avalon
su - avalon
cd /var/www/avalon
yarn install

Precompile assets

Code Block
# as avalon
RAILS_ENV=production bundle exec rake assets:precompile

Restart Apache

Code Block
# as root
systemctl restart httpd

Install ImageMagick

Code Block
# as root
yum install imagemagick

Sidekiq

Avalon uses Sidekiq for background processing, which relies on Redis as its key-value store.

Install Redis

Code Block
# as root
yum install redis
systemctl start redis

Install Sidekiq

Code Block
# as root
wget https://raw.githubusercontent.com/mperham/sidekiq/master/examples/systemd/sidekiq.service -O /lib/systemd/system/sidekiq.service

Edit the following lines in sidekiq.service 

Code Block
WorkingDirectory=/var/www/avalon
ExecStart=/bin/bash -lc '/home/avalon/.rvm/gems/ruby-2.5.5/bin/bundle exec sidekiq -e production'
User=avalon
Group=avalon


Code Block
# as root
systemctl start sidekiq

Sidekiq logs to STDOUT. 

Warning
titletmp Error after uploading file

When ingesting a media file, you may encounter an error message saying that file:///tmp/filename can’t be accessed or located.  This may result from the protected temp file settings that are defaults in CentOS 7.  Fix by changing “true” to “false” for PrivateTmp in these files in /usr/lib/systemd/system:

sidekiq.service  PrivateTmp=false
nginx.service   PrivateTmp=false
httpd.service   PrivateTmp=false

Additional Configurations

Dropbox

Code Block
groupadd -r dropbox
useradd -r avalondrop
usermod -G dropbox avalon
mkdir -p /srv/avalon/dropbox
chown avalondrop:dropbox /srv/avalon/dropbox
chmod 2775 /srv/avalon/dropbox

Edit /etc/ssh/sshd_config

Code Block
# override default of no subsystems
Subsystem sftp internal-sftp
 
# Example of overriding settings on a per-user basis
#Match User anoncvs
# X11Forwarding no
# AllowTcpForwarding no
# ForceCommand cvs server
Match Group dropbox
ChrootDirectory /srv/avalon
X11Forwarding no
AllowTcpForwarding no
ForceCommand internal-sftp

Restart SSH

Code Block
service sshd restart

Batch ingest

To manually start a batch ingest job, run as avalon user

Code Block
rake avalon:batch:ingest

To make batch ingest run automatically whenever a manifest is present, you need to add a cron job. This cron job can be created by the whenever gem from reading config/schedule.rb. To preview, run

Code Block
whenever

this will translate content in schedule.rb to cron job syntax. Once verified, run the following to write job to crontab

Code Block
whenever --update-crontab

You should get the cron job automatically if you were deploying from Capistrano.

Authentication Strategy

Avalon comes with Persona by default but it can be configured to work with other authentication strategies by using the appropriate omniauth gems. The following example is applicable to Indiana University CAS, it may need some adjustments in order to work with other CAS implementation.

Add to Gemfile

Code Block
gem 'net-ldap'
gem 'omniauth-cas', :git => "https://github.com/cjcolvar/omniauth-cas.git"

Install new gems

Code Block
bundle install

Add to config/initializers/my-ldap.rb

Code Block
module Avalon
      MY_GUEST_LDAP = Net::LDAP.new
      MY_GUEST_LDAP.host = "eads.myuni.edu"
      MY_GUEST_LDAP.authenticate 'cn=******,ou=Accounts,dc=eads,dc=myuni,dc=edu', '******'

      GROUP_LDAP = Net::LDAP.new
      GROUP_LDAP.host = "ads.myuni.edu"
      GROUP_LDAP.authenticate 'cn=******,ou=Accounts,dc=ads,dc=myuni,dc=edu', '******'
      GROUP_LDAP_TREE = "dc=ads,dc=myuni,dc=edu"
end

Add config/initializers/user_auth_cas.rb

Code Block
require 'net/ldap'

User.instance_eval do
  def self.find_for_cas(access_token, signed_in_resource=nil)
    logger.debug "#{access_token.inspect}"
    #data = access_token.info
    username = access_token.uid
    email = nil

    user = User.where(:username => username).first

    unless user
      if email.nil?
        tree = "dc=ads,dc=myuni,dc=edu"
        filter = Net::LDAP::Filter.eq("cn", "#{username}")
        email = Avalon::GROUP_LDAP.search(:base => tree, :filter => filter, :attributes=> ["mail"]).first.mail.first
      end
      user = User.find_or_create_by_username_or_email(username, email)
      raise "Finding user (#{ user }) failed: #{ user.errors.full_messages }" unless user.persisted?
    end
    user
  end
end

Add to config/settings/production.local.yml

Code Block
auth:
  configuration:
    - :name: My University
      :logo: my_logo.png
      :provider: :cas
      :params:
        :host: cas.myuni.edu
        :login_url: /cas/login
        :service_validate_url: /cas/validate
        :logout_url: /cas/logout
        :ssl: true

Using the System

You should be able to visit the webpage with just the hostname (ie http://localhost)

Create an admin account

You can  create an account from the command line in the root of your avalon install:

Code Block
bundle exec rake avalon:user:create avalon_username=user@example.com avalon_password=password avalon_groups=administrator

Additional information

You can find specific information about using the system in the Collection Manager's Guide.  Sample content is available for your convenience.  Upload new items individually or by batch directly via SFTP using the avalondrop account you created above.

Configure additional feataures

Known Issues - a list of bugs, workarounds, and cautions.

...

Code Block
chkconfig --level 345 tomcat on
chkconfig --level 345 mariadb on
chkconfig --level 345 mysqldnginx on
chkconfig --level 345 sshd on
chkconfig --level 345 red5redis on
chkconfig --level 345 httpdsidekiq on
chkconfig --level 345 matterhornhttpd on