Page tree
Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 163 Next »

This documentation is for Release 5.x. For documentation on previous releases, please select from the options below.

  • Release 1.x version of this page: v.81
  • Release 2.x version of this page: v.87.
  • Release 3.0-3.1 version of this page: v.111.
  • Release 3.2 version of this page: v.116.
  • Release 4.0 version of this page: v.143.

 

These instructions provide a recipe for building your own all-in-one Avalon system from scratch on CentOS or Red Hat Enterprise Linux, version 6.x is supported, 7.x is experimental.  Please note that while an all-in-one installation as outlined here is certainly suitable for testing and demos,  a single, all-in-one, server may not be suitable for production environments.  For recipes on building each piece of the Avalon system on different hosts, (as would typically be done for a production environment), please see the instructions for (standalone) environments here:  Fedora (standalone)MySQL (standalone)Web (standalone)Solr (standalone)Red5 (standalone), and Matterhorn (standalone).

 

Ready the Installation Environment

Install EPEL

rpm -ivh http://linux.mirrors.es.net/fedora-epel/6/i386/epel-release-6-8.noarch.rpm

This package has libyaml-devel which is required by ruby and not provided by Redhat.

On CentOS 7, please use the EPEL 7 package

rpm -ivh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm


Make sure a valid hostname is resolvable

The default hostname is “avalon.dev”, so name the machine this and enter it into /etc/hosts

# hostname
avalon.dev
# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4 avalon.dev

Configure iptables 

The Avalon Media System requires several ports to be open to client browsers.

Here are the port settings that will need to be configured:

PortPurposeExternal?
80HTTP (Avalon)Yes
1935RTMP (red5)Yes
5080HTTP (red5)No
8983HTTP (Fedora/Solr)No
8080HTTP (Matterhorn)Yes

 

The preferred method is to create a shell script that will do the work for you. Here is an example script that you should look through and customize as needed: avalon-iptables-config.sh

If you're connected over ssh, it might kick you off.

Save your script to /etc/sysconfig/avalon-iptables-config.sh, make it executable and run it.

chmod +x /etc/sysconfig/avalon-iptables-config.sh
/etc/sysconfig/avalon-iptables-config.sh

 

If you run into connection issues you can disable the iptables, by running "service iptables stop". This will completely drop your firewall. When finished troubleshooting run "service iptables start".

Disable SELinux

echo 0 > /selinux/enforce 
vim /etc/selinux/config #change the value of `SELINUX` from `enforcing` to `permissive`

You may have to disable SELinux completely if there's Passenger installation problem

vim /etc/selinux/config #change the value of `SELINUX` to `disabled`

Add the Avalon repository 

Create the Avalon repository config file:

vim /etc/yum.repos.d/avalon-public.repo

Append the following code:

[avalon_public]
name=Avalon Public RHEL repository
baseurl=http://repo.avalonmediasystem.org/x86_64
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-avalon
cost=150

Install and place the Avalon GPG key in the proper location:

curl http://repo.avalonmediasystem.org/RPM-GPG-KEY-avalon -o /etc/pki/rpm-gpg/RPM-GPG-KEY-avalon

Install development libraries and packages for building Ruby

yum groupinstall "Development Tools"
yum install readline-devel zlib-devel libyaml-devel libffi-devel openssl-devel libxml2-devel libxslt-devel cmake

Tomcat

Install Apache Tomcat with the avalonrepo repository configured and enabled.

yum install tomcat-7.0.32
service tomcat start
vim /usr/local/tomcat/conf/server.xml

CentOS 7 user may need to switch to EPEL6 temporarily for tomcat dependencies

on line 70, change the Tomcat connector port from 8080 to 8983:

<Connector port="8983" protocol="HTTP/1.1"

Add Tomcat manager user

By default, no user has access to the Tomcat Manager App. 

vim /usr/local/tomcat/conf/tomcat-users.xml

Define a user in tomcat-users.xml with access to the manager-gui role. Below is a very basic example.

<tomcat-users>
  <role rolename="manager-gui"/>
  <user username="admin" password="tomcat" roles="manager-gui"/>
</tomcat-users>

Restart Tomcat

service tomcat restart

See if you can log in to the manager app at http://<server host name>:8983/manager/html

Try it out on your local machine and on another machine. If you can't reach the app from another machine, your iptables might need to be changed to allow access.

Solr

Avalon makes use of Solr through the Blacklight gem for faceting and relevance-based searching.

Download the solr tarball and unpack it

Download Solr from http://www.apache.org/dyn/closer.cgi/lucene/solr/4.2.0

wget http://archive.apache.org/dist/lucene/solr/4.2.0/solr-4.2.0.tgz
tar xvf solr-4.2.0.tgz
mv solr-4.2.0 /usr/local/solr
cd /usr/local/solr/dist/
cp solr-4.2.0.war /usr/local/tomcat/webapps/solr.war
mv solr-4.2.0.war /usr/local/solr/solr.war

Add XML configurations

Edit /usr/local/tomcat/conf/Catalina/localhost/solr.xml and add

<Context docBase="/usr/local/solr/solr.war" debug="0" crossContext="true">
 <Environment name="solr/home" type="java.lang.String" value="/usr/local/solr" override="true"/>
</Context>

Edit /usr/local/solr/solr.xml and insert

<?xml version="1.0" encoding="UTF-8" ?>
  <solr persistent="true">
    <cores defaultCoreName="avalon" adminPath="/admin/cores" zkClientTimeout="${zkClientTimeout:15000}" hostPort="8983" hostContext="solr">
    <core instanceDir="avalon/" name="avalon"/>
  </cores>
</solr>

Download and Install Avalon specific Solr files from github

mkdir /usr/local/solr/lib
mv /usr/local/solr/dist/* /usr/local/solr/lib/
mv /usr/local/solr/contrib /usr/local/solr/lib/
 
wget https://github.com/avalonmediasystem/puppet-solr/archive/master.zip
unzip master
mv puppet-solr-master/files/avalon /usr/local/solr
chown -R tomcat7:tomcat /usr/local/solr/
chown -R tomcat7:tomcat /usr/local/tomcat
service tomcat restart

If you have successfully installed Solr you should be able to access the dashboard page at http://<server host name>:8983/solr

If you can't access the dashboard, check the tomcat logs in /usr/local/tomcat/logs/. Catalina.out and localhost.<date>.log usually provide the best information.

MySQL

Avalon uses MySQL for storing search queries, user data and roles, and as a back end for asynchronously sending requests to Matterhorn.

Install MySQL server 

yum install mysql-server
service mysqld start

Create databases and users

Enter the mysql monitor

#mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
...etc...
mysql>

Create a database for the Fedora Commons Repository software and add a user to it 

create database fedora3;
create user 'fcrepo'@'localhost' identified by 'fcrepo';
grant all privileges on fedora3.* to 'fcrepo'@'localhost';
flush privileges; 

Create a database for the Avalon web application and add a user to it

create database rails;
create user 'rails'@'localhost' identified by 'rails';
grant all privileges on rails.* to 'rails'@'localhost';
flush privileges;

Check your work and exit

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| fedora3            |
| mysql              |
| rails              |
| test               |
+--------------------+
5 rows in set (0.00 sec)
mysql> select user, host from mysql.user;
+--------+--------------+
| user   | host         |
+--------+--------------+
| root   | 127.0.0.1    |
|        | 129.79.32.87 |
| root   | 129.79.32.87 |
|        | localhost    |
| fcrepo | localhost    |
| rails  | localhost    |
| root   | localhost    |
+--------+--------------+
7 rows in set (0.00 sec)
 
mysql> exit;
Bye

See documentation for your version of MySQL Server for detailed syntax (http://dev.mysql.com/doc/refman/5.1/en/create-database.html )

Fedora Commons Repository

Download and run the fcrepo installer
wget http://sourceforge.net/projects/fedora-commons/files/fedora/3.6.2/fcrepo-installer-3.6.2.jar/download -O fcrepo-installer-3.6.2.jar
java -jar fcrepo-installer-3.6.2.jar

Use the following values in place of defaults

SettingValue
Installation type
custom
Fedora home directory
/usr/local/fedora
Fedora administrator password
fedoraPassword 
Fedora server host
localhost
Fedora application server context
fedora
Authentication requirement for API-A
false
SSL availability
true
SSL required for API-A
false
SSL required for API-M
true
Servlet engine
existingTomcat
Tomcat home directory
/usr/local/tomcat
Tomcat HTTP port
8983
Tomcat shutdown port
8005
Tomcat Secure HTTP port
8443
Keystore file
default
Keystore password
changeit
Keystore type
JKS
Database
mysql
MySQL JDBC driver
included
Database username
fcrepo
Database password
fcrepo

Accept remaining defaults then add permissions for Tomcat and restart Tomcat

chown -R tomcat7:tomcat /usr/local/fedora
service tomcat restart

Red5 Media Server

Red5 is an open source alternative to Adobe Media Server. If using the Adobe Media Server you can skip to the next step.

Create a red5 user.

useradd red5

Download and install Red5.

wget http://yumrepo-public.library.northwestern.edu/red5-1.0.1.tar.gz
tar xvf red5-1.0.1.tar.gz
mv red5-server-1.0 /usr/local/red5

Download the init script and add it to the init.d directory.

wget https://raw.github.com/avalonmediasystem/avalon-installer/master/modules/red5/templates/red5_init_script.erb -O red5_init_script.sh
mv red5_init_script.sh /etc/rc.d/init.d/red5

Give Red5 permissions to the red5 directory and the init script.

chown -R red5:red5 /usr/local/red5
chmod +x /etc/rc.d/init.d/red5

FFmpeg

Installation prerequisites

Install prerequisite packages using yum and the Avalon repository (note: mediainfo is pinned to 0.7.61-1 because of a bug with time fragment formatting in most recent version 0.7.87-1 in epel):

yum install SDL-devel a52dec-devel bzip2-devel faad2-devel freetype-devel frei0r-plugins-devel \
	gsm-devel imlib2-devel lame-devel libdc1394-devel libraw1394-devel librtmp-devel libtheora-devel \
	libva-devel libfaac-devel libvdpau-devel libstdc++-devel libvorbis-devel libvpx-devel \
	mediainfo-0.7.61-1 opencore-amr-devel opencv-devel openjpeg-devel openssl-devel schroedinger-devel \
	speex-devel texi2html vo-aacenc-devel x264-devel xvidcore-devel yasm zlib-devel

If CentOS 7 complains about libdc1394-devel not found

sudo yum -v --enablerepo=epel clean expire-cache
sudo yum --enablerepo=epel install libdc1394-devel

Install rpmdev-setuptree

yum install rpmdevtools

Install ffmpeg srpm

The following commands need to run under a user other than root. Change to the red5 user to continue.

su - red5

Run the ffmpeg installer

rpmdev-setuptree
rpm -ivh https://github.com/avalonmediasystem/avalon-installer/blob/master/files/ffmpeg/ffmpeg-2.4.2-1.el6.src.rpm?raw=true

# Retrieving https://github.com/avalonmediasystem/avalon-installer/blob/master/files/ffmpeg/ffmpeg-2.4.2-1.el6.src.rpm
#   1:ffmpeg                 warning: user makerpm does not exist - using root
# warning: group makerpm does not exist - using root
########################################### [100%]
# warning: user makerpm does not exist - using root
# warning: group makerpm does not exist - using root
# Build ffmpeg binary as non-root and install as root

rpmbuild -bb rpmbuild/SPECS/ffmpeg24.spec

Log back in as root and finish the install.

su - root
rpm -ivh /home/red5/rpmbuild/RPMS/x86_64/ffmpeg-*.rpm

You can also build a more modern ffmpeg from source, but not all versions work. Version 3.1 is known to work - check out the release/3.1 branch and build from there. If your modern ffmpeg uses fdk_aac instead of libfaac, adjust the Matterhorn etc/encoding/avalon.properties accordingly.

HTTPD

Install and start the httpd service.

yum install httpd
service httpd start

With newer httpd you may need to in /etc/httpd/conf.d/10-mod_rewrite.conf, replace RewriteLock line with Mutex sem

Matterhorn

Install Matterhorn

Create a user for Matterhorn and then install Matterhorn

useradd matterhorn
 
wget https://github.com/avalonmediasystem/avalon-felix/archive/1.4.x.tar.gz
tar xvf 1.4.x.tar.gz
mv avalon-felix-1.4.x /usr/local/matterhorn

wget --no-check-certificate https://raw.github.com/avalonmediasystem/config-files/master/matterhorn/matterhorn_init.sh
mv matterhorn_init.sh /etc/init.d/matterhorn

chmod +x /etc/init.d/matterhorn
chown -R matterhorn:matterhorn /usr/local/matterhorn

Add avalon user and create avalon directory.

useradd avalon
mkdir /var/www/avalon
chown -R avalon:avalon /var/www/avalon

Create and configure the media_path (upload) and streaming directories.

mkdir -p /usr/local/masterfiles 
chown avalon:avalon /usr/local/masterfiles

mkdir -p /usr/local/red5/webapps/avalon/streams
mkdir /var/avalon
mkdir -p /var/www/avalon/public/streams

chown red5:avalon /usr/local/red5/webapps/avalon/streams
chmod 0775 /usr/local/red5/webapps/avalon/streams
ln -s /usr/local/red5/webapps/avalon/streams /var/avalon/rtmp_streams
 
mkdir /var/avalon/hls_streams
chown matterhorn:matterhorn /var/avalon/hls_streams/
ln -s /var/avalon/hls_streams/ /var/www/avalon/public/streams

chmod 0775 /var/avalon/hls_streams/

Configure Matterhorn

Download Matterhorn config and verify property values.

wget --no-check-certificate https://raw.github.com/avalonmediasystem/config-files/master/matterhorn/config.properties
vim config.properties 
And verify the configuration of the streaming directories
org.opencastproject.streaming.directory=/var/avalon/rtmp_streams
org.opencastproject.hls.directory=/var/avalon/hls_streams

Also check in etc/load/org.opencastproject.organization-mh_default_org.cfg

prop.avalon.stream_base=file:///var/avalon/rtmp_streams

Move the config to the appropriate spot

mv config.properties /usr/local/matterhorn/etc/

Add matterhorn user to the avalon group.

usermod -G avalon matterhorn

Optional, but recommended to avoid problems with batch ingest: Change number of processes available to matterhorn user.

Apache Passenger and Ruby

Change current user to avalon then install RVM and ruby 2.2.5

su - root
yum install sqlite-devel
su - avalon
curl -L https://get.rvm.io | bash -s stable --ruby=2.2.5

Source the RVM shell (as avalon user) or close the terminal and open it back up.

source /home/avalon/.rvm/scripts/rvm
rvm use 2.2.5

Install Passenger via Gem (as avalon user)

gem install passenger

Check to make sure passenger installed in the expected location (as avalon user)

$ passenger-config --root
/home/avalon/.rvm/gems/ruby-2.2.5/gems/passenger-5.0.17

Install Passenger apache module requirements (as root)

su - root
yum install curl-devel httpd httpd-devel apr-devel apr-util-devel

Build passenger for your version of Apache and Ruby (as avalon user)

su - avalon
passenger-install-apache2-module #copy the suggested Apache configuration file settings for later

Create an apache configuration file (as root)

su - root
vim /etc/httpd/conf.d/passenger.conf

Example contents of /etc/httpd/conf.d/passenger.conf, which may need to be changed based off of the current version of ruby and passenger:

LoadModule passenger_module /home/avalon/.rvm/gems/ruby-2.2.5/gems/passenger-5.0.17/buildout/apache2/mod_passenger.so
<IfModule passenger_module>
  PassengerRoot /home/avalon/.rvm/gems/ruby-2.2.5/gems/passenger-5.0.17
  PassengerDefaultRuby /home/avalon/.rvm/wrappers/ruby-2.2.5/ruby
  PassengerMaxPoolSize 30
  PassengerPoolIdleTime 300
  PassengerMaxInstancesPerApp 0
  PassengerMinInstances 3
  PassengerSpawnMethod smart-lv2
</IfModule>

Apache security configuration

wget --no-check-certificate https://raw.github.com/avalonmediasystem/config-files/master/sbin/avalon_auth -O /usr/local/sbin/avalon_auth
chmod +x /usr/local/sbin/avalon_auth
wget --no-check-certificate https://raw.github.com/avalonmediasystem/config-files/master/apache/10-mod_rewrite.conf -P /etc/httpd/conf.d/

Create a virtual host for avalon in /etc/httpd/conf.d/avalon.conf

wget --no-check-certificate https://raw.github.com/avalonmediasystem/config-files/master/apache/20-avalon.conf -P /etc/httpd/conf.d/
vim /etc/httpd/conf.d/20-avalon.conf

In 20-avalon.conf add this line inside the VirtualHost tag:

RailsEnv development

If using SSL, the following fix should be added to address BEAST, POODLE, RC4 issues (after the SSLEngine on)

SSLProtocol all -SSLv2 -SSLv3
SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:!RC4:+HIGH:+MEDIUM:-LOW

Restart apache.  With apache running, check passenger-status

service httpd restart
su - avalon
which passenger-status
#> ~/.rvm/gems/ruby-2.2.5/bin/passenger-status

Avalon

Grab Avalon code from github

cd ~
git clone git://github.com/avalonmediasystem/avalon.git
cd avalon
git checkout master #make sure you are in the master branch (should be by default)
su - root
chown avalon:avalon /var/www/avalon/public/
su - avalon
mv public/* /var/www/avalon/public/
rmdir public
mv  * /var/www/avalon/

Configure Avalon 

If using vim with default settings and pasting the the code below, it will automatically comment out the last line. To prevent that, enable paste using the command :set paste and then use just ctrl+shift+v instead of going into insert mode.

Create /var/www/avalon/config/setup_load_paths.rb and add:

if ENV['MY_RUBY_HOME'] && ENV['MY_RUBY_HOME'].include?('rvm')
   begin
     gems_path = ENV['MY_RUBY_HOME'].split(/@/)[0].sub(/rubies/,'gems')
     ENV['GEM_PATH'] = "#{gems_path}:#{gems_path}@global"
     require 'rvm'
     RVM.use_from_path! File.dirname(File.dirname(__FILE__))
   rescue LoadError
     raise "RVM gem is currently unavailable."
   end
 end
 # If you're not using Bundler at all, remove lines bellow
 ENV['BUNDLE_GEMFILE'] = File.expand_path('../Gemfile', File.dirname(__FILE__))
 require 'bundler/setup'
cd /var/www/avalon/config
cp authentication.yml.example authentication.yml

Configure database settings

vim database.yml

Replace database.yml with the correct values for your production environment

production:
  adapter: mysql2
  host: localhost
  database: rails
  username: rails
  password: rails
  pool: 5
  timeout: 5000

Install the mysql2 adapter 

yum install mysql-devel
su - avalon
gem install activerecord-mysql2-adapter
gem install mysql2
su - root
yum install cmake #<--will be required for rugged gem
vim /var/www/avalon/Gemfile

Add this line to the Gemfile

gem 'mysql2', '~>0.3.20'

If you are using mysql instead of sqllite (as is done throughout this document), then comment out the following lines in the Gemfile

#    gem 'activerecord-jdbcsqlite3-adapter'
#    gem 'jdbc-sqlite3'
#    gem 'sqlite3'

Run the bundle install

su - avalon
cd /var/www/avalon
gem update debugger-ruby_core_source
gem install bundler
bundle install

Finish configuring Avalon

Edit /var/www/avalon/config/solr.yml

  development:
    url: http://localhost:8983/solr/

Edit /var/www/avalon/config/fedora.yml

  development:
    user: fedoraAdmin
    password: fedoraPassword
    url: http://127.0.0.1:8983/fedora

Create  /var/www/avalon/config/matterhorn.yml

development:
  url: http://matterhorn_system_account:CHANGE_ME@localhost:8080/

Create  /var/www/avalon/config/avalon.yml and base it off of /var/www/avalon/config/avalon.yml.example. Consult the documentation to customize this file for your installation. 

development:
  dropbox:
    path: '/var/avalon/dropbox/'
    upload_uri: 'sftp://localhost/var/avalon/dropbox'
    username: 'test'
    password: 'test'
    notification_email_address: ''
  domain:
    port: 80

Create the secrets.yml file:

cd /var/www/avalon/config
cp secrets.yml.example secrets.yml #<--need this file or the next rake command won't work
rake secret

 grab the output of rake secret and add it to secrets.yml where instructed.

More information: Configuration Files#config/secrets.yml

Create controlled_vocabulary.yml

cd /var/www/avalon/config
cp controlled_vocabulary.yml.example controlled_vocabulary.yml

 

Create the database using rake

su - avalon
cd /var/www/avalon
rake db:create

If you get an error message saying that you can't connect to the database, take a look at this post and follow some of the troubleshooting steps.

http://stackoverflow.com/questions/5376427/cant-connect-to-local-mysql-server-through-socket-var-mysql-mysql-sock-38

Run the database migrations

rake db:migrate

Set rails environment to development, if it has not defaulted to this. On the first line of /var/www/avalon/config/environment.rb make sure it says 'development'

ENV['RAILS_ENV'] ||= 'development'

Visit your new Avalon site!

You should be able to visit the webpage with just the hostname (ie http://localhost)

Click on "Sign in" in the upper right corner of the website main page. Set up a default identity with the following properties.

 

archivist1@example.com
password

 

This is a known identity with administrative privileges.

Start delayed_job

As avalon run

cd /var/www/avalon/
bundle exec rake delayed_job:start

Delayed Job handles background jobs.  DelayedJob logs to log/delayed_job.log in the avalon directory. 

Red5 Avalon Security Webapp

cd /usr/local/red5/webapps
wget https://github.com/avalonmediasystem/avalon-installer/raw/master/modules/avalon/files/red5/red5-avalon.tar.gz
tar xvzf red5-avalon.tar.gz
chown -R red5:red5 avalon/

Edit /usr/local/red5/webapps/avalon/WEB-INF/red5-web.properties

avalon.serverUrl=http://localhost/

Restart Red5

service red5 restart

If Red5 is installed and running you should be able to access http://avalon.dev:5080/

Dropbox

groupadd -r dropbox
useradd -r avalondrop
usermod -G dropbox avalon
mkdir -p /var/avalon/dropbox
chown avalondrop:dropbox /var/avalon/dropbox
chmod 2775 /var/avalon/dropbox

Edit /etc/ssh/sshd_config

# override default of no subsystems
Subsystem sftp internal-sftp
 
# Example of overriding settings on a per-user basis
#Match User anoncvs
# X11Forwarding no
# AllowTcpForwarding no
# ForceCommand cvs server
Match Group dropbox
ChrootDirectory /var/avalon
X11Forwarding no
AllowTcpForwarding no
ForceCommand internal-sftp

Restart SSH

service sshd restart

Batch ingest

To manually start a batch ingest job, run as avalon user

rake avalon:batch:ingest

To make batch ingest run automatically whenever a manifest is present, you need to add a cron job. This cron job can be created by the whenever gem from reading config/schedule.rb. To preview, run

whenever

this will translate content in schedule.rb to cron job syntax. Once verified, run the following to write job to crontab

whenever --update-crontab

You should get the cron job automatically if you were deploying from Capistrano.

Using the System

You can find specific information about using the system in the Collection Manager's Guide.  Sample content is available for your convenience.  Upload new items individually or by batch directly via SFTP using the avalondrop account you created above.

Configure additional feataures

Known Issues - a list of bugs, workarounds, and cautions.

Restarting the Server

Before you restart your Avalon server, you'll want to make sure all of the services necessary to run Avalon will start automatically after the restart. Run these commands once and you should be set:

chkconfig --level 345 tomcat on
chkconfig --level 345 mysqld on
chkconfig --level 345 sshd on
chkconfig --level 345 red5 on
chkconfig --level 345 httpd on
chkconfig --level 345 matterhorn on
  • No labels