Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Note
titleDon't follow these instructions

There is a script which does all of this and more automatically:

https://github.com/avalonmediasystem/avalon-packer#usage



These are the instructions for creating an end-user VM from scratch.

...

  • Using VirtualBox:
    • 1 Core
    • 3G RAM
    • 500G System Disk (VDI, Dynamically Allocated)
    • Set network to "Attached to Bridged Adapter"

Base OS Install

  • Centos 6.4 x86-64
  • English language and keyboard
  • hostname: localhost.localdomain (the default)
    • if it is localhost it should set it to the dhcp hostname provided.
  • eth0:
    • Clear "Device MAC address"
    • Connect automatically
    • IPv4 Settings
      • Automatic (DHCP)

  • set timezone to america/indiana/indianapolis, clock uses utc
  • root password is 'changeme'
  • partitioning:
    • custom layout
      • /dev/sda1:  500M ext4 mounted as /boot
      • /dev/sda2:  (remainder of disk) Physical volume
      • Volume Group:  vg_avalon
        • PVs: /dev/sda2
        • LV lv_swap 2G
        • LV lv_root (remainder of disk) ext4 mounted as /
  • Desktop install
  • Reboot / Firstboot
    • Leave all of the defaults, except...
    • Do Not create a new user
  • Log in as root
    • do not check "do not show me this again" to the "you are running as root" dialog.  Just close it.
    • yum update
    • open these ports in the firewall:
      • tcp/22
      • tcp/80
      • tcp/1935
      • tcp/18080
    • clean up eth0 configuration in /etc/sysconfig/network-scripts/ifcfg-eth0:
      • remove UUID
      • remove any DHCP_CLIENT_ID

        Code Block
        egrep -v '^(UUID|DHCP_CLIENT_ID)=' /etc/sysconfig/network-scripts/ifcfg-eth0 > /etc/sysconfig/network-scripts/ifcfg-eth0
  • OPTIONAL:
    • shut down the VM
    • at this point the vm is at "baseOS" and can be used as a cloning point
Note

There is a bug in virtual box VirtualBox with IPv6 handling over a bridged network.  When the bug is present the top transfer speeds for IPv6 are around 18kb/s.  Since RHEL (and derivatives) prefer IPv6 when looking up hosts, it is not uncommon to get an IPv6 host address for updates.  To disable IPv6 on the guest add these lines to /etc/sysctl.conf.  VirtualBox 4.2.12 was supposed to fix it but I'm not sure it has.

Code Block
net.ipv6.conf.all.disable_ipv6 = 1
 netnet.ipv6.conf.default.disable_ipv6 = 1

 


Avalon Install

  • OPTIONAL:
    • clone the baseOS image as puppet-YYYYMMDDYYMMDD
  • Use the installation notes from https://github.com/avalonmediasystem/avalon-installer Manual Installation of Avalon Using Puppet
    • When setting FACTER_* environment variables, set
      • FACTER_avalon_dropbox_password to "changeme"
      • FACTER_avalon_hostname public_address to "localhost.localdomain"
  • HACK:
    • add avalon to the dropbox group
    • verify that /var/avalon/dropbox is mode 2775
    maintain permissions for dropbox
    • create script /usr/local/bin/fix_dropbox_permissions with this content:

      Code Block
      languagebash
      #!/bin/bash
      chown -R avalon:dropbox /var/avalon/dropbox
      find /var/avalon/dropbox -type f -exec chmod 664 "{}" ";"
      find /var/avalon/dropbox -type d -exec chmod 2775 "{}" ";"

      and set the mode to 755.

    • set the cronjob to run every minute by adding this line:

      Code Block
      * * * * * /usr/local/bin/fix_dropbox_permissions

Demo Content

  • OPTIONAL:
    • clone the puppet image as democontent-YYMMDD
  • Register the archivist1@example.com user in Avalon, and log in as that user.
    • this account is removed when the system is prepared for delivery.
  • Create a 'Default Collection' collection with the archivist1@example.com user.
  • Get the demo fixtures bundle

    Code Block
    cd /tmp
    wget http://dlibwww.indianaavalonmediasystem.eduorg/~cjcolvardownloads/DemoFixturesBatch.tar.gz
        #We need a better place to serve this from
    tar xvzf DemoFixturesBatch.tar.gz
    chown -R avalondrop.avalon:dropbox DemoFixturesBatch
    chmod -R g+w DemoFixturesBatch
    mv DemoFixturesBatch /var/avalon/dropbox

    Monitor progress in Matterhorn (at http://<whatever>:18080 .  using the default username/password) or within Avalon.

  • Download the demo content thumbnails

    Code Block
    cd /tmp
    wget http://www.avalonmediasystem.org/downloads/DemoFixturesThumbnails.tar.gz
    tar -xzvf DemoFixturesThumbnails.tar.gz
    chown -R avalon:avalon DemoFixturesThumbnails
    mv DemoFixturesthumbnails /var/www/avalon/current/app/assets/images
  • Modify /var/www/avalon/current/app/views/catalog/_home_text.html.erb and replace sample content with something like:

    Code Block
    languagehtml/xml
     <ul class="thumbnails">
    	<li class="span3"><h5>Featured Video</h5><a href="<%= media_object_path('avalon:1') %>" class="thumbnail" <%><%= image_tag "thumbnail.name.pngDemoFixturesThumbnails/organClip.jpg", alt: "Featured Video" %></a></li>
    	<li class="span3"><h5>Featured Video</h5><a href="<%= media_object_path('avalon:2') %>" class="thumbnail" <%><%= image_tag "thumbnail.name.pngDemoFixturesThumbnails/lunchroomManners.jpg", alt: "Featured Video" %></a></li>
    	<li class="span3"><h5>Featured Video<Audio</h5><a href="<%= media_object_path('avalon:3') %>" class="thumbnail" <%><%= image_tag "thumbnail.name.pngDemoFixturesThumbnails/mahler.jpg", alt: "Featured VideoAudio" %></a></li>
    </ul>

    Thumbnail images should be stored in The values "avalon:1" to "avalon:3" should be changed to the appropriate container IDs

  • Reload the avalon application by running

    Code Block
    touch /var/www/avalon/current/
    app/assets/images
    tmp/restart.txt

    and verify the changes are correct.

  • Fix access permissions on all three items:
    • edit each item
      • click on the "Access Control" tab
      • click on the "Available to the general public" radio button
      • click Save
      • Click the 'Publish" button if it is available on the item view (Lunchroom Manners).
  • Remove source data

    Code Block
    rm /tmp/DemoFixturesBatch.tar.gz
    rm -rf /var/avalon/dropbox/DemoFixturesBatch

...

  • OPTIONAL:
    • clone the latest democontent image as dist-YYMMDD
  • Install avalon-vm-12.0-1.noarch.rpm to install the firstboot changes and tools
  • If IPv6 was disabled during install, re-enable it by removing the lines added to /etc/sysctl.conf during BaseOS.  If these lines are not removed, email will not work.
  • Prepare the disk for compacting
    • clean up anything that's left over from the build process

      Code Block
      rm -rf /root/Downloads/* /var/avalon/dropbox/* /home/makerpm/rpmbuild /opt/staging /root/avalon-installer-flat /root/flat.tar.gz
      yum clean all
      swapoff /dev/mapper/vg_avalon-lv_swap; dd if=/dev/zero of=/dev/mapper/vg_avalon-lv_swap bs=1M; mkswap /dev/mapper/vg_avalon-lv_swap
    • deleted files still take up space on the disk image.  The disk image (as of 4/24/13) is roughly 12G in size, with 7.8G of actual data.  Write zeros to a big zero file until the disk is full to clear out that unused datathe sectors which are unallocated but had data at one point.  Since the .VDI image is sparsely allocated, the disk image size will actually grow less that 1G even though nearly 500G has been written.

      Code Block
      dd if=/dev/zero of=/tmp/foo bs=1M countoflag=8192 oflags=direct; rm /tmp/foo
  • run /usr/share/avalon/dist-prep to prepare the machine for release
  • clear the command line history  (history -ccw)
  • shutdown the VM
  • Compact the disk image
    • On the host run:

      Code Block
      VBoxManage modifyhd avalon-vm-disk-YYMMDD.vdi --compact
  • Export the appliance
      • use a 0.9 version OVA
      • use a MANIFEST
    • copy the OVA to the distribution point

     

    Info

    Adding the demo batch increased the size of the VM download from 3.2G to 7.4G.  Partially its because we've make derivatives and there's really new content on the VM.  But for a bigger reason, any disk space that gets allocated is stored in the image file – the VM doesn't know what's being used and what isn't.  Since we download a largish package, untar it, and create a bunch of temp files, the VM disk image has a bunch of unused (from the OS point of view) disk that it still has to keep track of.  Future research should include how to 'unuse' the disk space.

    "zerofree" should do the trick, when combined with vboxmanager modifyhd <name> --compact

    Cleaning Unused Disk Space

    VirtualBox's VBoxManager tool supports a compact option when modifying hard drives.  Unfortunately it only supports VDI so I'm experimenting with using VDI instead of VMDK for the disk container format.   For the purposes of the release, there should be no functional difference, except that VDI is less efficient than VMDK (on the Base OS install the VDI is 6% larger).

    After a BaseOS install (as described above), the VDI disk image takes 4,304M of space.  A df on the system shows that 3405M is used for data, so there is nearly 1G of space which is a combination of filesystem overhead, VM sparse disk overhead, and non-zero-unused data. 

    The first step is to see if the filesystem can be compacted using

    Code Block
    VBoxManage modifyhd --compact avalon-vm-base-130419b.vdi

    After a few minutes it comes back and shows that 4303M is being used – we saved 1M by running it.  I'm not impressed.

    I know that I downloaded about 200M worth of update packages.  If the filesystem writes to the same places on the disk as it did before, a big dd should change that unused data to zeros:

    Code Block
    dd if=/dev/zero of=/tmp/foo bs=10M count=1024; rm /tmp/foo

    Afterward, the container was 4313M so the container grew by 10M – after writing 10G of zeros.  One would assume that the original deleted data is now zeros and the extra size is overhead of mapping a bunch of empty blocks.

    Compacting it gives us a new disk size:  4077M – we've saved 400M by writing a big bunch of zeros.  Hooray!

    I've written 4G of data to the image and it is currently 8176M on disk, even after erasing the files from the filesystem.

    I've written 8G of zeros to the disk

     

     

     

      • List the VMs and find the UUID of the Avalon VM

        Code Block
        VBoxManage list vms
      • Disable USB 2.0 in the VM to avoid a warning when users don't have the VirtualBox extensions loaded.

        Code Block
        VBoxManage modifyvm <machine-id> --usb off
      • VM Settings:

        NameValue
        Nameavalon-vm
        ProductAvalon Media System
        Product-URLhttp://www.avalonmediasystem.org
        VersionR2
        Code Block
        VBoxManage export <machine-id> --output avalon-vm.ova --product "Avalon Media System" --producturl http://www.avalonmediasystem.org --version R2



    • copy the OVA to the distribution point
      • Make sure to set the avalon-vm.ova symlink to the current release
      • Set the permissions to 644 (they're 600 by default)