Using Packer to create consistent Sugar platform environments

The "Works On My Machine" Problem.

How often have you reported a software bug and had it rejected with a terse response such as "could not reproduce"?

You press the issue with your colleague and the conversation usually goes something like this:

You:  Here, look at my screen.  You can see the problem happening right here.

Them: Whatever. It works on my machine. L8tr dude.

Doesn't this drive you crazy?

Ultimately, even if you are both running the exact same source code there are always factors related to configuration and platform environment that can affect how an application behaves.

The flexibility of Sugar that we all love also means that we are often running Sugar in multiple different places and in varying configurations. For example, your team could be running Sugar instances in the cloud or on-premise, on real or virtualized hardware, on different operating systems, database servers, etc.

How do you keep track of all these different configurations, let alone try to make them as consistent as possible to avoid the "works on my machine" problem?

Introducing Vagrant and Packer

Vagrant is a tool built by Hashicorp.  It is a tool designed to make managing development environments really easy.  If an existing Vagrant box exists, you are never more than two commands away from launching a development environment that you need.  But it is a tool designed specifically for developers.  It is not the right tool for hosting customer instances or for doing QA because it does not alone replicate a real operational environment.  Also, if a Vagrant box doesn't already exist for the platform you want to work with then you need to create one.  This is where Packer comes in.Packer is also built by Hashicorp, the same folks behind Vagrant.  Here is the definition of Packer straight off their website.
Packer is a tool for creating machine and container images for multiple platforms from a single source configuration

SugarCRM Engineering uses Packer as the base tool for creating all the images we use for development, continuous integration, and QA.

Packer can be used to create Virtualbox or VMWare images, Amazon AMIs, Vagrant boxes, or even Docker containers all from a single source.  Recall Cedric's recent post on using Docker containers to run Sugar.  There is even a Null builder that you could use to run the same provisioners used to build your images on existing machines.

Below we will explore how you can use Packer to build a consistent Sugar 7 ready LAMP stack for Vagrant or on any other platform of your choice.  The code from this post is all available in a Sugar 7 Ready template Github repository.

Using Packer to create a Sugar 7 LAMP stack

As a basis for these templates, you can start by following the Hashicorp tutorial for "Building Vagrant Boxes with Packer and Atlas".  This was a good starting place because it builds an Ubuntu 12.04 machine and publishes Vagrant boxes for it on Atlas which makes them available to anyone.  We just need to modify this example to install supported versions of Apache, MySQL, and Elasticsearch and configure them so they are ready for a Sugar 7 install.

You will need Packer installed locally or have an Atlas account in order to run Packer builds.

For local builds, you will need to install Packer.  You will also need to have Virtualbox installed to run the Virtualbox builds.  If you have a VMWare license then you need VMWare Desktop or Fusion installed in order to run the VMWare builds.

If you are using Atlas to run builds then you only need Git installed locally.

This example uses shell scripts to provision the system but Packer supports a variety of provisioning options such as Puppet, Chef or Ansible. It is worth mentioning that SugarCRM Engineering uses Puppet to centrally manage provisioning in our internal environment.  But we will keep things simple by focusing on using shell scripts for now.

Shell scripts for provisioning Sugar 7 dependencies

Most of the provisioning scripts are going to be boilerplate.  For example, the steps for installing a new operating system or configuring user accounts, etc, are going to be more dependent on the selected Operating System rather than being anything that is Sugar specific.  So we will show two scripts below that focus on installing Supported Platform dependencies.

This first script is used to install all the needed Sugar 7 software dependencies on a fresh Ubuntu 12.04 machine.

php54.sh

#!/bin/bash
#
# Setup the the box. This runs as root

apt-get -y update

# MySQL default username and password is "root"
echo "mysql-server-5.5 mysql-server/root_password password root" | debconf-set-selections
echo "mysql-server-5.5 mysql-server/root_password_again password root" | debconf-set-selections

apt-get -y install python-software-properties perl curl zip vim

# Add Java, php5.4, and Elasticsearch repos
add-apt-repository ppa:ondrej/php5-oldstable
add-apt-repository ppa:webupd8team/java -y
wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | apt-key add -
echo "deb http://packages.elastic.co/elasticsearch/1.4/debian stable main" | tee -a /etc/apt/sources.list
apt-get -y update

# Install Apache+php54 stack
apt-get -y install mysql-server php5-mysql php5-curl php5-gd php5-imap libphp-pclzip php-apc php5 apache2 php5-curl php5-dev php5-xdebug

#Install Elasticsearch and Java

# Auto-accept oracle license
echo debconf shared/accepted-oracle-license-v1-1 select true | debconf-set-selections
# Install Java 8, elasticsearch 1.4, then run it as a service
apt-get -y install oracle-java8-installer
apt-get -y install elasticsearch

# Set up Elasticsearch to run as a service
echo "Setting up Elasticsearch as a service"
update-rc.d elasticsearch defaults 95 10

# Update apache2 php.ini
sed -i 's/memory_limit = 128M/memory_limit = 512M/' /etc/php5/apache2/php.ini
sed -i 's/upload_max_filesize = 2M/upload_max_filesize = 20M/' /etc/php5/apache2/php.ini
sed -i 's/;date.timezone =/date.timezone = UTC/' /etc/php5/apache2/php.ini

# Update cli php.ini for cron
sed -i 's/;date.timezone =/date.timezone = UTC/' /etc/php5/cli/php.ini


This second script is used to configure Apache 2.2 web server on Ubuntu 12.04 to work with Sugar application.

apache22.sh

#!/bin/bash

# Add phpinfo to web dir as a convenience
echo "<?php phpinfo();" > /var/www/phpinfo.php

# We will have Apache run as Vagrant user because this will help prevent permissions issues on a dev setup
sed -i "s/export APACHE_RUN_USER=www-data/export APACHE_RUN_USER=vagrant/" /etc/apache2/envvars
chown -R vagrant /var/www/
usermod -a -G www-data vagrant

# Enable some important Apache modules
a2enmod headers expires deflate rewrite

# Enable DEFLATE on application/json - this helps speed up downloads of Sugar data
cat >> /etc/apache2/mods-enabled/deflate.conf <<DELIM
<IfModule mod_deflate.c>
AddOutputFilterByType DEFLATE application/json
</IfModule>
DELIM

# Update Apache config on web server to AllowOverride
sed -i "s/AllowOverride None/AllowOverride All/" /etc/apache2/sites-enabled/000-default

# Set up port 8080 virtualhost config to allow interactive installs of Sugar from Host system when using Vagrant
# Also make sure sugar directory has AllowOverride enabled
cat >> /etc/apache2/sites-available/sugar <<DELIM
Listen 8080
<VirtualHost *:8080>
ServerName localhost
DocumentRoot /var/www
</VirtualHost>
<Directory "/var/www/sugar">
Options Indexes FollowSymLinks MultiViews
AllowOverride All
Order allow,deny
Allow from all
</Directory>
DELIM
a2ensite sugar

# Restart apache22 when done
apachectl restart

Sugar 7 Ready template repository

The Sugar 7 Ready Packer templates are set up to build Virtualbox and VMWare images and Vagrant boxes.  But extending it to build Docker containers, Amazon AMIs, or any other container technology is possible and encouraged.

Trying it all out

Trying it out is easy if you have Virtualbox installed.  Virtualbox is a popular option because it is free.

Launching the Vagrant box

With Vagrant and Virtualbox installed, navigate to a clean Sugar development directory and run the following command.  By default, these boxes use 3GB of memory but you can customize this within your Vagrantfile.

vagrant init mmarum/sugar7-php54; vagrant up --provider virtualbox

When finished, your current directory will be running at http://localhost:8080/sugar/.  Visiting that URL will launch the Sugar installer.

If you have VMWare installed then you can launch the VMWare provider instead.

vagrant init mmarum/sugar7-php54; vagrant up --provider vmware_desktop

Downloading the Virtualbox Image

The same Packer template used to create the above Vagrant boxes was used to install a GUI desktop for running Sugar within a plain Virtualbox machine.  This is a great option for having an easy to use version of Sugar for training purposes.

Download it from here.  If you have Virtualbox installed, you can unzip it then import the image using Virtualbox  By default, it uses 4GB of memory but you can adjust this within Virtualbox before you launch it.

  • Comment originally made by Matthew Marum.

    Biggest performance bottleneck will be file system I/O with Virtualbox shared folder.  This is the default mechanism Vagrant uses for Synced Folders.

    If you set up NFS sharing on default share then the file I/O performance will be much better (4x faster to do Quick Repair) and I think the run away CPU usage will come down.

    For example,

      config.vm.network :private_network, ip: "10.0.0.10"

      config.vm.synced_folder ".", "/vagrant/", type: "nfs"

    https://www.vagrantup.com/docs/synced-folders/nfs.html

    My main issue is that NFS only works for Unix based hosts (MacOSX, Linux) and doesn't work on Windows.  You can use SMB instead for Windows but I don't have a Windows machine I can use to test this.  RSync is a cross platform solution but requires you have some extra tools installed and means you have to reload the box each time you want to deploy new code changes - which is probably reasonable.

    My goal was to provide something that will get people up and running quickly with minimal fuss.

  • Comment originally made by Caim Astraea.

    Hello o/

    Had a go with the vagrant box however ran into some issues with the CPU usage. I'm using a mac mini 2012 with 16GB ram and SSD on latest El Capitan. Latest version of vagrant and virtual box. VBoxHeadless will eat up to 170% cpu usage when apache is serving content after half an hour running it the cpu was at 80 degrees.

    Under the host, the two apache processes would spike up to 60% each and php also to 40% and sometimes the machine crashed.

    Accesing the vagrant box via privte networking instead of forwarded port

        config.vm.network "private_network", ip: "192.168.100.10"

    seemed to help a bit.

    Some people suggest to disable serial_logging and disable nested paging in virtual box but that doesn't seem to make any difference. Others point at opendirectoryd process as the culprit.

    I'll have more logs and snapshots once I get home if anyone is interested?