Skip navigation
All Places > Developer > Blog > 2017 > December
2017

TLDR 

Developing Sugar on Docker, get up to speed now!

 

The long version

 

What are Docker containers?

If containers for you are more than the two pictures below… then you are in the right place!

 

 

As you might be aware, there has been and there continues to be a lot of hype around containers. The hype really started with Docker, but containers have been around for a while (more or less), in a different shape or form. If you are not yet familiar with containers or are confused about the difference between a virtual machine and a container, this blog post from Docker might be useful to you.

 

Personally, I was deploying container-based virtualised single purpose hosting environments (now labelled micro-services?) for my customers, back when it was not so trendy. It was about 10 years ago when I was leveraging OpenVZ. OpenVZ is slightly different (eg: it has state/persistent storage for every container), but in the end you can run multiple lightweight containers that share the custom linux kernel of their host.

 

What Docker has managed to do differently is make infrastructure deployments repeatable and portable. An application can be shipped together with its Docker container directives, providing the same exact configuration across different host environments. The infrastructure looks more like code, which simplifies the process of automating and delivering software deployments. Teams using Docker found that the lines between development and operations blurred even further, and guess what? DevOps was born!

 

Docker has made it easy to get up and running with a technology for development purposes. A developer just needs Docker installed on his or her local machine (or on a linux VM) and a “Dockerfile”, to get an application up and running. If he or she needs to replace a single component of the infrastructure for testing purposes, they simply swap a few directives and re-build. The same application is executing on top of a slightly different platform. How awesome and easy is that?

 

With the always increasing pace of software development innovation required today, and with applications becoming more and more decoupled from each other (and at the same time integrated), development teams need to keep their deployment velocities up as well quickly deliver innovation across the board on a multitude of heterogeneous backends.

 

This need for a fast deployment velocity can be fulfilled with the aid of flexible containers and orchestration technologies when their delivery is seamlessly baked into the traditional software development pipeline. Obviously containers are not the silver bullet or the answer to every known question, but they are currently applicable to a vast set of different workloads and use cases.

 

Docker in production

At this stage, I do not have direct experience with Docker in production environments, but, as with any technology, there is always someone willing to share their horror stories…

 

Jokes aside; with the increase in popularity of orchestration, the extensive backing and support from various cloud providers and how lightweight it is, I am convinced Docker is the way to go for micro-services deployment and orchestration in production environments as well. As a matter of fact, container-technology adoption is constantly growing across the board and not just for micro-services.

 

Sugar on Docker

Enough said about the containers topic. Here it is, my attempt at creating a set of development stacks for Sugar with Docker.

 

I have to thank one of our long-standing partners Cédric Mourizard from Synolia for the initial work (read more in his previous blog post). The Sugar Docker setup has been polished here and there over the past 6 months, and it is now to a point where I only run Sugar locally with this solution.

 

I highly encourage you to give it a try. Have a look at how Docker can fit in your development process, and, if it does not yet fit into it, think about how it could help you streamline your overall software delivery pace and capabilities.

 

Flavours used

Let’s give a quick overview of the various components used. When using *nix, I’m a Debian kind of guy. So that’s that.

LAMP stack, Elasticsearch and Redis on top of Debian.

 

Stack components

It might evolve over time, but for now the main containers are:

 

  • Apache load balancer - Load balances requests between the cluster of Apache PHP web servers, round robin
  • Apache PHP web server - Web server(s)
  • MySQL database - Database
  • Elasticsearch - Sugar search engine
  • Redis - Two purposes: Sugar object caching service and PHP Session storage/sharing service
  • Cron - Sugar background scheduler processing. Note that this is enabled immediately and it will run cron.php as soon as the file is available, and it will attempt to do so every 60 seconds since its last run
  • Permission - Sets Sugar instance permissions correctly and then terminates

 

The application files are shared on the “cluster” (if you can call it that) through a directory based mounted volume to offer persistent storage. We also have various persistent storage directories within ./data/ for all the other software components that need them.

 

Based on the selected stack (within the ./stack/ directory) you will be able to pick and choose the various versions of the stack components available, and to decide if you want a load balancer and multiple web servers or just a single web server during initial development phases. Because the Docker containers are stateless, it is possible to run the same Sugar installation (stored on the persistent volume) either with PHP 7.1 or PHP 5.6 and swap between versions within seconds.

 

Current software versions

Currently, there are a few stack combinations available that run the supported platform for Sugar 7.9.

Containers are available with PHP 7.1/5.6, MySQL 5.7, Elasticsearch 1.7.5 and the latest Redis. When selecting a stack, it is possible to run it as a single web server or as two web servers behind a load balancer.

 

Sugar 7.10 is not available for on-site hosting but it can be downloaded for local development, so, I have prepared a single stack leveraging PHP 7.1, compatible with the required matching platform. The main difference in stack requirements between Sugar 7.10 and earlier versions is the Elasticsearch version 5.4 requirement. Elasticsearch 5.4 seems to be more memory hungry, so be prepared for that.

 

Finally, there is an additional stack for 7.9 upgrades, running both Elasticsearch 1.7.5 and 5.4, to allow future testing of upgrades to the Sugar Spring '18 release.

 

Getting up and running

Most of the details you need to get up and running are within the README.md of the git repository. In terms of where to run this setup, you have the option of leveraging docker-machine or to run your own VM, as currently the native Docker for Mac does not perform well with the mounted volumes for persistent storage.

 

I personally use a Debian VirtualBox VM on a Macbook. If you want to run the same VM as I do, I have prepared a VirtualBox appliance export. You can find the appliance on this repository, as well as more details on how to deploy it on your Mac. It runs a NFS share as well, so that you can mount the directory locally and code with your preferred editor.

 

During my process, I’ve also tried Vagrant to help with automatic deployment of the VM instead of building an appliance, but I did not obtain the same performance footprint (not even close). So I abandoned that route long ago.

 

Now you are up to speed with Sugar on Docker, and you might have realised how containers can help your business gain velocity.

 

Feel free to let us know your thoughts, by leaving some comments below!

Sugar v11 REST API

In order to support some cool new dashboard features, we made some enhancements to our Dashboard APIs in the Fall '17 release. So we added a new REST API version, v11, in that release in order to ensure clients and integrations that were using the existing v10 Dashboard APIs would continue to work properly. For more details on the v11 API, we recommend you check out a recording of our recent Fall '17 release developer webinar and the related migration guide.

 

Today we won't focus on version 11 specifically but instead on how the Sugar platform will be handling REST API versioning in the future. So read on!

Passing API version using HTTP Header

For REST v10 and our older web services frameworks, the API version was specified exclusively via the URL.

 

Example 1)

 

GET <sugar>/rest/v10/me

 

In the Fall '17 (7.10) release, we introduced an additional way to specify the API version: using an HTTP header. This change allows resource URLs to be treated as unique identifiers to resources regardless of the API version in use.

 

The REST API version can now be specified via HTTP Accept header. The REST API version should be specified in either the URL or in the HTTP Accept header but not in both locations at once. This new method of specifying the API version in the HTTP Accept header better adheres to RESTful architectural principles. By essentially encapsulating the API version as part of media type, a single resource at a given URL could offer multiple representations using different data formats and API versions.

 

Example 2)

 

GET <sugar>/rest/me
...
Accept: application/vnd.sugarcrm.core+json; version=42

 

We recommend that client start specifying the API version in the Accept header instead of as a component of the URL. Sugar clients will adopt use of the Accept header in the future.

Future use of semantic versioning in REST API

As you probably have noticed, the REST API version is a property of the Sugar REST API as a whole and not individual endpoints.

 

We plan to implement a semantic versioning policy with MAJOR.MINOR format for the REST API next year.

 

The version will be specified in the request Accept header as MAJOR.MINOR (ex. 11.1) whereas the version in the endpoint url will be specified as MAJOR_MINOR (ex. 11_1).

 

The Sugar REST API MAJOR version will be incremented for each Sugar release where compatibility is being broken on any endpoint. The Sugar REST API MINOR version will be incremented for each Sugar release where an API endpoint is changed or additional API endpoints are being added without breaking compatibility.

 

Not every new Sugar release will result in new REST API versions. Our intent is to only add new features or make breaking changes to our REST API only when it is not possible or prudent (ex. for performance reasons) to do so with the current API.

 

When there is a breaking change, older API versions should continue to work as-is. For example, if a parameter was removed in newer version, the old version should still support that parameter.

What are breaking and non-breaking changes?

We want to be clear about what we consider to be an API breaking change and what is not.

Breaking Changes

  • Removing or renaming an endpoint, a request parameter, or a fixed response field
    • For example, removing the GET /rest/v10/<module> endpoint or renaming the _acl field to _access_control.
  • Removing support for a HTTP request method on an endpoint
    • For example, changing from PUT to PATCH method for updating records.
  • Change in type for a request parameter
    • For example, a change from an Integer to Decimal value.
  • Change in response field format
    • For example, a change in encoding for a response field from plaintext to base64.
  • Semantic changes to request parameters
    • For example, a change from treating the % symbol as a literal instead of as a wildcard or a change in default value.
  • Semantic changes to an API endpoint
    • For example, an endpoint that was documented to return all records by default and now returns no records by default.

Non-breaking Changes

  • Metadata related changes in behavior
    • For example, when Sugar metadata (Vardefs, Viewdefs, etc.) changes are introduced during a Sugar release or as a result of a customization then REST API responses can vary as a result.
  • Adding a new endpoint
    • For example, adding a GET rest/Accounts/:id/map endpoint.
  • Adding a parameter to a request
    • For example, adding a boolean parameter called shorten_text that decides if long text fields are shortened to 200 characters.
  • Adding a field to a response
    • For example, adding a _self field with link to resource URI in GET responses.
  • Adding support for another HTTP request method
    • For example, adding support for PATCH method to an endpoint.

 

If you ever encounter a situation where there is a breaking change on a particular API version between future Sugar releases, then please report those issues via the Sugar Support portal.

Deprecation process for old API versions

Old REST API versions will continue to be supported until at least one year after they have been deprecated. We want to exercise good judgement on how quickly we remove old API versions but please understand that SugarCRM cannot indefinitely support an unlimited number of REST API versions.

What should you do?

  1. When building new Sugar clients or integrations or updating existing ones, adopt the latest available REST API version possible in order to take advantage of all the latest features.
  2. Use the HTTP Accept header to specify REST API version when using Sugar Fall '17 release or later.
  3. Monitor the release notes and documentation on each new Sugar release for deprecation notices on old API versions. Keep our customers safe by planning ahead!

RESTful APIs have always been a big part of the Sugar application, as well as the most appropriate touchpoint for integrating with other enterprise systems. Unfortunately, for a long time they weren't tested especially appropriately. Traditionally, REST APIs at Sugar were tested with PHPUnit, but only on the classes themselves - they weren't actually tested via direct use of HTTP. Worse, the tests were messy - they often assumed existing data was present, and when it wasn't, it was created and not deleted afterwards.

 

Thorn was envisioned as a solution to this problem. It's intended to test Sugar's RESTful API directly, abstract away unhelpful boilerplate, leave your database clean, and let you test like a user.

 

Thorn was open-sourced earlier this year after a brief internal testing period and is now available for any Sugar developer to use from github.com/sugarcrm/thorn. It is open source and available under the Apache License, version 2.0.

 

This initial post will focus on a technical and philosophical overview of Thorn. For usage information, see the Thorn website.

 

Key design elements of Thorn tests

Thorn has a few key design principles that informed how it was developed and how we test with it.

 

Test like Goldilocks

If all you're trying to prove with a test is that a REST API works, an end-to-end test with Selenium (or similar) is too expensive and too likely to fail for unrelated reasons to justify running it. Unit tests aren't particularly helpful either; most API bindings don't carry around all that much interesting functionality since they're mostly proxies to other code that do the real work.

 

Thorn is thus meant to test an API as an API, by using real HTTP requests and real data. The result is nimbler tests that nevertheless accurately reflect how Sugar's APIs are actually used.

 

Thorn does this by building on top of the excellent Chakram framework - an extension to the well-known Mocha test runner that makes it easy to make HTTP requests from within tests and make assertions about their responses.

 

In other words, Thorn is just right for testing REST APIs. (Cue groans from the readers.)

 

Take nothing but responses, leave nothing but logs

Tests that make implicit assumptions about the system under test have a number of issues. They can't be safely parallelized or run in isolation, because they might be assuming that one of the other tests in the suite provides that data.

 

Furthermore, a test you can only run once is a useless test, because if it fails, you may have to wipe out your entire environment and start over just to make one single code change. If the test fails again, this will get frustrating very quickly.

 

Thorn's Fixtures API provides the solution. Fixtures.create lets you decide what records should exist in the database before your test begins, and Fixtures.link can be used to create relationships between them. You only have to give values to fields you really care about - anything else necessary will be transparently filled in by Thorn. Finally, Fixtures.cleanup cleans up whatever mess in the database your fixtures made.

 

Another nice side effect is that a good Thorn test can run on any compatible Sugar instance - even an anonymized customer instance, in the case of partners or system integrators.

 

Think of a good Thorn test like a backpacking trip. You have to bring everything you need with you, so pack light and only bring what you need. And don't ruin the trail for the next camper to come through - clean up after your data.

 

Be the user

The only reason an API should exist is so a front-end (or a different service's back-end) can talk to it. But either way, ultimately APIs exist because some user felt it would be helpful for a service to do something. To that end, a good Thorn test is written with the users of the API (reflecting real usage by end users in real roles) in mind.

 

Mocha's traditional behavior-driven-development interface helps here, but Thorn's main advantage is the UserAgent interface. With Agent.as, you can specify exactly what user you want to perform an action as, then use Chakram's convenience methods (get, post, etc.) to perform exactly the same action that a Sugar user would do.

 

Your users expect your API to be honest. So when you’re writing a Thorn test, keep that in mind and keep them honest.

 

Remember your passwords

Authentication, session expiration, and re-authentication are some of the thorniest (pun intended) issues you'll experience while working with APIs.

 

Thorn provides solutions to all of these problems, with an aim of limiting manual developer intervention and undesired nondeterminism whenever possible. Authentication is performed implicitly by Thorn for each user involved in a test; there's no explicit login required.

 

Furthermore, if a user's request fails mid-flight for authentication reasons, the request is automatically retried and the test proceeds as normal.

 

You wouldn't want a user to have to login every time they go to another page of your app and Thorn doesn't expect that of you either.

 

Cool, how do I stop RESTing and start testing?

For now, check out the Thorn website. In a later post, I'll go into greater depth about good Thorn testing practices and show how you can test a sample REST API endpoint.