RESTful APIs have always been a big part of the Sugar application, as well as the most appropriate touchpoint for integrating with other enterprise systems. Unfortunately, for a long time they weren't tested especially appropriately. Traditionally, REST APIs at Sugar were tested with PHPUnit, but only on the classes themselves - they weren't actually tested via direct use of HTTP. Worse, the tests were messy - they often assumed existing data was present, and when it wasn't, it was created and not deleted afterwards.
Thorn was envisioned as a solution to this problem. It's intended to test Sugar's RESTful API directly, abstract away unhelpful boilerplate, leave your database clean, and let you test like a user.
Thorn was open-sourced earlier this year after a brief internal testing period and is now available for any Sugar developer to use from github.com/sugarcrm/thorn. It is open source and available under the Apache License, version 2.0.
This initial post will focus on a technical and philosophical overview of Thorn. For usage information, see the Thorn website.
Key design elements of Thorn tests
Thorn has a few key design principles that informed how it was developed and how we test with it.
Test like Goldilocks
If all you're trying to prove with a test is that a REST API works, an end-to-end test with Selenium (or similar) is too expensive and too likely to fail for unrelated reasons to justify running it. Unit tests aren't particularly helpful either; most API bindings don't carry around all that much interesting functionality since they're mostly proxies to other code that do the real work.
Thorn is thus meant to test an API as an API, by using real HTTP requests and real data. The result is nimbler tests that nevertheless accurately reflect how Sugar's APIs are actually used.
Thorn does this by building on top of the excellent Chakram framework - an extension to the well-known Mocha test runner that makes it easy to make HTTP requests from within tests and make assertions about their responses.
In other words, Thorn is just right for testing REST APIs. (Cue groans from the readers.)
Take nothing but responses, leave nothing but logs
Tests that make implicit assumptions about the system under test have a number of issues. They can't be safely parallelized or run in isolation, because they might be assuming that one of the other tests in the suite provides that data.
Furthermore, a test you can only run once is a useless test, because if it fails, you may have to wipe out your entire environment and start over just to make one single code change. If the test fails again, this will get frustrating very quickly.
Thorn's Fixtures API provides the solution. Fixtures.create lets you decide what records should exist in the database before your test begins, and Fixtures.link can be used to create relationships between them. You only have to give values to fields you really care about - anything else necessary will be transparently filled in by Thorn. Finally, Fixtures.cleanup cleans up whatever mess in the database your fixtures made.
Another nice side effect is that a good Thorn test can run on any compatible Sugar instance - even an anonymized customer instance, in the case of partners or system integrators.
Think of a good Thorn test like a backpacking trip. You have to bring everything you need with you, so pack light and only bring what you need. And don't ruin the trail for the next camper to come through - clean up after your data.
Be the user
The only reason an API should exist is so a front-end (or a different service's back-end) can talk to it. But either way, ultimately APIs exist because some user felt it would be helpful for a service to do something. To that end, a good Thorn test is written with the users of the API (reflecting real usage by end users in real roles) in mind.
Mocha's traditional behavior-driven-development interface helps here, but Thorn's main advantage is the UserAgent interface. With Agent.as, you can specify exactly what user you want to perform an action as, then use Chakram's convenience methods (get, post, etc.) to perform exactly the same action that a Sugar user would do.
Your users expect your API to be honest. So when you’re writing a Thorn test, keep that in mind and keep them honest.
Remember your passwords
Authentication, session expiration, and re-authentication are some of the thorniest (pun intended) issues you'll experience while working with APIs.
Thorn provides solutions to all of these problems, with an aim of limiting manual developer intervention and undesired nondeterminism whenever possible. Authentication is performed implicitly by Thorn for each user involved in a test; there's no explicit login required.
Furthermore, if a user's request fails mid-flight for authentication reasons, the request is automatically retried and the test proceeds as normal.
You wouldn't want a user to have to login every time they go to another page of your app and Thorn doesn't expect that of you either.
Cool, how do I stop RESTing and start testing?
For now, check out the Thorn website. In a later post, I'll go into greater depth about good Thorn testing practices and show how you can test a sample REST API endpoint.