Skip navigation
All Places > Developer > Blog
1 2 3 Previous Next

Developer

148 posts

One of the first things people do when they start thinking about implementing DevOps best practices is setting up continuous integration and continuous deployment (CI/CD) tooling.  So that’s just what we did with the Professor M School.  (Not sure what I’m talking about when I say the “Professor M School?”  Check out my earlier blog post Introducing Professor M’s School for Gifted Coders.)

 

We’ve set up CI/CD with not one but two popular tools:  Travis CI and Jenkins.  As you decide which tool to use, consider the following tradeoffs.  Travis CI is hosted online so you don’t have to set it up yourself.  Open source projects can leverage Travis CI for free, but closed source projects will have to pay.  Jenkins requires you to set up and manage the app yourself but is free for all projects.  

 

Travis CI Build
Automated build inside of Travis CI.

 

Jenkins Build

Automated build inside of Jenkins

 

We’ve set up the same basic flow for both CI/CD tools:

  • Run the automated tests
  • Build the Professor M module loadable packages

 

In addition, the Travis CI build is configured to post the Professor M module loadable packages to the GitHub releases page whenever a commit is pushed to the master branch.

 

You can learn more about how we’ve implemented CI/CD in the Professor M School’s Readme:

 

You can also explore the results of our latest Travis CI builds at https://travis-ci.org/sugarcrm/school/.

 

We hope you’ll be able to leverage the work we’ve done to setup CI/CD with Travis CI and Jenkins in your own projects!   

One of the biggest pains in developing a module loadable package for Sugar is figuring out how to track the package in source control.  You don’t want to keep your entire Sugar directory under source control--you just want your custom code in source control.  However, from a development perspective, this can be a huge pain.  If you do your development separate from the Sugar directory, every time you make an incremental change, you have to manually build and install the module loadable package, which is a process that can quickly become very time consuming.

 

We’ve developed a tool you can use called the Sugar Synchronizer.  The Synchronizer allows you to develop your Sugar customizations in your Sugar directory while still tracking your changes for your module loadable package in a separate directory.  You can see a live demo of the Sugar Synchronizer in the Sugar Winter '18 Technical Training

webinar recording (you may want to jump ahead to the 38:42 mark).

 

The Sugar Synchronizer is open source, and you can get a copy of the code in our GitHub repo:  https://github.com/sugarcrm/project-sync.  The repo’s Readme has instructions on how to use the Synchronizer.

 

If you have suggestions for improvement or find a bug, please create an issue in the GitHub repo.  Or, better yet, write some code and submit a pull request!  

 

We hope the Sugar Synchronizer makes developing module loadable packages easier for you!

We’ve been working behind the scenes to create a customized version of Sugar that we’re calling the Professor M School.  The essential pieces are ready to go, and we’re thrilled to announce that the repo for the Professor M School is now public and open source.  

 

So what is this the Professor M School?  Let’s begin by talking about the goals for this app:

 

  1. The app should be an authoritative example on upgrade-safe customizations.  We want you, the Sugar developers, to be able to use this app as a reference for what good, upgrade-safe customizations look like.

  2. The app should provide examples of DevOps best practices when developing Sugar customizations and integrations.  For example, we want to have examples of automated test suites and how to leverage continuous integration tools.

  3. The app should include common customizations so that SugarCRM’s engineers can use it for testing and validation.  When our engineers want to know if their changes are going to break common customizations, we want them to be able to use this app to find out.  Ultimately, we want to integrate the app’s automated tests into our internal continuous integration process.


With those goals in mind, we developed the scenario and use cases for the Professor M School.  The app is built around a fictitious university called Professor M’s School for Gifted Coders.  If you attended UnCon 2017, you might remember hearing about Professor M.  We’ve built on what we created for UnCon in order to create the Professor M School.  


Let’s talk about the scenario behind the Professor M School.  Professor M aka Professor Marum has created an exclusive not-for-profit school for gifted coders.  The school currently uses Sugar for the following use cases:

  • Managing applicants, current students, former students, and professors
  • Tracking super groups affiliated with students
  • Soliciting donations from alumni and alumni affiliated super groups.

Professor M will be expanding his use of Sugar to cover other use cases in the future.

To learn more about the Professor M scenario and see a demo of the customizations, check out this video I previously recorded (you might want to jump ahead to the 1:07 mark).

As I mentioned earlier, the Professor M School is open source.  Here are some ways that you can get involved:

  • “Watch” the Professor M School’s GitHub repo: https://github.com/sugarcrm/school.
  • Explore the code and use it as a resource as you develop your integrations and customizations.
  • Submit GitHub issues if you have ideas for improvement, a feature request, or a bug report.
  • Update the code and create a pull request.

 

We hope you find the Professor M School to be a useful resource!  Check back soon for more updates on the app!

Attention developers!  We’ve released Winter '18!  This is our second quarterly On-Demand only release, and we’re pretty excited about it.  Our Co-Founder & CMO, Clint Oram, discusses the highlights of this release from an end-user's perspective in the video below:

 

Matt Marum and I recently hosted a webinar where we gave an overview of the big things developers need to know about this release:

 

The slides from the webinar are available here .

 

If you’re looking for the CliffsNotes version, I’ve got you covered:

  1. Sugar has a new skin, and it’s fabulous.  No JavaScript changes were made as part of this reskinning.  For more information, check out Matt’s blog post: A new Sugar UX coming in Winter '18 .

    Before:
    Old look
    After:
    New look

  2. The Input Validation Framework that has been implemented since Sugar 7.7.1 is now enforced by default.  Input validation violations will cause FATAL errors.  See Matt’s blog post for more details:  How to fix Sugar input validation failures .  

    As part of enforcing the Input Validation Framework, all custom platforms must be registered.  If you have a customization or integration that uses our REST API, this probably affects you.  Check out my blog post  that has links to get more information about the change and a tutorial to help you register your custom platform.
  3. You now have the option of turning on denormalized team security, which can greatly improve List View performance.  Check out the Developer Release Notes or the webinar recording above for more details.
  4. We’ve added PhaserIO as a new library.  As a result, we’re seeing outstanding performance in the new Product Catalog Dashlet and great potential for speeding up other parts of the app.  Check out the Developer Release Notes for a list of all library changes.
  5. We’ve been working on an Exemplar App (https://github.com/sugarcrm/school) and the Sugar Synchronizer (https://github.com/sugarcrm/project-sync), which we hope will become key resources for everyone.  Watch the blog for more details coming soon!

 

This blog post promised to have just about everything you need to know about the Sugar Winter ‘18 release.  Below are some resources that have the rest of the details.

 

We hope you’re as excited about this release as we are!

What is the Input Validation Framework?

The Input Validation Framework is a centralized security framework used to validate that external input is provided in the expected format. Validating user input is critical because it is the basis for generated HTML and REST API output, used in the construction of SQL queries, used to access files, etc. Malicious input can be used to compromise the security of a Sugar instance and the underlying infrastructure. The validation framework provides better control than the pre-existing input sanitization techniques applied to user input within Sugar.

 

Fixing Input Validation problems

Sugar’s input validation was enabled but only logging failures as warnings into the sugarcrm.log file since the 7.7.1 release. With the Winter ‘18 release, the validation.soft_fail configuration setting will now default to false. This means that input validation checks will be strictly enforced by default instead of simply logging warning messages.

 

For more information about using validation constraints in your own custom code, check out the Sugar Developer Guide, the previous UnCon Security presentation, or the Winter '18 webinar.

 

Identifying input validation violations

Beginning in the Winter ‘18 release, input validation violations are reported as PHP Fatal errors in the PHP error log and Critical error messages in sugarcrm.log file. Invalid requests will generally be terminated with an HTTP 500 error as well.

 

Let’s walk through an example of an input validation violation.

 

In this example, we have a customization that is accessing the Sugar instance at the following URL: sugarcrm/ult/#bwc/index.php?module=xxxxx.

 

If xxxxx is not a registered module, there will be a FATAL error in the PHP error log.

For example,

[15-Jan-2018 22:04:09] WARNING: [pool www] child 1940 said into stderr: "NOTICE: PHP message: PHP Fatal error:  Uncaught exception 'Sugarcrm\Sugarcrm\Security\InputValidation\Exception\ViolationException' with message 'Violation for REQUEST -> module' in /srv/www/<instance>/prebuild/test/ult/sugarcrm/src/Security/InputValidation/Request.php:280"
[15-Jan-2018 22:04:09] WARNING: [pool www] child 1940 said into stderr: "Stack trace:"
[15-Jan-2018 22:04:09] WARNING: [pool www] child 1940 said into stderr: "#0 /srv/www/<instance>/prebuild/test/ult/sugarcrm/src/Security/InputValidation/Request.php(209): Sugarcrm\Sugarcrm\Security\InputValidation\Request->handleViolations('REQUEST', 'module')"...

 

The sugarcrm.log will also show an error.

For example,

Tue Jan 16 06:05:18 2018 [1941][-none-][CRITICAL] InputValidation: [REQUEST] module -> Invalid module xxxxx

 

From the end user perspective, these fatal errors will cause pages to appear blank or stalled during loading. If you inspect the browser console when this happens, you will likely see HTTP errors logged.

 

Fixing input validation violations

The steps to take to fix the violation are going to depend on the exact constraint that you are violating. Luckily the exceptions you find in the PHP error log and the messages in the sugarcrm.log will indicate the constraint that you are violating.

 

In the above example, attempting to access a BWC view for an unregistered custom module “xxxxx” is causing the failure. If you make sure that the module you are using is properly registered in Sugar’s module list, or modify your request to use an existing Sugar module, then this will fix this issue since the constraint is checking if the module name is a known modules installed in this Sugar instance.

 

There are basic Symfony framework PHP constraints that we use. These do not need an in depth explanation. The basic Symfony constraints can be used to validate URLs, date time formats, etc. Check out the Symfony documentation for these basic constraints.

 

In Sugar's implementation, we add our own set of validation constraints. These constraints extend from Symfony's Constraint and ConstraintValidator classes and are located in the ./src/Security/Validator/Constraints directory of your Sugar installation. Each Constraint is enforced using a related Validator class. You can inspect the Validator classes to see the precise logic that needs to be followed. For example, the Bean/ModuleName constraint is implemented by the Bean/ModuleNameValidator class. This constraint is used to ensure that the input is a string that matches the name of a valid module with an associated SugarBean class.

 

One of the basic rules that most of these constraints implement is that an input type should be a scalar. This means we expect most user input to be integer, decimal, string, or boolean values.

 

Sugar Constraints

The list provided below may not be exhaustive.

  • Bean/ModuleName
    • Must be a string. The Module’s SugarBean class must be retrievable using BeanFactory. For example, this is used in legacy Workflow.
  • ComponentName
    • Must start with a letter and may only consist of letters, numbers, hyphens and underscores. For example, this is used throughout Studio and Module Builder for module names, package names, and Sugar Logic functions.
  • Delimited
    • By default, must be comma delimited scalar values. For example, this is used in conjunction with other constraints.
  • DropdownList
    • Dropdown lists must only consist of letters, numbers and underscores. For example, this is applied to dropdown lists during module install.
  • File
    • File must exist in an allowed file path and must not include null characters or any directory traversals. For example, this is used by FileLoader utility class.
  • Guid
    • Must be a string with alphanumeric or dash characters. For example, this is used with SugarMVC Views.
  • InputParameters
    • General input validation for GET, POST, REQUEST superglobals. Must be scalar values and strings should not contain null characters.
  • JSON
    • Must be a string that is JSON decodable. Returns decoded PHP associative array. For example, used in Configure Navigation Bar Quick Create.
  • Language
    • Must be a locale code that matches an installed language. For example, this is applied where languages can be selected in UI.
  • LegacyCleanString
    • Reimplements input sanitization rules previously used with clean_string() function.
  • Mvc/ModuleName
    • Must be a string. A more permissive test than Bean/ModuleName that allows any module name that can be found in global module lists. For example, this is used throughout SugarMVC Views.
  • PhpSerialized
    • Must be a PHP serialized string that does not contain objects. Returns unserialized value.
  • Platform
    • Must be a string that is under 128 characters and contains alphanumeric and dash characters. Must be a known Platform value such as “base”, “mobile”, “portal”, etc., or one registered using Platform extension. For example, this is used to validate REST API login requests.
  • Sql/OrderBy
    • Must be a string that represents a valid ORDER BY SQL clause. For example, this is used in SugarMVC list views.
  • Sql/OrderDirection
    • Must be a string that is uses SQL keywords “ASC” or “DESC”. For example, this is used in SugarMVC list views.
  • SugarLogic/FunctionName
    • Must be a string that only uses word characters and dashes. For example, this is used in Sugar Studio.

 

If you encounter a failure, then you should be able to identify the constraint that you are violating from your error logs. By looking at log messages that appear immediately before the failure, and possibly some exploration in the Sugar user interface, you will be able to identify the failing request. From there, you can use the information above to alter the request to comply with the necessary constraints.

 

This is not a substitute for Sugar field level validation!

An important thing to note about the Input Validation Framework is that it is not the only input validation mechanism available. For example, there is still separate field level validation that occurs when users create or edit records. Sidecar customizations or Studio configuration can still be used to perform validation of user provided input for each field of a record in a user friendly way. For example, Sugar will let you save a record without filling out required fields and will display a user friendly warning that tells user what to correct. The Input Validation Framework is centered around security and does not provide the same usability than you would get validating user input using Sidecar or Studio settings.

If you have a REST API integration with Sugar, you are likely using a custom platform as part of that integration.  Beginning in the Winter ‘18 release, Sugar is enforcing a rule that all custom platforms must be registered in each Sugar instance where a custom platform is used.  For more details on the rule itself, see Matt’s earlier blog post: Unknown platforms to be restricted in Winter ‘18 release.

 

The bad news?  This change that requires all custom platforms to be registered probably affects you.

 

The good news?  There is an easy fix, and I’ve written a tutorial that walks you step-by-step through how to implement the fix.

 

Check out the tutorial here!

The Sugar Winter '18 release is fast approaching! Hopefully you are as excited about the new Sugar UX as I am!

 

As usual, Lauren Schaefer and I will be hosting a developer webinar to help get you and your customers prepared for the new Winter '18 (7.11) release. Please read below to learn more about the agenda and do not forget to register! We run these webinars twice, so no matter where you are in the world, we hope that one of these times will be convenient for you.

 

Winter '18 Technical Webinar Agenda

 

Technical Training AgendaWebinar Times
  1. Impacts of new interface
  2. Sugar Exemplar app (more below)
  3. Restricted REST API platforms (more below)
  4. Removal of some LESS files
  5. Emails subpanel changes
  6. Input Validation Framework (now the default)

Monday, January 22nd 4-5:00 PM PT
Tuesday, January 23rd 8-9:00 AM PT

Register Here

 

We will be posting the webinar recordings and deck to the Partner Portal for those who are unable to attend the live sessions.

 

Sneak peak of Sugar Exemplar app


The new Sugar Exemplar app is designed to demonstrate best practices for developing Sugar customizations (think DevOps and QA!) and a solid example of what upgrade-safe customizations should look like. We’ll tell you more about the app and how you can get involved during the webinar!

 

Reminder about restricted REST API Platforms


ISVs and developers who use a platform string to connect to a Sugar instance will need to update their existing API integrations. For more information - please refer to this blog post on how to to proceed.

 

Get on our developer mailing list!

 

Remember to opt-in for Sugar Developer communications to make sure you don't miss this and other important updates.

TLDR 

Developing Sugar on Docker, get up to speed now!

 

The long version

 

What are Docker containers?

If containers for you are more than the two pictures below… then you are in the right place!

 

 

As you might be aware, there has been and there continues to be a lot of hype around containers. The hype really started with Docker, but containers have been around for a while (more or less), in a different shape or form. If you are not yet familiar with containers or are confused about the difference between a virtual machine and a container, this blog post from Docker might be useful to you.

 

Personally, I was deploying container-based virtualised single purpose hosting environments (now labelled micro-services?) for my customers, back when it was not so trendy. It was about 10 years ago when I was leveraging OpenVZ. OpenVZ is slightly different (eg: it has state/persistent storage for every container), but in the end you can run multiple lightweight containers that share the custom linux kernel of their host.

 

What Docker has managed to do differently is make infrastructure deployments repeatable and portable. An application can be shipped together with its Docker container directives, providing the same exact configuration across different host environments. The infrastructure looks more like code, which simplifies the process of automating and delivering software deployments. Teams using Docker found that the lines between development and operations blurred even further, and guess what? DevOps was born!

 

Docker has made it easy to get up and running with a technology for development purposes. A developer just needs Docker installed on his or her local machine (or on a linux VM) and a “Dockerfile”, to get an application up and running. If he or she needs to replace a single component of the infrastructure for testing purposes, they simply swap a few directives and re-build. The same application is executing on top of a slightly different platform. How awesome and easy is that?

 

With the always increasing pace of software development innovation required today, and with applications becoming more and more decoupled from each other (and at the same time integrated), development teams need to keep their deployment velocities up as well quickly deliver innovation across the board on a multitude of heterogeneous backends.

 

This need for a fast deployment velocity can be fulfilled with the aid of flexible containers and orchestration technologies when their delivery is seamlessly baked into the traditional software development pipeline. Obviously containers are not the silver bullet or the answer to every known question, but they are currently applicable to a vast set of different workloads and use cases.

 

Docker in production

At this stage, I do not have direct experience with Docker in production environments, but, as with any technology, there is always someone willing to share their horror stories…

 

Jokes aside; with the increase in popularity of orchestration, the extensive backing and support from various cloud providers and how lightweight it is, I am convinced Docker is the way to go for micro-services deployment and orchestration in production environments as well. As a matter of fact, container-technology adoption is constantly growing across the board and not just for micro-services.

 

Sugar on Docker

Enough said about the containers topic. Here it is, my attempt at creating a set of development stacks for Sugar with Docker.

 

I have to thank one of our long-standing partners Cédric Mourizard from Synolia for the initial work (read more in his previous blog post). The Sugar Docker setup has been polished here and there over the past 6 months, and it is now to a point where I only run Sugar locally with this solution.

 

I highly encourage you to give it a try. Have a look at how Docker can fit in your development process, and, if it does not yet fit into it, think about how it could help you streamline your overall software delivery pace and capabilities.

 

Flavours used

Let’s give a quick overview of the various components used. When using *nix, I’m a Debian kind of guy. So that’s that.

LAMP stack, Elasticsearch and Redis on top of Debian.

 

Stack components

It might evolve over time, but for now the main containers are:

 

  • Apache load balancer - Load balances requests between the cluster of Apache PHP web servers, round robin
  • Apache PHP web server - Web server(s)
  • MySQL database - Database
  • Elasticsearch - Sugar search engine
  • Redis - Two purposes: Sugar object caching service and PHP Session storage/sharing service
  • Cron - Sugar background scheduler processing. Note that this is enabled immediately and it will run cron.php as soon as the file is available, and it will attempt to do so every 60 seconds since its last run
  • Permission - Sets Sugar instance permissions correctly and then terminates

 

The application files are shared on the “cluster” (if you can call it that) through a directory based mounted volume to offer persistent storage. We also have various persistent storage directories within ./data/ for all the other software components that need them.

 

Based on the selected stack (within the ./stack/ directory) you will be able to pick and choose the various versions of the stack components available, and to decide if you want a load balancer and multiple web servers or just a single web server during initial development phases. Because the Docker containers are stateless, it is possible to run the same Sugar installation (stored on the persistent volume) either with PHP 7.1 or PHP 5.6 and swap between versions within seconds.

 

Current software versions

Currently, there are a few stack combinations available that run the supported platform for Sugar 7.9.

Containers are available with PHP 7.1/5.6, MySQL 5.7, Elasticsearch 1.7.5 and the latest Redis. When selecting a stack, it is possible to run it as a single web server or as two web servers behind a load balancer.

 

Sugar 7.10 is not available for on-site hosting but it can be downloaded for local development, so, I have prepared a single stack leveraging PHP 7.1, compatible with the required matching platform. The main difference in stack requirements between Sugar 7.10 and earlier versions is the Elasticsearch version 5.4 requirement. Elasticsearch 5.4 seems to be more memory hungry, so be prepared for that.

 

Finally, there is an additional stack for 7.9 upgrades, running both Elasticsearch 1.7.5 and 5.4, to allow future testing of upgrades to the Sugar Spring '18 release.

 

Getting up and running

Most of the details you need to get up and running are within the README.md of the git repository. In terms of where to run this setup, you have the option of leveraging docker-machine or to run your own VM, as currently the native Docker for Mac does not perform well with the mounted volumes for persistent storage.

 

I personally use a Debian VirtualBox VM on a Macbook. If you want to run the same VM as I do, I have prepared a VirtualBox appliance export. You can find the appliance on this repository, as well as more details on how to deploy it on your Mac. It runs a NFS share as well, so that you can mount the directory locally and code with your preferred editor.

 

During my process, I’ve also tried Vagrant to help with automatic deployment of the VM instead of building an appliance, but I did not obtain the same performance footprint (not even close). So I abandoned that route long ago.

 

Now you are up to speed with Sugar on Docker, and you might have realised how containers can help your business gain velocity.

 

Feel free to let us know your thoughts, by leaving some comments below!

Sugar v11 REST API

In order to support some cool new dashboard features, we made some enhancements to our Dashboard APIs in the Fall '17 release. So we added a new REST API version, v11, in that release in order to ensure clients and integrations that were using the existing v10 Dashboard APIs would continue to work properly. For more details on the v11 API, we recommend you check out a recording of our recent Fall '17 release developer webinar and the related migration guide.

 

Today we won't focus on version 11 specifically but instead on how the Sugar platform will be handling REST API versioning in the future. So read on!

Passing API version using HTTP Header

For REST v10 and our older web services frameworks, the API version was specified exclusively via the URL.

 

Example 1)

 

GET <sugar>/rest/v10/me

 

In the Fall '17 (7.10) release, we introduced an additional way to specify the API version: using an HTTP header. This change allows resource URLs to be treated as unique identifiers to resources regardless of the API version in use.

 

The REST API version can now be specified via HTTP Accept header. The REST API version should be specified in either the URL or in the HTTP Accept header but not in both locations at once. This new method of specifying the API version in the HTTP Accept header better adheres to RESTful architectural principles. By essentially encapsulating the API version as part of media type, a single resource at a given URL could offer multiple representations using different data formats and API versions.

 

Example 2)

 

GET <sugar>/rest/me
...
Accept: application/vnd.sugarcrm.core+json; version=42

 

We recommend that client start specifying the API version in the Accept header instead of as a component of the URL. Sugar clients will adopt use of the Accept header in the future.

Future use of semantic versioning in REST API

As you probably have noticed, the REST API version is a property of the Sugar REST API as a whole and not individual endpoints.

 

We plan to implement a semantic versioning policy with MAJOR.MINOR format for the REST API next year.

 

The version will be specified in the request Accept header as MAJOR.MINOR (ex. 11.1) whereas the version in the endpoint url will be specified as MAJOR_MINOR (ex. 11_1).

 

The Sugar REST API MAJOR version will be incremented for each Sugar release where compatibility is being broken on any endpoint. The Sugar REST API MINOR version will be incremented for each Sugar release where an API endpoint is changed or additional API endpoints are being added without breaking compatibility.

 

Not every new Sugar release will result in new REST API versions. Our intent is to only add new features or make breaking changes to our REST API only when it is not possible or prudent (ex. for performance reasons) to do so with the current API.

 

When there is a breaking change, older API versions should continue to work as-is. For example, if a parameter was removed in newer version, the old version should still support that parameter.

What are breaking and non-breaking changes?

We want to be clear about what we consider to be an API breaking change and what is not.

Breaking Changes

  • Removing or renaming an endpoint, a request parameter, or a fixed response field
    • For example, removing the GET /rest/v10/<module> endpoint or renaming the _acl field to _access_control.
  • Removing support for a HTTP request method on an endpoint
    • For example, changing from PUT to PATCH method for updating records.
  • Change in type for a request parameter
    • For example, a change from an Integer to Decimal value.
  • Change in response field format
    • For example, a change in encoding for a response field from plaintext to base64.
  • Semantic changes to request parameters
    • For example, a change from treating the % symbol as a literal instead of as a wildcard or a change in default value.
  • Semantic changes to an API endpoint
    • For example, an endpoint that was documented to return all records by default and now returns no records by default.

Non-breaking Changes

  • Metadata related changes in behavior
    • For example, when Sugar metadata (Vardefs, Viewdefs, etc.) changes are introduced during a Sugar release or as a result of a customization then REST API responses can vary as a result.
  • Adding a new endpoint
    • For example, adding a GET rest/Accounts/:id/map endpoint.
  • Adding a parameter to a request
    • For example, adding a boolean parameter called shorten_text that decides if long text fields are shortened to 200 characters.
  • Adding a field to a response
    • For example, adding a _self field with link to resource URI in GET responses.
  • Adding support for another HTTP request method
    • For example, adding support for PATCH method to an endpoint.

 

If you ever encounter a situation where there is a breaking change on a particular API version between future Sugar releases, then please report those issues via the Sugar Support portal.

Deprecation process for old API versions

Old REST API versions will continue to be supported until at least one year after they have been deprecated. We want to exercise good judgement on how quickly we remove old API versions but please understand that SugarCRM cannot indefinitely support an unlimited number of REST API versions.

What should you do?

  1. When building new Sugar clients or integrations or updating existing ones, adopt the latest available REST API version possible in order to take advantage of all the latest features.
  2. Use the HTTP Accept header to specify REST API version when using Sugar Fall '17 release or later.
  3. Monitor the release notes and documentation on each new Sugar release for deprecation notices on old API versions. Keep our customers safe by planning ahead!

RESTful APIs have always been a big part of the Sugar application, as well as the most appropriate touchpoint for integrating with other enterprise systems. Unfortunately, for a long time they weren't tested especially appropriately. Traditionally, REST APIs at Sugar were tested with PHPUnit, but only on the classes themselves - they weren't actually tested via direct use of HTTP. Worse, the tests were messy - they often assumed existing data was present, and when it wasn't, it was created and not deleted afterwards.

 

Thorn was envisioned as a solution to this problem. It's intended to test Sugar's RESTful API directly, abstract away unhelpful boilerplate, leave your database clean, and let you test like a user.

 

Thorn was open-sourced earlier this year after a brief internal testing period and is now available for any Sugar developer to use from github.com/sugarcrm/thorn. It is open source and available under the Apache License, version 2.0.

 

This initial post will focus on a technical and philosophical overview of Thorn. For usage information, see the Thorn website.

 

Key design elements of Thorn tests

Thorn has a few key design principles that informed how it was developed and how we test with it.

 

Test like Goldilocks

If all you're trying to prove with a test is that a REST API works, an end-to-end test with Selenium (or similar) is too expensive and too likely to fail for unrelated reasons to justify running it. Unit tests aren't particularly helpful either; most API bindings don't carry around all that much interesting functionality since they're mostly proxies to other code that do the real work.

 

Thorn is thus meant to test an API as an API, by using real HTTP requests and real data. The result is nimbler tests that nevertheless accurately reflect how Sugar's APIs are actually used.

 

Thorn does this by building on top of the excellent Chakram framework - an extension to the well-known Mocha test runner that makes it easy to make HTTP requests from within tests and make assertions about their responses.

 

In other words, Thorn is just right for testing REST APIs. (Cue groans from the readers.)

 

Take nothing but responses, leave nothing but logs

Tests that make implicit assumptions about the system under test have a number of issues. They can't be safely parallelized or run in isolation, because they might be assuming that one of the other tests in the suite provides that data.

 

Furthermore, a test you can only run once is a useless test, because if it fails, you may have to wipe out your entire environment and start over just to make one single code change. If the test fails again, this will get frustrating very quickly.

 

Thorn's Fixtures API provides the solution. Fixtures.create lets you decide what records should exist in the database before your test begins, and Fixtures.link can be used to create relationships between them. You only have to give values to fields you really care about - anything else necessary will be transparently filled in by Thorn. Finally, Fixtures.cleanup cleans up whatever mess in the database your fixtures made.

 

Another nice side effect is that a good Thorn test can run on any compatible Sugar instance - even an anonymized customer instance, in the case of partners or system integrators.

 

Think of a good Thorn test like a backpacking trip. You have to bring everything you need with you, so pack light and only bring what you need. And don't ruin the trail for the next camper to come through - clean up after your data.

 

Be the user

The only reason an API should exist is so a front-end (or a different service's back-end) can talk to it. But either way, ultimately APIs exist because some user felt it would be helpful for a service to do something. To that end, a good Thorn test is written with the users of the API (reflecting real usage by end users in real roles) in mind.

 

Mocha's traditional behavior-driven-development interface helps here, but Thorn's main advantage is the UserAgent interface. With Agent.as, you can specify exactly what user you want to perform an action as, then use Chakram's convenience methods (get, post, etc.) to perform exactly the same action that a Sugar user would do.

 

Your users expect your API to be honest. So when you’re writing a Thorn test, keep that in mind and keep them honest.

 

Remember your passwords

Authentication, session expiration, and re-authentication are some of the thorniest (pun intended) issues you'll experience while working with APIs.

 

Thorn provides solutions to all of these problems, with an aim of limiting manual developer intervention and undesired nondeterminism whenever possible. Authentication is performed implicitly by Thorn for each user involved in a test; there's no explicit login required.

 

Furthermore, if a user's request fails mid-flight for authentication reasons, the request is automatically retried and the test proceeds as normal.

 

You wouldn't want a user to have to login every time they go to another page of your app and Thorn doesn't expect that of you either.

 

Cool, how do I stop RESTing and start testing?

For now, check out the Thorn website. In a later post, I'll go into greater depth about good Thorn testing practices and show how you can test a sample REST API endpoint.

SugarCRM is on a mission to empower our users to delight their customers. To that end, we are pleased to introduce the first phase of visual restyling, and give some tips on how to work with the new UX.

Our process in developing the new UX

Sugar 7 introduced the Sidecar framework with massive user interface improvements (called SugarUX) and a modernized architecture. Since then, we have been working hard on the next iteration of SugarUX built on the Sidecar framework. Our guiding principles came from our customers and their feedback on Sidecar. A competitive analysis and feedback from customers revealed three concerns; cluttered complicated UX, inefficient unguided layout, and a drab outdated user interface.

 

The SugarCRM User Experience team (led by Brian Ng) launched a rigorous heuristic evaluation of the web and mobile Sugar products. Issues were identified as we evaluated key use cases and new solutions were documented. This thorough understanding of the product, customers, and the industry has helped us identify which issues to address first.

 

Dark gray text on light gray background led to poor legibility. Heavy blacks and gray linen added to the dated look and feel.

Sugar Mobile had a dark and drab visual design. Buttons for key actions were positioned differently on key screens.

 

Establishing Goals

With an understanding of the problems, we set goals for ourselves.

  • Improve design for over 100 usability issues, prioritized by impact to the user's ability to get their job done.
  • Establish a clean and modern visual design language, with rules documented in a style guide.
  • Ensuring our products are accessible by all, meeting Web Content Accessibility Guidelines (WCAG) compliancy.

 

Guiding Principles for our new Design

We established three design principles: Clear, Consistent and Efficient. These principles guided all of our decision making. Establishing principles helped to resolve differences of opinions and to focus on user frustrations.We determined what colors and fonts would positively affect our use cases and still reflect our brand.

 

 

We have adopted Open Sans as the Sugar UX base font. It’s an open source font optimized for web and mobile interfaces with text heavy layouts. It’s known for its open, neutral type shape and friendly appearance. For colors, we have selected a color palette that is friendlier, modern and brighter.

 

We also selected a set of colors and fonts that would positively affect our use cases and still reflect the SugarCRM brand. Finally, we tested concepts in the real world, with real users in key flows to observe how they played out in practice.

 


 

Mobile has a bright appearance and color placed in strategic locations to help users identify groups of information.

 

Results

  • First revealed at SugarCon to over 1000 attendees where customers gave feedback.
  • A new visual design language, documented in a Style Guide (coming soon), empowering customers and partners to implement SugarUX consistently on our product.
  • Launch of the new SugarUX on our Sugar Mobile apps, available already on the the Apple App Store and Google Play.

 

 

Coming in Winter ‘18!

 

The first phase of our new UX is coming in Winter '18 release and we want to make sure the Sugar Developer community is first to give it a test drive. If you have built any Sidecar components or customizations (Dashlets, custom views or layouts, etc.) then you should verify that they display properly in new UX. As with all code level customizations, it is up to Sugar developers to make sure that you fix your code so that your customizations display properly in the new UX.

 

From implementation perspective, these changes have been made entirely in CSS where we've adjusted the styling for existing CSS classes. So, areas of concern would be in places where you have overridden the styling for Sugar's core CSS classes or have introduced your own CSS through the use of ./custom/themes/custom.less. The majority of customizations that simply use our existing CSS classes are expected to adopt new styling without errors.

 

As usual, we will be running an invitation only technical Preview program for the Winter '18 (On-Demand only) release. If you want to make sure that your organization is considered for this preview then please e-mail developers@sugarcrm.com and mention your interest in test driving the new Sugar UX!

 

Building your new Skin test plan

We have referred to this effort as the “new skin” because this first phase primarily involves changes to CSS and no significant changes to default HTML structure for any of our Sidecar Components.

 

How do you make sure your customizations are still pixel perfect? What follows below are some tips for you and your team to build a test plan.

 

Exploratory Testing

We recommend devoting a fixed amount of time to doing some exploratory testing in order to make sure your customizations look great in the new skin. How much time? It depends on how much UI you have. For example, it should not take you longer than a few minutes to test a custom Dashlet and uncover any bugs that you may want to address.

 

Guidelines for Testers

If you are testing, what are you looking for? Here are some tips.

  • Customizations only. Testing should focus on customizations that your team created.
  • CSS overrides. If you use a ./custom/themes/custom.less file or include your own CSS files, then you’ll need to pay special attention to these changes since they will likely need to be updated.
  • Scrolling. Are you able to scroll and are there things that are inaccessible even with scrolling?
  • Ellipsifying. When you shrink the window size down or reduce the available space for data on custom list views, record views, etc, do they ellipsify and show tooltips on hover?
  • Responsiveness. Resizing the browser window or switching to and from responsive-mode. Observe what happens to elements on page.
  • Orientation on tablets. Alternating display orientation between landscape and portrait modes.
  • Toggle things. Open/close drawers, menus, edit and detail mode on record views, etc.
  • Settings. Change system and profile settings that may alter how things look like date/time format, currency, etc.

 

Comparisons with out of the box Sugar

This post includes some example screenshots of what the new skin looks like out of the box. Comparing your customizations to the out of the box look and feel will help you identify things that you may need to change in your customizations. We encourage testers to keep two windows open while testing to compare the out of the box behavior in one window with your customizations in another window more easily.

We’re getting a head start on our New Year’s Resolutions!  We’ve given our developer community a fresh, new look, and we feel ten pounds lighter.  Let me give you a tour!

 

First, we’ve migrated our blog from developer.sugarcrm.com to the Developer Community at https://community.sugarcrm.com/community/developer/pages/dev-blog.  We’re really excited about this change as the community is now your one-stop-shop for everything related to developing on Sugar.  We’ll temporarily be putting new posts on both blogs, but please update your bookmarks now.

 

Second, we’ve redesigned the Developer Community to feel more modern and make it easier to find content.  

When you navigate to the community’s home page, you’ll notice 4 new buttons at the top:

Home page buttons

 

If you’re new to developing on Sugar (or you just need a refresher), click Getting Started with Sugar.  On this page you’ll learn how to get setup, trained, and certified.  You’ll also learn how to integrate with and create add-ons for Sugar.

Getting Started

 

If you want to leverage the Sugar Mobile Application Configuration Service or our Mobile SDK, click Getting Started with Mobile.  This subspace has its own documentation as well as a place for you to ask and answer questions.  If you’re a mobile developer, you’ll definitely want to follow the Mobile Developers space.

 

If you’re feeling helpful, click Answer a Question to browse questions from your fellow community members and share your expertise.

 

If you want to learn something new, click Try a Tutorial.  You’ll be able to get your hands in the code on a variety of topics including the REST API, Advanced Workflow, the Mobile SDK, and Quotes.

Tutorials

 

If you continue scrolling down the home page, you’ll see Recent Activity with all of the latest community content as well as a form where you can sign up for Developer News to receive the latest Sugar Developer news, webinars, and surveys straight to your e-mail inbox (I highly recommend signing up if you haven’t already!).

 

If you browse the home page’s right column, you’ll see a set of helpful widgets.  At the top, you’ll find a search box where you can search the developer community.  Next, you’ll see our latest Tweets (you may want to follow us on Twitter if you haven’t already).  If you continue scrolling, you’ll find the Developer Tools widget.  This is one of my favorite pieces of the community as it has all of our favorite resources listed in one place!

Developer Tools Widget

 

Be sure to check out the last widget:  Top Participants.  We hope to see your name there!

 

We hope you enjoy our new look!  Have suggestions on how we can improve the community?  Let us know in the comments below!

Sugar uses platforms to support the needs of multiple Sugar clients.  The Sugar REST API uses the platform parameter to indicate which platform is being used.  If you’d like a refresher on what the platform parameter is and how to use it, check out this blog post.  In Sugar 7.9, we added a new Platform extension that we advised developers to start using in the Sugar 7.9 Migration Guide.  The Platform extension allows you to indicate a particular custom platform should be allowed when the disable_unknown_platforms configuration setting is on.

 

Changes coming in Winter '18 release

In the Winter '18 release, we will be preventing REST API access to Sugar from unknown platform types. Sugar has a configuration setting disable_unknown_platforms that controls whether or not unregistered platforms are allowed to be used when logging in using the REST API. The current default value for disable_unknown_platforms is false. In the Winter '18 release, we will be changing the default to true, which is how it is already reflected in the documentation. If your integration uses a custom platform, this custom platform will need to be registered in each Sugar instance or your integration will break!

 

How do I know if I'm affected? New

In order to avoid conflicting with end-user sessions, some REST API integrations specify a different “platform” during login. Developers have often employed this technique to prevent integrations from interrupting or conflicting with active end-user sessions.

 

Below is an example of a login request that uses a custom platform:

 

POST /rest/v10/oauth2/token
{
  "grant_type":"password",
  "client_id":"sugar",
  "client_secret":"",
  "username":"{{username}}",
  "password":"{{password}}",
  "platform":"<SOME VALUE>"
}

 

Registering a new platform for an integration

 

Integrations must register any custom platforms they plan to use. For compatibility with Sugar On-Demand, we recommend creating a Module Loadable package that includes a simple Platform extension.

 

./custom/Extension/application/Ext/Platforms/<integration name>.php

<?php

/*

* A valid platform name requires:

* - Max length of 127 characters

* - Valid characters are: a-z, A-Z, 0-9 - (hypen) _ (underscore)

*/


$platforms[] = '<integration platform name>';

We recognize that is a change in assumption since it involves an installation of a package where previously no package had to be installed at all. We are working on an alternative approach as we roll out additional Identity Management (IdM) functionality that would allow for more convenient configuration of integrations.

 

Example Module Loadable Package

An example module loadable package has been added to the UnCon github repository. This package can be used as a template for those needing help understanding how to construct a package that will enable their API integration.

 

https://github.com/sugarcrm/uncon/tree/2017/custom-platform

Are you ready to build an integration with Sugar but not sure where to start?  You've come to the right place!

 

When you want to access or interact with information stored in Sugar, the REST API is a great place to start.  In this tutorial, you'll learn how to authenticate to the Sugar REST API.  Then you'll learn how to perform create, read, update, and delete (aka CRUD) operations.

 

Watch the video tutorial below or view the text-based tutorial at bit.ly/tutorial_rest.

 

Have you ever found yourself wishing you could create a custom Sidecar user interface within your Sugar instance? Maybe you want to allow users to visit a URL that displays a custom view.

 

It turns out that creating a linkable URL (or route) to within the Sugar client is fairly simple. In this post, I'll walk you through how to implement a new route in your Sugar instance that displays an alert.

 

What is a Route?

 

Sidecar utilizes the Backbone Router as the method for controlling client-side navigation within the Sugar application. The Router will take the current URL in the browser (the route) and call the appropriate callback function.  The callback could do something like display an alert or load a Sidecar Layout or View.  The Router will pass information included in the route like a module name and a record identifier to the callback function.

 

Most of the routes are already set up for you. For example, visiting a Sugar instance URL that ends with "#Home" will load the default dashboard layout. Visiting a URL that ends with "#Accounts/<id>" will load the Account record view for the record with the appropriate identifier.

 

Creating your own route

 

When you decide to create your own route, the first thing you'll want to do is decide what your custom route should be. In this example, we'll use "helloWorld" as our custom route, meaning that the full URL will be http://yourSugarInstance/#helloWorld.

 

Next, you need to decide what should happen when someone accesses the route. Let's display an alert message that says, "Hello, fabulous person!"

 

Now that we know what we want to do, it's time to code!

 

Create a new file in your Sugar directory:  ./custom/javascript/customRoutes.js:

SUGAR.App.events.on("router:init", function(){
    //Register the route #helloWorld
    SUGAR.App.router.route("helloWorld", "Hello World", function() {
        SUGAR.App.alert.show('message-id', {
            level: 'info',
            messages: 'Hello, fabulous person!',
            autoClose: false
        });
    });
})

 

This code registers the custom "helloWorld" route during the "router:init" event.  This code could exist in a file with any name at any location.

 

The next thing we need to do is create a JSGrouping extension that will tell Sugar we have created custom routes.

 

Create a new file in your Sugar directory: ./custom/Extension/application/Ext/JSGroupings/myCustomRoutes.php:

<?php
foreach ($js_groupings as $key => $groupings) {
    $target = current(array_values($groupings));
    //if the target grouping is found
    if ($target == 'include/javascript/sugar_grp7.min.js') {
        //append the custom JavaScript file
        $js_groupings[$key]['custom/javascript/customRoutes.js'] = 'include/javascript/sugar_grp7.min.js';
    }
}

 

Before we can try our custom route, we need to run Quick Repair.  Login to Sugar as an administrative user and navigate to Administration > Repair > Quick Repair and Rebuild.

 

Now we're ready to test.  Navigate to http://yourSugarInstance/#helloWorld.  Unless you missed a step, the alert you created will appear!Note:  you may need to refresh the page once to pull in the updated JavaScript in order for the alert to appear. It's a good idea to disable your browser cache when doing JavaScript development too.

 

If you want to take things a step further and display an iframe for a custom route as seen in the screenshot below, check out the Canvas IFrame Building Block example.

 

For more detailed information on creating custom routes including how to designate new route patterns, see the Developer Guide.