Skip navigation
All Places > Developer > Blog
1 2 3 4 5 Previous Next

Developer

181 posts

Post originally written by avlasovsugar.

 

Alex Vlasov is an experienced Sugar Developer and contractor that works on the SugarCRM Engineering platform team. In this post, Alex shares advice to Sugar Developers for preparing their Sugar customizations for PHP 5.6 and PHP 7.

 

The Sugar team has been preparing core Sugar code for PHP 5.6 and PHP 7 support with an eye on deprecating support for older PHP versions. This means there are some actions that the Sugar Developer community needs to take in order to prepare their code for latest versions of PHP.

 

Summary of core Sugar app changes

 

In order to support newer versions of PHP, we made some changes to internal core classes that Sugar Developers need to know about. Many of these changes were made in Sugar 7.7.1 when we added PHP 5.6 support. Other changes outlined below are in upcoming Sugar releases.

 

PHP 7 deprecated features removed from core Sugar code

  1. Removed static calls to non-static methods
  2. Removed PHP4-style constructors

 

Read the full list of features deprecated in PHP7.

 

Additional changes to address core Sugar code hygiene and to adopt new PHP 7 features

  1. Fixed incompatible child class methods signatures
  2. Updated 3rd party libraries to support latest versions of PHP
  3. Changes to support PHP 7.0 new uniform variable syntax
  4. Adopted PHP 7's CSPRNG API but added random_compat library to support older PHP versions

 

Actions for Sugar Developers

 

In order to properly upgrade custom code, two main tasks need to be performed:

 

Make your code compatible with PHP 5.6 and PHP 7.0

 

To make your code compatible with PHP 5.6 and 7.0, use the following checklist.

  1. Remove static calls to non-static methods within your PHP code
  2. Remove any use of PHP4-style constructors within your PHP code
  3. Remove other deprecated PHP functionality in your PHP code (PHP 5.6 deprecated features, PHP 7 deprecated features)

 

For example,

 

class Foo {

public function __construct() { // Yes.
// ...
}

public function Foo() { // No.
// ...
}

    public static function aStaticMethod() {
        // ...
    }

    public function aNormalMethod() {
    // ...
    }

}

Foo::aStaticMethod(); // Yes.

Foo::aNormalMethod(); // No.

 

 

 

For additional detail, refer to PHP migration guides for PHP 5.6 and PHP 7.0.

 

Make your code compatible with changes to Sugar PHP APIs

 

To make custom code compatible with updated Sugar PHP APIs, use the following checklist.

  1. (7.7.1) Do not use old PHP4 constructors in Sugar PHP classes, they are removed as part of PHP 7.0 support.
  2. (7.7.1) If there is code that extends Sugar PHP classes, make sure child methods signatures are correct
  3. (7.7.1) 3rd party libraries were moved:
    1. parsecsv library was moved to the vendor directory
  4. (future) 3rd party libraries updates:
    1. Elastica library will be upgraded, so make sure you do not use deprecated Elastica APIs.
    2. Removal of mcrypt support and Crypt_Blowfish library, so make sure you are not calling these libraries directly.
  5. (future) Significant changes to be made to the following Sugar PHP APIs:
    1. ActivitiesApi::getQueryObject() will become protected and function arguments are changed
    2. RelationshipHandler class will stop extending Relationship class
    3. SearchForm class will not extend EditView class anymore
    4. Quarter*TimePeriod::buildLeaves() methods will have function arguments changed
    5. PMSEEngineFilterApi::addFieldFilter() method will be renamed

Sugar is a fantastic product that is built on a powerful platform! As Director of Solutions Consulting for the Americas and an architect by trade, I am often asked to describe Sugar's platform for a variety of audiences. I have a standardized presentation that I give that provides an overview of Sugar's platform from a beginner's point of view. It covers the topics that I get asked about the most.

 

Looking for a concise overview of the components in the Sugar platform and the key highlights that makes Sugar so popular? Then this is the video for you! 

 

The topics it covers are:

  • Platform layers (top to bottom look)
  • Overview of Sugar's framework
  • Mobility
  • Security (Authentication and Access Control)
  • Open Cloud Strategy (hosting)

 

I hope you all find this valuable. As the platform grows, this presentation will continue to evolve.

 

 

I want to know what you think! Please leave feedback in comments below.

Please don’t hate me. It’s not really my fault. I am sorry to say that every single Sugar developer needs to care about the General Data Protection Regulation (GDPR) and data privacy in general. It is a sign of the times and part of the interrelationship of technology and modern society that impacts the software we implement today and in the future.

 

I can explain but first let me provide some context.

 

What is the General Data Protection Regulation (GDPR)?

 

First off, I am not a lawyer or a GDPR specialist. If you really want to understand what GDPR is and what steps you need to take to be compliant, then I suggest reading the Regulation for yourself or hiring counsel. GDPR is a complex set of rules, and there’s still a lot of debate about what it all really means. Other interpretations, including yours, may be different from what I describe below.

 

Many of our colleagues in the European Union are well acquainted with the GDPR which requires full compliance by May 25th, 2018. But for those of you who do not live in the European Union (EU), I highly recommend reading Data Privacy and GDPR in Sugar by Deepak Deolalikar. It covers some GDPR basics as well as provides an overview of some of the features planned in Sugar to help our customers comply with GDPR.

 

In short, the spirit of GDPR is about allowing individuals (data subjects) to have control over their own personal information. GDPR ensures that personal information or Personally Identifiable Information (PII) of EU citizens are processed responsibly. EU citizens have a right to privacy that in many cases means you must comply with their requests to access, restrict the use of, or delete the personal information that is stored about them.

 

The scope is broad and the penalties are severe. Any organization that stores personal information about EU citizens, even if that organization is based outside the EU such as in the United States, is subject to this regulation.

Penalties for failure to comply with these regulations could result in fines that start at €20 Million or 4% of total revenue if that value is greater.

 

Are you paying attention now?

 

Data Privacy is yet another software globalization requirement

 

When we build software, we make sure that it meets all sorts of requirements. We design for performance, scalability, and security. We work hard to ensure a positive user experience and high quality software. At SugarCRM, we also focus on making sure that all our products are ready for the global marketplace. That means that over the years we have invested in translations into dozens of languages and support for multiple currencies, numerous date and time formats, right-to-left languages, Unicode character encodings (though we’re still working on emojis), and 508 compliance.

 

The fact is that GDPR is just one example of a data privacy regulation at a time when there are dozens more government bodies all over the world considering and implementing new data privacy regulations. UTF-8 character encoding and a translatable UI are no longer enough to make sure your software is ready for the global marketplace! The responsible collection and processing of personal information is now an additional obligation for all software developers.

 

With the upcoming Sugar Spring ‘18 (cloud) and 8.0.0 (on premise) release, we are now investing in data privacy features. These features will help provide our customers with the tools they need to comply with GDPR and many other data privacy rules or regulations.

 

These data privacy features and other data privacy concerns will impact Sugar customizations and integrations that collect or process personally identifiable information (PII).

 

Future installments on Data Privacy and GDPR

 

We will be exploring several data privacy topics in the coming weeks. Here is a quick summary.

 

Collecting explicit consent before storing data in Sugar

 

GDPR has strict requirements for consent. Unless you have an existing lawful basis for using personal information, you need to collect positive and unambiguous consent from the data subject even before data is stored in Sugar. For example, a web to lead form with a pre-selected opt-in checkbox is not going to cut it in the EU. We will be exploring ways to make sure you are collecting explicit consent using techniques like double opt-in (DOI) or confirmed opt-in (COI).

 

Managing PII data and the right to erasure in Sugar

 

There are new Sugar APIs being added for working with PII as well as features that allow Sugar customers to comply with data subject requests like the right to access their own data and the right to erasure. In particular, external systems that integrate with Sugar may need to identify and implement their own erasure measures in compliance with a data subject request.

 

Improved change log and attribution of changes using Data Sources

 

We are implementing some enhancements in Sugar’s change log functionality to allow for finer grain tracking and proper attribution for changes to PII. We will be exploring these platform level enhancements including the ability to define new data sources (for example, an integration) that are responsible for changes to data.

 

What do you think?

 

What are some of your biggest data privacy or GDPR concerns? Let us know and we’ll try to address them in future posts.

 

Also, please follow this space and this blog to be sure you are notified when the next installment is posted!

What is the Sugar Metadata API?

As you probably know, Sugar metadata encompasses the settings, data model (Vardefs), and visual layouts (Viewdefs) used by the Sugar application. The majority of changes made to Sugar using Sugar Studio, Module Builder, or even custom code are implemented by modifying metadata.

 

How can I know that the custom field or custom modules that you need for your integration to work exists in Sugar?

How can I know what are the allowed values are valid for a Sugar dropdown field?

How can I know if a field is required?

How can I know the display label for a given field?

How can I know what fields are available in Sugar that I can map back to an external data source?

 

The answers to all these questions and many more can be found in Sugar metadata.

 

On the server side, metadata is implemented as an extensible set of PHP associative arrays. But how does this information get sent out to Sugar clients like your web browser, mobile app, or even the Outlook plug-in? This is accomplished via the Sugar metadata endpoint of the Sugar REST API.

 

GET /rest/v11/metadata HTTP/1.1
Host: localhost:8080
OAuth-Token: ...
Content-Type: application/json

 

The full metadata response comes back as a very large JSON object. It's usually a few megabytes in size (about 3MB in a stock version of Sugar Winter '18) but can be much larger depending on customizations. Make sure compression is enabled on your web server if you are going to be using it over a slow connection!

 

 

The value to using the Metadata API is that you can discover changes in Sugar configuration and Sugar customizations. New custom fields, relationships, modules, changes to Sugar views and layouts, and much more are all represented in Sugar metadata responses.

 

From a REST API perspective, the only way to accurately know the current data model for any given Sugar instance is to use the Metadata API.

 

Since the Metadata API is so important, here are three important tips to keep in mind to make development easier.

 

Tip 1: Retrieve only what you need

Metadata API calls can generate a huge response depending on your Sugar instance configuration and the customizations that may exist. You should specify filters on your metadata request in order to narrow the response to the data that you actually need. Narrowing what is returned will improve performance on both server side in terms of generating the response but also on the client side where you need to parse the JSON response.

 

For example, if you are integrating with only the Accounts module then you may only need to concern yourself with that part of the metadata. Specify a type_filter and module_filter as URL parameters to only return module metadata for the Accounts module. Other supported type_filter options are server_info, config, relationships, labels, languages, currencies, views, layouts, full_module_list, and etc.

 

GET /rest/v11/metadata?type_filter=modules&module_filter=Accounts HTTP/1.1
Host: localhost:8080
OAuth-Token: ...
Content-Type: application/json

 

Tip 2: Cache metadata, update only when it changes

You can safely cache the metadata that you need if you keep track of the _hash value returned in the metadata response.

 

You should present that hash value in the X-Metadata-Hash header on subsequent Sugar REST API requests to ANY endpoint. If the metadata hash is out of date then Sugar will return a 412 PRECONDITION FAILED HTTP response. You should then retrieve updated metadata from the Metadata API and store the updated metadata hash value.

 

GET /sugar/rest/v11/Accounts HTTP/1.1
Host: localhost:8080
OAuth-Token: ...
Content-Type: application/json
X-Metadata-Hash: 705fb99230e2ec60b59f481c6831a7cf

 

Also worth mentioning, many of our API resources (such as records and our metadata) implement HTTP caching. If an API response contains an E-Tag HTTP header then you can safely cache the response body as well as the E-Tag value. If you present this value in the If-None-Match HTTP header properly on a subsequent HTTP GET request for that resource, you will get an empty 304 NOT MODIFIED response back if it is safe to continue using the cached version of the previous response. This is another approach to reduce needless processing.

 

GET /rest/v11/metadata?type_filter=modules&module_filter=Accounts HTTP/1.1
Host: localhost:8080
OAuth-Token: ...
Content-Type: application/json
If-None-Match: 705fb99230e2ec60b59f481c6831a7cf

 

This allows you to update your local copy of the metadata as soon as it changes and precisely no more often than that.

 

Tip 3: Different users and platforms will retrieve different responses

If a user is assigned a role that doesn't have access to a particular module then the metadata for that module will not be returned when that user accesses the metadata API. Makes sense, right? 

 

Sugar also support things like role based views which will provide alternative UI layout metadata based upon the current user's role.

 

Different platforms will have different metadata responses as well. The mobile platform retrieves module metadata that reflects the needs of our Sugar Mobile app. Mobile UI components look a lot different from what you see when you access Sugar from your browser. The metadata that is retrieved is significantly different to accommodate these changes.

 

For example, the mobile client uses separate edit and detail views for representing records instead of a single record view like you find in Sugar web client. This is represented in the mobile metadata response.

 

One of the first things people do when they start thinking about implementing DevOps best practices is setting up continuous integration and continuous deployment (CI/CD) tooling.  So that’s just what we did with the Professor M School.  (Not sure what I’m talking about when I say the “Professor M School?”  Check out my earlier blog post Introducing Professor M’s School for Gifted Coders.)

 

We’ve set up CI/CD with not one but two popular tools:  Travis CI and Jenkins.  As you decide which tool to use, consider the following tradeoffs.  Travis CI is hosted online so you don’t have to set it up yourself.  Open source projects can leverage Travis CI for free, but closed source projects will have to pay.  Jenkins requires you to set up and manage the app yourself but is free for all projects.  

 

Travis CI Build
Automated build inside of Travis CI.

 

Jenkins Build

Automated build inside of Jenkins

 

We’ve set up the same basic flow for both CI/CD tools:

  • Run the automated tests
  • Build the Professor M module loadable packages

 

In addition, the Travis CI build is configured to post the Professor M module loadable packages to the GitHub releases page whenever a commit is pushed to the master branch.

 

You can learn more about how we’ve implemented CI/CD in the Professor M School’s Readme:

 

You can also explore the results of our latest Travis CI builds at https://travis-ci.org/sugarcrm/school/.

 

We hope you’ll be able to leverage the work we’ve done to setup CI/CD with Travis CI and Jenkins in your own projects!   

One of the biggest pains in developing a module loadable package for Sugar is figuring out how to track the package in source control.  You don’t want to keep your entire Sugar directory under source control--you just want your custom code in source control.  However, from a development perspective, this can be a huge pain.  If you do your development separate from the Sugar directory, every time you make an incremental change, you have to manually build and install the module loadable package, which is a process that can quickly become very time consuming.

 

We’ve developed a tool you can use called the Sugar Synchronizer.  The Synchronizer allows you to develop your Sugar customizations in your Sugar directory while still tracking your changes for your module loadable package in a separate directory.  You can see a live demo of the Sugar Synchronizer in the Sugar Winter '18 Technical Training

webinar recording (you may want to jump ahead to the 38:42 mark).

 

The Sugar Synchronizer is open source, and you can get a copy of the code in our GitHub repo:  https://github.com/sugarcrm/project-sync.  The repo’s Readme has instructions on how to use the Synchronizer.

 

If you have suggestions for improvement or find a bug, please create an issue in the GitHub repo.  Or, better yet, write some code and submit a pull request!  

 

We hope the Sugar Synchronizer makes developing module loadable packages easier for you!

We’ve been working behind the scenes to create a customized version of Sugar that we’re calling the Professor M School.  The essential pieces are ready to go, and we’re thrilled to announce that the repo for the Professor M School is now public and open source.  

 

So what is this the Professor M School?  Let’s begin by talking about the goals for this app:

 

  1. The app should be an authoritative example on upgrade-safe customizations.  We want you, the Sugar developers, to be able to use this app as a reference for what good, upgrade-safe customizations look like.

  2. The app should provide examples of DevOps best practices when developing Sugar customizations and integrations.  For example, we want to have examples of automated test suites and how to leverage continuous integration tools.

  3. The app should include common customizations so that SugarCRM’s engineers can use it for testing and validation.  When our engineers want to know if their changes are going to break common customizations, we want them to be able to use this app to find out.  Ultimately, we want to integrate the app’s automated tests into our internal continuous integration process.


With those goals in mind, we developed the scenario and use cases for the Professor M School.  The app is built around a fictitious university called Professor M’s School for Gifted Coders.  If you attended UnCon 2017, you might remember hearing about Professor M.  We’ve built on what we created for UnCon in order to create the Professor M School.  


Let’s talk about the scenario behind the Professor M School.  Professor M aka Professor Marum has created an exclusive not-for-profit school for gifted coders.  The school currently uses Sugar for the following use cases:

  • Managing applicants, current students, former students, and professors
  • Tracking super groups affiliated with students
  • Soliciting donations from alumni and alumni affiliated super groups.

Professor M will be expanding his use of Sugar to cover other use cases in the future.

To learn more about the Professor M scenario and see a demo of the customizations, check out this video I previously recorded (you might want to jump ahead to the 1:07 mark).

As I mentioned earlier, the Professor M School is open source.  Here are some ways that you can get involved:

  • “Watch” the Professor M School’s GitHub repo: https://github.com/sugarcrm/school.
  • Explore the code and use it as a resource as you develop your integrations and customizations.
  • Submit GitHub issues if you have ideas for improvement, a feature request, or a bug report.
  • Update the code and create a pull request.

 

We hope you find the Professor M School to be a useful resource!  Check back soon for more updates on the app!

Attention developers!  We’ve released Winter '18!  This is our second quarterly On-Demand only release, and we’re pretty excited about it.  Our Co-Founder & CMO, Clint Oram, discusses the highlights of this release from an end-user's perspective in the video below:

 

Matt Marum and I recently hosted a webinar where we gave an overview of the big things developers need to know about this release:

 

The slides from the webinar are available here .

 

If you’re looking for the CliffsNotes version, I’ve got you covered:

  1. Sugar has a new skin, and it’s fabulous.  No JavaScript changes were made as part of this reskinning.  For more information, check out Matt’s blog post: A new Sugar UX coming in Winter '18 .

    Before:
    Old look
    After:
    New look

  2. The Input Validation Framework that has been implemented since Sugar 7.7.1 is now enforced by default.  Input validation violations will cause FATAL errors.  See Matt’s blog post for more details:  How to fix Sugar input validation failures .  

    As part of enforcing the Input Validation Framework, all custom platforms must be registered.  If you have a customization or integration that uses our REST API, this probably affects you.  Check out my blog post  that has links to get more information about the change and a tutorial to help you register your custom platform.
  3. You now have the option of turning on denormalized team security, which can greatly improve List View performance.  Check out the Developer Release Notes or the webinar recording above for more details.
  4. We’ve added PhaserIO as a new library.  As a result, we’re seeing outstanding performance in the new Product Catalog Dashlet and great potential for speeding up other parts of the app.  Check out the Developer Release Notes for a list of all library changes.
  5. We’ve been working on an Exemplar App (https://github.com/sugarcrm/school) and the Sugar Synchronizer (https://github.com/sugarcrm/project-sync), which we hope will become key resources for everyone.  Watch the blog for more details coming soon!

 

This blog post promised to have just about everything you need to know about the Sugar Winter ‘18 release.  Below are some resources that have the rest of the details.

 

We hope you’re as excited about this release as we are!

What is the Input Validation Framework?

The Input Validation Framework is a centralized security framework used to validate that external input is provided in the expected format. Validating user input is critical because it is the basis for generated HTML and REST API output, used in the construction of SQL queries, used to access files, etc. Malicious input can be used to compromise the security of a Sugar instance and the underlying infrastructure. The validation framework provides better control than the pre-existing input sanitization techniques applied to user input within Sugar.

 

Fixing Input Validation problems

Sugar’s input validation was enabled but only logging failures as warnings into the sugarcrm.log file since the 7.7.1 release. With the Winter ‘18 release, the validation.soft_fail configuration setting will now default to false. This means that input validation checks will be strictly enforced by default instead of simply logging warning messages.

 

For more information about using validation constraints in your own custom code, check out the Sugar Developer Guide, the previous UnCon Security presentation, or the Winter '18 webinar.

 

Identifying input validation violations

Beginning in the Winter ‘18 release, input validation violations are reported as PHP Fatal errors in the PHP error log and Critical error messages in sugarcrm.log file. Invalid requests will generally be terminated with an HTTP 500 error as well.

 

Let’s walk through an example of an input validation violation.

 

In this example, we have a customization that is accessing the Sugar instance at the following URL: sugarcrm/ult/#bwc/index.php?module=xxxxx.

 

If xxxxx is not a registered module, there will be a FATAL error in the PHP error log.

For example,

[15-Jan-2018 22:04:09] WARNING: [pool www] child 1940 said into stderr: "NOTICE: PHP message: PHP Fatal error:  Uncaught exception 'Sugarcrm\Sugarcrm\Security\InputValidation\Exception\ViolationException' with message 'Violation for REQUEST -> module' in /srv/www/<instance>/prebuild/test/ult/sugarcrm/src/Security/InputValidation/Request.php:280"
[15-Jan-2018 22:04:09] WARNING: [pool www] child 1940 said into stderr: "Stack trace:"
[15-Jan-2018 22:04:09] WARNING: [pool www] child 1940 said into stderr: "#0 /srv/www/<instance>/prebuild/test/ult/sugarcrm/src/Security/InputValidation/Request.php(209): Sugarcrm\Sugarcrm\Security\InputValidation\Request->handleViolations('REQUEST', 'module')"...

 

The sugarcrm.log will also show an error.

For example,

Tue Jan 16 06:05:18 2018 [1941][-none-][CRITICAL] InputValidation: [REQUEST] module -> Invalid module xxxxx

 

From the end user perspective, these fatal errors will cause pages to appear blank or stalled during loading. If you inspect the browser console when this happens, you will likely see HTTP errors logged.

 

Fixing input validation violations

The steps to take to fix the violation are going to depend on the exact constraint that you are violating. Luckily the exceptions you find in the PHP error log and the messages in the sugarcrm.log will indicate the constraint that you are violating.

 

In the above example, attempting to access a BWC view for an unregistered custom module “xxxxx” is causing the failure. If you make sure that the module you are using is properly registered in Sugar’s module list, or modify your request to use an existing Sugar module, then this will fix this issue since the constraint is checking if the module name is a known modules installed in this Sugar instance.

 

There are basic Symfony framework PHP constraints that we use. These do not need an in depth explanation. The basic Symfony constraints can be used to validate URLs, date time formats, etc. Check out the Symfony documentation for these basic constraints.

 

In Sugar's implementation, we add our own set of validation constraints. These constraints extend from Symfony's Constraint and ConstraintValidator classes and are located in the ./src/Security/Validator/Constraints directory of your Sugar installation. Each Constraint is enforced using a related Validator class. You can inspect the Validator classes to see the precise logic that needs to be followed. For example, the Bean/ModuleName constraint is implemented by the Bean/ModuleNameValidator class. This constraint is used to ensure that the input is a string that matches the name of a valid module with an associated SugarBean class.

 

One of the basic rules that most of these constraints implement is that an input type should be a scalar. This means we expect most user input to be integer, decimal, string, or boolean values.

 

Sugar Constraints

The list provided below may not be exhaustive.

  • Bean/ModuleName
    • Must be a string. The Module’s SugarBean class must be retrievable using BeanFactory. For example, this is used in legacy Workflow.
  • ComponentName
    • Must start with a letter and may only consist of letters, numbers, hyphens and underscores. For example, this is used throughout Studio and Module Builder for module names, package names, and Sugar Logic functions.
  • Delimited
    • By default, must be comma delimited scalar values. For example, this is used in conjunction with other constraints.
  • DropdownList
    • Dropdown lists must only consist of letters, numbers and underscores. For example, this is applied to dropdown lists during module install.
  • File
    • File must exist in an allowed file path and must not include null characters or any directory traversals. For example, this is used by FileLoader utility class.
  • Guid
    • Must be a string with alphanumeric or dash characters. For example, this is used with SugarMVC Views.
  • InputParameters
    • General input validation for GET, POST, REQUEST superglobals. Must be scalar values and strings should not contain null characters.
  • JSON
    • Must be a string that is JSON decodable. Returns decoded PHP associative array. For example, used in Configure Navigation Bar Quick Create.
  • Language
    • Must be a locale code that matches an installed language. For example, this is applied where languages can be selected in UI.
  • LegacyCleanString
    • Reimplements input sanitization rules previously used with clean_string() function.
  • Mvc/ModuleName
    • Must be a string. A more permissive test than Bean/ModuleName that allows any module name that can be found in global module lists. For example, this is used throughout SugarMVC Views.
  • PhpSerialized
    • Must be a PHP serialized string that does not contain objects. Returns unserialized value.
  • Platform
    • Must be a string that is under 128 characters and contains alphanumeric and dash characters. Must be a known Platform value such as “base”, “mobile”, “portal”, etc., or one registered using Platform extension. For example, this is used to validate REST API login requests.
  • Sql/OrderBy
    • Must be a string that represents a valid ORDER BY SQL clause. For example, this is used in SugarMVC list views.
  • Sql/OrderDirection
    • Must be a string that is uses SQL keywords “ASC” or “DESC”. For example, this is used in SugarMVC list views.
  • SugarLogic/FunctionName
    • Must be a string that only uses word characters and dashes. For example, this is used in Sugar Studio.

 

If you encounter a failure, then you should be able to identify the constraint that you are violating from your error logs. By looking at log messages that appear immediately before the failure, and possibly some exploration in the Sugar user interface, you will be able to identify the failing request. From there, you can use the information above to alter the request to comply with the necessary constraints.

 

This is not a substitute for Sugar field level validation!

An important thing to note about the Input Validation Framework is that it is not the only input validation mechanism available. For example, there is still separate field level validation that occurs when users create or edit records. Sidecar customizations or Studio configuration can still be used to perform validation of user provided input for each field of a record in a user friendly way. For example, Sugar will let you save a record without filling out required fields and will display a user friendly warning that tells user what to correct. The Input Validation Framework is centered around security and does not provide the same usability than you would get validating user input using Sidecar or Studio settings.

 

In Sugar 8 / Spring '18, Sugar administrators can now configure API platforms using the Administration panel. The Platform extension is still available if you want to register a custom API platform as part of a Module Loadable Package.

If you have a REST API integration with Sugar, you are likely using a custom platform as part of that integration. Beginning in the Winter ‘18 release, Sugar is enforcing a rule that all custom platforms must be registered in each Sugar instance where a custom platform is used.  For more details on the rule itself, see Matt’s earlier blog post: Unknown platforms to be restricted in Winter ‘18 release.

 

The bad news?  This change that requires all custom platforms to be registered probably affects you.

 

The good news?  There is an easy fix, and I’ve written a tutorial that walks you step-by-step through how to implement the fix.

 

Check out the tutorial here!

The Sugar Winter '18 release is fast approaching! Hopefully you are as excited about the new Sugar UX as I am!

 

As usual, Lauren Schaefer and I will be hosting a developer webinar to help get you and your customers prepared for the new Winter '18 (7.11) release. Please read below to learn more about the agenda and do not forget to register! We run these webinars twice, so no matter where you are in the world, we hope that one of these times will be convenient for you.

 

Winter '18 Technical Webinar Agenda

 

Technical Training AgendaWebinar Times
  1. Impacts of new interface
  2. Sugar Exemplar app (more below)
  3. Restricted REST API platforms (more below)
  4. Removal of some LESS files
  5. Emails subpanel changes
  6. Input Validation Framework (now the default)

Monday, January 22nd 4-5:00 PM PT
Tuesday, January 23rd 8-9:00 AM PT

Register Here

 

We will be posting the webinar recordings and deck to the Partner Portal for those who are unable to attend the live sessions.

 

Sneak peak of Sugar Exemplar app


The new Sugar Exemplar app is designed to demonstrate best practices for developing Sugar customizations (think DevOps and QA!) and a solid example of what upgrade-safe customizations should look like. We’ll tell you more about the app and how you can get involved during the webinar!

 

Reminder about restricted REST API Platforms


ISVs and developers who use a platform string to connect to a Sugar instance will need to update their existing API integrations. For more information - please refer to this blog post on how to to proceed.

 

Get on our developer mailing list!

 

Remember to opt-in for Sugar Developer communications to make sure you don't miss this and other important updates.

TLDR 

Developing Sugar on Docker, get up to speed now!

 

The long version

 

What are Docker containers?

If containers for you are more than the two pictures below… then you are in the right place!

 

 

As you might be aware, there has been and there continues to be a lot of hype around containers. The hype really started with Docker, but containers have been around for a while (more or less), in a different shape or form. If you are not yet familiar with containers or are confused about the difference between a virtual machine and a container, this blog post from Docker might be useful to you.

 

Personally, I was deploying container-based virtualised single purpose hosting environments (now labelled micro-services?) for my customers, back when it was not so trendy. It was about 10 years ago when I was leveraging OpenVZ. OpenVZ is slightly different (eg: it has state/persistent storage for every container), but in the end you can run multiple lightweight containers that share the custom linux kernel of their host.

 

What Docker has managed to do differently is make infrastructure deployments repeatable and portable. An application can be shipped together with its Docker container directives, providing the same exact configuration across different host environments. The infrastructure looks more like code, which simplifies the process of automating and delivering software deployments. Teams using Docker found that the lines between development and operations blurred even further, and guess what? DevOps was born!

 

Docker has made it easy to get up and running with a technology for development purposes. A developer just needs Docker installed on his or her local machine (or on a linux VM) and a “Dockerfile”, to get an application up and running. If he or she needs to replace a single component of the infrastructure for testing purposes, they simply swap a few directives and re-build. The same application is executing on top of a slightly different platform. How awesome and easy is that?

 

With the always increasing pace of software development innovation required today, and with applications becoming more and more decoupled from each other (and at the same time integrated), development teams need to keep their deployment velocities up as well quickly deliver innovation across the board on a multitude of heterogeneous backends.

 

This need for a fast deployment velocity can be fulfilled with the aid of flexible containers and orchestration technologies when their delivery is seamlessly baked into the traditional software development pipeline. Obviously containers are not the silver bullet or the answer to every known question, but they are currently applicable to a vast set of different workloads and use cases.

 

Docker in production

At this stage, I do not have direct experience with Docker in production environments, but, as with any technology, there is always someone willing to share their horror stories…

 

Jokes aside; with the increase in popularity of orchestration, the extensive backing and support from various cloud providers and how lightweight it is, I am convinced Docker is the way to go for micro-services deployment and orchestration in production environments as well. As a matter of fact, container-technology adoption is constantly growing across the board and not just for micro-services.

 

Sugar on Docker

Enough said about the containers topic. Here it is, my attempt at creating a set of development stacks for Sugar with Docker.

 

I have to thank one of our long-standing partners Cédric Mourizard from Synolia for the initial work (read more in his previous blog post). The Sugar Docker setup has been polished here and there over the past 6 months, and it is now to a point where I only run Sugar locally with this solution.

 

I highly encourage you to give it a try. Have a look at how Docker can fit in your development process, and, if it does not yet fit into it, think about how it could help you streamline your overall software delivery pace and capabilities.

 

Flavours used

Let’s give a quick overview of the various components used. When using *nix, I’m a Debian kind of guy. So that’s that.

LAMP stack, Elasticsearch and Redis on top of Debian.

 

Stack components

It might evolve over time, but for now the main containers are:

 

  • Apache load balancer - Load balances requests between the cluster of Apache PHP web servers, round robin
  • Apache PHP web server - Web server(s)
  • MySQL database - Database
  • Elasticsearch - Sugar search engine
  • Redis - Two purposes: Sugar object caching service and PHP Session storage/sharing service
  • Cron - Sugar background scheduler processing. Note that this is enabled immediately and it will run cron.php as soon as the file is available, and it will attempt to do so every 60 seconds since its last run
  • Permission - Sets Sugar instance permissions correctly and then terminates

 

The application files are shared on the “cluster” (if you can call it that) through a directory based mounted volume to offer persistent storage. We also have various persistent storage directories within ./data/ for all the other software components that need them.

 

Based on the selected stack (within the ./stack/ directory) you will be able to pick and choose the various versions of the stack components available, and to decide if you want a load balancer and multiple web servers or just a single web server during initial development phases. Because the Docker containers are stateless, it is possible to run the same Sugar installation (stored on the persistent volume) either with PHP 7.1 or PHP 5.6 and swap between versions within seconds.

 

Current software versions

Currently, there are a few stack combinations available that run the supported platform for Sugar 7.9.

Containers are available with PHP 7.1/5.6, MySQL 5.7, Elasticsearch 1.7.5 and the latest Redis. When selecting a stack, it is possible to run it as a single web server or as two web servers behind a load balancer.

 

Sugar 7.10 is not available for on-site hosting but it can be downloaded for local development, so, I have prepared a single stack leveraging PHP 7.1, compatible with the required matching platform. The main difference in stack requirements between Sugar 7.10 and earlier versions is the Elasticsearch version 5.4 requirement. Elasticsearch 5.4 seems to be more memory hungry, so be prepared for that.

 

Finally, there is an additional stack for 7.9 upgrades, running both Elasticsearch 1.7.5 and 5.4, to allow future testing of upgrades to the Sugar Spring '18 release.

 

Getting up and running

Most of the details you need to get up and running are within the README.md of the git repository. In terms of where to run this setup, you have the option of leveraging docker-machine or to run your own VM, as currently the native Docker for Mac does not perform well with the mounted volumes for persistent storage.

 

I personally use a Debian VirtualBox VM on a Macbook. If you want to run the same VM as I do, I have prepared a VirtualBox appliance export. You can find the appliance on this repository, as well as more details on how to deploy it on your Mac. It runs a NFS share as well, so that you can mount the directory locally and code with your preferred editor.

 

During my process, I’ve also tried Vagrant to help with automatic deployment of the VM instead of building an appliance, but I did not obtain the same performance footprint (not even close). So I abandoned that route long ago.

 

Now you are up to speed with Sugar on Docker, and you might have realised how containers can help your business gain velocity.

 

Feel free to let us know your thoughts, by leaving some comments below!

Sugar v11 REST API

In order to support some cool new dashboard features, we made some enhancements to our Dashboard APIs in the Fall '17 release. So we added a new REST API version, v11, in that release in order to ensure clients and integrations that were using the existing v10 Dashboard APIs would continue to work properly. For more details on the v11 API, we recommend you check out a recording of our recent Fall '17 release developer webinar and the related migration guide.

 

Today we won't focus on version 11 specifically but instead on how the Sugar platform will be handling REST API versioning in the future. So read on!

Passing API version using HTTP Header

For REST v10 and our older web services frameworks, the API version was specified exclusively via the URL.

 

Example 1)

 

GET <sugar>/rest/v10/me

 

In the Fall '17 (7.10) release, we introduced an additional way to specify the API version: using an HTTP header. This change allows resource URLs to be treated as unique identifiers to resources regardless of the API version in use.

 

The REST API version can now be specified via HTTP Accept header. The REST API version should be specified in either the URL or in the HTTP Accept header but not in both locations at once. This new method of specifying the API version in the HTTP Accept header better adheres to RESTful architectural principles. By essentially encapsulating the API version as part of media type, a single resource at a given URL could offer multiple representations using different data formats and API versions.

 

Example 2)

 

GET <sugar>/rest/me
...
Accept: application/vnd.sugarcrm.core+json; version=42

 

We recommend that client start specifying the API version in the Accept header instead of as a component of the URL. Sugar clients will adopt use of the Accept header in the future.

Future use of semantic versioning in REST API

As you probably have noticed, the REST API version is a property of the Sugar REST API as a whole and not individual endpoints.

 

We plan to implement a semantic versioning policy with MAJOR.MINOR format for the REST API next year.

 

The version will be specified in the request Accept header as MAJOR.MINOR (ex. 11.1) whereas the version in the endpoint url will be specified as MAJOR_MINOR (ex. 11_1).

 

The Sugar REST API MAJOR version will be incremented for each Sugar release where compatibility is being broken on any endpoint. The Sugar REST API MINOR version will be incremented for each Sugar release where an API endpoint is changed or additional API endpoints are being added without breaking compatibility.

 

Not every new Sugar release will result in new REST API versions. Our intent is to only add new features or make breaking changes to our REST API only when it is not possible or prudent (ex. for performance reasons) to do so with the current API.

 

When there is a breaking change, older API versions should continue to work as-is. For example, if a parameter was removed in newer version, the old version should still support that parameter.

What are breaking and non-breaking changes?

We want to be clear about what we consider to be an API breaking change and what is not.

Breaking Changes

  • Removing or renaming an endpoint, a request parameter, or a fixed response field
    • For example, removing the GET /rest/v10/<module> endpoint or renaming the _acl field to _access_control.
  • Removing support for a HTTP request method on an endpoint
    • For example, changing from PUT to PATCH method for updating records.
  • Change in type for a request parameter
    • For example, a change from an Integer to Decimal value.
  • Change in response field format
    • For example, a change in encoding for a response field from plaintext to base64.
  • Semantic changes to request parameters
    • For example, a change from treating the % symbol as a literal instead of as a wildcard or a change in default value.
  • Semantic changes to an API endpoint
    • For example, an endpoint that was documented to return all records by default and now returns no records by default.

Non-breaking Changes

  • Metadata related changes in behavior
    • For example, when Sugar metadata (Vardefs, Viewdefs, etc.) changes are introduced during a Sugar release or as a result of a customization then REST API responses can vary as a result.
  • Adding a new endpoint
    • For example, adding a GET rest/Accounts/:id/map endpoint.
  • Adding a parameter to a request
    • For example, adding a boolean parameter called shorten_text that decides if long text fields are shortened to 200 characters.
  • Adding a field to a response
    • For example, adding a _self field with link to resource URI in GET responses.
  • Adding support for another HTTP request method
    • For example, adding support for PATCH method to an endpoint.

 

If you ever encounter a situation where there is a breaking change on a particular API version between future Sugar releases, then please report those issues via the Sugar Support portal.

Deprecation process for old API versions

Old REST API versions will continue to be supported until at least one year after they have been deprecated. We want to exercise good judgement on how quickly we remove old API versions but please understand that SugarCRM cannot indefinitely support an unlimited number of REST API versions.

What should you do?

  1. When building new Sugar clients or integrations or updating existing ones, adopt the latest available REST API version possible in order to take advantage of all the latest features.
  2. Use the HTTP Accept header to specify REST API version when using Sugar Fall '17 release or later.
  3. Monitor the release notes and documentation on each new Sugar release for deprecation notices on old API versions. Keep our customers safe by planning ahead!

RESTful APIs have always been a big part of the Sugar application, as well as the most appropriate touchpoint for integrating with other enterprise systems. Unfortunately, for a long time they weren't tested especially appropriately. Traditionally, REST APIs at Sugar were tested with PHPUnit, but only on the classes themselves - they weren't actually tested via direct use of HTTP. Worse, the tests were messy - they often assumed existing data was present, and when it wasn't, it was created and not deleted afterwards.

 

Thorn was envisioned as a solution to this problem. It's intended to test Sugar's RESTful API directly, abstract away unhelpful boilerplate, leave your database clean, and let you test like a user.

 

Thorn was open-sourced earlier this year after a brief internal testing period and is now available for any Sugar developer to use from github.com/sugarcrm/thorn. It is open source and available under the Apache License, version 2.0.

 

This initial post will focus on a technical and philosophical overview of Thorn. For usage information, see the Thorn website.

 

Key design elements of Thorn tests

Thorn has a few key design principles that informed how it was developed and how we test with it.

 

Test like Goldilocks

If all you're trying to prove with a test is that a REST API works, an end-to-end test with Selenium (or similar) is too expensive and too likely to fail for unrelated reasons to justify running it. Unit tests aren't particularly helpful either; most API bindings don't carry around all that much interesting functionality since they're mostly proxies to other code that do the real work.

 

Thorn is thus meant to test an API as an API, by using real HTTP requests and real data. The result is nimbler tests that nevertheless accurately reflect how Sugar's APIs are actually used.

 

Thorn does this by building on top of the excellent Chakram framework - an extension to the well-known Mocha test runner that makes it easy to make HTTP requests from within tests and make assertions about their responses.

 

In other words, Thorn is just right for testing REST APIs. (Cue groans from the readers.)

 

Take nothing but responses, leave nothing but logs

Tests that make implicit assumptions about the system under test have a number of issues. They can't be safely parallelized or run in isolation, because they might be assuming that one of the other tests in the suite provides that data.

 

Furthermore, a test you can only run once is a useless test, because if it fails, you may have to wipe out your entire environment and start over just to make one single code change. If the test fails again, this will get frustrating very quickly.

 

Thorn's Fixtures API provides the solution. Fixtures.create lets you decide what records should exist in the database before your test begins, and Fixtures.link can be used to create relationships between them. You only have to give values to fields you really care about - anything else necessary will be transparently filled in by Thorn. Finally, Fixtures.cleanup cleans up whatever mess in the database your fixtures made.

 

Another nice side effect is that a good Thorn test can run on any compatible Sugar instance - even an anonymized customer instance, in the case of partners or system integrators.

 

Think of a good Thorn test like a backpacking trip. You have to bring everything you need with you, so pack light and only bring what you need. And don't ruin the trail for the next camper to come through - clean up after your data.

 

Be the user

The only reason an API should exist is so a front-end (or a different service's back-end) can talk to it. But either way, ultimately APIs exist because some user felt it would be helpful for a service to do something. To that end, a good Thorn test is written with the users of the API (reflecting real usage by end users in real roles) in mind.

 

Mocha's traditional behavior-driven-development interface helps here, but Thorn's main advantage is the UserAgent interface. With Agent.as, you can specify exactly what user you want to perform an action as, then use Chakram's convenience methods (get, post, etc.) to perform exactly the same action that a Sugar user would do.

 

Your users expect your API to be honest. So when you’re writing a Thorn test, keep that in mind and keep them honest.

 

Remember your passwords

Authentication, session expiration, and re-authentication are some of the thorniest (pun intended) issues you'll experience while working with APIs.

 

Thorn provides solutions to all of these problems, with an aim of limiting manual developer intervention and undesired nondeterminism whenever possible. Authentication is performed implicitly by Thorn for each user involved in a test; there's no explicit login required.

 

Furthermore, if a user's request fails mid-flight for authentication reasons, the request is automatically retried and the test proceeds as normal.

 

You wouldn't want a user to have to login every time they go to another page of your app and Thorn doesn't expect that of you either.

 

Cool, how do I stop RESTing and start testing?

For now, check out the Thorn website. In a later post, I'll go into greater depth about good Thorn testing practices and show how you can test a sample REST API endpoint.

SugarCRM is on a mission to empower our users to delight their customers. To that end, we are pleased to introduce the first phase of visual restyling, and give some tips on how to work with the new UX.

Our process in developing the new UX

Sugar 7 introduced the Sidecar framework with massive user interface improvements (called SugarUX) and a modernized architecture. Since then, we have been working hard on the next iteration of SugarUX built on the Sidecar framework. Our guiding principles came from our customers and their feedback on Sidecar. A competitive analysis and feedback from customers revealed three concerns; cluttered complicated UX, inefficient unguided layout, and a drab outdated user interface.

 

The SugarCRM User Experience team (led by Brian Ng) launched a rigorous heuristic evaluation of the web and mobile Sugar products. Issues were identified as we evaluated key use cases and new solutions were documented. This thorough understanding of the product, customers, and the industry has helped us identify which issues to address first.

 

Dark gray text on light gray background led to poor legibility. Heavy blacks and gray linen added to the dated look and feel.

Sugar Mobile had a dark and drab visual design. Buttons for key actions were positioned differently on key screens.

 

Establishing Goals

With an understanding of the problems, we set goals for ourselves.

  • Improve design for over 100 usability issues, prioritized by impact to the user's ability to get their job done.
  • Establish a clean and modern visual design language, with rules documented in a style guide.
  • Ensuring our products are accessible by all, meeting Web Content Accessibility Guidelines (WCAG) compliancy.

 

Guiding Principles for our new Design

We established three design principles: Clear, Consistent and Efficient. These principles guided all of our decision making. Establishing principles helped to resolve differences of opinions and to focus on user frustrations.We determined what colors and fonts would positively affect our use cases and still reflect our brand.

 

 

We have adopted Open Sans as the Sugar UX base font. It’s an open source font optimized for web and mobile interfaces with text heavy layouts. It’s known for its open, neutral type shape and friendly appearance. For colors, we have selected a color palette that is friendlier, modern and brighter.

 

We also selected a set of colors and fonts that would positively affect our use cases and still reflect the SugarCRM brand. Finally, we tested concepts in the real world, with real users in key flows to observe how they played out in practice.

 


 

Mobile has a bright appearance and color placed in strategic locations to help users identify groups of information.

 

Results

  • First revealed at SugarCon to over 1000 attendees where customers gave feedback.
  • A new visual design language, documented in a Style Guide (coming soon), empowering customers and partners to implement SugarUX consistently on our product.
  • Launch of the new SugarUX on our Sugar Mobile apps, available already on the the Apple App Store and Google Play.

 

 

Coming in Winter ‘18!

 

The first phase of our new UX is coming in Winter '18 release and we want to make sure the Sugar Developer community is first to give it a test drive. If you have built any Sidecar components or customizations (Dashlets, custom views or layouts, etc.) then you should verify that they display properly in new UX. As with all code level customizations, it is up to Sugar developers to make sure that you fix your code so that your customizations display properly in the new UX.

 

From implementation perspective, these changes have been made entirely in CSS where we've adjusted the styling for existing CSS classes. So, areas of concern would be in places where you have overridden the styling for Sugar's core CSS classes or have introduced your own CSS through the use of ./custom/themes/custom.less. The majority of customizations that simply use our existing CSS classes are expected to adopt new styling without errors.

 

As usual, we will be running an invitation only technical Preview program for the Winter '18 (On-Demand only) release. If you want to make sure that your organization is considered for this preview then please e-mail developers@sugarcrm.com and mention your interest in test driving the new Sugar UX!

 

Building your new Skin test plan

We have referred to this effort as the “new skin” because this first phase primarily involves changes to CSS and no significant changes to default HTML structure for any of our Sidecar Components.

 

How do you make sure your customizations are still pixel perfect? What follows below are some tips for you and your team to build a test plan.

 

Exploratory Testing

We recommend devoting a fixed amount of time to doing some exploratory testing in order to make sure your customizations look great in the new skin. How much time? It depends on how much UI you have. For example, it should not take you longer than a few minutes to test a custom Dashlet and uncover any bugs that you may want to address.

 

Guidelines for Testers

If you are testing, what are you looking for? Here are some tips.

  • Customizations only. Testing should focus on customizations that your team created.
  • CSS overrides. If you use a ./custom/themes/custom.less file or include your own CSS files, then you’ll need to pay special attention to these changes since they will likely need to be updated.
  • Scrolling. Are you able to scroll and are there things that are inaccessible even with scrolling?
  • Ellipsifying. When you shrink the window size down or reduce the available space for data on custom list views, record views, etc, do they ellipsify and show tooltips on hover?
  • Responsiveness. Resizing the browser window or switching to and from responsive-mode. Observe what happens to elements on page.
  • Orientation on tablets. Alternating display orientation between landscape and portrait modes.
  • Toggle things. Open/close drawers, menus, edit and detail mode on record views, etc.
  • Settings. Change system and profile settings that may alter how things look like date/time format, currency, etc.

 

Comparisons with out of the box Sugar

This post includes some example screenshots of what the new skin looks like out of the box. Comparing your customizations to the out of the box look and feel will help you identify things that you may need to change in your customizations. We encourage testers to keep two windows open while testing to compare the out of the box behavior in one window with your customizations in another window more easily.