Skip navigation
All Places > Developer > Blog > 2018 > March
2018

Please don’t hate me. It’s not really my fault. I am sorry to say that every single Sugar developer needs to care about the General Data Protection Regulation (GDPR) and data privacy in general. It is a sign of the times and part of the interrelationship of technology and modern society that impacts the software we implement today and in the future.

 

I can explain but first let me provide some context.

 

What is the General Data Protection Regulation (GDPR)?

 

First off, I am not a lawyer or a GDPR specialist. If you really want to understand what GDPR is and what steps you need to take to be compliant, then I suggest reading the Regulation for yourself or hiring counsel. GDPR is a complex set of rules, and there’s still a lot of debate about what it all really means. Other interpretations, including yours, may be different from what I describe below.

 

Many of our colleagues in the European Union are well acquainted with the GDPR which requires full compliance by May 25th, 2018. But for those of you who do not live in the European Union (EU), I highly recommend reading Data Privacy and GDPR in Sugar by Deepak Deolalikar. It covers some GDPR basics as well as provides an overview of some of the features planned in Sugar to help our customers comply with GDPR.

 

In short, the spirit of GDPR is about allowing individuals (data subjects) to have control over their own personal information. GDPR ensures that personal information or Personally Identifiable Information (PII) of EU citizens are processed responsibly. EU citizens have a right to privacy that in many cases means you must comply with their requests to access, restrict the use of, or delete the personal information that is stored about them.

 

The scope is broad and the penalties are severe. Any organization that stores personal information about EU citizens, even if that organization is based outside the EU such as in the United States, is subject to this regulation.

Penalties for failure to comply with these regulations could result in fines that start at €20 Million or 4% of total revenue if that value is greater.

 

Are you paying attention now?

 

Data Privacy is yet another software globalization requirement

 

When we build software, we make sure that it meets all sorts of requirements. We design for performance, scalability, and security. We work hard to ensure a positive user experience and high quality software. At SugarCRM, we also focus on making sure that all our products are ready for the global marketplace. That means that over the years we have invested in translations into dozens of languages and support for multiple currencies, numerous date and time formats, right-to-left languages, Unicode character encodings (though we’re still working on emojis), and 508 compliance.

 

The fact is that GDPR is just one example of a data privacy regulation at a time when there are dozens more government bodies all over the world considering and implementing new data privacy regulations. UTF-8 character encoding and a translatable UI are no longer enough to make sure your software is ready for the global marketplace! The responsible collection and processing of personal information is now an additional obligation for all software developers.

 

With the upcoming Sugar Spring ‘18 (cloud) and 8.0.0 (on premise) release, we are now investing in data privacy features. These features will help provide our customers with the tools they need to comply with GDPR and many other data privacy rules or regulations.

 

These data privacy features and other data privacy concerns will impact Sugar customizations and integrations that collect or process personally identifiable information (PII).

 

Future installments on Data Privacy and GDPR

 

We will be exploring several data privacy topics in the coming weeks. Here is a quick summary.

 

Collecting explicit consent before storing data in Sugar

 

GDPR has strict requirements for consent. Unless you have an existing lawful basis for using personal information, you need to collect positive and unambiguous consent from the data subject even before data is stored in Sugar. For example, a web to lead form with a pre-selected opt-in checkbox is not going to cut it in the EU. We will be exploring ways to make sure you are collecting explicit consent using techniques like double opt-in (DOI) or confirmed opt-in (COI).

 

Managing PII data and the right to erasure in Sugar

 

There are new Sugar APIs being added for working with PII as well as features that allow Sugar customers to comply with data subject requests like the right to access their own data and the right to erasure. In particular, external systems that integrate with Sugar may need to identify and implement their own erasure measures in compliance with a data subject request.

 

Improved change log and attribution of changes using Data Sources

 

We are implementing some enhancements in Sugar’s change log functionality to allow for finer grain tracking and proper attribution for changes to PII. We will be exploring these platform level enhancements including the ability to define new data sources (for example, an integration) that are responsible for changes to data.

 

What do you think?

 

What are some of your biggest data privacy or GDPR concerns? Let us know and we’ll try to address them in future posts.

 

Also, please follow this space and this blog to be sure you are notified when the next installment is posted!

What is the Sugar Metadata API?

As you probably know, Sugar metadata encompasses the settings, data model (Vardefs), and visual layouts (Viewdefs) used by the Sugar application. The majority of changes made to Sugar using Sugar Studio, Module Builder, or even custom code are implemented by modifying metadata.

 

How can I know that the custom field or custom modules that you need for your integration to work exists in Sugar?

How can I know what are the allowed values are valid for a Sugar dropdown field?

How can I know if a field is required?

How can I know the display label for a given field?

How can I know what fields are available in Sugar that I can map back to an external data source?

 

The answers to all these questions and many more can be found in Sugar metadata.

 

On the server side, metadata is implemented as an extensible set of PHP associative arrays. But how does this information get sent out to Sugar clients like your web browser, mobile app, or even the Outlook plug-in? This is accomplished via the Sugar metadata endpoint of the Sugar REST API.

 

GET /rest/v11/metadata HTTP/1.1
Host: localhost:8080
OAuth-Token: ...
Content-Type: application/json

 

The full metadata response comes back as a very large JSON object. It's usually a few megabytes in size (about 3MB in a stock version of Sugar Winter '18) but can be much larger depending on customizations. Make sure compression is enabled on your web server if you are going to be using it over a slow connection!

 

 

The value to using the Metadata API is that you can discover changes in Sugar configuration and Sugar customizations. New custom fields, relationships, modules, changes to Sugar views and layouts, and much more are all represented in Sugar metadata responses.

 

From a REST API perspective, the only way to accurately know the current data model for any given Sugar instance is to use the Metadata API.

 

Since the Metadata API is so important, here are three important tips to keep in mind to make development easier.

 

Tip 1: Retrieve only what you need

Metadata API calls can generate a huge response depending on your Sugar instance configuration and the customizations that may exist. You should specify filters on your metadata request in order to narrow the response to the data that you actually need. Narrowing what is returned will improve performance on both server side in terms of generating the response but also on the client side where you need to parse the JSON response.

 

For example, if you are integrating with only the Accounts module then you may only need to concern yourself with that part of the metadata. Specify a type_filter and module_filter as URL parameters to only return module metadata for the Accounts module. Other supported type_filter options are server_info, config, relationships, labels, languages, currencies, views, layouts, full_module_list, and etc.

 

GET /rest/v11/metadata?type_filter=modules&module_filter=Accounts HTTP/1.1
Host: localhost:8080
OAuth-Token: ...
Content-Type: application/json

 

Tip 2: Cache metadata, update only when it changes

You can safely cache the metadata that you need if you keep track of the _hash value returned in the metadata response.

 

You should present that hash value in the X-Metadata-Hash header on subsequent Sugar REST API requests to ANY endpoint. If the metadata hash is out of date then Sugar will return a 412 PRECONDITION FAILED HTTP response. You should then retrieve updated metadata from the Metadata API and store the updated metadata hash value.

 

GET /sugar/rest/v11/Accounts HTTP/1.1
Host: localhost:8080
OAuth-Token: ...
Content-Type: application/json
X-Metadata-Hash: 705fb99230e2ec60b59f481c6831a7cf

 

Also worth mentioning, many of our API resources (such as records and our metadata) implement HTTP caching. If an API response contains an E-Tag HTTP header then you can safely cache the response body as well as the E-Tag value. If you present this value in the If-None-Match HTTP header properly on a subsequent HTTP GET request for that resource, you will get an empty 304 NOT MODIFIED response back if it is safe to continue using the cached version of the previous response. This is another approach to reduce needless processing.

 

GET /rest/v11/metadata?type_filter=modules&module_filter=Accounts HTTP/1.1
Host: localhost:8080
OAuth-Token: ...
Content-Type: application/json
If-None-Match: 705fb99230e2ec60b59f481c6831a7cf

 

This allows you to update your local copy of the metadata as soon as it changes and precisely no more often than that.

 

Tip 3: Different users and platforms will retrieve different responses

If a user is assigned a role that doesn't have access to a particular module then the metadata for that module will not be returned when that user accesses the metadata API. Makes sense, right? 

 

Sugar also support things like role based views which will provide alternative UI layout metadata based upon the current user's role.

 

Different platforms will have different metadata responses as well. The mobile platform retrieves module metadata that reflects the needs of our Sugar Mobile app. Mobile UI components look a lot different from what you see when you access Sugar from your browser. The metadata that is retrieved is significantly different to accommodate these changes.

 

For example, the mobile client uses separate edit and detail views for representing records instead of a single record view like you find in Sugar web client. This is represented in the mobile metadata response.