Skip navigation
All Places > Developer > Blog > Authors Matt Marum
1 2 3 Previous Next

Developer

111 Posts authored by: Matt Marum Employee

Those who have been around a while will know that SugarCRM Inc. was built around a single product called “Sugar”. Sugar is such a part of our DNA today that we often use “Sugar” and “SugarCRM” terms interchangeably. I’m sure that many Sugar Developers out there would be surprised to learn that we sell more than just Sugar at SugarCRM!

 

Sugar has been good to us but the way apps were built in 2004 and how apps are developed today are much different. We’ve been thinking about how we want to build, deploy, and maintain new applications and services today and in the future. This has led us to a long term investment to develop a framework for new cloud-based services that are tightly integrated with Sugar.

 

The tip of the spear is a new set of identity and access management services that we call Cloud Identity. Coming to a Sugar instance near you in June 2019!

 

What is SugarCRM’s Cloud Identity service?

 

Cloud Identity is a set of user authentication and access management microservices that will improve how we manage Sugar cloud users today. Regardless of the SugarCRM products and services you are using, you will be able to manage your end users in one place. This will make life easier for Sugar administrators while allowing SugarCRM engineers and Sugar Developers to more easily integrate Sugar and other applications and services including those built by partners and ISVs in the future.

 

It will offer improved OAuth 2.0 support, leverages OpenID Connect, and will also address a frequent customer request to support SAML Web Single Sign On (SSO) with the MS Outlook Plug-In.

 

Cloud Identity also supports the concept of Service Accounts which can be used to implement secure Server to Server integrations much easier without involving end user credentials.

 

It also provides a new Cloud Settings user interface that Sugar administrators will use to manage end users and some subtle but important changes to end user login experience.

 

We plan to start on-boarding new Sugar cloud customers in some regions to Cloud Identity service in June 2019.


Frequently Asked Questions

 

We’ve just recently finished up our Cloud Identity Beta where we asked a handful of partners and customers to test drive the new system. We received a lot of valuable input that we used to improve these services. We also received a number of questions from Sugar Developers that we wanted to share below:

 

How do I know if a given Sugar instance is using Cloud Identity?

SugarConfig settings are updated in order to connect a Sugar instance to Cloud Identity services. This will be done automatically for our Sugar cloud customers. Cloud Identity does not currently support Sugar on-premise installations. In practice, you can check the public metadata REST endpoint for any given Sugar cloud instance to see if it is operating in “IDM mode”.

 

GET /rest/v11_4/metadata/public

 

The response will include:

 

{

   …

   “config” : {

      …

      “idmModeEnabled”: true | false

      …

   }

   …

}

 

This setting will be true if Cloud Identity is in use.

 

How to find the Tenant ID for a given Sugar instance?

In most cases, you will not need to know the specific Tenant ID for your Sugar instance in the Sugar cloud. However, it can be useful to know if you need to login to Cloud Settings console directly.

 

 

For example, it can be used with the tenant_hint URL parameter to allow more seamless logins to Cloud Settings.

 

https://login-us-west-2.service.sugarcrm.com/?tenant_hint={TENANT_ID}

 

The Tenant ID can be retrieved from a Sugar instance’s SugarConfig settings or from the public metadata API.

 

GET /rest/v11_4/metadata/public

 

The response will include:

 

{

   …

   “config” : {

      …

      “tenant”: “srn:cloud:iam:us-west-2:{TENANT_ID}:tenant”

      …

   }

   …

}

 

The tenant config property above is an example of a Sugar Resource Name (SRN). SRNs are used to uniquely identify resources across the Sugar cloud ecosystem. You’ll see more SRNs as you adopt the new platform and we roll out more features.

 

How do I deploy a backup of a Sugar cloud instance that uses Cloud Identity?

If you have a backup of a Sugar cloud instance that was configured to use Cloud Identity then it will not likely work without some configuration changes. This is because your local system will not have permission to access the identity services running in the Sugar cloud.

 

Take the following steps to disable IDM before working with the backup.

 

Update config_override.php to set:

 

<?php

$sugar_config['idm_mode']['enabled'] = false;

 

In a terminal, remove cache/ directory contents:

 

$ rm -rf [path to sugar]/cache/*

 

How does Users module change with Cloud Identity?

User information is now federated across Cloud Identity services and the Sugar instance Users module. When a Sugar instance is connected to Cloud Identity, there will be specific User fields that are no longer owned by the Sugar instance. For example, the user name or e-mail address will be modifiable by a new Cloud Settings user interface and not within Sugar.

 

Updates that are made in Cloud Settings are pushed to Sugar user records automatically via REST APIs and also at the time of each user's login. If you have customizations that write to Users module, these changes could be overwritten by Cloud Identity services.

 

 

At this time, the following Users module fields are owned by Cloud Identity services and should only be updated via new Cloud Settings user interface.

 

  • first_name
  • last_name
  • address_street
  • address_city
  • address_state
  • address_postalcode
  • address_country
  • email_addresses
  • phone_work
  • title
  • department
  • status
  • is_admin
  • preferred_language
  • user_name

 

Custom fields on the Users module are not overwritten.

 

What are changes when authenticating via Cloud Identity?

Ultimately, Cloud Identity will provide more options for authentication and authorization. This will be the area of biggest impact for Sugar integrations.

 

Web Users

When an end user navigates to a Sugar URL and they are not yet logged in they will be redirected to a login service. This login service will have a different URL than their Sugar instance. However, once logged in they will be redirected back to their Sugar instance.

 

REST API integrations

If authenticating using REST API, the endpoint behavior is essentially unchanged. What has changed is the format for OAuth access and refresh tokens. Instead of using UUIDs, Cloud identity uses encrypted tokens that are base64 encoded. If you are storing access or refresh tokens, they can now be up to 255 characters long. However, there is no hard limit on their size and they may get longer over time.

 

Ex.

"SyeIRxNeMkRy8IBluxttQ8DzrXEt4CQp2vzoWVQFJqw.xaZOVhc8hdB630ZoLBt9LOHePZb6j6uRRKGkKgNe3RI"

 

What functionality or customizations are not supported when using Cloud Identity?

Much of the responsibility for handling authentication is now delegated to Cloud Identity services. This means that many customizations to Sugar’s authentication mechanisms are no longer supported.

 

In particular, if you have created any custom SugarOAuth2Storage* classes then these will not work when Cloud Identity is enabled. The responsibility for generating and managing authentication tokens is now the responsibility of our Sugar cloud service instead of the Sugar instance.

 

If you feel like an essential authentication feature is missing in Cloud Identity services, then please let us know by filing an enhancement request via the Support Portal.

 

When using SAML Web SSO, should I move my integration users to my external identity provider?

Some customers that have wanted to use SAML with Sugar have held off until the SugarCRM Outlook Plug-In supported it. For Sugar customers using the new Cloud Identity service, the Outlook Plug-In will now support SAML Web SSO for the first time.

 

If you’re using integration users, we recommend leaving them as local user accounts instead of moving them to your SAML identity provider. The same advice applies when using LDAP an external identity provider too. SAML Web SSO isn’t well suited for REST API integrations since it requires credentials to be entered into a browser which makes automated authentication challenging. Your SAML or LDAP identity provider should only be responsible for managing employee access to Sugar.

 

This is also a use case where Service Accounts will come into play. Service Accounts will allow for the separation of responsibility between regular user accounts and services accessing Sugar data for integration purposes. This is a feature we are planning to implement in the future.

Matt Marum

Sugar 9 has arrived!

Posted by Matt Marum Employee Apr 9, 2019

Hey Sugar devs!

 

Spring is in the air (for the northern hemisphere)! This can mean only two things: allergy season has started and a new Sugar On-Site release is now available.

 

Let me introduce to you Sugar 9.0!

 

See below for all the information Sugar Developers need to know. Sugar end users can be directed over to one of the official Sugar release announcements: the Sugar Spring '19 announcement is for cloud customers and the Sugar 9.0 announcement is for on-premise customers.

 

In our recent developer webinar we gave an overview of the important changes coming in the Sugar 9.0 On-Site and Sugar Spring ‘19 cloud releases:

 

 

The slides from that webinar are available here.

 

Did you miss the webinar? Want to make sure you’re the first to know about important Sugar product changes? Make sure you are signed up for Sugar Developer News!

 

Here's a quick overview of what can be expected if upgrading from Winter ‘19

  • Group Outbound Email Support
  • SugarBPM™ improvements
  • Performance and scalability improvements

 

In addition to the above, here are some of the many additional updates you’ll see if upgrading from Sugar 8.0.

  • Reporting improvements including report scheduling
  • Data Privacy improvements including double opt-in
  • New Quotes configuration panel
  • Emoji support on MySQL (utf8mb4 conversion)
  • Updates to JS libraries including upgrading to jQuery 3
  • Added PHP 7.3 support
  • Updated Microsoft stack support for those hosting on Windows
  • More performance and scalability improvements

 

Check out the below resources for the rest of the details.

Impatience is a virtue

Larry Wall (the inventor of Perl) is credited with identifying the three great virtues of a programmer: "laziness, impatience, and hubris."

 

These virtues are usually shared with a wink. For example, a programmer's "laziness" will drive them to get as much work done as possible with the least amount of effort. They say that we'll write automation scripts (in Perl probably) or follow the DRY principle to remove all redundant and unnecessary effort.

 

I find the "impatience" and "hubris" virtues more interesting.

 

The virtue of "impatience" is meant to make us "write programs that don't just react to your needs, but actually anticipate them." It also means that we demand a lot from our software. We want it to be as fast, efficient, and proactive as possible. It can be just as satisfying to double the speed of a program as it was to create "v1.0" in the first place.

 

 

Impatience is a virtue that I think we share with our Sugar users. We all want Sugar to be responsive and, at best, to be able to provide an insight that anticipates every Sugar user's need. That's why we have a dedicated team of Performance Engineers that work cross-functionally with Product, Operations, and Support teams with the goal to make Sugar fast. REAL FAST.

 

The last virtue is "hubris" which means that programmers should take great pride in the code they write and the work they do. So let me exercise a little hubris by bragging about what the Performance team has accomplished over the last year.

 

Performance Testing Apparatus

We've leveraged our experience of running the Sugar cloud to put together an exhaustive  REST API performance test suite of over 15,000 API requests using JMeter. These requests exercise commonly used Sugar functions such as loading List Views, viewing and updating Record Views, performing mass updates, and much more. We combine that by using Tidbit to load the Sugar database with 60GB of data. This data set includes a half million Accounts, two million Contacts, and millions more interaction records.

 

The combination of JMeter test suite and Tidbit data set serves as the basis for testing and comparing different Sugar versions and build to build. The results below were gathered by running this same test suite against multiple Sugar versions - 8.0.1, 8.0.2, 8.3.0, and a pre-release build of Sugar 9.0.0.

 

Remember: If you are a Sugar Customer or Partner and you want to have closer look at our JMeter performance test tools, you can sign up to get access to Sugar Test Tools. Tidbit is an open source project available on our Github organization.

 

3x to 7x faster Filter API performance since July 2018

The Filter API is one of the most commonly used Sugar REST APIs. It is used to load List Views, Dashlets, Subpanels, and is commonly used by integrations that need to synchronize data between Sugar and an external system.

 

This API can introduce some performance challenges because it allows access to many records at once using arbitrary filter criteria and sort order. The performance of this API can vary widely depending on the data set, the database schema, and the options being used. So this API has been a particular area of focus for our performance engineers that having been identifying bottlenecks and eliminating them one by one.

 

As a result, Filter API performance in Sugar 9 has improved by a factor of 3x to 7x as compared to Sugar 8.0.1 which was released less than a year ago in July 2018.

 

 

Average API response time cut by more than half since July 2018

The average Sugar 9.0 response time across the 15,000 requests in our test suite is a fraction of what it was in 8.0.1. It is 40% faster than 8.0.2 and 30% faster than even our Winter '19 release. This means that no matter how you are using or integrating with Sugar, you will see significant responsiveness gains by upgrading to Sugar 9.0 (on-premise) or Spring '19 (cloud) when they come out.

 

ReleaseAverage Response Time
8.0.11.59s
8.0.21.03s
8.3.0903ms
9.0.0 (pre-release)596ms

 

Increased throughput by 70% on same hardware

If you are using Sugar on-premise, there is a substantial increase in request throughput which can translate to lower hosting costs since more users can be hosted on less hardware. Our testing shows that Sugar 9.0 should process 70% more requests in the same amount of time as the latest Sugar 8.0.2 on-premise release using the same hardware. This should help you stretch hosting dollars further.

 

 

Our work is never done

This is a really exciting time to be part of the SugarCRM ecosystem. We are adding many new products and capabilities to the SugarCRM portfolio and our Performance team is here to make sure that all these products meet and exceed our customer's performance expectations.

Hello Sugar Developers!

 

Are you excited about Sugar 9.0.0?

 

We want to make sure your customizations and integrations are ready for the Sugar 9.0 (on-premise) and Spring '19 (cloud) releases, so we’re hosting two webinars just for you!

 

What we will be covering

 

We’ll discuss the big changes that are likely to impact you including the following:

  • Review of changes in Sugar 8.1Sugar 8.2, and Sugar 8.3
  • Updated Sugar on-premise supported platforms (PHP 7.3, SQL Server 2017)
  • Introducing new REST API version 11.5 11.4 endpoints
  • API performance and scalability enhancements

 

Webinar Information

 

We are holding two sessions to accommodate various geographical locations. Please choose ONE of the following times below.

 

Monday, March 25th 4:30 - 5:30 PM PT OR Tuesday, March 26th 7:00 - 8:00 AM PT (Choose one)

 

Register Now

 

As always, we will be posting the webinar recordings to the Sugar Community for those who are unable to attend the live sessions.

Hey Sugar devs!

 

Just like clockwork, it is time for another Sugar cloud quarterly release. If you are on-premise, then you will get these features and many more in the upcoming Sugar 9.0 release in the Spring.

 

Check out the official release announcement for information that your end users will care about.

 

For Developers, we recently hosted a developer webinar we gave an overview of the important changes developers need to know:

 

 

The slides from the webinar are available here.

 

Here's a quick overview of what developers can expect in Winter '19 release.

  • We've added a bunch of new great features like the Comment Log and Product Catalog Quick Picks dashlet
  • We've enhanced the export options available in Reports module
  • We've also made a number of enhancements to Advanced Workflow including the ability to use old and new values in email templates
  • We've put in groundwork for future PHP 7.3 support
  • We've also upgraded to jQuery 3.3.x
  • And our REST API version increased to 11.4 with some additional API endpoints

 

Check out the below resources that have the rest of the details.

 

We hope you’re as excited about this release as we are!

As of this writing, all current Sugar versions (7.9 extended, 8.0, and Sugar cloud) supports and uses PHP 7.1. This PHP version was released in 2016 and entered security support on December 1st 2018. While we’ve made great strides in supporting newer PHP versions over the last few years, we have tended to be a couple years away from the latest and greatest.

 

So what is today’s latest and greatest? Well PHP 7.3 was just released on December 6th 2018. Therefore PHP 7.3 will be the next version that Sugar will support. Over the last few months our engineers have been laying the groundwork for that support in our upcoming Sugar 9.0 release.

 

Some of that work involves adding cumulative code compatibility changes for PHP 7.2 and PHP 7.3. The details of these changes can be found in the PHP 7.2 migration guide and the PHP 7.3 migration guide.

 

However, our intent is to certify support for PHP 7.1 and PHP 7.3 for Sugar 9.0. We have no plan to support PHP 7.2 now or in future. By skipping 7.2, we are able to spend more time innovating while still allowing you all to access the latest and greatest features and performance that new PHP versions offer.

 

Library updates in Sugar Winter ‘19 release

In the upcoming Sugar Winter ‘19 release, the following PHP libraries are updated. Many of these were updated to newer versions that were compatible with PHP 7.3.

 

  • doctrine/dbal: 2.7.1 → 2.8.0 (changelog)
  • ramsey/uuid: 2.9.0 → 3.8.0 (changelog)
  • symfony/{cache,console,framework-bundle,security-core,security-csrf,translation,validator}: 3.4.8 → 3.4.16 (changelog)
  • tedivm/jshrink: 1.1.0 → 1.3.1 (changelog)
  • onelogin/php-saml: v2.11 → v3.0 (changelog)

 

Additionally, we are making minor changes to the following vendored libraries for PHP 7.3 compatibility:

 

  • Smarty
  • XTemplate
  • HTMLPurifier

 

If you have custom code that uses any of the above libraries, you should verify that your customizations are still compatible.

 

Looking ahead to PHP 7.3

Finally, here are some additional PHP resources that you can review today to help you get ready to use PHP 7.3.

 

 

Let us know what you think in the comments below!

Hey Developers!

 

While we loved seeing everybody who came out to SugarCon this year, we know that a lot of you weren't able to make it. The good news is that we recorded all the developer sessions at SugarCon last October. They are all published today in the SugarCon space but we wanted to make sure you didn't miss them.

 

Check out the table below to links to the recordings, slides, and speakers.

 

 

We hope to see you all at future SugarCons and other Sugar Developer events!

Hello Sugar Developers!

 

We want to make sure your customizations and integrations are ready for the Sugar Winter ‘19 release, so we’re hosting two webinars just for you!

 

Webinar Topics

 

We’ll discuss the big changes that are likely to impact you including the following:

 

  • New REST API version: v11_4
  • New Comments Log field type and APIs
  • Enhancements to Reports module
  • Upgraded PHP and JavaScript libraries including jQuery

 

Webinar Schedule

 

We are holding two sessions to accommodate various geographical locations. Please choose ONE of the following times below.

 

Thursday, December 13th 7:00 - 8:00 AM PT

OR

Thursday, December 13th 5:00 - 6:00 PM PT

 

Register Now!

 

As always, we will be posting the webinar recordings to the Developer community  for those who are unable to attend the live sessions.

We are fast approaching the Winter '19 release into the Sugar cloud that is planned for January 2019. Some of the changes coming in this release will be upgrades to the jQuery library and associated plug-ins that are commonly used by user interface customizations to Sugar.

 

jQuery libraries upgraded from v1.12.4 to v3.3.1

 

In the Sugar Winter '19 release, we have made a significant upgrade to the jQuery versions used by Sidecar and SugarMVC (BWC) modules. 

jquery: v1.12.4 -> v3.3.1 (changelog) (Upgrade Guide)
jquery-migrate: v1.4.1 -> v3.0.1 (changelog)

 

jQuery Migrate should help maintain compatibility for custom code using older jQuery APIs. Custom code that uses deprecated jQuery APIs should be updated as soon as possible.

 

As part of the jQuery upgrade, the following jQuery plug-ins needed to be updated to be compatible with v3.3.1.


include/javascript/jquery/markitup: v1.1 -> v1.1.15 (changelog)
include/javascript/pmse/lib/jquery.layout-latest.js: v1.3.0 -> v1.6.3 (changelog)
include/javascript/jquery/jquery.timepicker.js: 1.8.8 -> 1.11.14 (changelog)

 

If you've added jQuery plug-ins to Sugar as part of your customizations, you should make sure they are compatible with jQuery 3.3.1 as well! Check out the jQuery Plugin Registry to make sure the version of the plug-in you are using is compatible with jQuery 3.3.1. If not, you may need to upgrade to a newer version of your plug-in or remove it.

 

 

This plug-in is compatible with jQuery 3.3.1.  SWEET!

 

 

We've also upgraded the Bootstrap JS library from v2.0.0 to v2.2.1 (changelog) and also moved it to a new location.

 

include/javascript/jquery/bootstrap/bootstrap.min.js (v2.0.0) has been removed
include/javascript/twitterbootstrap/bootstrap.min.js (v2.2.1) has been added 

 

If you happen to refer to the old file directly, you'll need to update your links to point to the the new location.

 

Other JavaScript library updates

While jQuery is probably the most impactful JavaScript change in this release, there are other JS libraries that you should know about.

 

Sidecar library updates

We've also made some changes to these other two Sidecar libraries. If you happen to be using them, please check out their change logs to ensure that you're still compatible.

 

sugarcrm/ventana: v1.1.1 -> v1.1.2 (changelog)
moment: v2.11.2 -> v2.22.2 (changelog)

 

Removal of JIT Chart library

The deprecated JIT chart library has been removed in this Sugar release. The JIT chart library was deprecated in Sugar 7.10 and has been replaced by Sucrose charts. Please ensure you have removed any code that uses JIT charts prior to upgrading.

 

Don't miss the Winter '19 Sugar Developer Webinar!

On Thursday, December 13, come join us in one of our live webinars about all the enhancements coming in Winter '19 to the Sugar cloud.

We can probably all agree that automated tests are great, but sometimes we put off writing them because the initial first step of getting everything working correctly can be really difficult.  

 

Good news, Sugar developers!  Writing PHPUnit tests for your customizations just got easier in Sugar Fall '18!  Our engineering team has created a test runner specifically for your custom PHPUnit tests.

 

If you do not have access to the SugarCRM provided unit test suite, submit the Developer Tools Request.

 

The first thing you'll need to do is add the SugarCRM provided unit test suite to your Sugar directory. Be sure to select the branch in the unit test suite that matches the version of your Sugar instance (needs to be 8.2 or higher). Follow the instructions in the branch's readme to add the tests to your Sugar directory, install the necessary dependencies, and update the file permissions.  

 

Now you're ready to write your tests!  Add your new PHPUnit test classes to the custom/tests/unit-php directory. You may need to create it yourself if it doesn't already exist.

 

For example, create a new file named CustomLogicHookTest.php and place it in the custom/tests/unit-php directory.  Paste the following in to the new file:

<?php
    //custom/tests/unit-php/CustomLogicHookTest.php
   
    namespace Sugarcrm\SugarcrmTestsUnit\custom;
   
    use PHPUnit\Framework\TestCase;
   
    /**
     * @coversDefaultClass \CustomLogicHook
     */

    class CustomLogicHookTest extends TestCase
    {
       
        /**
         * @covers ::aFunctionInsideTheCustomLogicHook
         */

        public function testCustomHook()
        {
            // Wow! This is a horrible test.
            // You would never do something like this just to increase your test  
            // coverage, right?
            $this->assertTrue(true);
        }
    }

In a shell, navigate to the Sugar provided unit tests directory by doing something like the following:

cd tests/unit-php

Then you can run the tests by executing

php ../../vendor/bin/phpunit --testsuite custom

or by using gulp (gulp installation instructions are available here)

gulp test:unit:php --suite custom

The above test is extremely simple and doesn't test anything. If you'd like to see an example of a larger test suite that tests Sugar customizations, check out Professor M's School for Gifted Coders.  We've recently updated our PHPUnit test suite so we can use the new test runner.  Check out the pull request associated with this change for the details.

This blog by SugarCRM Technical Account Manager and Solution Architect Angel Magana is reposted with his permission. See Angel's Blog: SugarCRM: Related Data Interactions for the original post from Angel's blog. In this post, Angel explains some best practices for working with related records in PHP that can help improve performance of your customization.

Fetching related records

 

Often times a customization we are working on requires us to interact with related data, usually records that have a parent-child relationship with one another. For example, a customization might require us to determine if a given Account record has 1 or more related Contacts. Or for that matter, we may need the collection of ID values that represent all the Calls linked to a Contact.

 

In either example, the path most often followed could be described with the following pseudo-code:

 

1. Instantiate the parent SugarBean object (e.g. Account/Contact record object)
2. Load the corresponding relationship to access the related data objects
3. Retrieve the related SugarBean objects representing the related objects

 

See below for a PHP snippet illustrating the above:

 

<?php 

//Some other code

//Instantiate parent record SugarBean object
$ParentBean = BeanFactory::retrieveBean('Accounts', 'some_account_id_value');

//Load the pertinent relationship using Link field name
$ParentBean->load_relationship('contacts');

//Retrieve related Contact(s) SugarBean object(s)
$RelatedContacts = $ParentBean->contacts->getBeans();

//The rest of our code


Line 12 in the above example would effectively create an array of SugarBean objects representing the linked Contacts. From that point forward, determining whether or not the Account has any linked Contacts becomes a simple matter of checking the size of the array. 

 

For the linked Calls example, the code would be very similar, except we would load the 'calls' relationship and the ID values we want would be in the SugarBean object array that results from executing the getBeans() method.

 

Do you need full Beans or just the IDs?

 

Now, all of the above would function just fine, but let us consider a few things about this approach.

 

First, upon executing the getBeans() method, we are asking Sugar to retrieve the complete SugarBean object for all the related records of a given type. However, for both examples, simply retrieving the ID value would have sufficed.

 

That is, for our purposes, a list of ID values in the $RelatedContacts array is just as good as an array of all the SugarBean objects. That array would still allow us to properly answer the question of whether there are or are not any linked contacts, solely based on the number of ID value entries the array contains.

 

In the case of the second example, we specifically only need the ID values of the related Calls, but getBeans() gives us everything pertaining to each of those calls. This tells us there is a lot of overhead we can trim.

 

From a performance standpoint, anytime we can optimize our code to retrieve only what we need, we help minimize wait times a user experiences. 

 

How do we then change our code to only give us the ID values?

 

It is actually a quite simple.

 

<?php 

//Some other code

//Instantiate parent record SugarBean object
$ParentBean = BeanFactory::retrieveBean('Accounts', 'some_account_id_value');

//Load the pertinent relationship using Link field name
$ParentBean->load_relationship('contacts');

//Retrieve related Contact(s) IDs ONLY.
$RelatedContacts = $ParentBean->contacts->get();

//The rest of our code

 

Examining the Link2 class in Sugar reveals a get() method which specifically only returns ID values of related data. Using get() in place of getBeans() would allow us to achieve the goal described in our examples and also reduce the performance overhead.

 

Side note: A similar method exists in the TeamSet class, named getTeamIds(), to retrieve the ID values of Teams that compose a TeamSet, without needing to retrieve all of the Team SugarBean objects. Also be aware that get() will return all records, irrespective of team assignments on the related records. 

Hello Sugar Developers!

 

We want to make sure your customizations and integrations are ready for the upcoming Sugar Summer ‘18 release, so we’re hosting two webinars just for you! The Summer '18 release is just a couple short weeks away.

 

What we will be covering:

We’ll discuss the big changes that are likely to impact you including the following:

  • Changes to Login screen and authentication classes
  • Emoji support
  • Built-in support for Double Opt-In (DOI)
  • Improvements to Reports and Advanced Workflow
  • SugarCon Update

 

Webinar Information:

We are holding 2 sessions to accommodate various geographical locations. Please choose ONE of the following times below.


Monday, July 2nd, 5:00 - 6:00 PM PT 
OR 
Tuesday, July 3rd 8:00 - 9:00 AM PT 
(Choose one)

 

Register Here

 

We will be posting the webinar recordings to Developer community for those who are unable to attend the live sessions.

 

P.S. Did you know you can apply for free passes to SugarCon by speaking at the conference or becoming a Sugar Scholar? Get the details on Two fabulous ways to earn a FREE pass to SugarCon 2018.

We have hit a tipping point in world wide Emoji usage. We really must work like we live, because the perfect two character combination of pizza and beer and others like it are starting to appear in business emails and other business communications. In particular, marketers have started to use emojis in subject lines of emails as a way to catch people's attention.

 

As Rich Green put it:

It's nearing the end of the second decade of the third millennium.

If people want to use poop emoji's in business, so be it.

 

Many Sugar customers have started running into an issue with emojis when using MySQL databases or the Sugar cloud. What could be so challenging about a poop emoji, you may ask? For that you may need a quick lesson on Unicode.

 

Character encoding and the beginnings of Unicode

Many of you are programmers, so you likely know that computers run entirely on (binary) numbers. As you type away on your keyboard, the stream of characters that gets stored in memory is a sequence of numbers based on the character encoding in use. "A" could be stored as 65, while "a" could be stored as 97, and so on. Programmers are obsessed with efficiency, so they would create many different character encodings for different alphabets or character sets to only use as little space as possible.

 

This became a real challenge as we started to share documents between systems via e-mail or the world wide web. Identifying and using the right character encoding became very important and also very difficult. If you didn't know the character encoding then you had to guess and if you guessed wrong then the text would not be displayed properly at all.

 

Encoding gone wrong. By Wikimedia Foundation (Wikimedia Foundation) [<a href="https://creativecommons.org/licenses/by-sa/3.0">CC BY-SA 3.0</a> or <a href="http://www.gnu.org/copyleft/fdl.html">GFDL</a>], <a href="https://commons.wikimedia.org/wiki/File:Mojibakevector.png">via Wikimedia Commons</a>

 

Fortunately, a group of companies, organizations, and individuals decided to come together to form the Unicode consortium. The Unicode consortium's primary mission has been to standardize character encoding into the Unicode standard that is widely used today. The most common Unicode character encoding is UTF-8 which is a variable length character encoding that can store any Unicode code point (which includes a growing number of emojis) into a number that is 1 to 4 bytes long.

 

MySQL's utf8 and utf8mb4 encodings

Programmers are obsessed with efficiency. Many programmers will not use 4 bytes if they think 3 bytes will do. When MySQL first implemented UTF-8 years ago, they decided that supporting the Basic Multilingual Plane (BMP) was good enough since it contained all the characters and most symbols used in modern languages. These characters require no more than 3 bytes. Adding support for the Supplementary Multilingual Plane (SMP) which includes emojis would mean that some characters would need to be stored in 4 bytes, which was just one byte too many for somebody.

 

Whatever the reason, MySQL's first implementation of UTF-8 that they called utf8 did not fully support UTF-8. When they fully implemented UTF-8 years later, they called it utf8mb4. This has been a source of confusion for many years since MySQL's utf8 character set seems to work fine until you run into certain special characters like emojis. Trying to store an emoji in MySQL's utf8 results in failure and the loss of data.

 

Sugar Summer '18 migrates from utf8 to utf8mb4

Sugar Summer '18 will use utf8mb4 for MySQL. This will allow users to import and display records and emails that contain emoji and other characters in the SMP. This feature will also be rolled into a future on-premise release.

 

The Sugar Summer '18 release will be available for Sugar cloud customers in a few short weeks.

 

Emojis in Sugar!

 

This character set and collation will be set automatically for new instances of Sugar and updated during the upgrade of existing instances of Sugar.  

 

If you have changed the default DB collation and you are in the Sugar cloud or on-premise using a MySQL database, then you should ensure that your collation is utf8mb4 compatible prior to upgrade. The collation can be set by the admin in Admin > Locale > Collation or by modifying config_override.php. If your collation is utf8mb4 compatible, the upgrade will automatically migrate the collation to utf8mb4. For example, if you have set your collation to be utf8_swedish_ci, the upgrade will migrate the collation to utf8mb4_swedish_ci. If no collation is set, Sugar will use the default utf8mb4_general_ci.

 

Database tables with very large row sizes (for example, custom tables with a large number of custom fields) may be unable to be automatically upgraded. The upgrader will notify you if a table would exceed the single-row size supported by MySQL (65,535 bytes) upon conversion to utf8mb4. In order to reduce the row size, we recommend the following:

  • Remove any/all fields that are not being used.
  • Reduce the lengths for char/varchar fields (e.g. size of longest existing value plus some padding).
  • Replace large varchar fields with text fields. Text fields are roughly 10 bytes, so significant size reduction exists when text fields can replace large varchar fields (e.g. VARCHAR(255).

For years SugarCRM has published database schema information for each new Sugar release at apidocs.sugarcrm.com. Generated documentation, like our database schema reference, has largely taken a back seat to more formal documentation like the Sugar Developer Guide and Sugar Developer training curriculum over at Sugar University.

 

Over the past couple of releases, we have improved the Sidecar framework code to have better structure and code documentation while preserving compatibility. One of the benefits has been a significant improvement in the quality of our generated Sidecar JavaScript documentation when using tools like JSDoc.

 

Anyone can now view the documentation at anytime for each new Sugar release without having to manually generate the documentation!

 

If you visit SugarCRM Generated Documentation, you will find the Sugar 8.0.0 Sidecar framework documentation available today. We will be publishing the generated Sidecar framework documentation for each new release in the future.

 

We plan to continue to invest in the quality and quantity of generated API references that we publish for each release.

Sugar Summer '18 introduced confirmation links to allow customers to gather affirmative consent for using email addresses in campaigns. Check out Sugar Summer '18 (8.1.0) Overview for Sugar Developers for more details.

 

As discussed in Why SugarCRM developers should care about Data Privacy and GDPR, data privacy is an important topic that we are going to keep discussing here on the developer blog. Today is another installment specifically focused on best practices for getting consent to collect and process personal information.

 

General Data Protection Regulation (GDPR) has strict requirements for consent. However, consent to personal data processing is a big topic for regulation in many countries and especially applies to email marketing. Under GDPR, unless you have an existing lawful basis for using personal information, you need to collect explicit consent from the data subject when storing and processing their information in Sugar. By explicit consent, we mean it was freely given, informed, affirmative, and unambiguous. For example, a web to lead form with a pre-selected opt-in checkbox is not going to cut it in the EU, Australia, or Canada.

 

Let's look at a couple different ways of collecting explicit consent.

 

Collecting explicit consent

 

Collecting explicit consent is particularly important for marketing campaigns and business development. For other CRM use cases (like customer support), you may already have another lawful basis that gives you the right to process customer information.

 

Consent really just boils down to allowing someone to knowingly and freely give their permission for you to use their personal information. You could collect consent over the phone, written in an email, or over a handshake in person.

 

It is possible to collect explicit consent with a properly designed web form as well. There are lots of resources out on the web with advice for building compliant Single Opt-In (SOI) web forms.

 

However, a common practice for collecting consent is to use Double Opt-In (DOI) which is also known as Confirmed Opt-In (COI).

 

Double Opt-In (aka Confirmed Opt-In)

 

We all know web forms aren’t perfect. Sometimes users aren’t sure what they are signing up for themselves. Maybe they mistyped their email address and now somebody else is signed up for that mailing list. Or worse, somebody is a troll and intentionally signing them up for email or services that they don’t want. There is a better way and it is called double opt-in.

 

Double opt-in (or Confirmed Opt-In) essentially means that someone who puts their personal information into a web form must confirm twice. A user enters their information on a form and affirms that they want to subscribe to a service. They then get a confirmation email, text message, or some other communication where they can re-confirm their intention. This is typically done by providing a link in an email that users must then click to confirm.

 

Double opt-in is an easy way to ensure you have a clean mailing list or lead list full of people who went through the trouble to confirm twice.

 

Many of our customers use marketing tools such Act-On, Marketo, and others to manage their mailing lists and the necessary opt-in processes. As an added feature to Sugar’s Marketing capabilities, we are planning to include DOI/COI in a subsequent release. Meanwhile, here is how you can achieve DOI/COI with Sugar 8 or Spring ‘18.

 

Adding Double Opt-In to Sugar 8.0.0

 

If you have access to the Sugar 8 Preview then this is something you can try today. Otherwise you will need to wait until the official release in a couple short weeks.

 

This customization implements double opt-in behavior for specific email addresses stored within Sugar. The steps below involve Sugar Studio configuration changes as well as code level customizations. This example allows for any email address to be subject of a confirmed opt-in flow without any specific dependencies on Campaigns, web to lead forms, Leads, Contacts, etc. Note that since this customization is focused on the confirmation step of double opt-in so for this reason we prefer the term “Confirmed Opt-In”.

 

Once you have followed the steps below, you will be able to generate confirmation messages for new email addresses. For each email address that needs to be confirmed, there will be an associated URL that, when clicked, will confirm and opt-in the associated email address.

 

Step 1) Configure your lead forms or integrations to opt-out by default

 

You need to take some steps to make sure that email addresses are not prematurely opted-in within Sugar.

 

In Sugar administration, visit the “System Email Settings” panel and make sure “Opt-out new email addresses by default” is checked. You may need to enter SMTP Mail Server settings in order to save this change on this page. This is a new setting in Sugar 8.0.0.

 

 

If you are using Sugar web-to-lead forms, they should be updated to remove the email opt-in checkbox that Sugar includes by default if it still exists. I’d recommend updating the form to let the user know that a confirmation email will be sent separately.

 

 

If you are using some other system (like a lead portal or marketing automation tool) to capture contact information, they should be configured to not send back email opt-out information to Sugar. Or alternatively they could be configured to set email opt-out to true by default.

 

Step 2) Add Studio configuration changes to Data Privacy module

 

In this example, we will reuse the Data Privacy module (new in Sugar 8.0.0) to track email confirmation activities. If you haven’t already, you may need to add the Data Privacy module to the list of displayed modules in the Administration panel first.

 

Add a new “Confirmed Opt In” option to the dataprivacy_type_dom used by the Data Privacy module’s type field. This will be used to differentiate email confirmations from other Data Privacy activities.

 

 

Also add two new custom fields that are visible only for Confirmed Opt-In activities. To do this, I used the following visibility formula on these custom fields.

 

equal($type, "Confirmed Opt In")

 

The first is an email_address_to_confirm_c text field that stores the record ID for the email address that needs to be confirmed.

 

 

The second is a URL field called confirmation_url_c that will use a generated URL. The default value is designed to pass email_address_to_confirm_c field and the Data Privacy id field as parameters to a custom entry point that we will create later. The URL will vary based upon the site URL for your Sugar instance. For my development environment, I used:

http://localhost:8080/sugar/index.php?entryPoint=confirmEmail&email={email_address_to_confirm_c}&id={id}

 

 

Make sure to add these two new fields to your Data Privacy record layout as well which will help us with testing. Remember to Save and Deploy after adding them.

 

 

Step 3) Add the ConfirmedOptInManager class and related Entry Point

 

The following class includes two optional logic hooks as well as a required method that will be used by a new Entry Point. We are choosing to use a custom Entry Point here instead of a REST API endpoint because the Sugar REST API is designed to only return JSON responses. This makes for a poor usability experience for the person who is trying to confirm their email address and sees a page full of gobbledygook in their browser. Typically, people will expect to see a nicely formatted web page with a confirmation message. An entry point allows us to return an HTML response that matches user expectations.

 

The createConfirmedOptIn() method can be used as an after_save hook on the Leads module to automatically create a confirmed opt-in activity whenever new Lead records are created.

 

The sendConfirmOptInMessage() method is a stub that can be used as an after_save hook on the Data Privacy module to automatically send confirmation emails to end users. This could also be orchestrated using an Advanced Workflow.

 

The last confirm() method is the most important one. This method is called directly by a custom public entry point shown further below. This is what determines if the request is a valid one and then updates the email address record. It first loads the DataPrivacy bean, then compares the email address ID stored there with the one provided in the URL. It then updates the EmailAddress and DataPrivacy beans accordingly.

 

./custom/Extension/modules/Leads/Ext/LogicHooks/createConfirmOptInRecord.php

 

<?php
// Copyright 2018 SugarCRM Inc.  Licensed by SugarCRM under the Apache 2.0 license.
$hook_array['after_save'][] = array(
  1,
  'create_confirmation_record',
  null,
'Sugarcrm\\Sugarcrm\\custom\\ConfirmedOptIn\\ConfirmedOptInManager',
  'createConfirmOptInRecord',
);

 

./custom/Extension/modules/DataPrivacy/Ext/LogicHooks/sendConfirmOptInMessage.php

 

<?php
// Copyright 2018 SugarCRM Inc.  Licensed by SugarCRM under the Apache 2.0 license.
$hook_array['after_save'][] = array(
  1,
  'send_confirmation_message',
  null,
'Sugarcrm\\Sugarcrm\\custom\\ConfirmedOptIn\\ConfirmedOptInManager',
  'sendConfirmOptInMessage',
);

 

./custom/src/ConfirmedOptIn/ConfirmedOptInManager.php

<?php
// Copyright 2018 SugarCRM Inc.  Licensed by SugarCRM under the Apache 2.0 license.
namespace Sugarcrm\Sugarcrm\custom\ConfirmedOptIn;

/**
*
* Manages confirmed opt-in workflow for Email addresses
*
* Class ConfirmedOptInManager
*/

class ConfirmedOptInManager
{

    /**
     * After Save logic hook on Leads module (or any other Person module with an email address)
     *
     * Used to automatically create email confirmation records in DataPrivacy module.
     *
     * @param \SugarBean $bean SugarBean with an email address that needs to be confirmed.
     * @param string $event Logic hook event name. Expected to be 'after_save'.
     * @param array $arguments
     */

    public function createConfirmOptInRecord(\SugarBean $bean, string $event, array $arguments)
    {
        //If this is a newly created record
        if (!$arguments['isUpdate'] && $bean->load_relationship('emails')) {
            $address = $bean->emailAddress->getPrimaryAddress($bean);
            //And a primary email address is set
            if (!empty($address)) {
                //Create a new 'Confirmed Opt In' DataPrivacy activity
                $dpb = \BeanFactory::newBean('DataPrivacy');
                $dpb->name = "Confirm $address";
                //Note: This assigned user for Data Privacy activity MUST have the Data Privacy Manager role!
                $dpb->assigned_user_id = $bean->assigned_user_id;
                //We will store ID for primary address here.
                $dpb->email_address_to_confirm_c = (new \SugarEmailAddress())->getGuid($address);
                $dpb->type = 'Confirmed Opt In';
                $dpb->save();
            }
        }
    }


    /**
     *
     * After save logic hook on Data Privacy module
     *
     * Can be modified to send the confirmation message to the appropriate email address.
     * You should make sure not to send multiple messages for the same Data Privacy request.
     *
     * @param \DataPrivacy $bean DataPrivacy bean that has been saved
     * @param string $event Logic hook event name. Expected to be 'after_save'.
     * @param array $arguments
     */

    public function sendConfirmOptInMessage(\DataPrivacy $bean, string $event, array $arguments)
    {
        //Should only continue if this is a newly created Confirmed Opt In request
        if ($bean->type === 'Confirmed Opt In' && !$arguments['isUpdate']) {
            $log = \LoggerManager::getLogger();
            $eab = \BeanFactory::retrieveBean('EmailAddresses', $bean->email_address_to_confirm_c);
            if (!empty($bean->email_address_to_confirm_c) && !empty($eab->email_address)) {
                $log->info(
                    "Send out confirmation to $eab->email_address with link to $bean->confirmation_url_c."
                );
                // You could update to send message using mailer, SMS gateway, whatever.
            }
        }
    }

    /**
     *
     * Opt-in the email address if the email associated with given requestID matches the given emailAddressID
     *
     * Used with `confirmEmail` public custom entry point
     *
     * @param string $requestId
     * @param string $emailAddressId
     * @return bool
     */

    public function confirm(string $requestId, string $emailAddressId): bool
    {
        global $current_user, $timedate;
        // Since this is a public endpoint, we need to select a current user to associate with this action.
        // For our purposes, we will use the User assigned to the Data Privacy request since the person confirming
        // their email address is likely not a current Sugar user.
        // Note: For this to work, assigned/current user must be admin or have Data Privacy Manager role!
        if ($current_user && empty($current_user->id)) {
            $q = new \SugarQuery();
            $q->from(\BeanFactory::newBean('DataPrivacy'), array('team_security' => false));
            $q->where()->equals('id', $requestId);
            $q->select(array('assigned_user_id'));
            $results = $q->execute();
            //If no assigned user, then default Admin user will be used.
            $assignedUserId = empty($results) ? 1 : $results[0]['assigned_user_id'];
            $current_user = \BeanFactory::retrieveBean('Users', $assignedUserId);
        }

        $confirmRequest = \BeanFactory::retrieveBean('DataPrivacy', $requestId);
        //Make sure email address matches the one associated with the request ID
        if ($confirmRequest && $confirmRequest->email_address_to_confirm_c === $emailAddressId) {
            //If already closed, then no need to re-confirm.
            if ($confirmRequest->status === 'Closed') {
                return true;
            }
            $addrBean = \BeanFactory::retrieveBean('EmailAddresses', $emailAddressId);
            $addrBean->opt_out = false; //opt in!
            $addrBean->save();
            $confirmRequest->status = 'Closed';
            $currentTime = $timedate->getNow();
            $confirmRequest->resolution = 'Confirmation received on ' . $currentTime->format(DATE_RFC850);
            $confirmRequest->date_closed = $currentTime->asDbDate();
            $confirmRequest->save();
            return true;
        }
        return false;
    }

}

 

The last file is our public custom entry point, which we install with the help of the EntryPointRegistry extension. This will allow us to create a publicly accessible URL on our Sugar instance that can be used to easily display HTML for a user readable response.

 

./custom/Extension/application/Ext/EntryPointRegistry/confirmEmail.php

 

<?php
// Copyright 2018 SugarCRM Inc.  Licensed by SugarCRM under the Apache 2.0 license.
$entry_point_registry['confirmEmail'] = array(
'file' => 'custom/src/ConfirmedOptIn/ConfirmEmailEntrypoint.php',
'auth' => false
);

 

Since this is a public entry point, we should take extra care to validate user input. We will use the InputValidation framework to assert that the URL parameters are both the GUIDs that we expect for record IDs.

 

./custom/src/ConfirmedOptIn/ConfirmEmailEntrypoint.php

 

<?php
// Copyright 2018 SugarCRM Inc.  Licensed by SugarCRM under the Apache 2.0 license.
if(!defined('sugarEntry') || !sugarEntry) die('Not A Valid Entry Point');

use Sugarcrm\Sugarcrm\Security\InputValidation\InputValidation;
use Sugarcrm\Sugarcrm\custom\ConfirmedOptIn\ConfirmedOptInManager;

$manager = new ConfirmedOptInManager();
$validator = InputValidation::getService();

$recordId = $validator->getValidInputGet('id', 'Assert\Guid');
$emailId = $validator->getValidInputGet('email', 'Assert\Guid');

if ($manager->confirm($recordId, $emailId)) {
    echo "<h1>Confirmed!</h1>";
} else {
    echo "<h1>Unable to confirm.</h1>";
}

 

This page will just show “Confirmed!” if the email address was properly confirmed and “Unable to confirm.” if the confirmation failed which could be due to badly formed request or because the email address or data privacy record IDs didn’t already exist in the system. You could update this entrypoint to return whatever HTML you want or perhaps even to redirect the person’s browser to some other page.

 

For example,

 

header("Location: http://www.example.com/success");

 

If you happen to see a blank page, then input validation has probably failed. You can check for a fatal message in sugarcrm.log for more details on the failure.

 

Step 4) Try it out!

 

Create a new Lead record that includes a new email address.

 

New Lead record

 

Switch over to Data Privacy module and you will find a new Data Privacy record like the one below.

 

 

For convenience, we can just click the confirmation URL itself to trigger the confirmation check of using an email to do it. In a real implementation, it may make sense to hide this URL from Sugar end users so they do not accidentally click on it since we expect someone who does not use Sugar to click it themselves.

 

 

Nice, it was confirmed! If the request isn’t quite right or the email address is not in the system then you will get the “Unable to confirm” message instead.

 

If I return to my Data Privacy record, then I can see it has been completed and that the email address on my original Lead record has been opted in.

 

 

There is a lot that could or should be done to enhance this example for production use. But you can see that the pattern is straightforward to implement using existing Sugar platform extensions and APIs.