Skip navigation
All Places > Developer > Blog
1 2 3 4 Previous Next

Developer

194 posts

We can probably all agree that automated tests are great, but sometimes we put off writing them because the initial first step of getting everything working correctly can be really difficult.  

 

Good news, Sugar developers!  Writing PHPUnit tests for your customizations just got easier in Sugar Fall '18!  Our engineering team has created a test runner specifically for your custom PHPUnit tests.

 

If you do not have access to the SugarCRM provided unit test suite, submit the Developer Tools Request.

 

The first thing you'll need to do is add the SugarCRM provided unit test suite to your Sugar directory. Be sure to select the branch in the unit test suite that matches the version of your Sugar instance (needs to be 8.2 or higher). Follow the instructions in the branch's readme to add the tests to your Sugar directory, install the necessary dependencies, and update the file permissions.  

 

Now you're ready to write your tests!  Add your new PHPUnit test classes to the custom/tests/unit-php directory. You may need to create it yourself if it doesn't already exist.

 

For example, create a new file named CustomLogicHookTest.php and place it in the custom/tests/unit-php directory.  Paste the following in to the new file:

<?php
    //custom/tests/unit-php/CustomLogicHookTest.php
   
    namespace Sugarcrm\SugarcrmTestsUnit\custom;
   
    use PHPUnit\Framework\TestCase;
   
    /**
     * @coversDefaultClass \CustomLogicHook
     */

    class CustomLogicHookTest extends TestCase
    {
       
        /**
         * @covers ::aFunctionInsideTheCustomLogicHook
         */

        public function testCustomHook()
        {
            // Wow! This is a horrible test.
            // You would never do something like this just to increase your test  
            // coverage, right?
            $this->assertTrue(true);
        }
    }

In a shell, navigate to the Sugar provided unit tests directory by doing something like the following:

cd tests/unit-php

Then you can run the tests by executing

php ../../vendor/bin/phpunit --testsuite custom

or by using gulp (gulp installation instructions are available here)

gulp test:unit:php --suite custom

The above test is extremely simple and doesn't test anything. If you'd like to see an example of a larger test suite that tests Sugar customizations, check out Professor M's School for Gifted Coders.  We've recently updated our PHPUnit test suite so we can use the new test runner.  Check out the pull request associated with this change for the details.

Hey there Sugar Developers! I’m Niegel Leoncio (@niegel), CRM Manager for Bishop-Wisecarver. This year is my 2nd SugarCon and I was privileged to be chosen as a Sugar Scholar.  I attended the session "Sugar, DevOps, and IBM" led by Vineel Nirogi and Julian Coldrey and I wanted to share a few take away that I took from this session.

 

It’s easy to get distracted by the different festivities within the conference, and this year, even distractions outside of the conference is much greater since we are in Vegas. That’s why every time I attend a big event like SugarCon, I would define a goal of why I am attending and will focus my energy towards that goal. This year, it was finding the right tools and best practices now that I am enhancing my development skills. Vineel and Julian’s session is the Las Vegas jackpot that I was waiting for! Although it was the last session on the last day, it’s packed with lot of great nuggets.

 

Julian started the session by painting the picture of the challenges that they have in the beginning. It’s surprising that a company as large as IBM, with 60,000 users worldwide, 5,000+ business partners and 200+ integrations and customizations, struggles with similar issues that smaller companies like us have, except on a larger scale. They struggle with delayed delivery and about 100 code defects per release. Their challenge is – “How do we deliver value to our customers radically faster?”

Julian shared this high-level strategy in approaching this problem.

  • Setting up a goal
  • Identifying the current state
  • Learn from teams better than you
  • Design your roadmap
  • Execute the plan

 

Challenging the status quo and thinking outside the box is a principle that they embraced. They asked themselves, what can we do to shorten the time of code deployment from 45 days to 1 day. It’s an impossible task for some, but their team is ready to break barriers. As of this writing, they were able to move from a 45-day release to a weekly release with about less than 10 defects per release. Impressive!

How did they do that? Vineel took over and talked about how DevOps changed Everything. Since the definition of DevOps differs from everyone, Vineel defined DevOps as a Culture that induces sub-conscious way of continuously redesigning the working style as a Team to increase the happiness index in Software Development Cycle. He further explained that DevOps is influenced by 4 areas. These 4 areas comprise of:

  • Tools
  • Culture
  • Practices
  • Behavior

Vineel also shared some of the DevOps tools that they are using in IBM:

  • Github® - For Source Code Management
  • Jenkins® - For Continuous Integration
  • SonarQube® - For Source Code Analysis
  • PHP Analysis - PhpUnit, PhpLint, SugarLint, Jasmine
  • Zabbix® - For Production Monitoring
  • Slack® - For Communication & Feedback
  • Ansible® - To convert Infrastructure as a Code
  • Docker® - For Containerizing the application

 

For someone who is brand new into Development, knowing tools that are used by experts helps me to get a good foundation in my learning curve. Learning from their journey from identifying their challenges to achieving what they have accomplished gives me a confidence in my quest in the dev world.

 

Check out the session’s slides below to see more of what they covered.
Sugar, DevOps, and IBM

Does the idea of upgrading to the latest version of Sugar keep you awake at night? 

 

Matt Marum discussed how the development community can best be prepared for Sugar upgrades. As a reminder, Sugar cloud instances are upgraded quarterly, the current release is Fall 2018, and on-premise deployments are provided with yearly upgrades, the current release is Sugar 8. 

 

Steps to upgrade success

Preventive Care

Utilizing the latest information on the developer blogs and developer guide ensures that we won't be surprised as SugarCRM updates the Sugar platform. We, as developers, need to keep up to date. We perform preventative care to ensure that we are ready for the latest updates.

Items to be reviewed by for each release

  • Broken customizations and integrations 
  • Configuration based changes within SugarCRM
  • Custom code
  • Restrictions in Sugar Cloud with Custom Code

 

Release upgrade resources

 

Planning and testing

  • Use a staging environment - Sugar Enterprise customers have a sandbox included in their subscription. Be sure to utilize the sandbox. For all customers, it is possible to request a staging environment after GA has been announced and before the actual upgrade occurs. Be sure to leverage this staging environment to test all customizations to ensure success.

  • Heathcheck Tool - Sugar's silent upgrade application has the ability to run just the healthcheck portion of the upgrade script. Be sure to utilize this tool to determine which of your customizations need to be addressed as part of the upgrade.

  • If you get an "F" then you can’t upgrade - each item that has an F (RED) grade has a knowledge base article available on the Sugar Support website.
  • Document all customization and configurations - Be sure to have a list of all of your customizations so that you can run a full test of everything in the staging environment. It may not be necessary to test every customization for each upgrade but be sure to keep the list of changes up to date. 
  • If you utilize the list above and know the critical use cases, at a minimum it would be best to test each critical use cases before the final upgrade so that there are no surprises.

 

Follow the Sugar Developer blog and sign up for Sugar Developer news to make sure you get the latest info as it's available from SugarCRM!

 

For the slides, check out Preparing for Sugar Upgrades!

Hello!  I'm Justin Kuehlthau (@justinkuehlthau), Director of the Sugar Practice at Technology Advisors (Technology Advisors, Inc.).  This was my 8th SugarCon and this year I was lucky enough to be selected as a Sugar Scholar! 

 

I learned a lot about the future of security and authentication in Sugar from Jelle Vink, Distinguished Engineer at SugarCRM.  Jelle discussed Integration Best Practices and demoed a proof of concept integration between Android Assistance and Sugar!

 

Jelle started by covering some of the current pain points when integrating with Sugar:

  • No streamlined federated identity capabilities
  • LDAP & SAML have limited support on Mobile and Plugins
  • Credentials stored or used everywhere
    • Sugar Web Interface, Mobile, Outlook Plugin, Integrations, etc.
  • No fine grained authorization controls

 

Jellle introduced Cloud Identity Services, which will be available to Sugar cloud customers via a limited Beta launching near the end of October. Cloud Identity Services have many improved features including:

  • Federated identity solution
  • Industry standard OpenID Connect
  • Build on top of OAuth2
  • Enterprise SSO support
  • Full audit capabilities
  • Finely control what access each different application has

 

I'm very excited to see where this takes us.  Here is a video of Jelle's proof of concept integration with Android Assistant.

 

You can check out Jelle's slides here!

Hi peeps! My name is Kristján Geir Mathiesen and I work for Origo, an Icelandic Sugar partner. I´ve enjoyed attending SugarCon for the last three years - this year was just as much fun as ever

 

Me being from Europe and the year being 2018, GDPR has been on my mind and to do list for the past 18 months. When I looked over the SugarCon agenda and saw that David Wheeler was doing a presentation on data privacy, I just knew I had to attend it.

 

 

 Below are some of the highlights that stood out for me.

 

  • Consent should always be stored with audits and how it was obtained - basis for the processing is additionally very important.
  • Basis for processing must be checked and re-checked since the basis for processing might be revoked at any time! Make sure the basis are kept up to date.
  • Great idea to audit anything that relates to data privacy - including the source - to make any post-work easier to perform.
  • Sugar recommends: Don´t erase records, erase content of records.

  • New ErasureFieldList function was added for manual erase (see code in slides).
  • Personally Identifiable Information (PII) is tricky and can be delusive, like the 2006 paper reveals in this slide:

 

 

David´s presentation is called "Deep Dive into Data Privacy Architecture."  You can check out his slides here.

Hello!  I'm Justin Kuehlthau (@justinkuehlthau), Director of the Sugar Practice at Technology Advisors (Technology Advisors, Inc.).  This was my 8th SugarCon, and this year I was lucky enough to be selected as a Sugar Scholar! 

 

One of the interesting sessions I attended was presented by Jorge Arroyo, VP of Engineering and Fellow at SugarCRM, and Alexey Klimko, Senior Software Engineer at SugarCRM.  This session covered planning and architecting for and troubleshooting performance in Sugar.

 

This break out session started with Jorge discussing performance in Sugar and ways it can be improved.  My favorite quote of the presentation was:

Solutions must be architected and designed for performance.  The #1 element of user experience is performance.  You must plan for it during all phases of your project.

Main Takeaways

Jorge and Alexey split the presentation into two main parts:  performance and troubleshooting.

 

Performance by Jorge

List Views

List view performance is determined by the database.  The key to list view performance is indices.  If you’re going to sort on a field, you should consider having an index on it.  On the other hand, you do not want too many indices.  As such, you must be judicious with the fields you allow your users to sort on.

 

When typing in a search, Sugar list views automatically search by your filter after a 300ms pause.  If you type "Arr", pause for more than 300ms, type "oy", pause for more than 300ms, type "a" and then stop in order to search for Arroya, you’ve actually performed 3 searches for 1 search.  The pause delay can be modified via Sugar Config or you can disable the type-ahead search.  List views and search queries are 80% of the load on your instance.

 

Record Views

To improved Record view performance, only show the information you need.  Every chart in a dashlet in the Intelligence Pane is a report that runs.

 

Save action performance is affected by Workflows, Logic Hooks or external APIs.  With poor planning, 1 record save can easily be turned into many record saves.

 

Home Page Dashboards

Be careful when designing your Home Page.  Every chart is a report that runs and every report can potentially be looking at several different modules.  Only show what you need.  If you’re deploying a default Home Page, don’t deploy a default Home Page to all users that has a lot of Reports on it.

 

Troubleshooting by Alexey

Pay attention to where the issue may be.  A slow client machine cannot run Sugar well because Sugar uses a lot of JavaScript.  Sugar will not run well if there is slow network performance.  PHP runs on the web server, which must be setup properly on a correctly scaled machine.  Finally, it all requires good code.

 

Things To Check

Use Chrome Developer Tools to determine if Sugar is actually the issue as opposed to the Network.  PC/Browser?  What is the DB Server doing?  (Are other databases hosted on the DB Server?)  What are the slow queries in Sugar?  Review the DB statistics.  Get as many details and facts as possible: Which view? Which action? Specific user? Any page? Specific time? Specific IP? Logs? HW usage graphs? Data volume? 

 

When all else fails, ask Sugar Support for help.

 

Use Sugar's XHPROF!

SugarCRM XHProf viewer is an extended viewer based on the standard xhprof viewer by Facebook that shows some additional information like sql and elastic queries, their timing and stack traces. (https://github.com/sugarcrm/xhprof-viewer)

 

Get their slides

I tried to summarize Jorge and Alexey's big points above, but you can check out their full slide deck here.

In our recent release of SugarCRM Mobile 6.2, we made a change to the way our mobile app talks to the Sugar server.  

 

We made this change because Apple deprecated their UIWebView component (since iOS 8) in favor of their optimized WkWebView.  In addition to providing performance improvements, WKWebView also provides additional security features and a more responsive user experience.  But WKWebView also added enforcement of cross-origin resource sharing (CORS), so the mobile team needed to add a native proxy that runs locally on your mobile device at localhost:8081. 

 

This proxy allows the SugarCRM Mobile app to work around these restrictions and can introduce connectivity issues for some SugarCRM Mobile users. Most users of SugarCRM Mobile are not affected, but if you are using SugarCRM Mobile with Enterprise Mobility Management (EMM) infrastructure (i.e. Airwatch with App Tunneling configuration, or VPN) then you may have already noticed connectivity issues after upgrading to 6.2. For example, some customers have reported an issue with the app hanging on the splash screen.

 

If you are affected, you will need to adjust the whitelist in your EMM or Mobile Device Management (MDM) tool to allow traffic to localhost:8081. Please consult your EMM / MDM documentation for steps necessary to perform this configuration change.

 

Below is a diagram showing how the new proxy should work in an EMM infrastructure.

If you are an on-premise customer using our Mobile SDK, you have the option of enabling CORS support in Sugar by adjusting your .htaccess configuration to send the correct CORS headers (and removing the proxy). Sugar Enterprise customers with access to the SugarCRM Mobile SDK can build a custom app with adjusted the proxy settings for their own environment.

 

Please leave any questions or feedback below and we'd be happy to help!

Hi everyone! I’m Megan Sheehan, business analyst and trainer at Technology Advisors, and proud Sugar Scholar at sugarcon 2018.

 

Lauren Schaefer (@Lauren_Schaefer) gave the first developer-focused presentation this year, “Dissecting Professor M for Best Practices,” and it was great! Lauren did an awesome job engaging the audience with polls and interactive Q&A while sharing insights into how and why developers should engage with professor m.

 

The developer community is hopefully already familiar with the basics of Professor M, but Lauren explained how we can use the Professor M repo for examples of both customization and DevOps best practices. Even better, we can contribute our own code examples to the repo so that Sugar Engineering will test against them as part of their testing process – meaning your customizations will be even more ‘upgrade safe’ than ever!

 

Check out Lauren’s slides for all of the details.

SugarCon and Sugar Scholars

Posted by lschaefer Employee Oct 16, 2018

We had a fabulous time meeting so many of you in Las Vegas for SugarCon 2018 last week!  We learned, we laughed, and we danced.  It doesn't get much better than that!

 

We introduced a new program at this year's SugarCon:  Sugar Scholars.  The Sugar Scholars program is a competitive program that partners and customers applied to.  Sugar Scholars were selected based on their advocacy of Sugar all year long.  The Sugar Scholars received a scholarship to attend SugarCon and helped us run the SugarCon Developer Track.  You may have seen them greeting you at the Developer Lunches, keeping our speakers on time, and giving lightning talks (how fun were those???).

 

One way they are helping behind the scenes is blogging about each of the breakout sessions in the Developer Track.  Over the next few weeks, you'll see their posts in this blog.  Be sure to check out their posts as they will be including a high level overview of each session as well as a link to the speaker's slide deck.

 

Let me introduce you to our fabulous group of Sugar Scholars.  

From left to right, they are...

  • Niegel Leoncio, CRM Manager, Bishop Wisecarver Corp
  • Kristjan Geir Mathiesen, Systems Analyst, Origo hf
  • Justin Kuehlthau, Director, SugarCRM Practice, Technology Advisors, Inc.
  • Megan Sheehan, Business Analyst & Trainer, Technology Advisors, Inc.
  • Jeff Bickart, Director of Operations, Faye Business Systems Group 
  • Me--not actually a Sugar Scholar, but they let me jump in their picture anyway. :-)

 

Whether you were able to join us in Las Vegas or not, you can still join in on the fun on social media.  Be sure to check out the SugarCon Photo Album on the SugarCRM Facebook Page.  You can also discover some notable moments on the SugarCRM Developers Twitter page and the SugarCRM Twitter page.

 

If you have feedback from this year's conference, we'd love to hear it!  Let us know in the comments below how we can improve for next year.

 

We're already looking forward to SugarCon 2019!

Hey there, developers!  We've officially released Fall '18!  This release is just for Sugar Cloud (formerly known as Sugar On-Demand).  Don't worry on-prem users--you'll get these changes in our upcoming 9.0 release! 

 

This release has a ton of great features including new quotes configuration options for administrators, new data visualizations, and enhancements to Advanced Workflow that allow you to automatically send email alerts to a wider audience and to a more selective set of recipients.

 

Our Co-Founder & CMO, Clint Oram, discusses the highlights of this release from an end-user's perspective in the video below: 

 

 

I recently hosted a webinar where I gave an overview of the big things developers need to know about this release:

 

The slides from the webinar are available here.

 

If you're looking for the high-level overview, I've got you covered!

  • We've upgraded several libraries including Handlebars, Underscore, jQuery, jQuery Migrate, and jQuery UI.  You can get the details in my blog post You'll never guess which libraries are changing in the upcoming Sugar release.
  • Administrators now have a web interface where they can configure the Quotes module.  This means less coding for you!  Ensure any fields that your code customizations rely on are visible or included in a SugarLogic formula for a visible, calculated field. 
  • If user subscription limits are enforced on a Sugar subscription key, SAML and LDAP JIT user provisioning will not allow a customer to accidentally exceed the number of allowed active users.
  • Our REST API version increased to 11.3.
  • We have improved our support for OAuth 2.0 by adding support for bearer tokens.  Get all of the details with examples in my blog post The Sugar REST API adds support for OAuth 2.0 bearer tokens in Fall '18. 

 

This blog post promised to have just about everything you need to know about the Fall '18 release.  Below are some resources that have the rest of the details.

 

We hope you’re as excited about this release as we are!

You’ve been asking for it.  We’ve been listening.

 

We're improving our support of OAuth 2.0 standards by adding support for RFC 6750 bearer tokens in the Sugar Fall ‘18 release.  

 

So what's changing?  You’ll still authenticate to get the access token the same way.  For example, you might authenticate with a call similar to the following:

 

curl -X POST \
 https://mysugarinstance/rest/v11_3/oauth2/token \
 -d '{
  "grant_type":"password",
  "client_id":"sugar",
  "client_secret":"",
  "username":"admin",
  "password":"asdf",
  "platform":"base"
}'

 

And get something like the following in the response body:

{
   "access_token": "37cfa546-055b-43c9-8273-466fbf4b3235",
   "expires_in": 3600,
   "token_type": "bearer",
   "scope": null,
   "refresh_token": "671d8117-c8b7-4dee-92d6-4f309d717090",
   "refresh_expires_in": 1209600,
   "download_token": "eeb702a9-ec35-4a92-9619-d914172b5eac"
}

 

In versions prior to the Sugar Fall ‘18 release, you would include the access_token you received in response to the authentication call as an OAuth-Token in the header of subsequent requests.  For example, if you wanted to get a list of the Accounts, you would make a call like the following:

 

curl -X GET \
 https://mysugarinstance/rest/v11_3/Accounts \
 -H 'OAuth-Token: 37cfa546-055b-43c9-8273-466fbf4b3235'

 

Beginning in Sugar Fall ‘18, you can now follow OAuth 2.0 standards and pass the access_token as a Bearer Token.  Following the same example as above where we retrieved a list of the Accounts, you can now make a call like the following:

 

curl -X GET \
 https://mysugarinstance/rest/v11_3/Accounts \
 -H 'Authorization: Bearer 37cfa546-055b-43c9-8273-466fbf4b3235'

 

The difference is small.  You’re still passing an access token in the header of each request--you’re just changing the format of that header.  

 

So why are we making this change?  If you have integrated with other OAuth 2.0 resources, you’ll be able to reuse more of that code and/or behavior.  This change also enhances the interoperability of our APIs with standard tools and libraries. Plus, following industry standards is always a good thing.

We're just one month away from SugarCon, and we can't wait to meet you there in Las Vegas!  If you haven't registered already, you can still do so at sugarcon.com.  

 

The SugarCon agenda is packed full of keynotes, breakout sessions, training, certification exams, networking opportunities, food, and, of course, drinks.  Below is the agenda we've been preparing just for developers.  Check out the full SugarCon agenda for even more.

 

Wednesday, October 10

7:30 AM - 9:00 AMBreakfast (for all conference attendees)
9:00 AM - 12:00 PMOpening Remarks & Morning Keynotes (for all conference attendees)
12:00 PM - 1:30 PMDeveloper Lunch
1:30 PM - 2:30 PMDissecting Professor M for Best Practices
2:30 PM - 3:30 PMPerformance is a Feature
3:30 PM - 4:30 PMPreparing for Sugar Upgrades
4:30 PM - 5:30 PMAsk Me Anything Networking (for all conference attendees)
5:30 PM - 7:00 PMDevelopers Happy Hour & Lightning Talks
7:00 PMParty! (for all conference attendees)

 

Thursday, October 11

7:30 AM - 9:00 AMBreakfast (for all conference attendees)
9:00 AM - 12:00 PMOpening Remarks & Morning Keynotes (for all conference attendees)
12:00 PM - 1:30 PMDeveloper Lunch
1:30 PM - 2:30 PMDeep Dive into Data Privacy Architecture
2:30 PM - 3:30 PMIntegration Best Practices
3:30 PM - 4:30 PMSugar, DevOps, and IBM
4:30 PM - 6:30 PMDeveloper Happy Hour & Ask the Experts
Certification Exams

 

P.S.  You can still vote for the topics you'd like to discuss at our developer lunches here!

The new dates for Fall '18 developer webinars are September 18 and 19. We apologize for any inconvenience!

Hello Sugar Developers!

 

We want to make sure your customizations and integrations are ready for the Sugar Fall '18 release, so we’re hosting two webinars just for you!

 

What we will be covering:
We’ll discuss the big changes that are likely to impact you including the following:


Webinar Information:
We are holding two sessions to accommodate various geographical locations. Please choose ONE of the following times below.


Tuesday, Sept 18th 5:00 - 6:00 PM PT
OR
Wednesday, September 19th 8:00 - 9:00 AM PT
(Choose one)

 

Register Here

 

We will be posting the webinar recordings to the Sugar Community for those who are unable to attend the live sessions.

Introduction

Does your Sugar implementation extensively leverage Team security to apply data visibility rules? Do you have modules containing a few hundred thousand records?

 

If the answer is yes to at least one of these two questions and you are on Sugar version 7.11 or above, there is a great news for you! We have been developing a performance improvement for the above scenarios called Teams Denormalisation that can be enabled on your system.

 

Team security in a nutshell

Team security prevents a user from seeing records not associated with one or more of his/her teams. Administrator type users are exempt from visibility rules.

 

For example, Image 1 represents a Team security scenario. In this example, Jane is part of the Yoga Team, so she can view both the Yoga budget and the Yoga schedule records. Since she is not a member of the Accounting Team, she cannot view the Company budget record.

 

 

Image 1: End-user representation of a Teams visibility scenario

 

Team security technical implementation

To allow granular control over the single record visibility, a record can have multiple Teams associated with it. The unique combination of Teams is referred to as a "Team Set".

 

For example, Image 2 represents a Team security scenario that includes Team Sets. In this example, the Project1 test plan record is related to the Team Set Project1 and QA Automation (including the Project1 and QA Automation Teams). The record can be seen by the three users David, Esther and Ruslan, as they are all either members of Project1 and/or QA Automation Teams. The Project1 backlog record is related to the Project1 Team Set (including only the Project1 Team). The record is visible to David and Esther, but it is not visible to Ruslan.

 

 

Image 2: Back-end representation of a Teams visibility scenario, with Team Sets

 

More details of the Teams inner workings can be found within the Developer Guide.

 

High level description

Teams denormalisation is designed to simplify and therefore improve performance of runtime SQL system queries that are executed to display only records visible to the current user.

 

Detailed description

Teams denormalisation denormalises (and therefore duplicates) the information about which users are part of which Team Set. With this approach, the system can immediately determine if the current user accessing the system has an overlap on the Team Set for the module's records.

 

As denormalisation requires data duplication, every time a change on a Team Set happens, it is reflected either on the SQL table team_sets_users_1 or on team_set_users_2. Basically, the current SQL denormalisation table tracks which user ids are members of each Team Set.

 

Teams denormalisation has a few components. The main components that can be useful for debugging and supporting this functionality are the following:

  1. A scheduler job to complete background actions.
  2. Two additional SQL tables (team_sets_users_1 and team_sets_users_2). Only one of the two tables is considered active by the system at any one time.
  3. A configuration string on the config SQL table to determine: if the feature is enabled, what SQL table is currently considered active, and if the system knows if the current denormalised table is up to date or not.
  4. Two Sugar CLI commands to check the status of the system or to help rebuild the data.

 

Once the functionality is enabled, the background scheduler job will populate with data the SQL table that will be considered active. Upon completion, it will set the database configuration options accordingly, to let the system know that it is ready to use the denormalisation.

 

While waiting for the background scheduler job to finish, the system will automatically keep using the standard subquery based Teams lookup (not causing interruption of service).

 

A similar approach happens during rebuilds. The SQL table currently active is queried until the secondary SQL table is fully populated, and ready to be switched over.

 

When Teams denormalisation should be enabled

The Teams denormalisation feature is highly recommended for systems that are leveraging out-of-the-box Team security for data visibility purposes and have a decent size team set quantity.  As a base guideline, if there are more than 100,000 unique Team Sets combinations, the functionality should be considered. The functionality will be most beneficial for modules that have the largest number of records (over a few hundred thousand).

 

A possible drawback of Teams denormalisation is that the visibility of records can be applied with time delay, if the system is configured to propagate the Teams structure changes in the background (ie: not synchronously) or when a full SQL table rebuild is required.

 

It is not recommend to enable Teams denormalisation for customers with customised visibility rules that therefore do not use the standard product functionality as the system will most likely not be able to interpret the changes applied.

 

It is important to note that direct interactions with the database via SQL can corrupt visibility rules. For example, when a database restore is needed on an instance with Teams denormalisation enabled, you should forcefully initiate a full rebuild of the Teams denormalisation data after the database restore is complete.

 

Enabling Teams denormalisation

Complete the following steps to enable Teams denormalisation:

  1. Enable the Teams denormalisation scheduler job (Rebuild Denormalized Team Security Data) that is disabled by default
    1. Tune the interval of execution based on need or leave to the default 15 minutes
  2. Set the relevant settings in config_override.php 
  3. Wait for the Teams denormalisation scheduler to run and to complete the initial denormalisation. Alternatively, if available/possible, run the Sugar CLI command:
    ./bin/sugarcrm team-security:rebuild

 

Notes 

If the denormalisation is enabled, disabled and finally re-enabled through the config_override.php options and actions related to Teams have been completed through the framework while the denormalisation was disabled, the system will not enable the functionality immediately. The system only re-enables the functionality once the denormalised SQL table is brought again up to date. The denormalised SQL table can be brought up to date either by the next Teams denormalisation scheduler run, or by manually executing the repair Sugar CLI command where applicable.

 

Configuration settings

The use_denorm and inline_update settings can be applied to config_override.php to enable and configure Teams denormalisation. Detailed explanation of the configuration settings can be found below.

 

use_denorm

$sugar_config['perfProfile']['TeamSecurity']['default']['use_denorm'] = true;

From the Developer Guide:

This setting is used to enable or disable Team Security denormalization feature. Team Security denormalization can significantly improve SQL query performance when there are a large number of Teams in a Sugar instance. The cost is that this feature relies on a separate Team security denormalization table that needs to be updated as records change to ensure Sugar's Team security visibility rules are applied properly.

inline_update

$sugar_config['perfProfile']['TeamSecurity']['inline_update'] = true;

From the Developer Guide:

When denormalization is in use (perfProfile.TeamSecurity.default.use_denorm config set to true), this setting will allow the denormalization table to be updated in real time as records are updated in the system. If this setting is disabled, the denormalization table must be scheduled by the Sugar administrator via Schedulers > Rebuild Denormalized Team Security Data.

 

More details about inline_update

When Teams denormalisation is enabled and inline_update is set to "true", every action that affects Teams structures is completed synchronously.

 

If the option inline_update is set to to "false", the system will still synchronously add new Team Sets combinations (created at the module's record level). It will not perform synchronously actions that are completed on the admin area by an Administrator user such as:

  • adding users to a team (either directly or through reports to changes)
  • removing users from a team (either directly or through reports to changes)
  • deleting users (and therefore removing the user from a team)

The above actions can be expensive as the system needs to modify many entries on the denormalisation table currently in use. When the option is set to false, the action will be completed only when the scheduler executes its next run for the Teams denormalisation job. At that point the records will become visible/invisible according to the changes completed on the admin area.

 

The recommended approach is to start by setting the flag to "true", to make sure that all the changes are applied real-time. The save time for the Administrator(s) might increase slightly when performing changes that affect the Teams structure.

 

If this approach does not work for the specific scenario, it is possible to change the inline_update setting to "false" and be aware of the time delay required for the new visibility changes to be applied to the system.

 

Logger

For logging purposes, there is a dedicated logger channel provided for the Teams denormalisation functionality.

The denormalisation logger channel can be configured to "info" on config_override.php with the following configuration setting:

$sugar_config['logger']['channels']['denorm']['level'] = 'info';

 

All the available log levels are documented within the Sugar Developer guide.

 

Sugar CLI commands

Sugar has two Sugar CLI commands that are relevant to Teams denormalisation:

  • Status - Displays status information of the denormalised data
    • ./bin/sugarcrm team-security:status
  • Rebuild - Rebuilds the Teams denormalisation data when needed
    • ./bin/sugarcrm team-security:rebuild

 

Count unique Team Sets

It is possible to know how many unique Team Sets combinations exist within the system, by running the following SQL query:

 

SELECT
  COUNT(id)
FROM
  team_sets
WHERE
  deleted = '0';

 

Database configuration

The current database configuration settings for the Teams denormalisation can be read from the database with the following SQL query:

 

SELECT
    *
FROM
    config
WHERE
    category = 'team_security'

 

Sample SQL output showing the database settings of Teams denormalisation:

 

category: team_security
   name: denormalization_state
  value: {"rebuild_running":false,"active_table":"team_sets_users_1","up_to_date":true}
platform:

 

The information retrieved by the SQL query is also used by the system, and it can be easily displayed by the status Sugar CLI command wherever applicable.

 

Force repair

In cases where the denormalisation SQL table is not current (eg: data has been manually manipulated or data is of sync due to a database restore etc.), it is possible to enforce the re-build by running the Sugar CLI command to update the Teams denormalised data, by providing the additional parameter --ignore-up-to-date as shown below:

 

./bin/sugarcrm team-security:rebuild --ignore-up-to-date

 

The Sugar CLI command will truncate the content of the secondary table and complete a full alignment of the data. Then it will swap the SQL table the functionality is leveraging by updating the database configuration settings. This process removes the need for a downtime period during rebuild as the system is either attempting to point to an up-to-date SQL table or fall back to the standard functionality.

 

When the rebuild command is run without the parameter --ignore-up-to-date, the system is unable to understand if there are discrepancies introduced via SQL. The command will only understand discrepancies due to changes in Teams structure/user structure that were generated through the framework.

Hey there, developers!

 

We are hosting lunchtime table topics just for developers on both days of SugarCon.  After you pick up your lunch, you'll be able to browse tables that are labeled with various topics and select one that interests you.  We'll have a facilitator at each table to get the conversation going.  Everyone, regardless of experience level, is invited!  You're welcome to share your knowledge, ask questions, or just sit and listen.  This is your chance to do a little networking with other developers and learn something that may not be covered in the breakout sessions.

 

We need your help in deciding what the table topics will be!  I've added comments to this blog post with some ideas we have.  If we missed a topic you're interested in discussing, add a comment below with your suggestion.

 

Most importantly, we want you to vote!  Click the Like button on topics below to vote for your favorites.

 

We can't wait to see you at SugarCon on October 10 and 11!  If you haven't registered yet, you can register at sugarcon.com

 

P.S.  Don't forget to submit your proposal to give a 5 minute lightning talk at SugarCon by Tuesday, August 21, 2018!