Data integration has come a long way over the last couple decades. In this three-part series, we'll provide a brief recap of the history, a perspective on how data integrations are being disrupted today, and how businesses can best take advantage of the disruption to drive effective sales and marketing operations through Sugar.
Part 1 - A Brief History of Data Integration
Data integration is a powerful concept which, believe it or not, pre-dates the Internet. It’s a history full of acronyms and with far too many protocols and platforms to thoroughly document here, so our intent is to give you a good sampling of its evolution over the past few decades.
We’re in the midst of a true breaking of the status quo of data integration. As all of the evolution over the past decades has been about “building a better mousetrap,” but all the while focussed on IT as the gatekeeper for data integration.
Our history builds to that today, for the first time, integrations can be managed by business users. We’re breaking through the “IT as the gatekeeper” paradigm.
Our history starts with:
EDI (Electronic Data Interchange)
Data integration pre-dates the Internet as a mechanism for the structured exchange of data between two computers. The origins of EDI are attributed to developments in military logistics. A key moment was the 1948 Berlin airlift where large volumes of data and information about transported goods required transfer over a baud teletype modem.
EDI exchanges remain in place today, typically for highly secure, high-volume data exchange between business partners. They tend to support larger enterprises and require ongoing management from IT teams to oversee data exchange and uptime.
ETL (Extract, Transform, Load)
Early data integration methods, dating back to the 1970s, used the extract-transform-load process for data integration. These are heavily structured data integration processes with three discrete, sequential steps:
- Data extraction - bringing in data from multiple data sources
- Data transformation - adjusting the format and structure of the data to prepare it to be queried and analyzed
- Data loading - moving the data into a final target database, such as an operational data store, data mart, or data warehouse
ETL is an IT-intensive process with powerful but complex tools from vendors such as Oracle, IBM, SAP, SAS, Cognos, and many more.
Because of the rigid sequence required to get data into an accessible format, ETL systems face challenges around scalability and real-time data access. There are also strong dependencies on system and process design, as improperly designed ETL systems run into significant issues.
SOAP (Simple Object Access Protocol)
SOAP became a standard protocol for web services, leveraging XML to integrate systems in the early 2000s.
SOAP was originally designed by a team at Microsoft lead by Dave Winer in 1998, and version 1.2 of the specification became a W3C recommendation in 2003.
SOAP became a common method for developers to write data connections between systems using a common XML data format. Generally speaking, SOAP is effective for building a one-to-one bespoke integration (e.g. inter-bank communications) but is not optimal for scalability or speed of implementation.
REST (Representational State Transfer)
Computer scientist Roy Fielding introduced the term representational state transfer in 2000 as part of his doctoral dissertation at the University of California, Irvine. In between 2008 and 2010, REST reached a tipping point in popularity where it exceeded SOAP and has dominated SOAP in popularity since.
REST has become the standard for most publicly available APIs due to its scalability and speed to implement. REST web services are stateless, which delivers performance, reliability, and scalability. REST services re-use components that can be managed and updated independent of the greater system, even while it is running - a significant advancement from the rigidity of ETL.
That said, the stateless nature of REST also represents a challenge for how it is implemented in connecting systems. REST has no memory, so it relies on the connecting systems to manage audit trails and data history.
iPaaS (Integration Platform as a Service)
Over the past five years, iPaaS emerged with dozens of vendors providing integration tools to IT teams to develop, execute, and manage integration flows between multiple disparate applications. The technical breakthrough for iPaaS was allowing IT teams to leverage standard cloud services so that they don’t require hardware or middleware to manage their data integrations.
iPaaS represented an advancement in the data integration space, although it’s one whose benefits were largely directly at mid-sized to large IT teams who gain efficiencies through using iPaaS technology instead of developing their own customer integrations through the aforementioned protocols or many others.
Integration Platform for Business Users - Automated Integrations for Marketing, Sales & Support
The alphabet soup of acronyms leading up to this point represent an evolution in the capabilities around integrating data, but they remain in the old paradigm of requiring IT resources to manage data integrations.
Empowering business users to manage their data integrations breaks - dare we say, shatters - the status quo of data integrations.
Marketing, sales, and support users prefer not to be reliant on IT to create and update integrations for the systems they use every day.
Is there a new system feeding leads for the SDR/BDR team? Or a new data field that you want to give sales visibility to for context around leads? How about a new data field tracking the effectiveness of marketing programs through to sales results?
These are all adjustments these business teams want to make themselves. Businesses are moving too fast to be stuck in IT development queues for updating their data integrations.
In our next article on Breaking the Status Quo of Data Integration, we’ll share our perspective on what it takes for an automated data integration to work for business users today.