Adobe Experience Platform Real-Time CDP: Digital Data Ingestion

From capturing cart abandonment and browsing interests to tracking newsletter sign-ups, an organization must efficiently collect and act upon user behavior data to actually drive impact. Unlocking the potential of Customer Data Platforms (CDPs) relies heavily on digital analytics data. However, when evaluating CDPs, organizations often overlook this critical aspect. 

Although at first glance many of these capabilities may seem similar, today’s most popular CDPs employ diverse approaches to gathering and incorporating this data from digital properties. Digital analytics was my first career step into the martech world, so I was naturally curious to understand what these differences are. In exploring these platforms, I’ve delved deep to make sense of it all so I can explain the ways you can ingest digital analytics data, and what the recommended methods and best practices for each platform are. To kick-off this series exploring digital data ingestion for CDPs, I want to start with the first CDP I ever worked with – Adobe’s Real Time CDP (AEP RTCDP).

Data Collection and Analytics Data Sources

Adobe provides two main methods for collecting and ingesting digital analytics data to the platform: The Web SDK and Source Connectors.

The Web SDK

Adobe’s Experience Platform Web SDK is a client-side JavaScript library (alloy.js) that allows customers of Adobe Experience Cloud to collect and send data to various services in the Experience Cloud (and AEP RTCDP) through the Adobe Experience Platform Edge Network using datastreams. Data from the Web SDK can also be server-side forwarded on the Edge to 3rd Party destinations. The Web SDK library is free to the general public, but Adobe Launch Tag Management System access requires a license for an Adobe product like Analytics or RTCDP.

Adobe also provides a Mobile SDK with similar functionality for App data collection and the Server-Side API for data collection on IoT/OTT devices or a complete server-side implementation on Web/Mobile. To keep things simple, today I will only focus on client-side implementations and not the Server-Side API. Please note that anything in this article explicitly referencing the AEP Web SDK likely also applies to the AEP Mobile SDK – but I have listed just the Web SDK for brevity.

The Web (and Mobile) SDK can be deployed and managed using any Tag Management System but is most easily configured through Adobe Launch. Data collected using the Web SDK is formatted in Adobe’s Experience Data Model (XDM), Adobe’s standardized framework for organizing and exchanging customer experience data between Adobe and non-Adobe applications, and is sent downstream in real-time as a JSON object. Payloads are based on a schema defined within AEP or Launch and are composed of standard (automatically collected) and custom fields for page, event, account, or user-level data, which can also support array and hierarchical data. 

The Web SDK does not have differentiated types of tracking calls. Still, developers can configure payloads to include specific fields like eventType or webPageDetails.pageViews.value to indicate what event types are being collected. For those who are familiar with Adobe Analytics, you can perform the same types of custom data collection with the Web SDK, just in a different format.

The Web SDK automatically assigns users Adobe’s cookie-based ECID, but can also be configured to set First Party device identifiers (FPID) which can be used to re-identify users after their cookies are expired or cleared. XDM also supports the collection of durable user identifiers (e.g., customerID, email address) within a Web SDK payload which can be used as identities and for profile unification within AEP.

Source Connectors

In addition to the Web SDK, AEP RTCDP also supports data ingestion (including historical data) for Adobe Analytics and MixPanel data through native source connectors if your site is already tagged to collect data for one of these tools. Although these have a slightly slower ingestion latency than the Web SDK, using connectors prevents the need to perform any additional tagging if your data collection and data model are already suitable to fit your CDP use cases. 

In the case of the Adobe Analytics source connector, you can ingest data from multiple report suites into AEP, but different datasets are recommended for each unless your analytics solution designs (e.g., props/eVars) match up between report suites or you perform custom data mapping at the time of ingestion. Adobe still recommends, however, that all customers eventually move to the Web SDK for their data collection if they are using Adobe Analytics appmeasurement.js (legacy implementation).

If your site is already tagged with Google Analytics / Google Data Layer, historical data can be ingested to the platform through the BigQuery connector, with ongoing data streaming using the Web SDK deployed via Launch with the help of the Google Data Layer extension if the Google Data Layer is present in your site.

Other analytics platforms like Amplitude currently do not have a native source connector, but data can be ingested to the platform using Adobe’s Streaming Ingestion (HTTP) API or through any of Adobe’s database source connectors.

On the platform side, AEP RTCDP can support the ingestion of millions of events per hour from streaming source connectors and the Web SDK with a latency of less than 1 min to profile and less than 15 mins to the AEP Data Lake.

FAQs

Can the Web SDK and Analytics Source Connector coexist within AEP?

The Web SDK allows you to stream data simultaneously to Adobe Analytics and directly into the platform, so if you utilize the AEP datastream, I recommend that you turn off the Analytics source connector due to redundancy. If historical analytics data is required for your use cases, the Adobe Analytics connector should be used to backfill with the caution that this dataset will be automatically deleted if / when the AA Source Connector is turned off. To avoid data loss, you should utilize AEP Query Service (now called Data Distiller) to make a copy of the historical analytics data in a different dataset which will not be deleted when the connector is turned off.

Can the Web SDK (alloy.js) and Adobe Analytics (appmeasurement.js) Libraries coexist on a site? 

Yes, Adobe recommends a dual implementation approach for customers migrating onto the Web SDK. The Web SDK also allows for the migration of existing visitor ECIDs from the Visitor ID service with a toggle within the Launch UI so that identities are not reset.

Does the Web SDK support multiple datastreams to AEP?

Yes – in fact there is a fantastic recent article covering use cases for this exact topic

What if I use the Google data layer on my site?

Adobe Launch (and thereby the Web SDK) works best reading from the Adobe Client Data Layer, but Adobe’s recently-released Google Data Layer extension in Launch allows you to use a Google data layer in your implementation.

What is the data format for the Analytics Source Connector?

Data streamed into the platform using the source connector will be auto-mapped to the Adobe Analytics ExperienceEvent Full Extension schema field group. The Analytics source connector also enables you to apply filters to selectively include or exclude data from ingestion to the Real-Time Customer Profile (although all data still makes it to the Data Lake). In comparison, data streamed from the Web SDK directly into AEP will be pre-configured in XDM format (matching the custom schema set up to receive its data). Unfortunately, there is no way to filter which data makes it to the profile if the dataset is profile-enabled.

Can I use the Web SDK to send data to only Adobe Analytics and then use the Source Connector to ingest data into AEP?

Yes, this is possible. However, whether it is the right option for you depends on considerations for things like data latency for your use cases, the number of report suites, and desired schema design (e.g., generic vs. AA-specific (i.e., eVar/prop) fields) for your digital analytics data.

What if I do not use Adobe Analytics?

If you do not utilize Adobe Analytics or use a different digital Analytics tool, you have several options to collect and ingest your digital analytics data to AEP

1. Leverage existing tagging/analytics: Utilize other source connectors (e.g., Mixpanel) to ingest data or send your analytics data to a Warehouse (e.g., BigQuery) first and ingest using a source connector for the DW
   1. Considerations: Intermediate step (Data Warehouse) causing latency in data availability to AEP 
2. (Recommended) Tag site/app with Web SDK and stream data directly to platform: Keep existing tagging/analytics and implement the Web SDK through Launch in parallel to stream data needed for use cases directly into the platform
   1. Considerations: Requires maintaining a second TMS is Launch isn’t already in use by your org.

Summary

Unlike many other CDPs, Adobe provides methods to natively ingest data from your existing analytics data sources in near real-time without the need to retag your site/app. This can be a tremendous time savings, especially if your site is already tagged with Adobe Analytics. Adobe does, however, recommend that all Adobe Analytics customers eventually move their analytics tagging to the Web SDK, so an AEP implementation may present a good opportunity to also perform this migration. The Web SDK also offers a number of other advantages that are not discussed here. 

Navigating the ins, outs, and nuances of AEP can be tricky, but the team at Shift Paradigm is here to help you make sense of it all and set you up with the right data collection model for your business.

In future articles, I’ll explore various CDPs, how they ingest digital analytics data, and recommended methods and best practices for each.