Batch Delta Feed File Upload

You can upload the delta file segments feed using the FTP upload command. This allows the delta feed for the segment feed to be processed, enabling a subset of users and their segments to be updated in the UPS from the delta file. In this upload method, the batch delta file is processed, and only users with modifications or updates in their profiles or segments are updated in the UPS, rather than all users in the file. This differs from the full file upload, in which all users in the feed file are uploaded.

Note: Processing a delta feed file is similar to processing a full feed file, with a slight difference.

Presently delta files are processed for modified/updated users only. To remove users, you may still need to use full feed files as users cannot be removed using delta files.

Assumptions

The client is expected to send a full feed file to update all users' information at once and periodically send incremental delta feed files.

Processing Delta Feed File

What is a full feed file?

The full feed file consists of all users' details and each user will have full segments and non-segments batch data, user attributes, user preferences and/or user linking details in a single file.

What is a delta feed file?

The delta feed file contains only updated user details. Each user will have full segment and non-segment batch data, user attributes, user preferences, and/or user linking details in a single file.

What is a user-linking file?

The user linking file is a delimited text file and it has a special Unicode character as a delimiter.

Preparing Delta Feed File for Upload

  • Syntax and payload structure in delta feed files are similar to full feed files.
  • Delta files should include only updated users.
  • When preparing batch segment delta feed files, only updated and/or new users' segment details are added.
  • When preparing batch non-segment files and user linking files, only updated users' and/or new users' details are added.

User Linking File Upload

  • To establish a solid foundation for user linking, start by sending a Full User-Linking Feed File. This file serves as a baseline, ensuring that all user IDs are correctly linked within the platform.

  • Consider the Full User-Linking Feed File as a reset—it refreshes all previous linking information.

  • After sending the Full File, you can use Delta User-Linking Feed Files to make incremental updates. These files update specific user-linking information without affecting the entire system.

  • For efficient processing, it is recommended to split Delta files into smaller chunks, with each file no larger than 50 MB.

Other details concerning the processing of these files are provided in the table below.

Delta Feed File vs. Full Feed File

  Full Feed File Delta Feed File
Data Processing

After the full feed file is uploaded to the FTP server, all the users' details will be available in the database.

UPS system checks for the new incoming file version with the file available in the database. Only the latest file is processed and persisted in the database.

FTP server stores the incoming batch data into a Zstandard compressed file.

After the delta feed file is uploaded to the FTP server, only a set of users' details are updated in the database.

UPS system checks for the incoming file version with the available full feed file and delta feed file version. Only the latest files are processed and persisted in the database.

FTP server stores the incoming batch data into a Zstandard compressed file.

Note: The delta feed files are uploaded in the same way as mentioned in the Full Feed File column.
Data Restore The client sends a full feed file to sync UPS data with client data. The client sends the full feed file whenever UPS data needs to be synced with client data or to restore the missing data.
UPS Reader

Serves the latest full feed file users details.

Stalled full feed file versioned user details are ignored.

Serves the latest full feed file and/or subsequent all delta files users details.

Stalled full feed file and/or delta feed file versioned user details are ignored.
Data Compaction For the same API Key and attribute type, if multiple files are uploaded using the FTP command, the compactor removes the stalled versioned users' data from the database.

For the same API Key and attribute type, if multiple delta files are uploaded using the FTP command, the compactor does not remove the stalled delta feed file versioned users' data from the database.

Unless a full feed file is uploaded using the FTP command, all the existing full feed files and delta feed files users' details will be present in the database.

After a full feed file is uploaded using the FTP command, the compactor removes all the stalled versioned full files and/or delta files users' data from the database.

Sample Data Files

Batch Segment File

rr-segments

Copy 
{"userId":"00351934487736654412608034933597340001","value":{"Carta-11":"Carta-IDEAPIUEasyCasa","RFM-MEDIO":"RFM-MEDIO","CLTVtot-NULL":"CLTVtot-NULL","CLTVprog-NULL":"CLTVprog-NULL","Bricoleur-NULL":"Bricoleur-NULL"}}
{"userId":"00351934487736654412608034933597340002","value":{"Carta-12":"Carta-IDEAPIUEasyCasa","RFM-NULL":"RFM-NULL","CLTVtot-NULL":"CLTVtot-NULL","CLTVprog-NULL":"CLTVprog-NULL","Bricoleur-NULL":"Bricoleur-NULL"}}

Batch User-attributes File

rr-userattributes

Copy 
{"userId": "00351934487736654412608034933597340001","value": {"like" : ["prod102710181","prod102480201"],"disLike" : ["prod10150001","prod101520076"]}}
{"userId": "00351934487736654412608034933597340002","value": {"like" : ["prod102710182","prod102480202"],"disLike" : ["prod10150002","prod101520077"]}}

Syntaxes and Commands to Process and Upload Batch Files

Process and Upload File Commands' Syntax

The following syntaxes are used to process the batch files.

Uploading Full and Delta Feed File by Client

Client uploads full and delta feed files to the FTP server, using the following FTP command:

Copy 
put <FileName>

Processing Batch Files

Batch Segments Full Feed File

After uploading the full feed file, run the following command to process the batch segments full feed file:

Copy 
site upsbatch <FileName> -attribute <attributeType>

Batch Segments Feed File

After uploading the delta feed file, run the following command to process the batch segments delta feed file:

Copy 
site upsbatch <FileName> -attribute <attributeType> -loadtype delta

User Linkings Full Feed File

After uploading the full feed file, run the following command to process the user linkings full feed file:

Copy 
site upslink <FileName>

User Linkings Delta Feed File

After uploading the delta feed file, run the following command to process the user linkings delta feed file:

Copy 
site upslink <FileName> -loadtype delta

Upload Commands

Run the following commands to upload and process batch files with the required changes in file names:

To upload and process batch segments full feed file:

Copy 
put segment1.json
site upsbatch segment1.json -attribute rr-segments

To upload and process batch segments delta feed file:

Copy 
put segment2.json
site upsbatch segment2.json -attribute rr-segments -loadtype delta

To upload and process user linkings full feed file:

Copy 
put userlink1.txt
site upslink userlink1.txt

To upload and process user linkings delta feed file:

Copy 
put userlink2.txt
site upslink userlink2.txt -loadtype delta

FAQs

What happens if a customer provides a delta feed file without first sending a full initial feed file ?

While the feed process can intelligently process incoming delta feed files, ensuring all user data is correctly updated in the database, it's not recommended to start with a delta feed file. It's best to send a complete full feed file first

Note: It is recommended that you always send the delta feed file on top of the full feed file.

What happens when a customer uploads a full feed file as a delta feed file?

Sending a delta feed file instead of a full initial feed file can cause issues with user data. The system may not properly clean up old data, leading to potential inconsistencies.

What happens when a customer uploads a delta feed file as a full feed file?

If a delta feed file is sent as the first feed instead of a full initial feed file, the system might retain old versions of user data that should have been cleaned up. This can cause issues and complicate data management.

To fix the data inconsistencies, the customer should:

  • Resend the full initial feed file.

  • Resend any delta feed files in the same order they were originally sent.

  • Finally, upload the actual delta feed file that caused the issue.

Does the User profile Feed process support Bi-directional(Grouped) user-linking delta feed files?

Yes, the User Profile Feed process supports both Bi-directional (Grouped) and Uni-directional (Explicit) mapping with user-linking delta feed files.

How does the User profile system data sync with customer data?

The customer should send a full feed file to ensure all user data is up-to-date and matches the customer's records.