Enterprise Data Integration

Revision History

Date Changes
July 26th 2018 Initial EDI Documentation
August 8th 2018 Document version history
SQS-Upshot Implementation Notes
SQS Input / Output Sample Payload Updates
August 27th 2018 Added new event types for inputs and added mandatory/not-mandatory meta data for various attributes.
November 26th 2019 Document re-organized to accommodate newer input and output types added to the stack.
November 26th 2019 Added AWS Kinesis input and output
December 12th 2019 Added S3 input
Added info on configuring 3rd party access to AWS resources.

Streaming Data Integration

When large sets of data are to be pushed/pulled as events occur or new data is received or ‘near real-time" processing is required, the following mechanisms are used.

AWS SQS

SQS, short for "Simple Queueing Service", abstracts queues in a SaaS model with uptime and delivery guarantees. Used when multiple sources can send in-to and read from queues to take advantage of parallel and / or near real-time processing.

Currently, Upshot does not create queues for your use. Upshot accesses existing queues created and managed by you in your own AWS account. You can setup SQS inputs to be read / ingested through External Inputs/Outputs section in your Upshot account.

Securing your AWS SQS Queues The SQS queues are managed by the application developers themselves. To secure access to these SQS queues and allow Upshot read/write from/to them, please refer to AWS guidelines here..

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-security.html

Permissions Needed:

SQS Queue Usage Read Permission Write Permission
External Input Yes No
External Output No (Can be Yes, but not needed) Yes

Upshot DOES NOT need permissions to manage SQS Queue lifecycle, like creating or deleting queues. Upshot uses AWS SDKs to access AWS resources and would need access to appropriate credentials for the resource being accessed. The IAM credentials can be configured on your Upshot Account dashboard.

  1. If your account has two apps App 1 and App 2, one External Input CANNOT be shared between them. Each app MUST have its own input queue.
  2. You can share External Output queues, if your SQS receiver application can handle the messages appropriately based on the "appId" field.
  1. Since the same queue is used for input and output, Upshot will read back data that it has sent out intended for your consumption. You MIGHT not receive the data, and the message could be dropped since it does not conform to Upshot’s External Input format as specified in this document.

AWS Kinesis

https://docs.aws.amazon.com/streams/latest/dev/controlling-access.html https://docs.aws.amazon.com/streams/latest/dev/security-best-practices.html
Kinesis Stream Usage Read Permission Write Permission
External Input Yes No
External Output No (Can be Yes, but not needed) Yes

Managing Cross Account Access for AWS Resources

When using EDI with Upshot, all resources used are owned by you and not by Upshot. And since Upshot needs access to your resources, there are security best practices defined by AWS on how to provide access to 3rd parties.

Below is a list of few tutorials and material that describe those best practices and things to take care of while providing us access to your resources.

  1. https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html
  2. https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html
  3. https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html
  1. Create an AWS resource in your AWS account.
  2. Generate an "externalID" to use with Upshot.
  3. Get access to Upshot’s AWS Account ID.
  4. Create an IAM role granting permissions needed, with a condition that only users from Upshot AWS account with the above created "externalID" can access it.
  5. Use the ARN of the above created IAM role and the AWS resource to configure an Upshot external input or output.

Streaming Input Definition

  1. The JSON string MUST NOT contain nested objects in it.
  2. The JSON string MUST be encoded in "UTF-8".
  3. Pre-defined keys that MUST be present in the payload are…
S.No Key Name Data Type Description
1 appId String The ID used by Upshot SDKs to identify the "app" from which the events are received. This is available through your Upshot account dashboard. This attribute IS REQUIRED / MANDATORY.
2 ownerId String Every Application on Upshot is governed by an "account" or the "owner". This is available through your Upshot account dashboard. This attribute IS REQUIRED / MANDATORY.
3 appuid String An ID, which your application uses to identify a user uniquely within the system. This MUST be unique across all users, not exceeding 64 bytes.
This attribute IS NOT REQUIRED for server side events. Will be ignored if sent.
This attribute IS MANDATORY for other event types. This should be an empty string (""), and NOT "null"… if this field is not applicable for this event.
4 userId String Upshot creates a unique id for every install of an application on mobile devices. Every time, the application is installed (users could do this by un-installing earlier, resetting a device, etc). This might not always be available to the integrators, in which case this COULD be equal to "appuid", not exceeding 64 bytes.
This attribute IS NOT REQUIRED for server side events. Will be ignored if sent.
This attribute IS MANDATORY for other event types. This MUST NOT be an empty string (""), and MUST NOT be "null".
5 eventId String Every event that is sent into SQS MUST have a unique id. This MUST be a "string" (not exceeding 64 bytes).
This attribute IS NOT REQUIRED for server side events. Will be ignored if sent.
This attribute IS MANDATORY for other event types. This MUST NOT be an empty string (""), and MUST NOT be "null".
6 sessionId String For mobile and web apps, this identifies the user session. For enterprise integrations, when this is not available, any "string" (not exceeding 64 bytes) can be supplied.
This attribute IS NOT REQUIRED for server side and user profile events. Will be ignored if sent.
7 platform String For mobile and web apps, this identifies the platform through which this session is in play. For mobile platforms, valid values can be "iOS", "Android", "Unity_iOS", "Unity_Android" and for browser based sessions valid values could be "Desktop web", Mobile web". For non-mobile / non-browser platforms, any appropriate string value can be sent.
This attribute IS NOT REQUIRED for server side and user profile events. Experimental for server side events and will be ignored if sent for user profile events.
8 version String This represents the OS version of the platform on which this event was generated (on which the session occurred).
This attribute IS NOT REQUIRED for server side and user profile events. Experimental for server side events and will be ignored if sent for user profile events.
9 appVersion String This is the version of the application from which the event was generated.
This attribute IS NOT REQUIRED for server side and user profile events. Experimental for server side events and will be ignored if sent for user profile events.
10 eventName String This is the name of the "event" generated. Special characters aren’t allowed except ‘_’. Event name MUST start with alphabet and can not be "null".
This attribute IS NOT REQUIRED for user profile events. Will be ignore if sent.
This attribute IS MANDATORY for other event types..
11 startTime Unix Timestamp This is the time at which the event "started". This timestamp should include time upto "milliseconds".
This attribute IS NOT REQUIRED and will be ignored for user profile events, MANDATORY for others.
12 endTime Unix Timestamp This is the time at which the event "ended". This timestamp should include time upto "milliseconds". This can be the same value as the "startTime", if this event is not a timed event. Timed events are ones which take a while to complete. Ex: A page view is a timed event, while a button click is not!
This attribute IS NOT REQUIRED and will be ignored for user profile events, MANDATORY for others.
13 tzoffset Integer This value is the difference in "milliseconds" from UTC to the User’s timezone. Ex: If user’s timezone is UTC, this value is 0. If the user’s timezone is -0700 (Mountain Time Zone), then this value is -25200000. If the user’s timezone is +0530 (Indian Standard Time), then this value is +19800000.
This attribute IS NOT REQUIRED and will be ignored for user profile events, MANDATORY for others.
14 eventType String This describes what data this event represents. This attribute MAY be present and is optional. Valid values for this are…
a) userProfile - Used when you need to update a user’s profile attribute.
b) userEvent - Used to send an "event" that a user has performed through a system that does not have the Upshot SDK integrated.
c) serverEvent - Used to send an "event" that has occurred on the application server side, and not attributed to a user. Ex: A service update available, a new app released to the store, a service down, a service notification, etc.
This attribute IS NOT REQUIRED / MANDATORY. If this attribute is not present, the default value will be "userEvent".
NOTE : Currently only "userEvent" works in production and the others are experimental and not available for general use in your applications/environments.
15 attrType String This specifies if the user profile attribute is a custom attribute or Upshot default attribute. Values for this key are case-sensitive and MUST be one of the below…
"upshot" - for Upshot default attributes.
"externalID" - a set of identities of the user provided to Upshot to identify a user across various social or other integrations that are in place already or added in future. Upshot has a set of pre-defined externalIDs, but you can add more identities as needed, similar to adding custom profile attributes. Identities (externalIDs) and Custom attributes are treated differently in Upshot.
"custom" - for custom user profile attributes.
This attribute IS MANDATORY for "userProfile" events. Not required for others.
NOTE : Currently only "userEvent" works in production and the others are experimental and not available for general use in your applications/environments.
S.No Description
1 Special characters won’t be allowed except ‘_’
2 Key should start with an alphabet
3 Keys must be "string"
4 Key and Value should be valid JSON types. Value should be "String" or "Number", if not they will be converted into "String" with a best case effort.
5 Total bytes of all custom keys and their values should not exceed 2048 bytes.
6 If any of the above is violated, the key-value pair or the entire event can be dropped.
Ex 1 : If an event has a custom attribute named "98States", it will not be saved since it starts with a digit. But, all other attributes of this event are saved.
Ex 2 : If an Upshot pre-defined key (appuid, ownerId, etc) is not sent with an event, the entire event is dropped.

Upshot default user profile attributes

S.No Key Name Attribute Type Data Type Description
1 age upshot Number Cannot be less than 0, Cannot be greater than 120.
2 gender upshot String Possible values are "Male", "Female", "Unknown", "Other"
3 email upshot String MUST be a valid email address. No validations except for the email address format will be made on the given string.
4 phone upshot String MUST follow International Telephone number format. No validations except to check if the "+" prefix is provided in the given string is made. If the "+" prefix is not found, the given string will be rejected.
Ex: +15555555555
5 firstName upshot String First name of the user.
6 middleName upshot String Middle name of the user.
7 lastName upshot String Last name of the user.
8 userName upshot String Friendly name to address a user in messages. Can be used to personalize message content.
9 language upshot String Primary language the current user speaks/knows. When localising content, this language will be chosen.
10 occupation upshot String Occupation of the user.
11 occupation upshot String Occupation of the user.
12 maritalStatus upshot String Marital status of the user.
13 location upshot String Location of the user passed as latitude, longitude pairs
14 localeCode upshot String Locale Code of the user’s device on which the application is being used right now.
15 dobDay upshot Number Date component of the user’s date of birth.
16 dobMonth upshot Number Month component of the user’s date of birth.
17 dobYear upshot Number Year component of the user’s date of birth.
18 emailOptout upshot Number For GDPR compliance…
Set to 1, if the user has opted out of email communications.
Set to 0, if the user has opted in for email communications.
19 smsOptout upshot Number For GDPR compliance…
Set to 1, if the user has opted out of SMS communications.
Set to 0, if the user has opted in for SMS communications.
20 pushOptout upshot Number For GDPR compliance…
Set to 1, if the user has opted out of Mobile/Web Push Notifications.
Set to 0, if the user has opted in for Mobile/Web Push Notifications.
21 dataOptout upshot Number For GDPR compliance…
Set to 1, if the user has opted out of any level of data storage (all raw and aggregated data about this user will be deleted at a best case effort). Aggregated data which cannot be attributed to a user/identity will remain.
Set to 0, if the user has opted in for storage of all forms of data.
22 ipOptout upshot Number For GDPR compliance…
Set to 1, if the user has opted out of IP address capture.
Set to 0, if the user has opted in for IP address capture.
23 appUID externalID String A way to uniquely identify this user in Upshot irrespective of the number of their devices / times this application is installed. This identity is how the application identifies this user in its systems.
A way to uniquely identify this user in Upshot irrespective of the number of their devices / times this application is installed and ACROSS apps (created on Upshot dashboard). This is useful companies / enterprises / business units.
24 enterpriseID externalID String Ex: A company has a HR Portal, Project Management applications, etc. Each of them might identify a user either by phone number or email-id as per their requirements. But this ID is how a user can be UNIQUELY identified across all those applications. This could be an LDAP ID / SSO ID /etc or something much more complex based on how these applications are connected internally within the enterprise.
25 advertisingID externalID String An identifier provided for the marketing/advertiser agencies to identify a mobile device / or a user using the device.
26 apnsToken externalID String APNS Push Token for iOS.
27 gcmToken externalID String GCM Push Token for Android.
28 fcmToken externalID String FCM Push Token for Android or FireFox web browser.
29 baiduToken externalID String Baidu Push Token for sending push notifications in China over the Baidu network.
30 facebookID externalID String User’s Facebook identifier.
31 googleplusID externalID String User’s GooglePlus identifier.
32 twitterID externalID String User’s Twitter identifier.
33 foursquareID externalID String User’s FourSquare identifier.
34 linkedinID externalID String User’s LinkedIn identifier.
35 pinterestID externalID String User’s Pinterest identifier.

Sample Document

    
         {
            'appId': 'b67803c7-909f-4da7-8d99-3b0ccc9d05fa',
            'ownerId': 'afdd77cf-2b04-4515-b5c7-d23df203249d',
            'appuid': '98638d79ac5d419da0cf48bb3cf842a6',
            'userId': '98638d79ac5d419da0cf48bb3cf842a6',
            'eventId': '2320be78-7212-4263-b17a-cd5d80cb941c',
            'sessionId': 'e1022e83-e231-4b1a-a6fe-2cf57518f790',
            'platform': 'Desktop web',
            'version': ’1703',
            'appVersion': '1.4.3',
            'eventName': 'BookReview',
            'startTime': 1527180007719,
            'endTime': 1527180007719,
            'tzoffset': -25200000,
            ‘eventType’: 'userEvent',
            'bookName': 'Name of the book being reviewed',
            'bookPrice': 25.49,
            'bookReview': ‘Text Review for the book by the buyer',
            'bookReviewStars': 3
    
          }
        

Error Logs and Streams

Upshot can read data from external data sources of clients’ AWS accounts such as SQS and Kinesis. While reading and processing data from the said sources, there might be errors in the read data like invalid data file format, invalid data type of individual fields in the data object, non-ascii characters etc.Upshot will write back to the error streams configured in External Input section of Upshot dashboard with details of what went wrong with the input data. Errors stream shall be configured as an option, while configuring Input Stream. Kinesis Input Stream should have Kinesis as Errors stream and SQS should have SQS as Errors stream.

    
            {
               "type": "InputReadError",
               "inputName": "my-kinesis/my-sqs stream",
               "inputType": "Kinesis/SQS",
               "timestamp": 1581342890000,
               "payload": {
                   "eventType": "userEvent",
                   "startTime": 1527180007719,
                   "eventName": "Register",
                   "sessionId": "f1022e83-e231-4b1a-a6fe-2cf57518f790",
                   "appVersion": "1.4.3",
                   "appId": "1bb312b5-5350-4e4c-ae1a-96c4d7e4f96e",
                   "ownerId": "afdd77cf-2b04-4515-b5c7-d23df203249d",
                   "endTime": 1527180007719,
                   "tzoffset": "134578624578",
                   "userId": "E3EF46B6-7E3A-48B4-97D5-9C9C949825E6",
                   "eventId": "9320be88-7212-4263-b17a-cd5d80cb976c",
                   "segment": "Business Builder",
                   "version": "1703",
                   "platform": "Desktop web",
                   "mobilePhone": "6169999999"
               },
               "errorMessage": "appuid key is missing"
               “errorCode”: 100/101/102,
               “missingKeys”:"appuid"
           }
    
          
Error Codes:

Streaming Output Definition

The above sample streaming output document says, it was created because a server side rule (with ruleId = 5b57464f51d23d1e008b4567) was triggered for an Upshot campaign (with campaignId = 5b5751e051d23d1b008b4568). Every rule triggered is mapped to a user on a platform where possible. Attributes "actionTime" and "time" represent the times at which the event that caused the rule to be triggered was actually performed by the user and at which time Upshot processed the "trigger" respectively.

a) The JSON string WILL NOT contain nested objects in it.

b) The JSON string WILL be encoded in "UTF-8".

c) Pre-defined keys that WILL be present in the payload are…

S.No Key Name Data Type Description
1 appId String The ID used by Upshot SDKs to identify the "app" from which the events are received. This is available through your Upshot account dashboard. This attribute IS REQUIRED / MANDATORY.
2 type String Possible values are:
1. "server_rule" - Identifies that this output JSON payload is sent to SQS/Kinesis because a server side rule has been triggered.
2. "journey" - Identifies that this output JSON payload is sent to SQS/Kinesis because the user reached a specific node in an Upshot Journey.
3 userId String The install for which this output belongs to.
4 ruleId String Present if this output payload is generated on a server side rule trigger. This is the ID of the rule created on Upshot dashboard.
5 journeyId String Present if this output payload is generated from an Upshot Journey. This is the ID of the Journey created on Upshot dashboard.
6 campaignId String Present if this output payload is generated in response to a campaign. All activities (including server rules) on Upshot are always linked to a campaign. But not all outputs need to be in response to a campaign.
7 actionTime Unix Timestamp Depends on the reason for this output. If this for a server side rule, this will be the "startTime" of the event that triggered the rule. If the rule was triggered because of an update to a user profile, this time represents the time at which the rule was evaluated, etc.
8 time String Time at which the rule was triggered by Upshot and inserted into the SQS/Kinesis output, etc. This timestamp represents time up to seconds and not in milliseconds.
9 appuid String The "user" for which this output belongs to. This corresponds to the "appuid" key as received in an SDK event / SQS / Kinesis / Bulk S3 input.
10 platform String If this output was triggered because of an event, the platform on which this event occurred. This COULD be "null" if the rule was not triggered because of an event (triggered because an aggregate was updated, user profile was updated, etc).

Sample Document

Streaming Output Definition

      {
    
        "campaignId":"5b5751e051d23d1b008b4568",
        "platform":null,
        "ruleId":"5b57464f51d23d1e008b4567",
        "actionTime":1532585671,
        "time":1532585671,
        "appId":"5ac3235707d0ff74008b456f",
        "userId":"104c6b8d46ad96d1ffa7ff8a35d9f95f8767777",
        "appuid":"104c6b8d46ad96d1ffa7ff8a35d9f95f8767777",
        "type":"server_rule"
    
     }
    
            

AWS S3

All data that you intended to push to Upshot must be uploaded to an S3 bucket on your own AWS account as Upshot currently does not open its S3 buckets for writing data / uploading files.

Once a batch of data to be uploaded is ready, the file must be uploaded as a UTF-8 encoded JSON file whose contents are a JSON array. After the file is ready on S3, you will have to make an API call to Upshot to initiate the file import. The API call to be made is exposed through Upshot’s eAPI whose documentation is available at https://www.upshot.ai/documentation/enterpriseaccess/

The API endpoint to invoke is https://eapi.goupshot.com/v1/external/input. A sample request payload is listed below…

Request Payload

                {
                   "auth": 
                       {
        
                          "appId": "8360baca-d361-4b5a-8139-87a4638394ec",
                          "accountId": "f3bf1d6f-5771-41f7-a6ff-640d3af4805e",
                          "apiKey": "IqZI1yEHJ3u035VpDAl9C1kg1N0vXXUEmSmnq0494y97D"
        
                        },
        
                        
                    "receiveAt":
                         {
                          
                          "cbType": "email",
                          "cbValue": "your-email-address@your-domain"
                      
                         },
        
                  
                    "data":
                        {
        
                          "url": "https://your-s3-file-path-access-opened-to-upshot/file-name.json",
                          "fileType":"json",
                          "dataType":"userProfile",
                          "docVersion" : 1
        
                        }
        
                 }
                

Once the file provided is processed and events/user profile imported you will be informed about its completion through the callback mechanism provided in the "receiveAt" key. You can choose to receive either an email confirmation or an API callback to your server. The above example uses email confirmation.

Value for "docVersion" in the above payload sample MUST ALWAYS be set to 1, no other values are supported. This key in the payload is optional for now. When newer versions of document types are accepted, this will be needed. It is advised to include the key always and set to 1.

The contents of the input file MUST have the exact same fields that a streaming input accepts, with the exception of the "eventType" field which defines the type of content in the file and is sent in the API request. All rules mentioned in the streaming input definition are applicable here too.

File Based Integration

Sample Documents

            [
                    {
                      "appVersion": "3.5.16",
                      "appId": "5ac3235707d0ff74008b456f",
                      "appuid": "5db49990ddeac1000ba0dadf",
                      "device_model": "iOS",
                      "endTime": 1576737454321,
                      "eventId": "FE79C1FE-B361-4A51-9564-0CF025D6D1B8",
                      "eventName": "RegisterPushNotifications",
                      "eventType": "userEvent",
                      "ownerId": "afdd77cf-2b04-4515-b5c7-d23df203249d",
                      "platform": "iOS",
                      "sessionId": "5C2527AF-D285-4CDA-BFA0-27F643F369C6",
                      "startTime": 1576737454321,
                      "tzoffset": "-18000000",
                      "userId": "FC1DF3A6-E8AD-4BD2-A51D-5D90FC301349",
                      "version": "13.3",
                      "registrationStatus": "True"
                    },
                           
                    {
                       "appVersion": "3.5.16",
                       "appId": "5ac3235707d0ff74008b456f",
                       "appuid": "5db49990ddeac1000ba0dadf",
                       "device_model": "iOS",
                       "endTime": 1576737453368,
                       "eventId": "9CFFE277-6533-4996-86C8-FE681220C2DC",
                       "eventName": "APIStatus",
                       "eventType": "userEvent",
                       "ownerId": "afdd77cf-2b04-4515-b5c7-d23df203249d",
                       "platform": "iOS",
                       "sessionId": "5C2527AF-D285-4CDA-BFA0-27F643F369C6",
                       "startTime": 1576737453368,
                       "tzoffset": "-18000000",
                       "userId": "FC1DF3A6-E8AD-4BD2-A51D-5D90FC301349",
                       "version": "13.3",
                       "apiName": "profile/user/mzbg1@yahoo.com",
                       "apiStatus": "Success"
                           
                     },
            
            
                     {
                        
                        "appVersion": "3.5.16",
                        "appId": "5ac3235707d0ff74008b456f",
                        "appuid": "5db49990ddeac1000ba0dadf",
                        "device_model": "iOS",
                        "endTime": 1576737453162,
                        "eventId": "7EA65E62-24BE-41ED-936D-839A61E6335E",
                        "eventName": "Purchase",
                        "eventType": "userEvent",
                        "ownerId": "afdd77cf-2b04-4515-b5c7-d23df203249d",
                        "platform": "iOS",
                        "sessionId": "5C2527AF-D285-4CDA-BFA0-27F643F369C6",
                        "startTime": 1576737453162,
                        "tzoffset": "-18000000",
                        "userId": "FC1DF3A6-E8AD-4BD2-A51D-5D90FC301349",
                        "version": "13.3",
                        "productName": "iPhone XR",
                        "productPrice": 899.99,
                        "productQty": 1,
                        "productDiscount": 159.00,
                        "productPurchaseDate": 1576737453162
            
                     }
            
            
             ]
                          
                          
             [
                    {
            
                        "appId": "8609fe12-13fc-453f-990c-6eaaab883ed5",
                        "ownerId": "afdd77cf-2b04-4515-b5c7-d23df203249d",
                        "userId": "E3EF46B6-7E3A-48B4-97D5-9C9C949825E6",
                        "appuid": "1122009483",
                        "key": "latestIP",
                        "attrType": "custom",
                        "type": "String",
                        "value": "202.153.43.131"
            
                     },
            
            
                     {
            
                        "appId": "8609fe12-13fc-453f-990c-6eaaab883ed5",
                        "ownerId": "afdd77cf-2b04-4515-b5c7-d23df203249d",
                        "userId": "E3EF46B6-7E3A-48B4-97D5-9C9C949825E6",
                        "appuid": "1122009483",
                        "key": "age",
                        "attrType": "upshot",
                        "type": "Numeric",
                        "value": 25
            
            
                    }
            
            
            ]