Data Exchange Service

The Data Exchange (DEX) service of Dataswift logs all activities in the PDA ecosystem; respond to PDA requests to create Data Debits; install Data Plugs when instructed; records the contract of data transactions; verify exchanges; quickly, accurately and securely sends and receives data between parties.

  • Enables the sharing of data between PDAs and applications

  • Data Transaction Logging

  • Reports statistics across the ecosystem

  • PDA Access brokering for applications and developers

  • Equivalent to swift in banking

DEX is the trust anchor on the Dataswift Platform, providing authoritative information about approved PDAs and applications as well as the data access rules agreed by the contracts between them; these range from Data Plugs for data acquisition to tools for transforming personal data, to complete custom personal data container management.

DEX provides APIs for PDAs to call on for verifying trustworthiness of entities interacting in the distributed network as well as for trusted parties to orchestrate their interactions with PDAs, such as Data Offer requests and unified data acquisition tooling.

DEX is not able to view any of the content or the data being transferred.

As part of DEX service, Dataswift provides API wrapper libraries (https://github.com/Hub-of-all-Things/dex-client-scala-play) and standalone microservices for scalable data acquisition across the distributed ecosystem.

For Data Acquirers, the DEX reports the ecosystem statistics as well as integrating datasets and data services for third parties into the system. It provides high-level, aggregated information about all available data, which can be used independently.

The core aspects are statistics logging and reporting, data plug management, Data Offer brokering and setup, and new PDA onboarding.

The Data Exchange (DEX) service consists of:

  • Data Exchange processes

  • Data transaction logging

  • Statistics across the ecosystem

  • PDA Access brokering for applications and developers

Dex logs all activities on the platform and responds to PDA requests to create Data Debits. It also

  • logs data contracts and transactions

  • verifies exchanges

  • securely sends and receives data between parties

It reports the ecosystem statistics as well as integrating datasets and data services for third parties into the system.

Authentication

DEX uses the same authentication methods as the PDA. When your DEX account is created, you are given a username and a password, which you use to retrieve an access_token:

{
"request": {
"method": "GET",
"header": [
{
"key": "username",
"value": "DEX_USERNAME",
"name": "username",
"description": ""
},
{
"key": "password",
"value": "DEX_PASSWORD",
"name": "password",
"description": ""
}
],
"body": {},
"url": "https://dex.hubofallthings.com/api/users/access_token",
"description": ""
}
}

You use this token for all subsequent API requests to DEX by including it as a X-Auth-Token header.

DEX Statistics

The PDAs in the ecosystem report aggregated metadata about their data operations to DEX. Specifically, they report how many specific values have been sent to the PDA by Data Plugs, as well as how many have been retrieved from the PDA with Data Debits. Furthermore, since Data Debits are the mechanisms for third-parties to request data from the individual, they also report operations on Data Debits:

  • creation

  • removal

  • enabling

  • disabling

This allows DEX to monitor commercial data exchanges for security, quality assurance and transparency.

Collecting available statistics

Current DEX statistics lookup for all available data points is a single, public (no authentication required) API call:

{
"request": {
"url": "https://dex.hubofallthings.com/stats",
"method": "GET",
"header": [],
"body": {},
"description": "DEX Data Statistics Lookup"
}
}

Which responds with aggregated statistics:

{
"response": [
{
"name": "Statistics",
"status": "OK",
"code": 200,
"header": [
{
"key": "Content-Type",
"value": "application/json",
"name": "Content-Type",
"description": ""
}
],
"body": [
{
"title": "Data Plugs",
"value": 7,
"updated": 1496407712243
},
{
"title": "Total Offer Claims",
"value": 70,
"updated": 1496407712248
},
{
"title": "Data Out",
"value": 24317796,
"updated": 1496407712246
},
{
"title": "Data In",
"value": 3118189,
"updated": 1496407712239
},
{
"title": "Top Datapoints",
"value": 17476332,
"children": {
"longitude in locationv1 of rumpel": {
"title": "longitude in locationv1 of rumpel",
"value": 607523,
"updated": 1496407713864
},
"...": "..."
},
"updated": 1496407713864
},
{
"title": "HATs",
"value": 387,
"updated": 1496407712236
}
]
}
]
}

Retrieving data points available in PDAs

The PDA store is very flexible in what data it accepts via its APIs for storage to limit the amount of setup required before a developer can start storing data in it, however that also means it may not be easy for others to find out what data is available for exchange. Therefore, DEX processes all incoming metadata to aggregate available data properties:

{
"request": {
"method": "GET",
"header": [],
"body": {},
"url": "https://dex.hubofallthings.com/stats/available-data",
"description": ""
}
}

Describing available data

While it is not possible to derive a description just from the fact that a specific datapoint is being exchanged, DEX provides means to add those descriptions via a (currently authorized for management accounts only) API:

{
"request": {
"url": "https://dex.hubofallthings.com/stats/available-data/descriptions",
"description": "",
"method": "POST",
"header": [
{
"key": "Content-Type",
"value": "application/json",
"description": ""
},
{
"key": "X-Auth-Token",
"value": "ACCESS_TOKEN",
"description": ""
}
],
"body": {
"mode": "raw",
"raw": [
{
"namespace": "dex",
"description": "Data stored by the Data Exchange in individual HATs",
"endpoints": [
{
"endpoint": "databuyer",
"description": "DataBuyer service related information",
"fields": [
{
"name": "merchants[]",
"description": "The list of merchants the user is following",
"count": 9
},
{
"name": "date",
"description": "The date this record was updated on",
"count": 17
}
]
}
]
}
]
}
}
}

Private endpoints

The other endpoints are used by automatic communication between systems, e.g. the PDAs reporting of statistical events.

Data Plug Management

Data Plugs are the microservices that bring data from other digital resources into the PDA. However, to ensure users' privacy in the distributed ecosystem, DEX requires Data Plugs to be registered before they can be plugged into a PDA.

DEX records some basic data about every plug:

Parameter

Description

Required/Generated

uuid

UUID of the Data Plug

Generated

providerId

date when the field was created

Generated

created

date when the Data Plug was registered

Generated

name

Alpha-numerical name of the Plug

Required

description

Textual description of what the Plug Does

Required

url

Address of the plug for users to visit to connect

Required

illustrationUrl

Logo of the plug, typically shown in the UI for connecting plugs

Required

passwordHash

Hash of the Data Plug's password, set up with a HAT when connecting

Required

approved

Whether or not the plug has been approved within the DEX

Required

To register a new Data Plug, you will need to contact us, providing the required details. Your plug will then be reviewed to check if it matches the platform's governance rules, e.g. only sending the personal data to users' PDAs and nowhere else. The review will also include a check on how the plug works, etc.

Listing Data Plugs

All Data Plugs are public and listed by DEX:

{
"request": {
"method": "GET",
"header": [
{
"key": "Content-Type",
"value": "application/json",
"description": ""
},
{
"key": "X-Auth-Token",
"value": "ACCESS_TOKEN",
"description": ""
}
],
"body": {},
"url": "https://dex.hubofallthings.com/api/dataplugs",
"description": ""
}
}

Providing a list of all Data Plugs and their details.

Connecting a Data Plug

Since PDAs are distributed and are likely to be created before you register your own Data Plug, PDAs do not necessarily recognise the plug and your authentication would fail initially. When your Data Plug is registered, you, as the owner of the plug, will receive a set of credentials to use with the DEX and your account is associated with the plug as the "owner" account. Before you can connect to a PDA, you need to ask DEX to connect:

{
"request": {
"method": "GET",
"header": [
{
"key": "X-Auth-Token",
"value": "ACCESS_TOKEN",
"description": ""
}
],
"body": {},
"url": "https://dex.hubofallthings.com/api/dataplugs/DATAPLUG_ID/connect",
"description": ""
}
}

Once DEX responds with 200 OK status and a message saying that the Data Plug has been connected, you can connect directly to the PDA with username matching the name of your plug and the password corresponding to the passwordHash.

Data Offer brokering and setup

An Offer is one side of a data exchange agreement between the individual PDA owner and any other entity requesting data. The other side of the offer is the Data Debit. For the DEX, an Offer simply describes what data is requested and for how long. Applications do not have rights to directly create a Data Debit on a PDA, but instead ask DEX to do so on their behalf when the user claims the Offer. It is important to note, however, that even DEX is not able to grant itself access to the data, but only request for data from a PDA by registering a Data Debit: the Data Debit is not active until the owner of the PDA explicitly enables the Data Debit through confirmation of the contract.

Offer details

Parameter

Description

offerId

Alphanumeric Offer ID — must be unique

title

Title of the Offer

description

Textual description of the Offer

starts

The (UNIX timestamp) Date and Time of when the Offer becomes available for claiming

expires

The Date and Time of when the Offer expires

collectionDuration

Duration of data collection (in milliseconds) from the moment a PDA claims the Offer

dataConditions

(Optional) conditions that need to be satisfied, i.e. contain data in the Data Bundle format

requiredData

The technical definition of the data required in the Data Bundle format

requiredMinUser

The minimum number of users required to release access to all claiming HATs, confirming exchange

requiredMaxUser

The maximum number of users allowed to claim the offer, e.g. to control the maximum campaign cost

status

Valid values include: "approved" and "rejected"

The format of the Data Bundle is covered in the guide on the power of PDA data bundling.

Register an Offer

To be able to register an Offer directly with DEX, your account will need to have MarketPlace role granted.

Offer is created by sending all the details as described above to DEX, for example:

{
"request": {
"method": "POST",
"header": [
{
"key": "X-Auth-Token",
"value": "ACCESS_TOKEN",
"description": ""
},
{
"key": "Content-Type",
"value": "application/json",
"description": ""
}
],
"body": {
"mode": "raw",
"raw": {
"offerId": "789d1dff-ce8d-4c6e-a201-e89ee9d7f5cf",
"title": "Location fetching offer",
"description": "Location fetch test",
"starts": 1507503600000,
"expires": 1510099200000,
"collectionDuration": 86400000,
"requiredData": {
"name": "789d1dff-ce8d-4c6e-a201-e89ee9d7f5cf",
"bundle": {
"iphone/locations": {
"endpoints": [
{
"endpoint": "iphone/locations",
"mapping": {
"accuracy": "accuracy",
"latitude": "latitude",
"longitude": "longitude",
"timestamp": "timestamp",
"lastUpdated": "lastUpdated"
},
"filters": []
}
]
}
}
},
"requiredMinUser": 1,
"requiredMaxUser": 10,
"status": "approved"
}
},
"url": "https://dex.hubofallthings.com/api/v2/offer",
"description": ""
}
}

Only after the Offer has been registered (DEX responds with status 201 CREATED), it can be claimed by users.

Claim an Offer

Claiming an offer through DEX, or rather registering a claim, can be done by either the PDA, or by the owner of the Offer (i.e. the account that has registered it):

  • Claim by PDA is done by sending a request with a fresh and appropriate Access Token (one that has been issued for

    the use with DEX) to GET /api/v2/offer/:id/claim, where :id is the Offer ID that is being claimed.

  • Claim by the Offer owner is done by sending a request with an Access Token issued to the owner account,

    to the endpoint PUT /api/v2/offer/:id/registerClaim?hat=address, where :id is the Offer ID and address is the PDA address.

An example of registering an offer claim is below:

{
"request": {
"method": "PUT",
"header": [
{
"key": "X-Auth-Token",
"value": "ACCESS_TOKEN",
"description": ""
}
],
"body": {},
"url": "https://dex.hubofallthings.com/api/v2/offer/OFFER_ID/registerClaim?hat=postman.hubat.net",
"description": "Uses Offer owner's credentials (the marketplace, from DEX point of view) to register an offer claim, triggering a data debit being created"
}
}

To which, if the claim is processed successfully, DEX responds with the details of the claim:

{
"response": [
{
"name": "Register claim for HAT",
"status": "OK",
"code": 200,
"header": [
{
"key": "Content-Type",
"value": "application/json",
"name": "Content-Type",
"description": "The mime type of this content"
}
],
"body": {
"offerId": "{{page.offerId}}",
"user": {
"address": "{{page.hat}}",
"publicKey": "-----BEGIN PUBLIC KEY-----nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAmbpD4OytlMcWRuGHgfy8nq9I9GT46Sh3FwuZQkicCMqMWaV9ukwo6mW+e4UciNh5acXf2qEnQdbRHCxNVe90GnjTzVCPcFvigE/Tn9icNzISludwA4/uiAbaFwJg9dvXVphQuxZdq5dEIDAvPcqwUknJBCX+CLBP1a0CWiB/ACbaVwYm2bZApZe52BLiw7ejvM6UvQoOjOYiRiVGJKdgUgmnWIruC+bMcbhbNpf/11M0+YCi/d51OSwup/nyEweoa6deTrMdFyMosnlcknEaWx9tnNPU3Agub9SNZVKkXTYgRXzoQu8k/BC331uKII1pi6atqLjQGkY4rY6fXJ3Db3NYIn9QIDAQABn-----END PUBLIC KEY-----n"
},
"relationship": "claim",
"confirmed": false,
"dataDebitId": "{{page.offerId}}",
"dateCreated": 1507651701435
}
}
]
}

List offer claims

The final step in the Offer Claim process after the claim and the user accepting the Data Debit on their side, is getting credentials issued by DEX for collecting data from confirmed PDAs. The call needs to be authenticated with the Offer owner's Access Token as well:

{
"request": {
"method": "GET",
"header": [
{
"key": "X-Auth-Token",
"value": "ACCESS_TOKEN",
"description": ""
}
],
"body": {},
"url": "https://dex.hubofallthings.com/api/v2/offer/OFFER_ID/claims",
"description": ""
}
}

The DEX response includes access credentials and the list of all confirmed claims:

{
"response": [
{
"name": "Register claim for HAT",
"status": "OK",
"code": 200,
"header": [
{
"key": "Content-Type",
"value": "application/json",
"name": "Content-Type",
"description": "The mime type of this content"
}
],
"body": {
"offerId": "{{page.offerId}}",
"user": {
"address": "{{page.hat}}",
"publicKey": "-----BEGIN PUBLIC KEY-----nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAmbpD4OytlMcWRuGHgfy8nq9I9GT46Sh3FwuZQkicCMqMWaV9ukwo6mW+e4UciNh5acXf2qEnQdbRHCxNVe90GnjTzVCPcFvigE/Tn9icNzISludwA4/uiAbaFwJg9dvXVphQuxZdq5dEIDAvPcqwUknJBCX+CLBP1a0CWiB/ACbaVwYm2bZApZe52BLiw7ejvM6UvQoOjOYiRiVGJKdgUgmnWIruC+bMcbhbNpf/11M0+YCi/d51OSwup/nyEweoa6deTrMdFyMosnlcknEaWx9tnNPU3Agub9SNZVKkXTYgRXzoQu8k/BC331uKII1pi6atqLjQGkY4rY6fXJ3Db3NYIn9QIDAQABn-----END PUBLIC KEY-----n"
},
"relationship": "claim",
"confirmed": false,
"dataDebitId": "{{page.offerId}}",
"dateCreated": 1507651701435
}
}
]
}

New PDA onboarding and account management

DEX acts as the trust anchor on the Platform, In addition to verifying Data Plugs and Offers, it also verifies that only certified PDAs participate in the exchange. What this means is that even though the core code is open-source and anyone could run their own copy, such PDAs may not comply with the security requirements or the governance rules of the platform e.g. if they modify data exchange mechanisms, produce fraudulent data etc. It is therefore the responsibility of PDA Platform Providers who provision PDAs as well as the HAT Community Foundation to certify the PDAs that are can participate on the platform.

DEX recognises HAT clusters by the domain they are addressed under (e.g. hubat.net for the testing purposes) and assumes that once the domain manager has been certified, ensuring that individual PDAs are compliant to platform rules is the responsibility of the Platform Provider managing the domain.

Registering individual PDAs

Individual PDAs still need to be registered with DEX before they can connect Data Plugs, claim Offers or report any statistics. Registering a PDA is done via an unauthenticated call:

{
"request": {
"method": "POST",
"header": [
{
"key": "Content-Type",
"value": "application/json",
"description": ""
}
],
"body": {
"mode": "raw",
"raw": {
"hatAddress": "postman.hubat.net"
}
},
"url": "https://dex.hubofallthings.com/api/users/registerHat",
"description": ""
}
}

At this point, DEX verifies that the PDA is up and running, retrieves its Public Key and stores it. The Public Key is used to verify validity of requests coming in from the PDA by checking if the attached Access Token matches the stored public key.

Result of this security model with DEX is that PDA addresses cannot easily be reused: if an address gets reused, the Private/Public key must be regenerated by a certified provisioning system, and requests from such PDAs will no longer be authorized. The decision to restrict PDA Address reuse has been made due to the fact that the address identifies the individual when logging in to other services, retrieving and exchanging data, etc. and the inherent risks do not justify the potential convenience gains.

Updating certified apps on a PDA

Similarly to Data Plugs, new certified applications may be registered within the ecosystem after a PDA gets created, resulting in the PDA not recognising the app. Certain information, including access details may also need to change throughout the lifetime of the app. Therefore, DEX has the ability to update the details on selected PDAs.

Properties of an app are:

Parameter

Description

title

Title (Name) of the app used for logging in

namespace

Namespace where the app can write its data

description

Textual description of the app

logoUrl

URL of the logo for the app

url

URL of the application to send the user to

authUrl

Relative path within the URL for authentication, empty string if the main URL used

browser

Boolean flag of whether or not the app is a full browser, i.e. allows user to manage their HAT fully

category

Category of the app, e.g. "app" or "dataPlug"

Registering of an application (or updating them) is currently only available for the administrative accounts, however is done via a single call:

{
"request": {
"method": "POST",
"header": [
{
"key": "X-Auth-Token",
"value": "ACCESS_TOKEN",
"description": ""
},
{
"key": "Content-Type",
"value": "application/json",
"description": ""
}
],
"body": {
"mode": "raw",
"raw": {
"title": "TestApp",
"namespace": "testing",
"description": "Test Application",
"logoUrl": "/assets/images/testapp.png",
"url": "https://example.com",
"authUrl": "/signin/hat",
"browser": true,
"category": "app",
"setup": true,
"loginAvailable": true
}
},
"url": "https://dex.hubofallthings.com/api/users/update-application?cluster=hubat.net&hatFilter=postman.hubat.net",
"description": ""
}
}