Raw Data I/O

One of the core design principles of the PDA has been its ability to store any structure data. In v2 of the API this ability was overhauled to simplify data input and minimise setup required before data can be saved.

Therefore, plain Data APIs consume any JSON:

  • individual JSON values are saved as-is in a data record

  • a data record is a natural grouping of data, expected to relate to the way data is grouped at the source

    (e.g. all data together with metadata of a calendar event)

  • records posted as an array get saved into separate records

  • records can be further grouped to form complex data relationships (e.g. calendar events linked to a separate set of

    event attendees, each stored as a record), either at the time of saving the data or afterwards by referring to individual records by their IDs

  • deleting of data records is supported by accepting a list of previously saved record IDs

The v2 API loosely replaces the notion of data source and table from the previous API version with namespace and endpoint, where:

  • each application that is authorized to write any data into a PDA, can do so only within one or more namespaces

    designated for it, such as Rumpel only writing to the rumpel namespace. Applications are not authorized to write into each other's namespaces

  • endpoints are defined by the application developer as required. Data is grouped under a combination of a namespace

    and an endpoint where the two together identify a set of data accessible via a URL path /api/v2.6/data/namespace/endpoint.

    An example would be /api/v2.6/data/rumpel/locations for all locations saved into a PDA by the Rumpel app

  • data can thus be stored and retrieved using permitted credentials for the individual endpoints

You will need to acquire an access_token before calling any data APIs. Please refer to the guide on HAT login for details on authentication.

Data Input

Data Input can happen in one of two ways:

  1. Saving one or more homogeneous data records into a single endpoint

  2. Saving multiple data records that can be potentially heterogeneous, related, or both

Saving homogeneous data records

A data entry request for a single endpoint (/hat/locations), with multiple records inserted in one go. The data provided in the example is merely an illustration — the PDA will accept any well-formed JSON data. Note the request is authenticated with an ACCESS_TOKEN obtained previously and the token has been issued for the namespace of hat specifically.

{
"request": {
"url": "https://{{page.hat}}/api/v2.6/data/hat/locations",
"method": "POST",
"header": [
{
"key": "Content-Type",
"value": "application/json",
"description": ""
},
{
"key": "X-Auth-Token",
"value": "ACCESS_TOKEN",
"description": ""
}
],
"body": {
"mode": "raw",
"raw": [
{
"id": 85998,
"name": "record 1",
"lastUpdated": "2017-04-23T16:11:17.000Z",
"data": {
"locations": {
"latitude": "51.671358277138",
"longitude": "0.101014673709963",
"accuracy": "10.0",
"timestamp": "2017-04-23T16:11:17+0000"
}
}
},
{
"id": 85997,
"name": "record 3",
"lastUpdated": "2017-04-23T16:18:04.000Z",
"data": {
"locations": {
"latitude": "51.6658257133844",
"longitude": "0.080477950927866",
"accuracy": "1414.0",
"timestamp": "2017-04-23T16:18:04+0000"
}
}
},
{
"id": 85996,
"name": "record 2",
"lastUpdated": "2017-04-23T16:12:58.000Z",
"data": {
"locations": {
"latitude": "51.674001392439",
"longitude": "0.100905202634514",
"accuracy": "1414.0",
"timestamp": "2017-04-23T16:12:58+0000"
}
}
}
]
},
"description": "Storing a ist of data records (locations in the example) in one go"
}
}

If the data gets saved, the response includes the full data, wrapped in the Record data structure:

  • endpoint field that includes the specific endpoint the record was assigned to

  • recordId — UUID of the data record

  • data — the data JSON structure exactly as submitted

{
"response": [
{
"name": "Data saved",
"status": "Created",
"code": 201,
"header": [
{
"key": "Content-Type",
"value": "application/json",
"name": "Content-Type",
"description": "The mime type of this content"
}
],
"body": [
{
"endpoint": "locations",
"recordId": "f8c011d1-98f0-4e5f-89a6-64b949724eba",
"data": {
"id": 185998,
"data": {
"locations": {
"accuracy": "10.0",
"latitude": "51.671358277138",
"longitude": "0.101014673709963",
"timestamp": "2017-04-23T16:11:17+0000"
}
},
"name": "record 1",
"lastUpdated": "2017-04-23T16:11:17.000Z"
},
"links": []
},
{
"endpoint": "locations",
"recordId": "0a79bef4-9a8e-4d19-937d-9a759fc2981b",
"data": {
"id": 185997,
"data": {
"locations": {
"accuracy": "1414.0",
"latitude": "51.6658257133844",
"longitude": "0.080477950927866",
"timestamp": "2017-04-23T16:18:04+0000"
}
},
"name": "record 3",
"lastUpdated": "2017-04-23T16:18:04.000Z"
},
"links": []
},
{
"endpoint": "locations",
"recordId": "b6ebcdc3-235e-4e8c-834c-eec08beb55cd",
"data": {
"id": 185996,
"data": {
"locations": {
"accuracy": "1414.0",
"latitude": "51.674001392439",
"longitude": "0.100905202634514",
"timestamp": "2017-04-23T16:12:58+0000"
}
},
"name": "record 2",
"lastUpdated": "2017-04-23T16:12:58.000Z"
},
"links": []
}
]
}
]
}

If there are any problems with your JSON (e.g. it is malformed), you will receive a Bad Request response code with the details of the error in the body of the response. If, for example, such data (as decided by its hash) already exists in the PDA, none of the data in the request will be saved and you will receive a message indicating so:

{
"response": [
{
"name": "Error: duplicate data received",
"status": "Bad Request",
"code": 400,
"header": [
{
"key": "Content-Type",
"value": "application/json",
"name": "Content-Type",
"description": ""
}
],
"body": {
"message": "Duplicate Data",
"cause": "Could not insert data — Duplicate data"
}
}
]
}

Note: all data entry operations are transactional, i.e. insertion of all data provided in the request succeeds or the whole request fails and no data gets inserted.

To illustrate the PDA's acceptance of any valid JSON data, here is another example, saving very different profile data in the same namespace, but a different endpoint:

{
"request": {
"url": "https://postman.hubat.net/api/v2.6/data/rumpel/profile",
"method": "POST",
"header": [
{
"key": "Content-Type",
"value": "application/json",
"description": ""
},
{
"key": "X-Auth-Token",
"value": "ACCESS_TOKEN",
"description": ""
}
],
"body": {
"mode": "raw",
"raw": {
"profile": {
"website": {
"link": "https://example.com",
"private": "false"
},
"nick": {
"private": "true",
"name": ""
},
"primary_email": {
"value": "testuser@example.com",
"private": "false"
},
"private": "false",
"youtube": {
"link": "",
"private": "true"
},
"address_global": {
"city": "London",
"county": "",
"country": "UK",
"private": "true"
},
"age": {
"group": "",
"private": "true"
},
"personal": {
"first_name": "",
"private": "false",
"preferred_name": "Test",
"last_name": "User",
"middle_name": "",
"title": ""
},
"blog": {
"link": "",
"private": "false"
},
"facebook": {
"link": "",
"private": "false"
},
"address_details": {
"no": "",
"street": "",
"private": "false",
"postcode": ""
},
"emergency_contact": {
"first_name": "",
"private": "true",
"relationship": "",
"last_name": "",
"mobile": ""
},
"alternative_email": {
"private": "true",
"value": ""
},
"fb_profile_photo": {
"private": "false"
},
"twitter": {
"link": "",
"private": "false"
},
"about": {
"body": "A short bio about me shown on my PDA",
"private": "false",
"title": "Me the Test User"
},
"mobile": {
"no": "",
"private": "true"
},
"gender": {
"type": "",
"private": "true"
}
}
}
},
"description": "Using Rumpel profile as an exmaple, can save the whole object as a single API call"
}
}

Heterogeneous data records can be inserted simultaneously using the batch-data API endpoint. One example where such data insertion is useful is PDA notes, where a text-based note is saved with an attached "nudge" — a time-based reminder connected to the note. It is important to keep notes and nudges logically separate, because both can be used independently, e.g. a nudge can be attached to certain locations, photos, posts...

{
"request": {
"url": "https://postman.hubat.net/api/v2.6/data-batch",
"method": "POST",
"header": [
{
"key": "Content-Type",
"value": "application/json",
"description": ""
},
{
"key": "X-Auth-Token",
"value": "ACCESS_TOKEN",
"description": ""
}
],
"body": {
"mode": "raw",
"raw": [
{
"endpoint": "rumpel/notable",
"data": {
"id": 84995,
"lastUpdated": "2017-04-23T14:21:51+01:00",
"data": {
"notablesv1": {
"authorv1": {
"phata": "postman.hubat.net"
},
"created_time": "2017-04-10T14:19:59:+01:00",
"shared": "true",
"shared_on": "twitter",
"message": "Showcasing the new HAT APIs",
"public_until": "2017-05-11T14:21:54+01:00",
"updated_time": "2017-04-23T14:21:58+01:00",
"kind": "note"
}
}
},
"links": [
{
"endpoint": "rumpel/nudge",
"data": {
"type": "time",
"nudge": "Share APIs with the world",
"time": "2017-04-30T14:22:52+01:00"
}
}
]
}
]
},
"description": "Sending multiple records to the HAT simultaneously, potentially for separate 'endpoints' as well as linked, all in one call"
}
}

The response will include the main data record with the note as well as a separate data record with the nudge, linked to the main one.

Data records do not necessarily need to be linked up when being inserted. This can be done later by referring to the records using their IDs:

{
"request": {
"url": "https://postman.hubat.net/api/v2.6/data-link?records=6dcff611-45cb-49a9-82ca-318b9e5a3c17",
"method": "POST",
"header": [
{
"key": "Content-Type",
"value": "application/json",
"description": ""
},
{
"key": "X-Auth-Token",
"value": "ACCESS_TOKEN",
"description": ""
}
],
"body": {
"mode": "raw",
"raw": "{}"
},
"description": "Links up data records for a notion of related data, e.g. a note associated with a specific nudge"
}
}

Data Retrieval

In this guide we only look at retrieving simple data that is only available to the owner of the PDA and hence usable only by certified dashboard applications. For other kinds of data access, jump to the section on consented data exchange where you will find further pointers.

In the simple case, data from the PDA is retrieved using a GET request to the /api/v2.6/data/namespace/endpoint.

The endpoint takes the following parameters:

Parameter

Default

Meaning

take

1000

Limit to how many records the requester will take

skip

0

How many records to skip — used together with take to implement data paging

orderBy

The field in the JSON structure according to which data should be ordered. If, for example, you wish to order data by time, we recommend to include UNIX timestamp in the data as a number

recordId

ID of the record when looking up data for one specific item

{
"request": {
"url": "https://postman.hubat.net/api/v2.6/data/hat/locations?take=3",
"method": "GET",
"header": [
{
"key": "Content-Type",
"value": "application/json",
"description": ""
},
{
"key": "X-Auth-Token",
"value": "ACCESS_TOKEN",
"description": ""
}
],
"body": {
"mode": "raw",
"raw": ""
},
"description": "getting the 3 most recent data records"
}
}

Which returns top 3 results sorted by default ordering:

{
"response": [
{
"id": "0ff8df0b-fd8f-45c1-a5d6-2f64d44804e0",
"name": "Sample data output",
"status": "OK",
"code": 200,
"header": [
{
"key": "Content-Type",
"value": "application/json",
"name": "Content-Type",
"description": "The mime type of this content"
}
],
"body": [
{
"endpoint": "rumpel/locations",
"recordId": "e965e022-6613-476a-a0cd-1f587a41b148",
"data": {
"id": 85998,
"data": {
"locations": {
"accuracy": "10.0",
"latitude": "51.671358277138",
"longitude": "0.101014673709963",
"timestamp": "2017-04-23T16:11:17+0000"
}
},
"name": "record 1",
"lastUpdated": "2017-04-23T16:11:17.000Z"
}
},
{
"endpoint": "rumpel/locations",
"recordId": "fcf1a26b-e49f-4457-915b-156e14140f38",
"data": {
"id": 85996,
"data": {
"locations": {
"accuracy": "1414.0",
"latitude": "51.674001392439",
"longitude": "0.100905202634514",
"timestamp": "2017-04-23T16:12:58+0000"
}
},
"name": "record 2",
"lastUpdated": "2017-04-23T16:12:58.000Z"
}
},
{
"endpoint": "rumpel/locations",
"recordId": "8f7afa92-39e2-48ab-8028-f5aebaa9918e",
"data": {
"id": 85997,
"data": {
"locations": {
"accuracy": "1414.0",
"latitude": "51.6658257133844",
"longitude": "0.080477950927866",
"timestamp": "2017-04-23T16:18:04+0000"
}
},
"name": "record 3",
"lastUpdated": "2017-04-23T16:18:04.000Z"
}
}
]
}
]
}

Data Modification

Updating

Data can be updated by sending a PUT request to the /api/v2.6/data endpoint with the complete Data Record structure in the request body:

{
"request": {
"url": "https://postman.hubat.net/api/v2.6/data",
"method": "PUT",
"header": [
{
"key": "Content-Type",
"value": "application/json",
"description": ""
},
{
"key": "X-Auth-Token",
"value": "ACCESS_TOKEN",
"description": ""
}
],
"body": {
"mode": "raw",
"raw": [
{
"endpoint": "hat/locations",
"recordId": "e965e022-6613-476a-a0cd-1f587a41b148",
"data": {
"id": 85998,
"data": {
"locations": {
"accuracy": "1000.0",
"latitude": "51.671358277138",
"longitude": "0.101014673709963",
"timestamp": "2017-04-23T16:11:17+0000"
}
},
"name": "record 1",
"lastUpdated": "2017-04-23T16:11:17.000Z"
}
}
]
},
"description": "Deleting a record happens via a request providing the record's ID"
}
}

The above request is expected to effectively update only the accuracy field of the record already stored, however in practice the provided data replaces the whole record already stored in the PDA.

Deleting

Data is deleted by sending a DELETE request to the /api/v2.6/data endpoint with IDs of the records to be deleted in query parameters:

{
"request": {
"url": "https://postman.hubat.net/api/v2.6/data?records=3474369e-2317-4ea2-9bc8-198700a1f9cb&records=6dcff611-45cb-49a9-82ca-318b9e5a3c17",
"method": "DELETE",
"header": [
{
"key": "Content-Type",
"value": "application/json",
"description": ""
},
{
"key": "X-Auth-Token",
"value": "ACCESS_TOKEN",
"description": ""
}
],
"body": {
"mode": "raw",
"raw": ""
},
"description": "Deleting a record happens via a request providing the record's ID"
}
}

If you are deleting multiple records in one go, just add a records parameter for each record ID. The response will contain an error if you are not authorized to delete any of the provided records or if any of the records do not exist — no records will be deleted in this case.

Consented Data Exchange

This guide has covered the basics of PDA Data I/O, namely inserting new data and retrieving it as-is. The points we have not covered include: