Skip to content

Commit 142e3ff

Browse files
wilwadewilwade
authored
Update serializations of numbers vs byte data (#114)
* Update serializations of numbers vs byte data * Dependabot updates * Add additional generic Data Type field for the announcement specs * Use UINT_64 for parquet types
1 parent d062138 commit 142e3ff

11 files changed

+6129
-11640
lines changed

‎package-lock.json

Lines changed: 6039 additions & 11564 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎pages/Announcements/Overview.md

Lines changed: 24 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -30,18 +30,36 @@ Most serializations use outside standards, but some require additional clarifica
3030

3131
### hexadecimal
3232

33+
Used to represent bytes.
34+
3335
- MUST use 0-9,a-f representation
3436
- MUST be lowercase
3537
- MUST be prefixed with a `0x`
36-
- MUST NOT have spaces
37-
- MUST NOT have any zero padding
38+
- MUST NOT have spaces or separators
39+
- MUST have two characters per byte in addition to the `0x` characters
3840

3941
| Bytes | Invalid | Valid |
4042
| --- | --- | --- |
41-
| 2 | `0x0123` | `0x123` |
42-
| 2 | `123h` | `0x123` |
43-
| 8 | `0x0000000000000abc` | `0xabc` |
44-
| 32 | `0x00003e34c4325f4461b9355027b314f3eb56d31af549f7da7bd9ef1ce951651e` | `0x3e34c4325f4461b9355027b314f3eb56d31af549f7da7bd9ef1ce951651e` |
43+
| 2 | `0x123` | `0x0123` |
44+
| 2 | `123h` | `0x0123` |
45+
| 2 | `0x0ABC` | `0x0abc` |
46+
| 8 | `0xabc` | `0x0000000000000abc` |
47+
| 32 | `0x3e34c4325f4461b9355027b314f3eb56d31af549f7da7bd9ef1ce951651e` | `0x00003e34c4325f4461b9355027b314f3eb56d31af549f7da7bd9ef1ce951651e` |
48+
49+
### decimal
50+
51+
Used to represent integers.
52+
Strings are used to avoid issues with different implementations of numbers.
53+
54+
- MUST use 0-9 representation
55+
- MUST NOT have spaces or separators
56+
- MUST be a string
57+
58+
| Invalid | Why | Valid |
59+
| --- | --- | --- |
60+
| `0x123` | Must be decimal | `"291"` |
61+
| 291 | Must be a string | `"291"` |
62+
| `291n` | `BigInt(291)` serialization appends an `n` | `"291"` |
4563

4664
## Duplicate Handling
4765

‎pages/Announcements/Signatures.md

Lines changed: 13 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -28,26 +28,24 @@ All parties interacting with Announcements should independently validate signatu
2828

2929
#### Example
3030

31-
```json
32-
{
33-
"announcementType": 1,
34-
"fromId": "0x12345",
35-
"contentHash": "0x67890",
36-
"url": "https://www.dsnp.org/",
37-
"createdAt": "2021-07-31T10:11:12"
38-
}
39-
```
31+
| Field | Value |
32+
| --- | --- |
33+
| announcementType | `1` |
34+
| contentHash | `0x9c22ff5f21f0b81b113e63f7db6da94fedef11b2119b4088b89664fb9a3cb658` |
35+
| createdAt | `1627726272000` |
36+
| fromId | `74565` |
37+
| url | `https://www.dsnp.org/` |
4038

4139
Expected serialization:
4240

4341
```
44-
\x19Ethereum Signed Message:\n64announcementType0x1contentHash0x67890createdAt0x17afc0bd600fromId0x12345urlhttps://www.dsnp.org/
42+
\x19Ethereum Signed Message:\n64announcementType1contentHash0x9c22ff5f21f0b81b113e63f7db6da94fedef11b2119b4088b89664fb9a3cb658createdAt1627726272000fromId74565urlhttps://www.dsnp.org/
4543
```
4644

4745
Serialization in hexadecimal:
4846

4947
```
50-
0x19457468657265756d205369676e6564204d6573736167653a0a3936616e6e6f756e63656d656e7454797065307831636f6e74656e7448617368307836373839306372656174656441743078313761666330626436303066726f6d49643078313233343575726c68747470733a2f2f7777772e64736e702e6f72672f
48+
0x19457468657265756d205369676e6564204d6573736167653a0a313531616e6e6f756e63656d656e745479706531636f6e74656e74486173683078396332326666356632316630623831623131336536336637646236646139346665646566313162323131396234303838623839363634666239613363623635386372656174656441743136323737323632373230303066726f6d4964373435363575726c68747470733a2f2f7777772e64736e702e6f72672f
5149
```
5250

5351
### Hashing
@@ -59,7 +57,7 @@ Serialization in hexadecimal:
5957
For the previous example, the resulting hexadecimal hash MUST match:
6058

6159
```
62-
0xe998171b9eedfe13a181aa158c7b2dbb739af9e5ca062cc5822e668be1314478
60+
0xabaae4d8fda61c1b9cf481ef784158ebae5cea36f5cb7d1242987553a6dc6aa8
6361
```
6462

6563

@@ -79,15 +77,15 @@ Private Key: `0xd9d3b5afb7765ffd9f047fd0d1d9b47d4d538b6a56f1cf29dc160ab9c6d30aa3
7977
```
8078
{
8179
v: '0x1c',
82-
r: '0xa34e5f6ba5f133cc1c8dfed613ad913f07dc5dff38c92278f9253c07ff43bd1d',
83-
s: '0x3f86a862db3db7223a2d2b530dd15cbdc450fb2394917f1f413f4a102822deca',
80+
r: '0x2e05b0f769b0344a58a06718f90f5d605878b6d5e9e14e1f235de24b399cfe42',
81+
s: '0x7135a0b704862a8bc2847c4bb9f78bb43f707d427f0ba19bb43f66d5666934c9',
8482
}
8583
```
8684

8785
The compressed form of the above being this (`r + s + v`):
8886

8987
```
90-
0xa34e5f6ba5f133cc1c8dfed613ad913f07dc5dff38c92278f9253c07ff43bd1d3f86a862db3db7223a2d2b530dd15cbdc450fb2394917f1f413f4a102822deca1c
88+
0x2e05b0f769b0344a58a06718f90f5d605878b6d5e9e14e1f235de24b399cfe427135a0b704862a8bc2847c4bb9f78bb43f707d427f0ba19bb43f66d5666934c91c
9189
```
9290

9391
## Validating a Signature

‎pages/Announcements/Types/Broadcast.md

Lines changed: 8 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -10,14 +10,14 @@ A Broadcast Announcement is a way to send a public message to everyone.
1010

1111
## Fields
1212

13-
| Field | Description | Serialization | Parquet Type | Bloom Filter |
14-
| ----- | ----------- | ------------- | ------------ | ------------ |
15-
| announcementType | Announcement Type Enum (`2`) | [hexadecimal](/Announcements/Overview#hexadecimal) | `INT32` | no |
16-
| contentHash | keccak-256 hash of content stored at URL | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | YES
17-
| createdAt | milliseconds since Unix epoch | [hexadecimal](/Announcements/Overview#hexadecimal) | `INT64` | no
18-
| fromId | id of the user creating the announcement | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | YES
19-
| url | content URL | [UTF-8](https://datatracker.ietf.org/doc/html/rfc3629) | `BYTE_ARRAY` | no
20-
| signature | creator signature | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | no
13+
| Field | Description | Data Type | Serialization | Parquet Type | Bloom Filter |
14+
| ----- | ----------- | --------- | ------------- | ------------ | ------------ |
15+
| announcementType | Announcement Type Enum (`2`) | enum | [decimal](/Announcements/Overview#decimal) | `INT32` | no |
16+
| contentHash | keccak-256 hash of content stored at URL | 32 bytes | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | YES
17+
| createdAt | milliseconds since Unix epoch | 64 bit unsigned integer | [decimal](/Announcements/Overview#decimal) | `UINT_64` | no
18+
| fromId | id of the user creating the announcement | 64 bit unsigned integer | [decimal](/Announcements/Overview#decimal) | `UINT_64` | YES
19+
| url | content URL | UTF-8 | [UTF-8](https://datatracker.ietf.org/doc/html/rfc3629) | `UTF8` | no
20+
| signature | creator signature | 65 bytes | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | no
2121

2222
## Field Requirements
2323

@@ -27,7 +27,6 @@ A Broadcast Announcement is a way to send a public message to everyone.
2727

2828
### contentHash
2929

30-
- MUST be 32 bytes in length
3130
- MUST be the [keccak-256 hash](https://keccak.team/files/Keccak-submission-3.pdf) of the bytes of the reference at the url
3231

3332
### createdAt

‎pages/Announcements/Types/GraphChange.md

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -10,14 +10,14 @@ A Graph Change Announcement is for publishing relationship state changes for a u
1010

1111
## Fields
1212

13-
| Field | Description | Serialization | Parquet Type | Bloom Filter |
14-
| ----- | ----------- | ------------- | ------------ | ------------ |
15-
| announcementType | Announcement Type Enum (`1`) | [hexadecimal](/Announcements/Overview#hexadecimal) | `INT32` | no |
16-
| changeType | Type of relationship change | [hexadecimal](/Announcements/Overview#hexadecimal) | `INT32` | no
17-
| createdAt | milliseconds since Unix epoch | [hexadecimal](/Announcements/Overview#hexadecimal) | `INT64` | no
18-
| fromId | id of the user creating the relationship | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | YES
19-
| objectId | id of the target of the relationship | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | YES
20-
| signature | creator signature | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | no
13+
| Field | Description | Data Type | Serialization | Parquet Type | Bloom Filter |
14+
| ----- | ----------- | --------- | ------------- | ------------ | ------------ |
15+
| announcementType | Announcement Type Enum (`1`) | enum | [decimal](/Announcements/Overview#decimal) | `INT32` | no |
16+
| changeType | Type of relationship change | enum | [decimal](/Announcements/Overview#decimal) | `INT32` | no
17+
| createdAt | milliseconds since Unix epoch | 64 bit unsigned integer | [decimal](/Announcements/Overview#decimal) | `UINT_64` | no
18+
| fromId | id of the user creating the relationship | 64 bit unsigned integer | [decimal](/Announcements/Overview#decimal) | `UINT_64` | YES
19+
| objectId | id of the target of the relationship | 64 bit unsigned integer | [decimal](/Announcements/Overview#decimal) | `UINT_64` | YES
20+
| signature | creator signature | 65 bytes | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | no
2121

2222
## Field Requirements
2323

‎pages/Announcements/Types/Profile.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -13,11 +13,11 @@ The reference content *MUST be of profile type*.
1313

1414
| Field | Description | Serialization | Parquet Type | Bloom Filter |
1515
| ----- | ----------- | ------------- | ------------ | ------------ |
16-
| announcementType | Announcement Type Enum (`5`) | [hexadecimal](/Announcements/Overview#hexadecimal) | `INT32` | no |
16+
| announcementType | Announcement Type Enum (`5`) | [decimal](/Announcements/Overview#decimal) | `INT32` | no |
1717
| contentHash | keccak-256 hash of content stored at URL | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | YES
18-
| createdAt | milliseconds since Unix epoch | [hexadecimal](/Announcements/Overview#hexadecimal) | `INT64` | no
19-
| fromId | id of the user creating the announcement | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | YES
20-
| url | Profile content URL | [UTF-8](https://datatracker.ietf.org/doc/html/rfc3629) | `BYTE_ARRAY` | no
18+
| createdAt | milliseconds since Unix epoch | [decimal](/Announcements/Overview#decimal) | `UINT_64` | no
19+
| fromId | id of the user creating the announcement | [decimal](/Announcements/Overview#decimal) | `UINT_64` | YES
20+
| url | Profile content URL | [UTF-8](https://datatracker.ietf.org/doc/html/rfc3629) | `UTF8` | no
2121
| signature | creator signature | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | no
2222

2323
## Field Requirements

‎pages/Announcements/Types/Reaction.md

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -10,14 +10,14 @@ A Reaction Announcement is for publishing emoji reactions to anything with a [DS
1010

1111
## Fields
1212

13-
| Field | Description | Serialization | Parquet Type | Bloom Filter |
14-
| ----- | ----------- | ------------- | ------------ | ------------ |
15-
| announcementType | Announcement Type Enum (`4`) | [hexadecimal](/Announcements/Overview#hexadecimal) | `INT32` | no |
16-
| createdAt | milliseconds since Unix epoch | [hexadecimal](/Announcements/Overview#hexadecimal) | `INT64` | no
17-
| emoji | the encoded reaction | [UTF-8](https://datatracker.ietf.org/doc/html/rfc3629) | `BYTE_ARRAY` | YES
18-
| fromId | id of the user creating the relationship | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | YES
19-
| inReplyTo | Target [DSNP Announcement URI](/Identifiers#dsnp-announcement-uri) | [UTF-8](https://datatracker.ietf.org/doc/html/rfc3629) | `BYTE_ARRAY` | YES
20-
| signature | creator signature | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | no
13+
| Field | Description | Data Type | Serialization | Parquet Type | Bloom Filter |
14+
| ----- | ----------- | --------- | ------------- | ------------ | ------------ |
15+
| announcementType | Announcement Type Enum (`4`) | enum | [decimal](/Announcements/Overview#decimal) | `INT32` | no |
16+
| createdAt | milliseconds since Unix epoch | 64 bit unsigned integer | [decimal](/Announcements/Overview#decimal) | `UINT_64` | no
17+
| emoji | the encoded reaction | UTF-8 | [UTF-8](https://datatracker.ietf.org/doc/html/rfc3629) | `UTF8` | YES
18+
| fromId | id of the user creating the relationship | 64 bit unsigned integer | [decimal](/Announcements/Overview#decimal) | `UINT_64` | YES
19+
| inReplyTo | Target [DSNP Announcement URI](/Identifiers#dsnp-announcement-uri) | UTF-8 | [UTF-8](https://datatracker.ietf.org/doc/html/rfc3629) | `UTF8` | YES
20+
| signature | creator signature | 65 bytes | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | no
2121

2222
## Field Requirements
2323

‎pages/Announcements/Types/Reply.md

Lines changed: 9 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -11,15 +11,15 @@ but includes an `inReplyTo` field for noting it as a reply to a given [DSNP Anno
1111

1212
## Fields
1313

14-
| Field | Description | Serialization | Parquet Type | Bloom Filter |
15-
| ----- | ----------- | ------------- | ------------ | ------------ |
16-
| announcementType | Announcement Type Enum (`3`) | [hexadecimal](/Announcements/Overview#hexadecimal) | `INT32` | no |
17-
| contentHash | keccak-256 hash of content stored at URL | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | YES
18-
| createdAt | milliseconds since Unix epoch | [hexadecimal](/Announcements/Overview#hexadecimal) | `INT64` | no
19-
| fromId | id of the user creating the announcement | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | YES
20-
| inReplyTo | Target [DSNP Announcement URI](/Identifiers#dsnp-announcement-uri) | [UTF-8](https://datatracker.ietf.org/doc/html/rfc3629) | `BYTE_ARRAY` | YES
21-
| url | content URL | [UTF-8](https://datatracker.ietf.org/doc/html/rfc3629) | `BYTE_ARRAY` | no
22-
| signature | creator signature | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | no
14+
| Field | Description | Data Type | Serialization | Parquet Type | Bloom Filter |
15+
| ----- | ----------- | --------- | ------------- | ------------ | ------------ |
16+
| announcementType | Announcement Type Enum (`3`) | enum | [decimal](/Announcements/Overview#decimal) | `INT32` | no |
17+
| contentHash | keccak-256 hash of content stored at URL | 32 bytes | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | YES
18+
| createdAt | milliseconds since Unix epoch | 64 bit unsigned integer | [decimal](/Announcements/Overview#decimal) | `UINT_64` | no
19+
| fromId | id of the user creating the announcement | 64 bit unsigned integer | [decimal](/Announcements/Overview#decimal) | `UINT_64` | YES
20+
| inReplyTo | Target [DSNP Announcement URI](/Identifiers#dsnp-announcement-uri) | UTF-8 | [UTF-8](https://datatracker.ietf.org/doc/html/rfc3629) | `UTF8` | YES
21+
| url | content URL | UTF-8 | [UTF-8](https://datatracker.ietf.org/doc/html/rfc3629) | `UTF8` | no
22+
| signature | creator signature | 65 bytes | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | no
2323

2424
## Field Requirements
2525

@@ -29,7 +29,6 @@ but includes an `inReplyTo` field for noting it as a reply to a given [DSNP Anno
2929

3030
### contentHash
3131

32-
- MUST be 32 bytes in length
3332
- MUST be the [keccak-256 hash](https://keccak.team/files/Keccak-submission-3.pdf) of the bytes of the reference at the url
3433

3534
### createdAt

‎pages/Announcements/Types/Tombstone.md

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -17,14 +17,14 @@ It is NOT possible to revert a tombstone.
1717

1818
## Fields
1919

20-
| Field | Description | Serialization | Parquet Type | Bloom Filter |
21-
| ----- | ----------- | ------------- | ------------ | ------------ |
22-
| announcementType | Announcement Type Enum (`0`) | [hexadecimal](/Announcements/Overview#hexadecimal) | `INT32` | no |
23-
| createdAt | milliseconds since Unix epoch | [hexadecimal](/Announcements/Overview#hexadecimal) | `INT64` | no
24-
| fromId | id of the user creating the announcement and tombstoned announcement | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | YES
25-
| targetAnnouncementType | target tombstoned announcement type | [hexadecimal](/Announcements/Overview#hexadecimal) | `INT32` | no |
26-
| targetSignature | target announcement signature to tombstone | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | YES
27-
| signature | creator signature | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | no
20+
| Field | Description | Data Type | Serialization | Parquet Type | Bloom Filter |
21+
| ----- | ----------- | --------- | ------------- | ------------ | ------------ |
22+
| announcementType | Announcement Type Enum (`0`) | enum | [decimal](/Announcements/Overview#decimal) | `INT32` | no |
23+
| createdAt | milliseconds since Unix epoch | 64 bit unsigned integer | [decimal](/Announcements/Overview#decimal) | `UINT_64` | no
24+
| fromId | id of the user creating the announcement and tombstoned announcement | 64 bit unsigned integer | [decimal](/Announcements/Overview#hecimal) | `UINT_64` | YES
25+
| targetAnnouncementType | target tombstoned announcement type | enum | [decimal](/Announcements/Overview#decimal) | `INT32` | no |
26+
| targetSignature | target announcement signature to tombstone | 65 bytes | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | YES
27+
| signature | creator signature | 65 bytes | [hexadecimal](/Announcements/Overview#hexadecimal) | `BYTE_ARRAY` | no
2828

2929
## Field Requirements
3030

‎pages/Identifiers.md

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -7,13 +7,13 @@ route: /Identifiers
77

88
## DSNP User Id
99

10-
- Is 8 bytes in size
10+
- 64 bit Unsigned Integer
1111
- MUST be registered in the [Identity Registry](/Identity/Registry)
12-
- MUST be serialized as [hexadecimal](/Announcements/Overview#hexadecimal)
12+
- MUST be serialized as [decimal](/Announcements/Overview#decimal)
1313

1414
## DSNP Content Hash
1515

16-
- Is 32 bytes in size
16+
- MUST be 32 bytes in size
1717
- MUST be a [keccak-256 hash](https://keccak.team/files/Keccak-submission-3.pdf) of the bytes of the content
1818
- MUST be serialized as [hexadecimal](/Announcements/Overview#hexadecimal)
1919

@@ -28,13 +28,13 @@ It is used to identify a user via a URI.
2828

2929
### Example
3030
```
31-
dsnp://0x1234567890abcdef
31+
dsnp://1311768467294899700
3232
```
3333

3434
| part | value |
3535
| ---- | ----- |
3636
| Scheme | `dsnp://` |
37-
| User Id | `0x1234567890abcdef` |
37+
| User Id | `1311768467294899700` |
3838

3939
## DSNP Announcement URI
4040

@@ -45,11 +45,11 @@ Any [Announcement Types](/Announcements/Overview#announcement-types) with a `fro
4545

4646
### Example
4747
```
48-
dsnp://0x1234567890/0x1234567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef
48+
dsnp://78187493520/0x1234567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef
4949
```
5050

5151
| part | value |
5252
| ---- | ----- |
5353
| Scheme | `dsnp://` |
54-
| User Id | `0x1234567890` |
54+
| User Id | `78187493520` |
5555
| Content Hash | `0x1234567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef` |

‎pages/index.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ route: /
55

66
# DSNP Specification
77

8-
## Version 0.9.2
8+
## Version 0.10.0
99

1010
Welcome to the Decentralized Social Networking Protocol (DSNP) specification!
1111
Here you can find a detailed documentation regarding the current state of the protocol, previous iterations and proposals for future extensions.

0 commit comments

Comments
 (0)