Skip to content

Commit c74d920

Browse files
florindragosflorindragos
committed
add transform docs
1 parent a89d118 commit c74d920

File tree

3 files changed

+355
-0
lines changed

3 files changed

+355
-0
lines changed

‎README.md

+2
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,8 @@ The default transformation template can be exported using `ds-load <plugin-name>
121121

122122
A custom transformation file can be provided when running the plugin in `exec` or `transform` mode via the `--template-file` parameter.
123123

124+
More information on the transformation template language can be found in the [tranform template docs](./docs/templates.md).
125+
124126
## Logs
125127

126128
Logs are printed to `stdout`. You can increase detail using the verbosity flag (e.g. `-vvv`).

‎docs/ds-load.png

13.2 KB
Loading

‎docs/templates.md

+353
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,353 @@
1+
# ds-load transformation templates
2+
3+
The ds-load tranformation templates use the go template syntax to output two arrays one for [objects](https://buf.build/aserto-dev/directory/docs/main:aserto.directory.common.v3#aserto.directory.common.v3.Object) and one for [relations](https://buf.build/aserto-dev/directory/docs/main:aserto.directory.common.v3#aserto.directory.common.v3.Relation).
4+
5+
As the data gets read from the source IDP and inserted into the directory, there are 3 stages the data is processed:
6+
7+
<img src="./ds-load.png">
8+
9+
1. the fetcher (`ds-load plugin fetch`) reads data and outputs a valid json representing the source data (e.g. a user)
10+
2. the transformer (`ds-load plugin transform`) applies the transformation template over the data received from the fetcher and outputs the source objects representation as two arrays (objects and relations)
11+
3. the publisher imports the output from the transform stage into the directory
12+
13+
## Auth0 use case
14+
This would be a process of tranforming a auth0 user object to it's directory representation of objects/relations.
15+
16+
### Transformer input
17+
The transformer input is actually the output of the fetch command. `ds-load transform` receives an array of json objects as input and applies the given template on each element of that array.
18+
19+
example of a fetcher output:
20+
```json
21+
[
22+
{
23+
"created_at": "2024-01-12T10:45:11.138Z",
24+
"email": "[email protected]",
25+
"email_verified": true,
26+
"identities": [
27+
{
28+
"connection": "Username-Password-Authentication",
29+
"isSocial": false,
30+
"provider": "auth0",
31+
"user_id": "65a118371e5f3e33399d0fa8"
32+
}
33+
],
34+
"name": "Rick Sanchez",
35+
"nickname": "rick",
36+
"picture": "https://www.topaz.sh/assets/templates/citadel/img/Rick%20Sanchez.jpg",
37+
"roles": [
38+
{
39+
"description": "admin",
40+
"id": "rol_r8zZczsSuq25J3tx",
41+
"name": "admin"
42+
},
43+
{
44+
"description": "evil genius",
45+
"id": "rol_RrcNi465Gkd69eLB",
46+
"name": "evil_genius"
47+
}
48+
],
49+
"updated_at": "2024-01-12T11:07:06.994Z",
50+
"user_id": "auth0|65a118371e5f3e33399d0fa8",
51+
"user_metadata": {
52+
"superuser": true
53+
}
54+
}
55+
]
56+
```
57+
58+
### Template
59+
60+
The transform template uses the [go template syntax](https://pkg.go.dev/text/template) to output a valid json that contains objects and relations.
61+
62+
For example, if we want an object type user with the key being the users email, we would add the following item in the objects array:
63+
```go
64+
{
65+
"id": "{{ $.email }}",
66+
"type": "user"
67+
}
68+
```
69+
Here, `{{ $.email }}` is a key at the root level of the input object (see [above](#transformer-input))
70+
***
71+
For an email identity object, we would do somethink like this:
72+
```go
73+
{
74+
"id": "{{ $.email }}",
75+
"type": "identity"
76+
}
77+
```
78+
***
79+
If we want to link these objects together using a relation, we need to add a new entry in the relations array:
80+
```go
81+
{
82+
"object_type": "identity",
83+
"object_id": "{{ $.email }}",
84+
"relation": "identifier",
85+
"subject_type": "user",
86+
"subject_id": "{{ $.email }}"
87+
88+
}
89+
```
90+
***
91+
If we want to iterate over a list of items (eg. roles) and add an object for each one:
92+
```go
93+
{{ range $i, $element := $.roles }}
94+
{{ if $i }},{{ end }}
95+
{
96+
"id": "{{$element.name}}",
97+
"type": "group",
98+
"properties":{}
99+
}
100+
{{ end }}
101+
```
102+
> note: we need to be careful with the commas. `{{ if $i }},{{ end }}` is handy since it will add a comma before all elements, except the first one.
103+
***
104+
Building on this, a basic transform template for this user would look like this:
105+
106+
```go
107+
{
108+
"objects": [
109+
{
110+
"id": "{{ $.email }}",
111+
"type": "user",
112+
"displayName": "{{ $.nickname }}",
113+
"created_at":"{{ $.created_at }}",
114+
"properties":{
115+
"email": "{{ $.email }}",
116+
"enabled": true,
117+
"picture": "{{ $.picture }}"
118+
{{ range $key, $value := $.user_metadata }}
119+
,"{{ $key }}": {{ marshal $value }}
120+
{{ end }}
121+
}
122+
},
123+
{
124+
"id": "{{ $.user_id }}",
125+
"type": "identity",
126+
"properties": {
127+
"verified": true
128+
}
129+
},
130+
{
131+
"id": "{{ $.email }}",
132+
"type": "identity",
133+
"properties": {
134+
"verified": {{ .email_verified }}
135+
}
136+
}
137+
{{ if $.roles }}, {{ end }}
138+
{{ range $i, $element := $.roles }}
139+
{{ if $i }},{{ end }}
140+
{
141+
"id": "{{$element.name}}",
142+
"type": "group",
143+
"properties":{}
144+
}
145+
{{ end }}
146+
],
147+
"relations":[
148+
{
149+
"object_type": "identity",
150+
"object_id": "{{ $.user_id }}",
151+
"relation": "identifier",
152+
"subject_type": "user",
153+
"subject_id": "{{ $.email }}"
154+
},
155+
{
156+
"object_type": "identity",
157+
"object_id": "{{ $.email }}",
158+
"relation": "identifier",
159+
"subject_type": "user",
160+
"subject_id": "{{ $.email }}"
161+
}
162+
{{ if $.roles }}, {{ end }}
163+
{{ range $i, $element := $.roles }}
164+
{{ if $i }},{{ end }}
165+
{
166+
"object_type": "group",
167+
"object_id": "{{$element.name}}",
168+
"relation": "member",
169+
"subject_type": "user",
170+
"subject_id": "{{$.email}}"
171+
}
172+
{{ end }}
173+
]
174+
}
175+
```
176+
177+
### Transformer ouput
178+
179+
The input for the transformer can vary from plugin to plugin, but the output needs to be in the same format in order to be consumed by ds-load:
180+
181+
```json
182+
[
183+
{
184+
"objects": [],
185+
"relations": []
186+
}
187+
]
188+
```
189+
***
190+
The above transformation template, using the given input from [Transformer input](#transformer-input) would render the following json result.
191+
192+
```json
193+
[
194+
{
195+
"objects": [
196+
{
197+
"type": "user",
198+
"id": "[email protected]",
199+
"displayName": "rick",
200+
"properties": {
201+
"email": "[email protected]",
202+
"enabled": true,
203+
"picture": "https://www.topaz.sh/assets/templates/citadel/img/Rick%20Sanchez.jpg",
204+
"superuser": true
205+
},
206+
"createdAt": "2024-01-12T10:45:11.138Z"
207+
},
208+
{
209+
"type": "identity",
210+
"id": "auth0|65a118371e5f3e33399d0fa8",
211+
"properties": {
212+
"verified": true
213+
}
214+
},
215+
{
216+
"type": "identity",
217+
"id": "[email protected]",
218+
"properties": {
219+
"verified": true
220+
}
221+
},
222+
{
223+
"type": "group",
224+
"id": "admin",
225+
"properties": {}
226+
},
227+
{
228+
"type": "group",
229+
"id": "evil_genius",
230+
"properties": {}
231+
}
232+
],
233+
"relations": [
234+
{
235+
"objectType": "identity",
236+
"objectId": "auth0|65a118371e5f3e33399d0fa8",
237+
"relation": "identifier",
238+
"subjectType": "user",
239+
"subjectId": "[email protected]"
240+
},
241+
{
242+
"objectType": "identity",
243+
"objectId": "[email protected]",
244+
"relation": "identifier",
245+
"subjectType": "user",
246+
"subjectId": "[email protected]"
247+
},
248+
{
249+
"objectType": "group",
250+
"objectId": "admin",
251+
"relation": "member",
252+
"subjectType": "user",
253+
"subjectId": "[email protected]"
254+
},
255+
{
256+
"objectType": "group",
257+
"objectId": "evil_genius",
258+
"relation": "member",
259+
"subjectType": "user",
260+
"subjectId": "[email protected]"
261+
}
262+
]
263+
}
264+
]
265+
```
266+
## Fetchers that output multiple object types
267+
268+
A fetch operation can, in some cases, output multiple object types.
269+
For example, the [google plugin](/plugins/google/pkg/app/assets/transform_template.tmpl) outputs an array oj users mixed with groups. In this case, the transform template needs to know how to handle both types.
270+
```go
271+
{
272+
"objects": [
273+
{{ if contains $.kind "admin#directory#user" }}
274+
// render user objects
275+
{{ end }}
276+
277+
{{ if contains $.kind "admin#directory#group" }}
278+
// render group objects
279+
{{ end }}
280+
],
281+
"relations":[
282+
{{ if contains $.kind "admin#directory#user" }}
283+
//render user relations
284+
{{ end }}
285+
286+
{{ if contains $.kind "admin#directory#group" }}
287+
// render group relations
288+
{{ end }}
289+
]
290+
}
291+
```
292+
This way, the transformer will output objects and relations for a user when receiving a kind containing `admin#directory#user` and objects and relations for a group when receiving a kind containing `admin#directory#group`
293+
## Helper functions
294+
295+
Since go template does not support some operations, we provide a list of [helper functions](../sdk/transform/functions.go)
296+
***
297+
### last
298+
`last i a`
299+
return true if index `i` is the last element in `a` where `i` is an int value and `a` can be a map or an array
300+
301+
example:
302+
```
303+
{{ range $i, $element := $.roles }}
304+
"last": "{{ last $i $.roles }}"
305+
{{ end }}
306+
```
307+
***
308+
### contains
309+
alias for `strings.Contains(str, substr string) bool`
310+
311+
example:
312+
```
313+
...
314+
{{ if contains $.kind "admin" }}
315+
...
316+
{{ end }}
317+
...
318+
```
319+
***
320+
### marshal
321+
does json.marshal on the received value and returns the json result. useful for escaping strings that contain special characters of saving an entire json object as a value
322+
323+
example:
324+
```
325+
"properties": {{ marshal $.Attributes }}
326+
```
327+
```
328+
{{ range $key, $value := $.Attributes }}
329+
{{ if $i }},{{ end }}
330+
"{{ $key }}": {{ marshal $value }}
331+
{{ end }}
332+
```
333+
***
334+
### fromEnv
335+
`fromEnv key ENV_VAR` returns a key/value pair with the value of the given ENV_VAR
336+
337+
```
338+
{{ fromEnv "userId" "USER_ID" }}
339+
```
340+
***
341+
### phoneIso3166
342+
returns the ISO3166 representation of a phone number
343+
344+
```
345+
"object_id": "{{ phoneIso3166 $.profile.mobilePhone }}
346+
```
347+
***
348+
### add
349+
returns the result of the addition of 2 numbers
350+
351+
```
352+
{{$index = add $index 1}}
353+
```

0 commit comments

Comments
 (0)