Transactional outbox and event audit records (#18)

* Update README 📚

* Event entity

* Create event when we create a Todo

* Refactor nuke function

* Create separate field for todo_id

* Create event when we delete a Todo

* Use xid rather than Dgraph uid in event table

* Use field constants

* Events store xid not Dgraph uid

* Fix event type and Dgraph type

Author: kevinmichaelchen

README.md

@@ -1,30 +1,54 @@
 ![go-dgraph-starter](https://raw.githubusercontent.com/graph-gophers/graphql-go/master/docs/img/logo.png)
 
+## Table of Contents
+1. [Introduction](#introduction)
+    1. [In Progress](#in-progress)
+    1. [Tools](#tools)
+        1. [Protocol Buffers](#protocol-buffers)
+        1. [gRPC](#grpc)
+        1. [GraphQL](#graphql)
+        1. [Dgraph](#dgraph)
+        1. [Sqlboiler](#sqlboiler)
+        1. [Meilisearch](#meilisearch)
+    1. [Patterns](#patterns)
+        1. [Cursor pagination](#cursor-pagination)
+        1. [Garden](#garden)
+        1. [Change Data Capture](#change-data-capture)
+
+TOC generated with
+```
+docker run -v $PWD:/app -w /app --rm -it pbzweihander/markdown-toc README.md --min-depth 1 
+```
+
 ## Introduction
 
 [![forthebadge](https://forthebadge.com/images/badges/60-percent-of-the-time-works-every-time.svg)](https://forthebadge.com)
 
 This is a Todo List app built with [Dgraph](https://dgraph.io/)-backed [gRPC](https://grpc.io/) and [GraphQL](https://graphql.org/) APIs, combined with a [NextJS](https://nextjs.org/) + [Chakra-UI](https://chakra-ui.com/) powered front-end.
 
 ### In Progress
+
 This project is still very much in progress.
 
 https://github.com/kevinmichaelchen/go-dgraph-starter/projects/1
 
 ### Tools
+
 #### Protocol Buffers
-Protocol buffers are a great choice for a language-neutral representation 
+
+Protocol buffers are a great choice for a language-neutral representation
 of your data models.
 
 They are binary, lean, and fast to serialize when compared with JSON.
 
-Their extensibility comes from the fact that you can 
+Their extensibility comes from the fact that you can
 add or remove fields in a way while maintaining backwards compatibilty.
 
 Code generation makes it easy to generate your models in any language.
 
 #### gRPC
-gRPC is a lean transport system (typically paired with HTTP/2) meant to 
+
+gRPC is a lean transport system (typically paired with HTTP/2) meant to
 reduce latency and payload size.
 
 HTTP/2 allows for long-lived connections (fewer handshakes).
@@ -34,11 +58,13 @@ Unlike REST, gRPC uses an HTTP POST method for all calls, and doesn't specify th
 gRPC is also binary, so it's not as easy to debug than something human-readable, like GraphQL. But it is excellent for internal calls between microservices.
 
 #### GraphQL
+
 GraphQL is an ideal query language for APIs, especially when consumed by web clients. It's typically human-readable (JSON). Clients can specify exactly which fields they want. GraphiQL is an amazing tool for gaining insight into what an API offers.
 
 It also can make you think about your data as a graph.
 
 #### Dgraph
+
 Dgraph is a popular open-source, fast, distributed graph database written in Golang.
 You get high throughput and low latency for deep joins and complex traversals.
 It offers a query language that is a superset of GraphQL.
@@ -50,25 +76,31 @@ Performance: SQL performance suffers the more joins you ask it to do.
 Flexibility / Complexity: Like NoSQL, the schema can be modified easily with time. Adding new relationships is as simple as adding a predicate. No need to create join tables. Ultimately, the graph model is more simpler / more intuitive.
 
 #### Sqlboiler
-It's not currently used in this project, but it's worth mentioning [Sqlboiler](https://github.com/volatiletech/sqlboiler) 
-since I believe it is by far the best SQL ORM for Golang due its "data-first" 
-approach: it auto-generates Go code based on your existing schema and as a 
+
+It's not currently used in this project, but it's worth mentioning [Sqlboiler](https://github.com/volatiletech/sqlboiler)
+since I believe it is by far the best SQL ORM for Golang due its "data-first"
+approach: it auto-generates Go code based on your existing schema and as a
 result you get extreme type safety.
 
 For services that aren't expected to have a whole lot of data relationships, SQL is still an excellent choice, and sqlboiler is an ideal ORM for keeping your persistence layer code strongly typed.
 
 #### Meilisearch
+
 We use [MeiliSearch](https://www.meilisearch.com/) as our "open source, blazingly fast and hyper relevant search-engine."
 
 Per their site, MeiliSearch is effective and accessible:
-> Efficient search engines are often only accessible to companies with the financial means and resources necessary to develop a search solution adapted to their needs. The majority of other companies that do not have the means or do not realize that the lack of relevance of a search greatly impacts the pleasure of navigation on their application, end up with poor solutions that are more frustrating than effective, for both the developer and the user. 
+
+> Efficient search engines are often only accessible to companies with the financial means and resources necessary to develop a search solution adapted to their needs. The majority of other companies that do not have the means or do not realize that the lack of relevance of a search greatly impacts the pleasure of navigation on their application, end up with poor solutions that are more frustrating than effective, for both the developer and the user.
 
 ### Patterns
+
 #### Cursor pagination
-* https://uxdesign.cc/why-facebook-says-cursor-pagination-is-the-greatest-d6b98d86b6c0
-* https://relay.dev/graphql/connections.htm
+
+- https://uxdesign.cc/why-facebook-says-cursor-pagination-is-the-greatest-d6b98d86b6c0
+- https://relay.dev/graphql/connections.htm
 
 #### Garden
+
 The problem: developers will not bother with running CI tests and integration tests locally; instead they'll push to their GitHub PR and let the CI system take care of it.
 
 Not only are they deprived of a production-like system on their local machines, but the feedback loop is too slow: having CircleCI run all the end-to-end tests takes too long.
@@ -78,4 +110,5 @@ Not only are they deprived of a production-like system on their local machines,
 For shops with hundreds or thousands of microservices, I can see the argument that it's excessive (or even impossible) to run the whole system on your local machine. That said, maybe there's a term for a subset of services that are related, and maybe it's worth running those on Garden.
 
 #### Change Data Capture
-In a distributed system, when events occur in one service, they need to eventually be broadcast to other services. Too often, I've seen an event get emitted inside a database transaction, regardless of whether that transaction succeeds or fails. The [transactional outbox](https://microservices.io/patterns/data/transactional-outbox.html) pattern offers a way to keep data across services in sync.
+
+In a distributed system, when events occur in one service, they need to eventually be broadcast to other services. Too often, I've seen an event get emitted inside a database transaction, regardless of whether that transaction succeeds or fails. The [transactional outbox](https://microservices.io/patterns/data/transactional-outbox.html) pattern offers a way to keep data across services in sync.

api/internal/app/app.go

@@ -38,7 +38,7 @@ func (a App) Run() {
 
 	// Drop all data and schema
 	log.Info().Msg("Dropping all Dgraph data...")
-	if err := db.Nuke(context.Background(), dgraphClient); err != nil {
+	if err := db.NukeDataAndSchema(context.Background(), dgraphClient); err != nil {
 		log.Fatal().Err(err).Msg("failed to nuke database")
 	}
 

api/internal/db/nuke.go

@@ -7,13 +7,13 @@ import (
 	"github.com/dgraph-io/dgo/v200/protos/api"
 )
 
-func Nuke(ctx context.Context, dgraphClient *dgo.Dgraph) error {
+func NukeDataAndSchema(ctx context.Context, dgraphClient *dgo.Dgraph) error {
 	return dgraphClient.Alter(ctx, &api.Operation{
 		DropAll: true,
 	})
 }
 
-func NukeData(ctx context.Context, dgraphClient *dgo.Dgraph) error {
+func NukeDataButNotSchema(ctx context.Context, dgraphClient *dgo.Dgraph) error {
 	return dgraphClient.Alter(ctx, &api.Operation{
 		DropOp: api.Operation_DATA,
 	})

api/internal/db/schema.go

@@ -28,6 +28,23 @@ const schema = `
     name
     created_at
   }
+
+  event_type: string @index(exact) .
+  event_at: datetime @index(hour) .
+  todo_id: string @index(exact) .
+  is_published_to_search_index: bool .
+  creator_id: string @index(exact) .
+
+  type TodoEvent {
+    event_type
+    event_at
+    is_published_to_search_index
+    todo_id
+    created_at
+    title
+    is_done
+    creator_id
+  }
 `
 
 func BuildSchema(ctx context.Context, dgraphClient *dgo.Dgraph) error {

api/internal/db/transaction.go

@@ -10,6 +10,32 @@ import (
 	"github.com/MyOrg/go-dgraph-starter/internal/configuration"
 )
 
+const (
+	eventTypeCreate = "create"
+	eventTypeUpdate = "update"
+	eventTypeDelete = "delete"
+
+	dgraphTypeUser      = "User"
+	dgraphTypeTodo      = "Todo"
+	dgraphTypeTodoEvent = "TodoEvent"
+
+	fieldDgraphType = "dgraph.type"
+
+	fieldID        = "id"
+	fieldCreatedAt = "created_at"
+	fieldCreator   = "creator"
+
+	fieldEventAt                     = "event_at"
+	fieldEventType                   = "event_type"
+	fieldTodoID                      = "todo_id"
+	fieldEventPublishedToSearchIndex = "is_published_to_search_index"
+
+	fieldName      = "name"
+	fieldTitle     = "title"
+	fieldDone      = "is_done"
+	fieldCreatorID = "creator_id"
+)
+
 type Transaction interface {
 	TodoTransaction
 }

api/internal/db/tx_todo_create.go

@@ -54,10 +54,10 @@ func (tx *todoTransactionImpl) CreateTodo(ctx context.Context, item *todoV1.Todo
 			// Insert user into database
 			if res, err := tx.tx.Mutate(ctx, &api.Mutation{
 				Set: []*api.NQuad{
-					nquadStr("_:newUser", "dgraph.type", "User"),
-					nquadStr("_:newUser", "id", item.CreatorId),
-					nquadStr("_:newUser", "name", "Alice"),
-					nquadStr("_:newUser", "created_at", nowStr),
+					nquadStr("_:newUser", fieldDgraphType, dgraphTypeUser),
+					nquadStr("_:newUser", fieldID, item.CreatorId),
+					nquadStr("_:newUser", fieldName, "Alice"),
+					nquadStr("_:newUser", fieldCreatedAt, nowStr),
 				},
 			}); err != nil {
 				return err
@@ -74,15 +74,26 @@ func (tx *todoTransactionImpl) CreateTodo(ctx context.Context, item *todoV1.Todo
 	// Insert into database
 	res, err := tx.tx.Mutate(ctx, &api.Mutation{
 		Set: []*api.NQuad{
-			nquadStr("_:todo", "dgraph.type", "Todo"),
-			nquadStr("_:todo", "id", item.Id),
-			nquadStr("_:todo", "title", item.Title),
-			nquadStr("_:todo", "created_at", nowStr),
-			nquadBool("_:todo", "is_done", item.Done),
-			nquadRel("_:todo", "creator", creatorUID),
+			nquadStr("_:todo", fieldDgraphType, dgraphTypeTodo),
+			nquadStr("_:todo", fieldID, item.Id),
+			nquadStr("_:todo", fieldTitle, item.Title),
+			nquadStr("_:todo", fieldCreatedAt, nowStr),
+			nquadBool("_:todo", fieldDone, item.Done),
+			nquadRel("_:todo", fieldCreator, creatorUID),
+
+			nquadStr("_:todoEvent", fieldDgraphType, dgraphTypeTodoEvent),
+			nquadStr("_:todoEvent", fieldEventType, eventTypeCreate),
+			nquadStr("_:todoEvent", fieldEventAt, nowStr),
+			nquadBool("_:todoEvent", fieldEventPublishedToSearchIndex, false),
+			nquadStr("_:todoEvent", fieldTodoID, item.Id),
+			nquadStr("_:todoEvent", fieldTitle, item.Title),
+			nquadStr("_:todoEvent", fieldCreatedAt, nowStr),
+			nquadBool("_:todoEvent", fieldDone, item.Done),
+			nquadStr("_:todoEvent", fieldCreatorID, item.CreatorId),
 		},
 	})
 
+	// Handle error
 	if err != nil {
 		return err
 	}

api/internal/db/tx_todo_delete.go

@@ -2,22 +2,34 @@ package db
 
 import (
 	"context"
+	"time"
 
 	"github.com/MyOrg/go-dgraph-starter/internal/obs"
 	todoV1 "github.com/MyOrg/go-dgraph-starter/pkg/pb/myorg/todo/v1"
 	"github.com/dgraph-io/dgo/v200/protos/api"
 )
 
 func (tx *todoTransactionImpl) DeleteTodo(ctx context.Context, id string) (*todoV1.DeleteTodoResponse, error) {
-	ctx, span := obs.NewSpan(ctx, "CreateTodo")
+	ctx, span := obs.NewSpan(ctx, "DeleteTodo")
 	defer span.End()
 
 	logger := obs.ToLogger(ctx)
 
+	nowStr := time.Now().UTC().Format(time.RFC3339)
+
+	// Perform deletion
 	res, err := tx.tx.Do(ctx, &api.Request{
 		Query: `
 		query getTodo($id: string) {
-			todo as var(func: eq(id, $id))
+			todo as var(func: eq(id, $id)) {
+				todo_id as id
+				todo_created_at as created_at
+				todo_title as title
+				todo_is_done as is_done
+				creator {
+					todo_creator_id as id
+				}
+			}
 		}
 		`,
 		Mutations: []*api.Mutation{
@@ -26,19 +38,35 @@ func (tx *todoTransactionImpl) DeleteTodo(ctx context.Context, id string) (*todo
 				// Otherwise, we'll assume there are no Todos with that ID,
 				// and we don't perform the Deletion, thus ensuring Idempotence.
 				Cond: `@if(eq(len(todo), 1))`,
+
+				// Instruct Dgraph to delete the Todo
 				Del: []*api.NQuad{
 					// The pattern S * * deletes all the known edges out of a node,
 					// any reverse edges corresponding to the removed edges,
 					// and any indexing for the removed data.
 					nquadAll("uid(todo)"),
 				},
+
+				// Insert event
+				Set: []*api.NQuad{
+					nquadStr("_:todoEvent", fieldDgraphType, dgraphTypeTodoEvent),
+					nquadStr("_:todoEvent", fieldEventType, eventTypeDelete),
+					nquadStr("_:todoEvent", fieldEventAt, nowStr),
+					nquadBool("_:todoEvent", fieldEventPublishedToSearchIndex, false),
+					nquadRel("_:todoEvent", fieldTodoID, "val(todo_id)"),
+					nquadRel("_:todoEvent", fieldTitle, "val(todo_title)"),
+					nquadRel("_:todoEvent", fieldCreatedAt, "val(todo_created_at)"),
+					nquadRel("_:todoEvent", fieldDone, "val(todo_is_done)"),
+					nquadRel("_:todoEvent", fieldCreatorID, "val(todo_creator_id)"),
+				},
 			},
 		},
 		Vars: map[string]string{
 			"$id": id,
 		},
 	})
 
+	// Handle error
 	if err != nil {
 		return nil, err
 	}