Optional vs. mandatory JSON properties during serialization and de-serialization

[danielweck]

Go lang's JSON (un)marshal-er includes a handy omitempty "annotation" for struct fields:
https://golang.org/pkg/encoding/json/
"The 'omitempty' option specifies that the field should be omitted from the encoding if the field has an empty value, defined as false, 0, a nil pointer, a nil interface value, and any empty array, slice, map, or string."

There is a similar default behaviour in ta-json, during serialization to JSON:
https://github.com/edcarroll/ta-json/blob/master/lib/methods/serialize.ts#L33

if ((value === null || value === undefined) || p.writeonly) {
   return;
}

...as well as during deserialization from JSON:
https://github.com/edcarroll/ta-json/blob/master/lib/methods/deserialize.ts#L41

if ((value === null || value === undefined) || p.readonly) {
    return;
}

So, I have been wondering whether an additional @JsonRequired() decoration could be useful for class members (or conversely: @JsonOptional(), but this would require changing the current default behaviour). In case of an "empty" value (to be clarified, perhaps not exactly the same definition as Go lang), a property would not be outputted to JSON, and deserialization may "fail" if the property is actually required ("failure" may be an error callback where the API consumer could act accordingly).

...which also brings the question of TypeScript's strictNullChecks and the ? question mark to denote optionality:
https://www.typescriptlang.org/docs/handbook/release-notes/typescript-2-0.html

Thoughts welcome :)

[danielweck]

PS: I originally tested a custom IPropertyConverter to find-out whether non-set (i.e. null or undefined ... or even falsy) class members could be detected during (de)serialization:

@JsonObject()
export class TestClazz {
    @JsonProperty("requiredProp")
    @JsonConverter(new NotOptional("required-field"))
    public RequiredMember: string;
}

class NotOptional implements IPropertyConverter {
    constructor(readonly name: string) {
    }

    public serialize(property: any): any {
        console.log(this.name + " >> " + property);
        if (!property) {
            console.log("NotOptional! (serialize)");
        }
        return property;
    }

    public deserialize(value: any): any {
        console.log(this.name + " << " + value);
        if (!value) {
            console.log("NotOptional! (deserialize)");
        }
        return value;
    }
}

...which prompted me to look at ta-json's source code (lib/methods/[de]serialize.ts) :)

[danielweck]

Note that it is possible to use the @OnDeserialized() decorator to detect non-set required class members. This approach works, but only for deserialization from JSON, and in a non-declarative manner (i.e. a single function must be added in each JSON-able class, to programmatically check for all possible empty fields):

@JsonObject()
export class TestClazz {
    @JsonProperty("requiredProp")
    public RequiredMember: string;

    @OnDeserialized()
    private _OnDeserialized() {
        if (!this.RequiredMember) {
            console.log("property must be set!");
        }
    }
}

[edcarroll]

As I understand it, you're suggesting a decorator that when used, throws an error if the decorated member isn't set? I'd prefer to use @JsonRequired() rather than @JsonOptional() as it's a breaking change as you've mentioned, and would end up littering the classes with extra decorators (though I suppose that is a matter of perspective depending on your use case...)

Thinking about this suggestion, it seems more in the realm of validation rather than serialization / parsing. Generalising this use case, perhaps a @JsonValidation() decorator that is defined as follows would be more fitting:

@JsonValidation(validators:JsonValidator[])

Decorated property is validated using the provided validators upon (de)serialization. JsonValidator is defined as follows:

type JsonValidator = (propertyValue:any, class:any) => boolean;

where propertyValue is the current value of the decorated member, and class is the value of the entire class being (de)serialized.

We can then provide a set of 'default' validators such as RequiredValidator which does a null/undefined comparison, and TruthyValidator which does a truthiness check. This gives us the flexibility to define what we want to be invalid, and allows us to avoid those non-declarative checks as you've exemplified above.

Let me know your thoughts on this!

[edcarroll]

Hi @danielweck, did you have any further thoughts on this - might a validation option be something you'd like to see in the library? Thanks 😄

[danielweck]

Sorry for the lack of feedback. I like your "validators" suggestion, instead of singled-out "required" / "optional" decorator.
At the moment I am quite happy with the way your lib works (support for optional / required is not critical to my use-case). I am currently using ta-json in an experimental project.

[danielweck]

FYI: I implemented the XML equivalent of ta-json, based on your original code. There's full support for XML namespaces, I use pre-loaded DOM (rather than incremental Sax), and expressions are matched using XPath (my own basic-but-fast implementation, with fallback to a "proper" ; albeit slower ; XPath lib for more complex expressions).
At the moment I only care about de-serialization / parsing from XML into in-memory data models. If time allows, I will also write the serialiser.
The code is open-source but I use it internally for an experimental project. At some hypothetical point in the future I will publish a proper NPM package with credits to ta-json :)

[danielweck]

Just as point of reference, TypeGoose combines both "required" and mix/max (etc.) validators in the @prop() decorator:

https://github.com/szokodiakos/typegoose/blob/master/README.md

(not saying that's how this should be implemented in ta-json, just wanted to share what others are doing)