This is an automated email from the ASF dual-hosted git repository. jsinovassinnaik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/unomi.git
The following commit(s) were added to refs/heads/master by this push: new f2b6fab9a UNOMI-625 : add documentation on json schema (#457) f2b6fab9a is described below commit f2b6fab9accc5d60b5e795b75df830687fcf8aa5 Author: jsinovassin <58434978+jsinovas...@users.noreply.github.com> AuthorDate: Fri Jul 8 17:01:46 2022 +0200 UNOMI-625 : add documentation on json schema (#457) * UNOMI-625 : add documentation on json schema --- .../asciidoc/images/process-creation-extension.png | Bin 0 -> 145044 bytes .../asciidoc/images/process-creation-schema.png | Bin 0 -> 136021 bytes manual/src/main/asciidoc/index.adoc | 2 + .../jsonSchema/extend-an-existing-schema.adoc | 87 +++++++ .../src/main/asciidoc/jsonSchema/introduction.adoc | 280 +++++++++++++++++++++ .../main/asciidoc/jsonSchema/json-schema-api.adoc | 138 ++++++++++ .../src/main/asciidoc/jsonSchema/json-schema.adoc | 21 ++ 7 files changed, 528 insertions(+) diff --git a/manual/src/main/asciidoc/images/process-creation-extension.png b/manual/src/main/asciidoc/images/process-creation-extension.png new file mode 100644 index 000000000..0c7357a85 Binary files /dev/null and b/manual/src/main/asciidoc/images/process-creation-extension.png differ diff --git a/manual/src/main/asciidoc/images/process-creation-schema.png b/manual/src/main/asciidoc/images/process-creation-schema.png new file mode 100644 index 000000000..4158eb50b Binary files /dev/null and b/manual/src/main/asciidoc/images/process-creation-schema.png differ diff --git a/manual/src/main/asciidoc/index.adoc b/manual/src/main/asciidoc/index.adoc index 2976cc0fa..111ba170e 100644 --- a/manual/src/main/asciidoc/index.adoc +++ b/manual/src/main/asciidoc/index.adoc @@ -38,6 +38,8 @@ include::request-examples.adoc[] include::configuration.adoc[] +include::jsonSchema/json-schema.adoc[] + include::migrations/migrations.adoc[] == Queries and aggregations diff --git a/manual/src/main/asciidoc/jsonSchema/extend-an-existing-schema.adoc b/manual/src/main/asciidoc/jsonSchema/extend-an-existing-schema.adoc new file mode 100644 index 000000000..d82501f97 --- /dev/null +++ b/manual/src/main/asciidoc/jsonSchema/extend-an-existing-schema.adoc @@ -0,0 +1,87 @@ +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// + +=== Extend an existing schema + +==== When a extension is needed? + +Apache Unomi provides predefined schemas to validate some known events such as a view event. + +The Apache Unomi JSON schemas are designed to consider invalid any properties which are not defined in the JSON schema. +So if an unknown property is part of the event, the event will be considered as invalid. + +This means that if your events include additional properties, you will need extensions to describe these. + +==== Understanding how extensions are merged in unomi + +An extension schema is a JSON schema whose id will be overridden and be defined by a keyword named *extends* in the *self* part of the extension. + +When sending an extension through the API, it will be persisted in Elasticsearch then will be merged to the targeted schema. + +What does “merge a schema” mean? +The merge will simply add in the *allOf* keyword of the targeted schema a reference to the extensions. +It means that to be valid, an event should be valid against the base schema and against the ones added in the *allOf*. + +Example of an extension to allow to add a new property in the view event properties: + +[source] +---- +{ + "$id": "https://vendor.test.com/schemas/json/events/dummy/extension/1-0-0", + "$schema": "https://json-schema.org/draft/2019-09/schema", + "self":{ + "vendor":"com.vendor.test", + "name":"dummyExtension", + "format":"jsonschema", + "extends": "https://unomi.apache.org/schemas/json/events/view/properties/1-0-0", + "version":"1-0-0" + }, + "title": "DummyEventExtension", + "type": "object", + "properties": { + "myNewProp": { + "type": "string" + } + } +} +---- + +When validating the events of type view, the extension will be added to the schema with the id *\https://unomi.apache.org/schemas/json/events/view/properties/1-0-0* like the following: + +[source] +---- +"allOf": [{ + "$ref": "https://vendor.test.com/schemas/json/events/dummy/extension/1-0-0" +}] +---- + +With this extension the property *myNewProp* can now be added to the event. + +[source] +---- +… +"properties": { + "myNewProp" : "newValue" +}, +… +---- + +Process when adding extension: + +image::process-creation-extension.png[pdfwidth=35%,align=center] + +==== How to add an extension through the API + +Since an extension is also a JSON schema, it is possible to add extensions by calling the endpoint to add a JSON schema. +By calling `POST {{url}}/cxs/jsonSchema` with the JSON schema in the payload of the request, the extension will be persisted and will be merged to the targeted schema. diff --git a/manual/src/main/asciidoc/jsonSchema/introduction.adoc b/manual/src/main/asciidoc/jsonSchema/introduction.adoc new file mode 100644 index 000000000..ce4848000 --- /dev/null +++ b/manual/src/main/asciidoc/jsonSchema/introduction.adoc @@ -0,0 +1,280 @@ +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// + +=== Introduction + +==== What is a JSON Schema + +https://json-schema.org/specification.html[JSON Schema] is a powerful standard for validating the structure of JSON data. +Described as a JSON object, a JSON schema file contains format, types, patterns, and more. +Used against JSON data, a JSON schema validates that the data is compatible with the specified schema. + +Example of a basic JSON schema that validates that the path property is a string property: + +[source] +---- +{ + "$id":"https://unomi.apache.org/schemas/json/example/1-0-0", + "$schema":"https://json-schema.org/draft/2019-09/schema", + "title":"Example of a basic schema", + "type":"object", + "properties":{ + "path":{ + "type":"string", + "$comment":"Example of a property." + } + } +} +---- + +[source] +---- +{ + "path": "example/of/path" //Is valid +} +---- + +[source] +---- +{ + "path": 100 // Is not valid +} +---- + +Apache Unomi is using json-schema-validator to integrate JSON schema. +The library and its source code is available at: https://github.com/networknt/json-schema-validator[https://github.com/networknt/json-schema-validator], you can refer to the feature’s pom.xml available at https://github.com/apache/unomi/blob/master/extensions/json-schema/services/pom.xml#L35[json-schema/service/pom.xml] to identify which version of the library is currently integrated. + +You can discover and play with JSON schema using online tools such as https://www.jsonschemavalidator.net/[JSON Schema Validator]. +Such tools allow you to validate a schema against JSON data (such as the example above), and can point to particular errors. +More details about JSON schema are available on the official specification’s website: https://json-schema.org/specification.html[https://json-schema.org/specification.html] + +==== Key concepts + +This section details concepts that are important to understand in order to use JSON schema validation with Apache Unomi. + +===== $id keyword + +The *$id* keyword: + +Each schema in Apache Unomi should have a *$id*, the *$id* value is an URI which will be used to retrieve the schema and must be unique. + +Example: + +[source] +---- +{ + "$id":"https://unomi.apache.org/schemas/json/example/1-0-0" +} +---- + +===== $ref keyword + +The *$ref* keyword allows you to reference another JSON schema by its *$id* keyword. +It’s possible to separate complex structures or repetitive parts of schema into other small files and use *$ref* to include them into several json schemas. + +Example with a person and an address: + +[source] +---- +{ + "$id": "https://example.com/schemas/address", + "type": "object", + "properties": { + "street_address": { "type": "string" }, + "city": { "type": "string" }, + "state": { "type": "string" } + } +} +---- + +[source] +---- +{ + "type": "object", + "properties": { + "first_name":{ "type": "string" }, + "last_name": { "type": "string" }, + "shipping_address": { + "$ref": "https://example.com/schemas/address" + }, + "billing_address": { + "$ref": "https://example.com/schemas/address" + } + } +} +---- + +More details about *$ref* can be found in the specifications: https://json-schema.org/understanding-json-schema/structuring.html#ref[https://json-schema.org/understanding-json-schema/structuring.html#ref] + +===== allOf keyword + +The allOf keyword is an array of fields which allows schema composition. +The data will be valid against a schema if the data are valid against all of the given subschemas in the allOf part and are valid against the properties defined in the schema. + +[source] +---- +{ + "$id": "https://unomi.apache.org/schemas/json/example/1-0-0", + "$schema": "https://json-schema.org/draft/2019-09/schema", + "type": "object", + "allOf": [ + { + "type": "object", + "properties": { + "fromAllOf": { + "type": "integer", + "$comment": "Example of allOf." + } + } + } + ], + "properties": { + "myProperty": { + "type": "string", + "$comment": "Example of a property." + } + } +} +---- + +Valid JSON: + +[source] +---- +{ + "myProperty": "My property", + "fromAllOf" : 10 +} +---- + +Invalid JSON: + +[source] +---- +{ + "myProperty": "My property", + "fromAllOf" : "My value" +} +---- + +It’s also possible to use a reference *$ref* in the *allOf* keyword to reference another schema. + +In Unomi, there is an example of using *$ref* in the *allOf* keyword to validate the properties which are defined in the event schema. +This schema contains properties common to all events. +It’s done in the the view event schema. +The file can be found on github: https://github.com/apache/unomi/blob/master/extensions/json-schema/services/src/main/resources/META-INF/cxs/schemas/events/view/view.json#L13[view.json] +More details about allOf can be found in the specifications: https://json-schema.org/understanding-json-schema/reference/combining.html#allof[https://json-schema.org/understanding-json-schema/reference/combining.html#allof] + +===== unevaluatedProperties keyword + +The *unevaluatedProperties* keyword is useful for schema composition as well as enforcing stricter schemas. +This keyword is similar to *additionalProperties* except that it can recognize properties declared in sub schemas. +When setting the *unevaluatedProperties* value to *false*, the properties which are not present in the properties part and are not present in the sub schemas will be considered as invalid. + +Example with the following schema: + +[source] +---- +{ + "$id": "https://unomi.apache.org/schemas/json/example/1-0-0", + "$schema": "https://json-schema.org/draft/2019-09/schema", + "type": "object", + "allOf": [ + { + "$ref": "https://unomi.apache.org/schemas/json/subschema/1-0-0" + } + ], + "properties": { + "myProperty": { + "type": "string", + "$comment": "Example of a property." + } + }, + "unevaluatedProperties": false +} +---- + +Sub schema: + +[source] +---- +{ + "$id": "https://unomi.apache.org/schemas/json/subschema/1-0-0", + "$schema": "https://json-schema.org/draft/2019-09/schema", + "type": "object", + "properties": { + "fromAllOf": { + "type": "string", + "$comment": "Example of allOf." + } + } +} +---- + +With the following data, the validation will fail because the property *myNewProperty* is not defined neither the *properties* part nor the *allOf* part. + +[source] +---- +{ + "myProperty": "My property", + "fromAllOf" : 10, + "myNewProperty": "another one" //Not valid +} +---- + +==== How are JSON Schema used in Unomi + +JSON Schema is used in Unomi to validate the data coming from the two public endpoints */contextRequest* and */eventCollector*. +Both endpoints have a custom deserializer which will begin by validating the payload of the request, then will filter invalid events present in this payload. +If an event is not valid it will not be processed by the system. +The internal events are not validated by JSON schema as they are not sent through the public endpoints. + +In Unomi, each event type must have an associated JSON schema. +To validate an event, Unomi will search for a schema in which the target of the schema is *events*, and with the name of the schema matching the event type. + +A custom keyword named *self* has to be present in the JSON schemas to store the information related to each schema. +The following example is the *self* part of the view event JSON schema. +Having the target set to *events* and the name set to *view*, this schema will be used to validate the events of type *view*. + +[source] +---- +… +"self":{ + "vendor":"org.apache.unomi", + "target" : "events", + "name": "view", + "format":"jsonschema", + "version":"1-0-0" +}, +… +---- + +Link to the schema on github: https://github.com/apache/unomi/blob/master/extensions/json-schema/services/src/main/resources/META-INF/cxs/schemas/events/view/view.json[view.json]. + +A set of predefined schema are present in Unomi, these schemas can be found under the folder : https://github.com/apache/unomi/tree/master/extensions/json-schema/services/src/main/resources/META-INF/cxs/schemas[extensions/json-schema/services/src/main/resources/META-INF/cxs/schemas]. + +These schemas will be loaded in memory at startup. +Each schema where the *target* value is set to *events*, will be used to validate events. +The others are simply used as part of JSON schema or can be used in additional JSON schemas. + +It’s possible to add JSON schemas to validate your own event by using the API, the explanations to manage JSON schema through the API are +in the <<Create / update a JSON schema to validate an event, Create / update a JSON schema to validate an event>> section. + +Contrary to the predefined schemas, the schemas added through the API will be persisted in Elasticsearch in the jsonSchema index. +Schemas persisted in Elasticsearch do not require a restart of the platform to reflect changes. + +Process of creation of schemas: + +image::process-creation-schema.png[pdfwidth=35%,align=center] + diff --git a/manual/src/main/asciidoc/jsonSchema/json-schema-api.adoc b/manual/src/main/asciidoc/jsonSchema/json-schema-api.adoc new file mode 100644 index 000000000..bec18024b --- /dev/null +++ b/manual/src/main/asciidoc/jsonSchema/json-schema-api.adoc @@ -0,0 +1,138 @@ +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// + +=== JSON schema API + +The JSON schema endpoints are private, so the user has to be authenticated to manage the JSON schema in Unomi. + +==== List existing schemas + +The REST endpoint GET `{{url}}/cxs/jsonSchema` allows to get all ids of available schemas and subschemas. + +List of predefined schemas: + +[source] +---- +[ + "https://unomi.apache.org/schemas/json/events/modifyConsent/properties/1-0-0", + "https://unomi.apache.org/schemas/json/item/1-0-0", + "https://unomi.apache.org/schemas/json/events/login/1-0-0", + "https://unomi.apache.org/schemas/json/events/modifyConsent/1-0-0", + "https://unomi.apache.org/schemas/json/consentType/1-0-0", + "https://unomi.apache.org/schemas/json/items/page/properties/1-0-0", + "https://unomi.apache.org/schemas/json/items/page/properties/attributes/1-0-0", + "https://unomi.apache.org/schemas/json/events/incrementInterest/1-0-0", + "https://unomi.apache.org/schemas/json/events/view/flattenProperties/1-0-0", + "https://unomi.apache.org/schemas/json/interests/1-0-0", + "https://unomi.apache.org/schemas/json/items/site/1-0-0", + "https://unomi.apache.org/schemas/json/items/page/properties/pageInfo/1-0-0", + "https://unomi.apache.org/schemas/json/rest/requestIds/1-0-0", + "https://unomi.apache.org/schemas/json/rest/eventscollectorrequest/1-0-0", + "https://unomi.apache.org/schemas/json/events/view/properties/1-0-0", + "https://unomi.apache.org/schemas/json/items/page/1-0-0", + "https://unomi.apache.org/schemas/json/URLParameters/1-0-0", + "https://unomi.apache.org/schemas/json/event/1-0-0", + "https://unomi.apache.org/schemas/json/timestampeditem/1-0-0", + "https://unomi.apache.org/schemas/json/events/updateProperties/1-0-0", + "https://unomi.apache.org/schemas/json/consent/1-0-0", + "https://unomi.apache.org/schemas/json/events/incrementInterest/flattenProperties/1-0-0", + "https://unomi.apache.org/schemas/json/events/view/1-0-0" +] +---- + +Custom schemas will also be present in this list once added. + +==== Read a schema + +It’s possible to get a schema by its id by calling the endpoint `POST {{url}}/cxs/jsonSchema/query` with the id of the schema in the payload of the query. + +Example: + +[source] +---- +curl --location --request POST 'http://localhost:8181/cxs/jsonSchema/query' \ +-u 'karaf:karaf' +--header 'Content-Type: text/plain' \ +--header 'Cookie: context-profile-id=0f2fbca8-c242-4e6d-a439-d65fcbf0f0a8' \ +--data-raw 'https://unomi.apache.org/schemas/json/event/1-0-0' +---- + +==== Create / update a JSON schema to validate an event + +It’s possible to add or update JSON schema by calling the endpoint `POST {{url}}/cxs/jsonSchema` with the JSON schema in the payload of the request. +If the JSON schema exists it will be updated with the new one. + +Example of creation: + +[source] +---- +curl --location --request POST 'http://localhost:8181/cxs/jsonSchema' \ +-u 'karaf:karaf' \ +--header 'Content-Type: application/json' \ +--header 'Cookie: context-profile-id=0f2fbca8-c242-4e6d-a439-d65fcbf0f0a8' \ +--data-raw '{ + "$id": "https://vendor.test.com/schemas/json/events/dummy/1-0-0", + "$schema": "https://json-schema.org/draft/2019-09/schema", + "self": { + "vendor": "com.vendor.test", + "name": "dummy", + "format": "jsonschema", + "target": "events", + "version": "1-0-0" + }, + "title": "DummyEvent", + "type": "object", + "allOf": [ + { + "$ref": "https://unomi.apache.org/schemas/json/event/1-0-0" + } + ], + "properties": { + "properties": { + "$ref": "https://vendor.test.com/schemas/json/events/dummy/properties/1-0-0" + } + }, + "unevaluatedProperties": false +}' +---- + +==== Deleting a schema + +To delete a schema, call the endpoint `POST {{url}}/cxs/jsonSchema/delete` with the id of the schema into the payload of the request + +Example: + +[source] +---- +curl --location --request POST 'http://localhost:8181/cxs/jsonSchema/delete' \ +-u 'karaf:karaf' \ +--header 'Content-Type: text/plain' \ +--header 'Cookie: context-profile-id=0f2fbca8-c242-4e6d-a439-d65fcbf0f0a8' \ +--data-raw 'https://vendor.test.com/schemas/json/events/dummy/1-0-0' +---- + +==== Error Management + +When calling an endpoint with invalid data, such as an invalid value for the *sessionId* property in the contextRequest object or eventCollectorRequest object, the server would respond with a 400 error code and the message *Request rejected by the server because: Invalid received data*. + +==== Details on invalid events + +If it’s an event which is incorrect the server will continue to process the request but will exclude the invalid events. +Running Apache Unomi with the logs in debug level will add to the logs the reason why events are rejected. +You can set the log level of the class validating the events to debug by using the following karaf command: + +[source] +---- +log:set DEBUG org.apache.unomi.schema.impl.SchemaServiceImpl +---- diff --git a/manual/src/main/asciidoc/jsonSchema/json-schema.adoc b/manual/src/main/asciidoc/jsonSchema/json-schema.adoc new file mode 100644 index 000000000..a4586a6ae --- /dev/null +++ b/manual/src/main/asciidoc/jsonSchema/json-schema.adoc @@ -0,0 +1,21 @@ +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// + +== JSON schemas + +include::introduction.adoc[] + +include::json-schema-api.adoc[] + +include::extend-an-existing-schema.adoc[]