This is an automated email from the ASF dual-hosted git repository.
jsinovassinnaik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/unomi.git
The following commit(s) were added to refs/heads/master by this push:
new ab7d2c099 UNOMI-646 : update documentation (#494)
ab7d2c099 is described below
commit ab7d2c0993f12b04dfb4566d49ef511757205523
Author: jsinovassin <58434978+jsinovas...@users.noreply.github.com>
AuthorDate: Thu Sep 8 17:23:03 2022 +0200
UNOMI-646 : update documentation (#494)
* UNOMI-646 : update documentation
---
manual/src/main/asciidoc/datamodel.adoc | 31 ++++---
.../main/asciidoc/images/data-model-overview.png | Bin 280344 -> 0 bytes
manual/src/main/asciidoc/images/data-model.png | Bin 0 -> 702077 bytes
manual/src/main/asciidoc/whats-new.adoc | 103 +++++++++++++++++----
4 files changed, 101 insertions(+), 33 deletions(-)
diff --git a/manual/src/main/asciidoc/datamodel.adoc
b/manual/src/main/asciidoc/datamodel.adoc
index 8f0485735..c4ac43a79 100755
--- a/manual/src/main/asciidoc/datamodel.adoc
+++ b/manual/src/main/asciidoc/datamodel.adoc
@@ -21,15 +21,17 @@ user profiles into segments along user-definable dimensions
or acted upon by alg
The following data model only contains the classes and properties directly
related to the most important objects of Apache Unomi.
There are other classes that are less central to the functionality but all the
major ones are represented in the diagram below:
-image::data-model-overview.png[]
+image::data-model.png[]
We will detail many of these classes in the document below.
=== Scope
-Scopes are not actually an object in the system, but simply unique strings
that are used to "classify" objects.
+Scopes are objects which simply contains unique strings that are used to
"classify" objects.
For example, when using scopes with a web content management system, a scope
could be associated with a site identifier or even a host name.
+In events, scopes are used to validate event. Events with scope which are
unknown by the system will be considered as invalid
+
____
Unomi defines a built-in scope (called `systemscope`) that clients can use to
share data across scopes.
@@ -204,12 +206,16 @@ Inherits all the fields from: <<Item>>
| scope | String | (Optional, event type specific) An identifier for a scope
+| persistent | Boolean | Defines if the event should be persisted or not
(default: true)
+
| source | <<Item>> | An Item that is the source of the event. For example a
web site, an application name, a web page
| target | <<Item>> | An Item that is the target of the event. For example a
button, a link, a file or a page
| properties | Map<String,Object> | Properties for the event. These will
change depending on the event type.
+| flattenedProperties | Map<String,Object> | Properties that will be persisted
as flattened. These will change depending on the event type.
+
|===
==== Event types
@@ -253,7 +259,7 @@ Inherits all the fields from: <<Item>>
| scores | Map<String,Integer> | A map of scores with the score identifier as
the key and the score total value as the value.
-| mergedWith | String | If merged with another profile, the profile identifier
to the master profile is stored here
+| @Deprecated mergedWith | String | If merged with another profile, the
profile identifier to the master profile is stored here
| consents | Map<String,<<Consent>>> | The consents for the profile, as a map
with the consent identifier as a key and the Consent object type as a value.
@@ -314,13 +320,13 @@ image::profile.png[]
"lists": [
"userListId"
],
-"goals": {
- "viewLanguagePageGoalTargetReached": "2020-02-10T19:30:31Z",
- "downloadGoalExampleTargetReached": "2020-02-10T15:22:41Z",
- "viewLandingPageGoalStartReached": "2020-02-10T19:30:27Z",
- "downloadGoalExampleStartReached": "2020-02-10T19:30:27Z",
- "optimizationTestGoalStartReached": "2020-02-10T19:30:27Z"
-}
+ "goals": {
+ "viewLanguagePageGoalTargetReached": "2020-02-10T19:30:31Z",
+ "downloadGoalExampleTargetReached": "2020-02-10T15:22:41Z",
+ "viewLandingPageGoalStartReached": "2020-02-10T19:30:27Z",
+ "downloadGoalExampleStartReached": "2020-02-10T19:30:27Z",
+ "optimizationTestGoalStartReached": "2020-02-10T19:30:27Z"
+ }
},
"segments": [
"leads",
@@ -394,7 +400,6 @@ In the following example a Persona represents a visitor
from Europe, that can be
"systemProperties": {},
"segments": [],
"scores": null,
- "mergedWith": null,
"consents": {}
}
----
@@ -786,7 +791,7 @@ Inherits all the fields from: <<MetadataItem>>
==== Example
-In this exmample we should the default `updateProperties` built-in rule that
matches the `updateProperties` event and
+In this example we can see the default `updateProperties` built-in rule that
matches the `updateProperties` event and
executes the built-in `updatePropertiesAction`
[source,json]
@@ -1001,7 +1006,7 @@ Inherits all the fields from: <<MetadataItem>>
In the following example a campaign that starts January 1st 31, 2020 at 8:38am
and finished on February 29th, 2020 at the same time has the following entry
condition: the session duration must be less or equal to 3000 milliseconds (3
seconds) and the profile has viewed the “about” page on the “digitall” website.
The cost of the campaign is USD 1’000’000 and the timezone is Europe/Zurich.
-The primary goal for the campaign is the goal we should as an example in the
Goal section.
+The primary goal for the campaign is the goal we should have as an example in
the Goal section.
[source,json]
----
diff --git a/manual/src/main/asciidoc/images/data-model-overview.png
b/manual/src/main/asciidoc/images/data-model-overview.png
deleted file mode 100755
index 8243b9e6e..000000000
Binary files a/manual/src/main/asciidoc/images/data-model-overview.png and
/dev/null differ
diff --git a/manual/src/main/asciidoc/images/data-model.png
b/manual/src/main/asciidoc/images/data-model.png
new file mode 100644
index 000000000..e63d771a4
Binary files /dev/null and b/manual/src/main/asciidoc/images/data-model.png
differ
diff --git a/manual/src/main/asciidoc/whats-new.adoc
b/manual/src/main/asciidoc/whats-new.adoc
index b04296865..97e5a1661 100644
--- a/manual/src/main/asciidoc/whats-new.adoc
+++ b/manual/src/main/asciidoc/whats-new.adoc
@@ -13,14 +13,14 @@
//
=== What's new in Apache Unomi 2.0
-Apache Unomi 2 is a new release focused on improving core functionalities and
robustness of the product.
+Apache Unomi 2 is a new release focused on improving core functionalities and
robustness of the product.
The introduction of tighter data validation with JSON Schemas required some
changes in the product data model, which presented an opportunity for
noticeable improvements in the overall performance.
This new release also introduces a first (beta) version of the Unomi GraphQL
API.
==== Scopes declarations are now required
-Scopes declarations are now required in Unomi 2. When submitting an event and
specifying a scope,
+Scopes declarations are now required in Unomi 2. When submitting an event and
specifying a scope,
that scope must already be declared on the platform.
Scopes can be easily created via the corresponding REST API (`cxs/scopes`)
@@ -39,8 +39,8 @@ curl --location --request POST
'http://localhost:8181/cxs/scopes' \
==== JSON Schemas
-Apache Unomi 2 introduces support for
https://json-schema.org/specification.html[JSON Schema] for all of its publicly
exposed endpoints.
-Data received by Apache Unomi 2 will first be validated against a known schema
to make sure it complies with an expected payload.
+Apache Unomi 2 introduces support for
https://json-schema.org/specification.html[JSON Schema] for all of its publicly
exposed endpoints.
+Data received by Apache Unomi 2 will first be validated against a known schema
to make sure it complies with an expected payload.
If the received payload does not match a known schema, it will be rejected by
Apache Unomi 2.
Apache Unomi 2 also introduces a set of administrative endpoints allowing new
schemas and/or schemas extensions to be registered.
@@ -51,11 +51,50 @@ More details about JSON Schemas implementation are
available in the <<JSON schem
The introduction of JSON schema required us to modify Apache Unomi data model,
One of the key differences is the removal of open maps.
-For example, the following event model in Apache Unomi 1.x:
+The properties field in the events objects provided by unomi are now
restricted by JSON schema.
+This means object properties must be declared in a JSON schema for an event to
be accepted.
+
+A new property, flattenedProperties has been introduced to the event object,
this property has been added to store the properties as
+flattened in Elasticsearch and should avoid mapping explosion for dynamic
properties.
+
+If there is dynamic properties that you want to send with your event, you
should use the flattenedProperties field of the event.
+
+It's also necessary to specify the format of the values which are added to
flattenedProperties by JSON schema but these value will be
+stored as flattened and will not create dynamic mapping contrary to the
properties field of the events.
+
+Here is an example for objects that used dynamic properties for URL parameters:
+
+The following event example in Apache Unomi 1.x:
[source]
----
{
- "TODO": "ADD JSON"
+ "eventType":"view",
+ "scope":"digitall",
+ "properties":{
+ "URLParameters":{
+ "utm_source":"source"
+ }
+ },
+ "target":{
+ "scope":"digitall",
+ "itemId":"30c0a9e3-4330-417d-9c66-4c1beec85f08",
+ "itemType":"page",
+ "properties":{
+ "pageInfo":{
+ "pageID":"30c0a9e3-4330-417d-9c66-4c1beec85f08",
+ "nodeType":"jnt:page",
+ "pageName":"Home",
+ ...
+ },
+ "attributes":{},
+ "consentTypes":[]
+ }
+ },
+ "source":{
+ "scope":"digitall",
+ "itemId":"ff5886e0-d75a-4061-9de9-d90dfc9e18d8",
+ "itemType":"site"
+ }
}
----
@@ -63,14 +102,38 @@ Is replaced by the following in Apache Unomi 2.x:
[source]
----
{
- "TODO": "ADD JSON"
+ "eventType":"view",
+ "scope":"digitall",
+ "flattenedProperties":{
+ "URLParameters":{
+ "utm_source":"source"
+ }
+ },
+ "target":{
+ "scope":"digitall",
+ "itemId":"30c0a9e3-4330-417d-9c66-4c1beec85f08",
+ "itemType":"page",
+ "properties":{
+ "pageInfo":{
+ "pageID":"30c0a9e3-4330-417d-9c66-4c1beec85f08",
+ "nodeType":"jnt:page",
+ "pageName":"Home",
+ ...
+ },
+ "attributes":{},
+ "consentTypes":[]
+ }
+ },
+ "source":{
+ "scope":"digitall",
+ "itemId":"ff5886e0-d75a-4061-9de9-d90dfc9e18d8",
+ "itemType":"site"
+ }
}
----
-Most objects were refactored as part of this new release.
-
-If using the default Apache 1.x data model, our Unomi 2 migration process will
handle the data model changes for you.
-If you are using custom events/objects, please refer to the detailed migration
guide for more details.
+If using the default Apache 1.x data model, our Unomi 2 migration process will
handle the data model changes for you.
+If you are using custom events/objects, please refer to the detailed migration
guide for more details.
==== Removal of the Web Tracker
@@ -82,8 +145,8 @@ use case.
==== GraphQL API - beta
-Apache Unomi 2.0 sees the introduction of a new (beta) GraphQL API.
-Available behind a feature flag (the API disabled by default), the GraphQL API
is available for you to play with.
+Apache Unomi 2.0 sees the introduction of a new (beta) GraphQL API.
+Available behind a feature flag (the API disabled by default), the GraphQL API
is available for you to play with.
More details about how to enable/disable the GraphQL API are available in the
<<GraphQL API,corresponding section>> of the documentation.
@@ -91,20 +154,20 @@ We welcome tickets/PRs to improve its robustness and
progressively make it ready
==== Migrate from Unomi 1.x
-To facilitate migration we prepared a set of scripts that will automatically
handle the migration of your data from Apache Unomi 1.5+ to Apache Unomi 2.0.
+To facilitate migration we prepared a set of scripts that will automatically
handle the migration of your data from Apache Unomi 1.5+ to Apache Unomi 2.0.
-It is worth keeping in mind that for Apache Unomi 2.0 we do not support “hot”
migration,
+It is worth keeping in mind that for Apache Unomi 2.0 we do not support “hot”
migration,
the migration process will require a shutdown of your cluster to guarantee
that no new events will be collected while data migration is in progress.
-Special caution must be taken if you declared custom events as our migration
scripts can only handle objects we know of.
+Special caution must be taken if you declared custom events as our migration
scripts can only handle objects we know of.
More details about migration (incl. of custom events) is available in the
corresponding section <<Migrations,corresponding section>> of the documentation.
==== Elasticsearch compatibility
-We currently recommend using Elasticsearch 7.17.5 with Apache Unomi 2.0,
+We currently recommend using Elasticsearch 7.17.5 with Apache Unomi 2.0,
this ensure you are on a recent version that is not impacted by the log4j
vulnerabilities (fixed in Elasticsearch 7.16.3).
-This version increase is releated to Apache Unomi 2.0 makeing use of a new
Elasticsearch field type
-called
https://www.elastic.co/guide/en/elasticsearch/reference/7.17/flattened.html[Flattened],
-and although it was available in prior versions of Elasticsearch, we do not
recommend using those
+This version increase is releated to Apache Unomi 2.0 makeing use of a new
Elasticsearch field type
+called
https://www.elastic.co/guide/en/elasticsearch/reference/7.17/flattened.html[Flattened],
+and although it was available in prior versions of Elasticsearch, we do not
recommend using those
due to the above-mentioned log4j vulnerabilities.
