[jira] [Commented] (NIFI-11627) Add Dynamic Schema References to ValidateJSON Processor
[
https://issues.apache.org/jira/browse/NIFI-11627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17771313#comment-17771313
]
Chuck Tilly commented on NIFI-11627:
Hi [~markap14]
{quote}Making use of a Controller Service would work well. We could have a
Controller Service that allows user-added properties where the values are JSON
Schemas, and then allow ValidateJson to be configured with a Controller Service
and take in the name of the schema, which would allow for Expression Language
to be used. So I believe that would give you exactly what you're looking for,
[~nwchuckster], no?
{quote}
Absolutely!!! That would be fantastic. Of course that is my preference so I
look forward to that.
{quote}...you cannot use Expression Language within a Parameter Context because
parameters' values are resolved before the processor ever even has access to
the property value. So, if you were to enter #\{${schema.name}} what would
happen is that NiFi would resolve that to a parameter named ${schema.name} and
the processor would be invalid, before it ever had any chance to even evaluate
Expression Language.
{quote}
My experience is that you can use them together with the UpdateAttribute
processor where I use this technique in several places. For example I use the
following expression in an UpdateAttribute rule to dynamically generate the
correct URL string as an attribute for each flowfile.
{color:#0747a6}{{${#\{'Toadol End Point Service'}:evaluateELString()}}}{color}
{quote}There are security policies that guard who is allow to reference
parameters, etc. and allowing dynamic creation of parameter names would violate
the security constraints.
{quote}
I was not aware of this. Is this true even if this process is contained in its
own process group with its own parameter context specific to only this context?
> Add Dynamic Schema References to ValidateJSON Processor
> ---
>
> Key: NIFI-11627
> URL: https://issues.apache.org/jira/browse/NIFI-11627
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.19.1
>Reporter: Chuck Tilly
>Assignee: Daniel Stieglitz
>Priority: Major
>
> For the ValidateJSON processor, add support for flowfile attribute references
> that will allow for a JSON schema located in the Parameter Contexts, to be
> referenced dynamically based on a flowfile attribute. e.g.
> {code:java}
> #{${schema.name}} {code}
>
> The benefits of adding support for attribute references are significant.
> Adding this capability will allow a single processor to be used for all JSON
> schema validation. Unfortunately, the current version of this processor
> requires a dedicated processor for every schema, i.e. 12 schemas requires 12
> ValidateJSON processors. This is very laborious to construct and maintain,
> and resource expensive.
> ValidateJSON processor (https://issues.apache.org/jira/browse/NIFI-7392)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-11627) Add Dynamic Schema References to ValidateJSON Processor
[
https://issues.apache.org/jira/browse/NIFI-11627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17770593#comment-17770593
]
Chuck Tilly commented on NIFI-11627:
[~exceptionfactory] A specific use-case would be every single time you want
to validate your data, which is always. Hence the need for this processor.
But storing the schema in the attributes is not the solution, you are right
about that. Rather, it would be better if the flow file contained a +reference+
to its schema. The schema itself is stored as a value in the Parameter
Contexts, and the ValidateJSON Processor uses this schema to perform the
validation. This is exactly the same paradigm that the AvroSchemaRegistry
uses, except the schemas are stored in a Controller Service instead of a
Parameter Context (the principle is the same though). So assuming you have a
flowfile with an attribute named "schema.name" (e.g. schema.name = blue-cars),
then the syntax for referencing a schema stored as a Parameter Context value
would be:
JSON Schema = #\{${schema.name}}
Within the Parameter Contexts there would be a value "blue-cars", and it would
contain the JSON schema for blue-cars.
The benefits are:
1) Only a single ValidateJSON processor is needed to perform all the
validations. This allows for clean and simple flows that are easy to manage,
and consume fewer resources.
2) This approach is consistent with how schema validation is done in Nifi
using schema registries. It is a well established pattern in NiFi.
> Add Dynamic Schema References to ValidateJSON Processor
> ---
>
> Key: NIFI-11627
> URL: https://issues.apache.org/jira/browse/NIFI-11627
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.19.1
>Reporter: Chuck Tilly
>Assignee: Daniel Stieglitz
>Priority: Major
>
> For the ValidateJSON processor, add support for flowfile attribute references
> that will allow for a JSON schema located in the Parameter Contexts, to be
> referenced dynamically based on a flowfile attribute. e.g.
> {code:java}
> #{${schema.name}} {code}
>
> The benefits of adding support for attribute references are significant.
> Adding this capability will allow a single processor to be used for all JSON
> schema validation. Unfortunately, the current version of this processor
> requires a dedicated processor for every schema, i.e. 12 schemas requires 12
> ValidateJSON processors. This is very laborious to construct and maintain,
> and resource expensive.
> ValidateJSON processor (https://issues.apache.org/jira/browse/NIFI-7392)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-11627) Add Dynamic Schema References to ValidateJSON Processor
[
https://issues.apache.org/jira/browse/NIFI-11627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763836#comment-17763836
]
Chuck Tilly commented on NIFI-11627:
This seems like an anti-pattern since the prescribed way of performing
validation in NiFi is through the use of a schema registry. Using a hack like
the Jolt spec is undocumented, and inconsistent with best practices for data
validation in NiFi.
> Add Dynamic Schema References to ValidateJSON Processor
> ---
>
> Key: NIFI-11627
> URL: https://issues.apache.org/jira/browse/NIFI-11627
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.19.1
>Reporter: Chuck Tilly
>Assignee: Daniel Stieglitz
>Priority: Major
>
> For the ValidateJSON processor, add support for flowfile attribute references
> that will allow for a JSON schema located in the Parameter Contexts, to be
> referenced dynamically based on a flowfile attribute. e.g.
> {code:java}
> #{${schema.name}} {code}
>
> The benefits of adding support for attribute references are significant.
> Adding this capability will allow a single processor to be used for all JSON
> schema validation. Unfortunately, the current version of this processor
> requires a dedicated processor for every schema, i.e. 12 schemas requires 12
> ValidateJSON processors. This is very laborious to construct and maintain,
> and resource expensive.
> ValidateJSON processor (https://issues.apache.org/jira/browse/NIFI-7392)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-11627) Add Dynamic Schema References to ValidateJSON Processor
Chuck Tilly created NIFI-11627:
--
Summary: Add Dynamic Schema References to ValidateJSON Processor
Key: NIFI-11627
URL: https://issues.apache.org/jira/browse/NIFI-11627
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions
Affects Versions: 1.19.1
Reporter: Chuck Tilly
For the ValidateJSON processor, add support for flowfile attribute references
that will allow for a JSON schema located in the Parameter Contexts, to be
referenced dynamically based on a flowfile attribute. e.g.
{code:java}
#{${schema.name}} {code}
The benefits of adding support for attribute references are significant.
Adding this capability will allow a single processor to be used for all JSON
schema validation. Unfortunately, the current version of this processor
requires a dedicated processor for every schema, i.e. 12 schemas requires 12
ValidateJSON processors. This is very laborious to construct and maintain, and
resource expensive.
ValidateJSON processor (https://issues.apache.org/jira/browse/NIFI-7392)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
