[
https://issues.apache.org/jira/browse/NIFI-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Kawamura updated NIFI-3921:
--------------------------------
Description:
Currently, any record writer has to retrieve schema from either SchemaRegistry
or Schema text. It doesn't work well with schema embedded Avro record as
incoming FlowFile, or used with any schema inferable readers such as CSVReader.
For example, ExecuteSQL outputs Avro record with schema embedded, and if user
wants to convert it to Json, they have to register the schema to SchemaRegistry.
I expect we don't have to touch registry in that use-case.
AvroReader has 'Use Embedded Avro Schema' option for 'Schema Access Strategy',
so processor such as ConvertRecord or QueryRecord can read record with embedded
schema, however since any writer doesn't have such option for 'Schema Access
Strategy', user have to register the schema in a SchemaRegistry for those
record operations to work properly.
I had to view a FlowFile which has embedded schema from NiFi content viewer and
copy the schema text then add it to registry. Plus extra UpdateAttribute is
needed to add 'schema.name' attribute. These extra work can be avoided.
If writers can utilize embedded schema, we can provide more seamless UX with
schema embedded data.
Probably we can use schema attached to record that Bryan Bende mentioned at
nifi-dev ML:
{quote}
When a reader produces a record it attaches the schema it used to the
record, but we currently don't have a way for a writer to use that
schema when writing a record, although I think we do want to support
that... something like a "Use Schema in Record" option as a choice in
the 'Schema Access Strategy' of writers.
{quote}
was:
Currently, any record writer has to retrieve schema from either SchemaRegistry
or Schema text. It doesn't work well with schema embedded Avro record as
incoming FlowFile.
For example, ExecuteSQL outputs Avro record with schema embedded, and if user
wants to convert it to Json, they have to register the schema to SchemaRegistry.
I expect we don't have to touch registry in that use-case.
AvroReader has 'Use Embedded Avro Schema' option for 'Schema Access Strategy',
so processor such as ConvertRecord or QueryRecord can read record with embedded
schema, however since any writer doesn't have such option for 'Schema Access
Strategy', user have to register the schema in a SchemaRegistry for those
record operations to work properly.
I had to view a FlowFile which has embedded schema from NiFi content viewer and
copy the schema text then add it to registry. Plus extra UpdateAttribute is
needed to add 'schema.name' attribute. These extra work can be avoided.
If writers can utilize embedded schema, we can provide more seamless UX with
schema embedded data.
> Utilize schema embedded in incoming FlowFile from record Writers
> ----------------------------------------------------------------
>
> Key: NIFI-3921
> URL: https://issues.apache.org/jira/browse/NIFI-3921
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Affects Versions: 1.2.0
> Reporter: Koji Kawamura
>
> Currently, any record writer has to retrieve schema from either
> SchemaRegistry or Schema text. It doesn't work well with schema embedded Avro
> record as incoming FlowFile, or used with any schema inferable readers such
> as CSVReader.
> For example, ExecuteSQL outputs Avro record with schema embedded, and if user
> wants to convert it to Json, they have to register the schema to
> SchemaRegistry.
> I expect we don't have to touch registry in that use-case.
> AvroReader has 'Use Embedded Avro Schema' option for 'Schema Access
> Strategy', so processor such as ConvertRecord or QueryRecord can read record
> with embedded schema, however since any writer doesn't have such option for
> 'Schema Access Strategy', user have to register the schema in a
> SchemaRegistry for those record operations to work properly.
> I had to view a FlowFile which has embedded schema from NiFi content viewer
> and copy the schema text then add it to registry. Plus extra UpdateAttribute
> is needed to add 'schema.name' attribute. These extra work can be avoided.
> If writers can utilize embedded schema, we can provide more seamless UX with
> schema embedded data.
> Probably we can use schema attached to record that Bryan Bende mentioned at
> nifi-dev ML:
> {quote}
> When a reader produces a record it attaches the schema it used to the
> record, but we currently don't have a way for a writer to use that
> schema when writing a record, although I think we do want to support
> that... something like a "Use Schema in Record" option as a choice in
> the 'Schema Access Strategy' of writers.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
