[ https://issues.apache.org/jira/browse/NIFI-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15325078#comment-15325078 ]
Daniel Cave edited comment on NIFI-1935 at 6/10/16 7:00 PM: ------------------------------------------------------------ I had to revisit the issues to remind myself of the cases/issues involved. Previously I had issues with it not properly reading from the attribute due to the way that attributes are interpreted when they are JSON (i.e. as a JSON string value with the element being the attribute name), however this may have been fixed. I retested it as of today based on 0.x, and ConvertJSONToAvro says it cannot find schema if you try to read it from an attribute (likely due to the same reason AttributesToJson fails to properly convert it, since it cant read avro.brinary and requires it to be a true string). Also, ConvertJSONToAvro doesn't register the data provenance, which is a bug and needs a ticket and fix for. However, the case for this processor is interrelated with use cases for InferAvroSchema (which also required the changes to SplitJSON). If I write the schema out as an attribute and the only use of that schema is to convert the json to avro, then you are correct that the existing processor is sufficient. However, writing the schema to an attribute presents other issues and limits its usefulness. My use cases for the schema also involve using the same schema to generate SQL/CQL/Hive/etc statements for ingestion as well as sending the avro to a schema registry and programatically creating RDDs. To do all this I need the schema as content in its proper JSON form. In theory, AttributesToJSON would do this, however it isn't language aware and will create JSON with { <attributeName> : <attributeValue> } where the attributeName is schema and the attributeValue would be the avro schema as a JSON string (there is a similar issue in using InvokeHTTP with the response as an attribute), however due to the way the schema exists in the attribute it actually returns an empty string (since the attribute is actually in avro.binary form). The processor seems to have been meant for simple attributes and not complex ones as putting an avro schema in one creates where the attribute contents itself is avro or JSON. EvaluateJsonPath is also an option, however again once you extract the avro schema from attribute you'll find that it's not in a proper format and isn't valid JSON or a valid schema anymore. Basically, there are four options to fix the issue: Do Infer twice (once to content and once to attribute, not desirable due to overhead ramifications on small devices with sub-second throughput and involves fixing the schema attribute issue), upgrade ConvertJSONToAvro to handle schemas from either content or attribute, make major changes to AttributeToJson or create a new complex version of it, or to split Convert into Convert and ConvertDynamic where one can accept the schema from a flowfile content (which has other use cases as well). I chose the latter as it created the least amount of backwards compatibility issues. That is not to say it is necessarily the best choice of the four in all cases, it was the lesser of evils for the community in my view for now. If you guys disagree or I've missed a better way to extract the schema then I'm certainly open to discussion and revisiting my design for dynamically handling everything from source to sink (any sink). Keep in mind, some of the sink processors require everything in flowfile content (PutCassandra), some hybrid (ExecuteSQL), and some all in attributes (PutHiveQL). So since those design inconsistent processors are already in public use, I have to be able to interpret the avro schema and create statements for any of the three in order to be able to handle the JSON and sink or transfer it into any source which means I need to be able to do varying kinds of parsing to it depending on the sink. Let me know what you think. Also, keep in mind in building this response to you I found three new bugs in at least three processors that need new tickets: evaluateAttributeExpressions() doesnt seem to be able to handle avro.binary (affects anything evaluating an attribute I assume, and may apply to normal blobs too), ConvertJsonToAvro isn't writing any data provenance on failure, InferAvroSchema writing JSON to attribute doesn't write in the right form. was (Author: daniel cave): I had to revisit the issues to remind myself of the cases/issues involved. Previously I had issues with it not properly reading from the attribute due to the way that attributes are interpreted when they are JSON (i.e. as a JSON string value with the element being the attribute name), however this may have been fixed. I retested it as of today based on 0.x, and ConvertJSONToAvro says it cannot find schema if you try to read it from an attribute (likely due to the same reason AttributesToJson fails to properly convert it, since it cant read avro.brinary and requires it to be a true string). Also, ConvertJSONToAvro doesn't register the data provenance, which is a bug and needs a ticket and fix for. However, the case for this processor is interrelated with use cases for InferAvroSchema (which also required the changes to SplitJSON). If I write the schema out as an attribute and the only use of that schema is to convert the json to avro, then you are correct that the existing processor is sufficient. However, writing the schema to an attribute presents other issues and limits its usefulness. My use cases for the schema also involve using the same schema to generate SQL/CQL/Hive/etc statements for ingestion as well as sending the avro to a schema registry and programatically creating RDDs. To do all this I need the schema as content in its proper JSON form. In theory, AttributesToJSON would do this, however it isn't language aware and will create JSON with { <attributeName> : <attributeValue> } where the attributeName is schema and the attributeValue would be the avro schema as a JSON string (there is a similar issue in using InvokeHTTP with the response as an attribute), however due to the way the schema exists in the attribute it actually returns an empty string (since the attribute is actually in avro.binary form). The processor seems to have been meant for simple attributes and not complex ones as putting an avro schema in one creates where the attribute contents itself is avro or JSON. EvaluateJsonPath is also an option, however again once you extract the avro schema from attribute you'll find that it's not in a proper format and isn't valid JSON or a valid schema anymore. Basically, there are four options to fix the issue: Do Infer twice (once to content and once to attribute, not desirable due to overhead ramifications on small devices with sub-second throughput and involves fixing the schema attribute issue), upgrade ConvertJSONToAvro to handle schemas from either content or attribute, make major changes to AttributeToJson or create a new complex version of it, or to split Convert into Convert and ConvertDynamic where one can accept the schema from a flowfile content (which has other use cases as well). I chose the latter as it created the least amount of backwards compatibility issues. That is not to say it is necessarily the best choice of the four in all cases, it was the lesser of evils for the community in my view for now. If you guys disagree or I've missed a better way to extract the schema then I'm certainly open to discussion and revisiting my design for dynamically handling everything from source to sink (any sink). Keep in mind, some of the sink processors require everything in flowfile content (PutCassandra), some hybrid (ExecuteSQL), and some all in attributes (PutHiveQL). So since those design inconsistent processors are already in public use, I have to be able to interpret the avro schema and create statements for any of the three in order to be able to handle the JSON and sink or transfer it into any source which means I need to be able to do varying kinds of parsing to it depending on the sink. Let me know what you think. Also, keep in mind in building this response to you I found three new bugs in at least three processors that need new tickets: evaluateAttributeExpressions() doesnt seem to be able to handle avro.binary (affects anything evaluating an attribute I assume, and may apply to normal blobs too), ConvertJsonToAvro is writing data provenance on failure, InferAvroSchema writing JSON to attribute doesn't write in the right form. > Added ConvertDynamicJsonToAvro processor > ---------------------------------------- > > Key: NIFI-1935 > URL: https://issues.apache.org/jira/browse/NIFI-1935 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions > Affects Versions: 1.0.0, 0.7.0 > Reporter: Daniel Cave > Assignee: Alex Halldin > Priority: Minor > Fix For: 1.0.0, 0.7.0 > > Attachments: > 0001-NIFI-1935-Added-ConvertDynamicJSONToAvro.java.-Added.patch > > > ConvertJsonToAvro required a predefined Avro schema to convert JSON and > required the presence of all field on the incoming JSON. > ConvertDynamicJsonToAvro functions similarly, however it now accepts the JSON > and schema as incoming flowfiles and creates the Avro dynamically. > This processor requires the InferAvroSchema processor in its upstream flow so > that it can use the original and schema flowfiles as input. These two > flowfiles will have the unique attribute inferredAvroId set on them by > InferAvroSchema so that they can be properly matched in > ConvertDynamicJsonToAvro. -- This message was sent by Atlassian JIRA (v6.3.4#6332)