[jira] [Commented] (NIFI-4441) Add MapRecord support inside avro union types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322867#comment-16322867
 ] 

ASF GitHub Bot commented on NIFI-4441:
--

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/2207
  
@frett27 @mattyb149 I think what is done in this PR is actually correct 
as-is. The issue you're running into, @mattyb149, I think is that the Avro 
Reader is returning a MapRecord object when it should in fact return a Map 
object (Any time a RecordFieldType is MAP, it should return a java.util.Map for 
the value). So I will address that as part of NIFI-4767. +1 merged to master.


> Add MapRecord support inside avro union types
> -
>
> Key: NIFI-4441
> URL: https://issues.apache.org/jira/browse/NIFI-4441
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Patrice Freydiere
>
> Using an avro union type that contain maps in the definition lead to errors 
> in loading avro records.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4441) Add MapRecord support inside avro union types

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322865#comment-16322865
 ] 

ASF subversion and git services commented on NIFI-4441:
---

Commit 5f7bd81af97523b6b25f206cd0810c148d8dcd4a in nifi's branch 
refs/heads/master from [~frett27]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=5f7bd81 ]

NIFI-4441: patch avro maps in union types. This closes #2207.

Signed-off-by: Mark Payne 


> Add MapRecord support inside avro union types
> -
>
> Key: NIFI-4441
> URL: https://issues.apache.org/jira/browse/NIFI-4441
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Patrice Freydiere
>
> Using an avro union type that contain maps in the definition lead to errors 
> in loading avro records.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4441) Add MapRecord support inside avro union types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322866#comment-16322866
 ] 

ASF GitHub Bot commented on NIFI-4441:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2207


> Add MapRecord support inside avro union types
> -
>
> Key: NIFI-4441
> URL: https://issues.apache.org/jira/browse/NIFI-4441
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Patrice Freydiere
>
> Using an avro union type that contain maps in the definition lead to errors 
> in loading avro records.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4441) Add MapRecord support inside avro union types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16207902#comment-16207902
 ] 

ASF GitHub Bot commented on NIFI-4441:
--

Github user frett27 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2207#discussion_r145189423
  
--- Diff: 
nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/test/java/org/apache/nifi/avro/TestAvroTypeUtil.java
 ---
@@ -239,4 +243,20 @@ public void testComplicatedRecursiveSchema() {
 Assert.assertEquals(recordASchema, 
((RecordDataType)recordBParentField.get().getDataType()).getChildSchema());
 }
 
+@Test
+public void testMapWithNullSchema() throws IOException {
+
+Schema recursiveSchema = new 
Schema.Parser().parse(getClass().getResourceAsStream("schema.json"));
+
+// Make sure the following doesn't throw an exception
+RecordSchema recordASchema = 
AvroTypeUtil.createSchema(recursiveSchema.getTypes().get(0));
+
+// check the fix with the proper file
+try(DataFileStream r = new 
DataFileStream<>(getClass().getResourceAsStream("data.avro"),
--- End diff --

OK let's change that to MapRecord, i'm sure it's reliable, because we use 
it on dev plateform, for the "instanceof Map" you suggest i'm not sure of the 
impact. 


> Add MapRecord support inside avro union types
> -
>
> Key: NIFI-4441
> URL: https://issues.apache.org/jira/browse/NIFI-4441
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Patrice Freydiere
>
> Using an avro union type that contain maps in the definition lead to errors 
> in loading avro records.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4441) Add MapRecord support inside avro union types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16207903#comment-16207903
 ] 

ASF GitHub Bot commented on NIFI-4441:
--

Github user frett27 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2207#discussion_r145189535
  
--- Diff: 
nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/test/java/org/apache/nifi/avro/TestAvroTypeUtil.java
 ---
@@ -239,4 +243,20 @@ public void testComplicatedRecursiveSchema() {
 Assert.assertEquals(recordASchema, 
((RecordDataType)recordBParentField.get().getDataType()).getChildSchema());
 }
 
+@Test
+public void testMapWithNullSchema() throws IOException {
+
+Schema recursiveSchema = new 
Schema.Parser().parse(getClass().getResourceAsStream("schema.json"));
+
+// Make sure the following doesn't throw an exception
+RecordSchema recordASchema = 
AvroTypeUtil.createSchema(recursiveSchema.getTypes().get(0));
+
+// check the fix with the proper file
+try(DataFileStream r = new 
DataFileStream<>(getClass().getResourceAsStream("data.avro"),
--- End diff --

and if you also tests it , it's OK


> Add MapRecord support inside avro union types
> -
>
> Key: NIFI-4441
> URL: https://issues.apache.org/jira/browse/NIFI-4441
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Patrice Freydiere
>
> Using an avro union type that contain maps in the definition lead to errors 
> in loading avro records.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4441) Add MapRecord support inside avro union types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16207881#comment-16207881
 ] 

ASF GitHub Bot commented on NIFI-4441:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2207#discussion_r145185126
  
--- Diff: 
nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/test/java/org/apache/nifi/avro/TestAvroTypeUtil.java
 ---
@@ -239,4 +243,20 @@ public void testComplicatedRecursiveSchema() {
 Assert.assertEquals(recordASchema, 
((RecordDataType)recordBParentField.get().getDataType()).getChildSchema());
 }
 
+@Test
+public void testMapWithNullSchema() throws IOException {
+
+Schema recursiveSchema = new 
Schema.Parser().parse(getClass().getResourceAsStream("schema.json"));
+
+// Make sure the following doesn't throw an exception
+RecordSchema recordASchema = 
AvroTypeUtil.createSchema(recursiveSchema.getTypes().get(0));
+
+// check the fix with the proper file
+try(DataFileStream r = new 
DataFileStream<>(getClass().getResourceAsStream("data.avro"),
--- End diff --

I have the following schema in my test flow: 
`{
 "type": "record",
 "name": "A","fields": [
   {"name": "a", "type": "string"},
   {"name": "c", "type": [ "null", {"type" : "map","values" : "string"} ] } 
 ]
}`

When I debug through the processor to AvroTypeUtil, I get a MapRecord as 
the type of "value", not Map. If it could be either, we could just check for 
either?


> Add MapRecord support inside avro union types
> -
>
> Key: NIFI-4441
> URL: https://issues.apache.org/jira/browse/NIFI-4441
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Patrice Freydiere
>
> Using an avro union type that contain maps in the definition lead to errors 
> in loading avro records.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4441) Add MapRecord support inside avro union types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16207882#comment-16207882
 ] 

ASF GitHub Bot commented on NIFI-4441:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2207#discussion_r145185287
  
--- Diff: 
nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/test/java/org/apache/nifi/avro/TestAvroTypeUtil.java
 ---
@@ -239,4 +243,20 @@ public void testComplicatedRecursiveSchema() {
 Assert.assertEquals(recordASchema, 
((RecordDataType)recordBParentField.get().getDataType()).getChildSchema());
 }
 
+@Test
+public void testMapWithNullSchema() throws IOException {
+
+Schema recursiveSchema = new 
Schema.Parser().parse(getClass().getResourceAsStream("schema.json"));
+
+// Make sure the following doesn't throw an exception
+RecordSchema recordASchema = 
AvroTypeUtil.createSchema(recursiveSchema.getTypes().get(0));
+
+// check the fix with the proper file
+try(DataFileStream r = new 
DataFileStream<>(getClass().getResourceAsStream("data.avro"),
--- End diff --

I can post my flow and sample Avro file if you'd like to see what I mean


> Add MapRecord support inside avro union types
> -
>
> Key: NIFI-4441
> URL: https://issues.apache.org/jira/browse/NIFI-4441
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Patrice Freydiere
>
> Using an avro union type that contain maps in the definition lead to errors 
> in loading avro records.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4441) Add MapRecord support inside avro union types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16207847#comment-16207847
 ] 

ASF GitHub Bot commented on NIFI-4441:
--

Github user frett27 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2207#discussion_r145175950
  
--- Diff: 
nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/main/java/org/apache/nifi/avro/AvroTypeUtil.java
 ---
@@ -691,6 +697,10 @@ private static boolean isCompatibleDataType(final 
Object value, final DataType d
 return true;
 }
 break;
+case MAP:
+if (value instanceof Map) {
--- End diff --

in debugging initially, the given value was MapRecord, 


> Add MapRecord support inside avro union types
> -
>
> Key: NIFI-4441
> URL: https://issues.apache.org/jira/browse/NIFI-4441
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Patrice Freydiere
>
> Using an avro union type that contain maps in the definition lead to errors 
> in loading avro records.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4441) Add MapRecord support inside avro union types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16207845#comment-16207845
 ] 

ASF GitHub Bot commented on NIFI-4441:
--

Github user frett27 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2207#discussion_r145175558
  
--- Diff: 
nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/test/java/org/apache/nifi/avro/TestAvroTypeUtil.java
 ---
@@ -239,4 +243,20 @@ public void testComplicatedRecursiveSchema() {
 Assert.assertEquals(recordASchema, 
((RecordDataType)recordBParentField.get().getDataType()).getChildSchema());
 }
 
+@Test
+public void testMapWithNullSchema() throws IOException {
+
+Schema recursiveSchema = new 
Schema.Parser().parse(getClass().getResourceAsStream("schema.json"));
+
+// Make sure the following doesn't throw an exception
+RecordSchema recordASchema = 
AvroTypeUtil.createSchema(recursiveSchema.getTypes().get(0));
+
+// check the fix with the proper file
+try(DataFileStream r = new 
DataFileStream<>(getClass().getResourceAsStream("data.avro"),
--- End diff --

my uderstanding was that the map presence cause the issue in union type,
 whether or not, the map is filled, (instance of map will be returned)


> Add MapRecord support inside avro union types
> -
>
> Key: NIFI-4441
> URL: https://issues.apache.org/jira/browse/NIFI-4441
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Patrice Freydiere
>
> Using an avro union type that contain maps in the definition lead to errors 
> in loading avro records.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4441) Add MapRecord support inside avro union types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16207844#comment-16207844
 ] 

ASF GitHub Bot commented on NIFI-4441:
--

Github user frett27 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2207#discussion_r145175249
  
--- Diff: 
nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/main/java/org/apache/nifi/avro/AvroTypeUtil.java
 ---
@@ -691,6 +697,10 @@ private static boolean isCompatibleDataType(final 
Object value, final DataType d
 return true;
 }
 break;
+case MAP:
+if (value instanceof Map) {
--- End diff --

The MapRecord was my initial post, you suggested to open to Map , type. 
the probleme was when the map is present in the union ("null", "record").


> Add MapRecord support inside avro union types
> -
>
> Key: NIFI-4441
> URL: https://issues.apache.org/jira/browse/NIFI-4441
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Patrice Freydiere
>
> Using an avro union type that contain maps in the definition lead to errors 
> in loading avro records.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4441) Add MapRecord support inside avro union types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16207793#comment-16207793
 ] 

ASF GitHub Bot commented on NIFI-4441:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2207#discussion_r145169320
  
--- Diff: 
nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/test/java/org/apache/nifi/avro/TestAvroTypeUtil.java
 ---
@@ -239,4 +243,20 @@ public void testComplicatedRecursiveSchema() {
 Assert.assertEquals(recordASchema, 
((RecordDataType)recordBParentField.get().getDataType()).getChildSchema());
 }
 
+@Test
+public void testMapWithNullSchema() throws IOException {
+
+Schema recursiveSchema = new 
Schema.Parser().parse(getClass().getResourceAsStream("schema.json"));
+
+// Make sure the following doesn't throw an exception
+RecordSchema recordASchema = 
AvroTypeUtil.createSchema(recursiveSchema.getTypes().get(0));
+
+// check the fix with the proper file
+try(DataFileStream r = new 
DataFileStream<>(getClass().getResourceAsStream("data.avro"),
--- End diff --

Also it could check the records to make sure the field values are what you 
expect (null vs not-null, e.g.)


> Add MapRecord support inside avro union types
> -
>
> Key: NIFI-4441
> URL: https://issues.apache.org/jira/browse/NIFI-4441
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Patrice Freydiere
>
> Using an avro union type that contain maps in the definition lead to errors 
> in loading avro records.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4441) Add MapRecord support inside avro union types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16207794#comment-16207794
 ] 

ASF GitHub Bot commented on NIFI-4441:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2207#discussion_r145169617
  
--- Diff: 
nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/main/java/org/apache/nifi/avro/AvroTypeUtil.java
 ---
@@ -691,6 +697,10 @@ private static boolean isCompatibleDataType(final 
Object value, final DataType d
 return true;
 }
 break;
+case MAP:
+if (value instanceof Map) {
--- End diff --

What kind of flow (which processors, e.g.) did you use to test this with? 
When I use ConvertRecord, value is a MapRecord not a Map, which causes this not 
to work. Perhaps we should check for both here?


> Add MapRecord support inside avro union types
> -
>
> Key: NIFI-4441
> URL: https://issues.apache.org/jira/browse/NIFI-4441
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Patrice Freydiere
>
> Using an avro union type that contain maps in the definition lead to errors 
> in loading avro records.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4441) Add MapRecord support inside avro union types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16207792#comment-16207792
 ] 

ASF GitHub Bot commented on NIFI-4441:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2207#discussion_r145169184
  
--- Diff: 
nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/test/java/org/apache/nifi/avro/TestAvroTypeUtil.java
 ---
@@ -239,4 +243,20 @@ public void testComplicatedRecursiveSchema() {
 Assert.assertEquals(recordASchema, 
((RecordDataType)recordBParentField.get().getDataType()).getChildSchema());
 }
 
+@Test
+public void testMapWithNullSchema() throws IOException {
+
+Schema recursiveSchema = new 
Schema.Parser().parse(getClass().getResourceAsStream("schema.json"));
+
+// Make sure the following doesn't throw an exception
+RecordSchema recordASchema = 
AvroTypeUtil.createSchema(recursiveSchema.getTypes().get(0));
+
+// check the fix with the proper file
+try(DataFileStream r = new 
DataFileStream<>(getClass().getResourceAsStream("data.avro"),
--- End diff --

Can you explain more about what's going on here, including what is in the 
data.avro file? When I run avro-tools tojson on it, I get the following:

```
java -jar avro-tools-1.8.1.jar tojson datasets/data.avro
{"a.A":{"o":{"a.O":{"hash":{"map":{}}
```

Perhaps it would be good to have a test file that has a record with a 
non-null value for hash, as well as a record with a null value for hash?


> Add MapRecord support inside avro union types
> -
>
> Key: NIFI-4441
> URL: https://issues.apache.org/jira/browse/NIFI-4441
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Patrice Freydiere
>
> Using an avro union type that contain maps in the definition lead to errors 
> in loading avro records.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4441) Add MapRecord support inside avro union types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16207686#comment-16207686
 ] 

ASF GitHub Bot commented on NIFI-4441:
--

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/2207
  
Reviewing...


> Add MapRecord support inside avro union types
> -
>
> Key: NIFI-4441
> URL: https://issues.apache.org/jira/browse/NIFI-4441
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Patrice Freydiere
>
> Using an avro union type that contain maps in the definition lead to errors 
> in loading avro records.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4441) Add MapRecord support inside avro union types

[Patrice Freydiere (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16204567#comment-16204567
 ] 

Patrice Freydiere commented on NIFI-4441:
-

any news ?

> Add MapRecord support inside avro union types
> -
>
> Key: NIFI-4441
> URL: https://issues.apache.org/jira/browse/NIFI-4441
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Patrice Freydiere
>
> Using an avro union type that contain maps in the definition lead to errors 
> in loading avro records.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4441) Add MapRecord support inside avro union types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200909#comment-16200909
 ] 

ASF GitHub Bot commented on NIFI-4441:
--

Github user frett27 commented on the issue:

https://github.com/apache/nifi/pull/2184
  
Thank's matt, new pull request associated : #2207, hope this will help the 
review


> Add MapRecord support inside avro union types
> -
>
> Key: NIFI-4441
> URL: https://issues.apache.org/jira/browse/NIFI-4441
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Patrice Freydiere
>
> Using an avro union type that contain maps in the definition lead to errors 
> in loading avro records.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4441) Add MapRecord support inside avro union types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200903#comment-16200903
 ] 

ASF GitHub Bot commented on NIFI-4441:
--

GitHub user frett27 opened a pull request:

https://github.com/apache/nifi/pull/2207

NIFI-4441 patch avro maps in union types

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x ] Does your PR title start with NIFI- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.

- [ x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x ] Have you written or updated unit tests to verify your changes?
- [x ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [x ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [x ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/frett27/nifi nifi-4441

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2207.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2207


commit 30b3596ac351405ea33d09b9737cece257d8ff54
Author: Patrice Freydiere 
Date:   2017-10-11T20:17:15Z

patch avro maps in union types




> Add MapRecord support inside avro union types
> -
>
> Key: NIFI-4441
> URL: https://issues.apache.org/jira/browse/NIFI-4441
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Patrice Freydiere
>
> Using an avro union type that contain maps in the definition lead to errors 
> in loading avro records.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4441) Add MapRecord support inside avro union types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200899#comment-16200899
 ] 

ASF GitHub Bot commented on NIFI-4441:
--

Github user frett27 closed the pull request at:

https://github.com/apache/nifi/pull/2184


> Add MapRecord support inside avro union types
> -
>
> Key: NIFI-4441
> URL: https://issues.apache.org/jira/browse/NIFI-4441
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Patrice Freydiere
>
> Using an avro union type that contain maps in the definition lead to errors 
> in loading avro records.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4441) Add MapRecord support inside avro union types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200605#comment-16200605
 ] 

ASF GitHub Bot commented on NIFI-4441:
--

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/2184
  
This branch looks out of whack with the current master, can you start with 
a fresh master and cherry-pick in your commits for this issue? It will help 
with merging and review, thanks!


> Add MapRecord support inside avro union types
> -
>
> Key: NIFI-4441
> URL: https://issues.apache.org/jira/browse/NIFI-4441
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Patrice Freydiere
>
> Using an avro union type that contain maps in the definition lead to errors 
> in loading avro records.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4441) Add MapRecord support inside avro union types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195060#comment-16195060
 ] 

ASF GitHub Bot commented on NIFI-4441:
--

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/2184
  
We cannot take a document/source/item from some place unless it has a 
clear/compatible license for it.  If there isn't one at all or isn't one that 
is favorable to our needs as an ASF project we cannot use it.  So either we use 
all originally created source or we cite it.  Does that make sense?


> Add MapRecord support inside avro union types
> -
>
> Key: NIFI-4441
> URL: https://issues.apache.org/jira/browse/NIFI-4441
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Patrice Freydiere
>
> Using an avro union type that contain maps in the definition lead to errors 
> in loading avro records.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4441) Add MapRecord support inside avro union types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190911#comment-16190911
 ] 

ASF GitHub Bot commented on NIFI-4441:
--

Github user frett27 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2184#discussion_r142597913
  
--- Diff: 
nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/test/java/org/apache/nifi/avro/TestAvroTypeUtil.java
 ---
@@ -238,5 +242,23 @@ public void testComplicatedRecursiveSchema() {
 // Make sure the 'parent' field has a schema reference back to the 
original top level record schema
 Assert.assertEquals(recordASchema, 
((RecordDataType)recordBParentField.get().getDataType()).getChildSchema());
 }
+
+@Test
+public void testMapWithNullSchema() throws IOException {
+
+Schema recursiveSchema = new Schema.Parser().parse(
+
"{\"type\":\"record\",\"name\":\"OSMEntity\",\"namespace\":\"org.osm.avro\",\"fields\":[{\"name\":\"osmtype\",\"type\":{\"type\":\"enum\",\"name\":\"OSMType\",\"symbols\":[\"NODE\",\"WAY\",\"POLYGON\",\"RELATION\"]}},{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"node\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"ANode\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"x\",\"type\":\"double\"},{\"name\":\"y\",\"type\":\"double\"},{\"name\":\"fields\",\"type\":[\"null\",{\"type\":\"map\",\"values\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"avro.java.string\":\"String\"}]}]}]},{\"name\":\"way\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"AComplex\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"geometry\",\"type\":\"bytes\"},{\"name\":\"fields\",\"type\":[\"null\",{\"type\":\"map\",\"values\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"avro.java.string\":\"String\"}]}]}]},{\"name\":\"polygon\",\"type\":[\"null\",\"AComplex\"]},{\"name\":\"rel\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"ARelation\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"fields\",\"type\":[\"null\",{\"type\":\"map\",\"values\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"avro.java.string\":\"String\"}]},{\"name\":\"related\",\"type\":[\"null\",{\"type\":\"array\",\"items\":{\"type\":\"record\",\"name\":\"ARelated\",\"fields\":[{\"name\":\"relatedId\",\"type\":\"long\"},{\"name\":\"type\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"}},{\"name\":\"role\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"}}]}}]}]}]}]}"
--- End diff --

OK


> Add MapRecord support inside avro union types
> -
>
> Key: NIFI-4441
> URL: https://issues.apache.org/jira/browse/NIFI-4441
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Patrice Freydiere
>
> Using an avro union type that contain maps in the definition lead to errors 
> in loading avro records.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4441) Add MapRecord support inside avro union types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190913#comment-16190913
 ] 

ASF GitHub Bot commented on NIFI-4441:
--

Github user frett27 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2184#discussion_r142597985
  
--- Diff: 
nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/main/java/org/apache/nifi/avro/AvroTypeUtil.java
 ---
@@ -691,6 +697,10 @@ private static boolean isCompatibleDataType(final 
Object value, final DataType d
 return true;
 }
 break;
+case MAP:
+if (value instanceof MapRecord) {
--- End diff --

OK, i'll have a dive on this


> Add MapRecord support inside avro union types
> -
>
> Key: NIFI-4441
> URL: https://issues.apache.org/jira/browse/NIFI-4441
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Patrice Freydiere
>
> Using an avro union type that contain maps in the definition lead to errors 
> in loading avro records.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4441) Add MapRecord support inside avro union types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190910#comment-16190910
 ] 

ASF GitHub Bot commented on NIFI-4441:
--

Github user frett27 commented on the issue:

https://github.com/apache/nifi/pull/2184
  
Hi, markap14, the definition is mine for the tests, and yes on top of the 
osm definition, i think there is no license associated issue. And yes could be 
more interesting for have a separate file for ease the reading.



> Add MapRecord support inside avro union types
> -
>
> Key: NIFI-4441
> URL: https://issues.apache.org/jira/browse/NIFI-4441
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Patrice Freydiere
>
> Using an avro union type that contain maps in the definition lead to errors 
> in loading avro records.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4441) Add MapRecord support inside avro union types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189935#comment-16189935
 ] 

ASF GitHub Bot commented on NIFI-4441:
--

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/2184
  
@frett27 thanks for your contribution! It looks like you're provided some 
test data that is used in a unit test. Any idea what the license would be on 
this test data? I'm guessing from looking at it that it was provided by 
osm.org? If licensing information is not available, it may make sense to just 
create some mock-up data instead?


> Add MapRecord support inside avro union types
> -
>
> Key: NIFI-4441
> URL: https://issues.apache.org/jira/browse/NIFI-4441
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Patrice Freydiere
>
> Using an avro union type that contain maps in the definition lead to errors 
> in loading avro records.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4441) Add MapRecord support inside avro union types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189940#comment-16189940
 ] 

ASF GitHub Bot commented on NIFI-4441:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2184#discussion_r142454463
  
--- Diff: 
nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/test/java/org/apache/nifi/avro/TestAvroTypeUtil.java
 ---
@@ -238,5 +242,23 @@ public void testComplicatedRecursiveSchema() {
 // Make sure the 'parent' field has a schema reference back to the 
original top level record schema
 Assert.assertEquals(recordASchema, 
((RecordDataType)recordBParentField.get().getDataType()).getChildSchema());
 }
+
+@Test
+public void testMapWithNullSchema() throws IOException {
+
+Schema recursiveSchema = new Schema.Parser().parse(
+
"{\"type\":\"record\",\"name\":\"OSMEntity\",\"namespace\":\"org.osm.avro\",\"fields\":[{\"name\":\"osmtype\",\"type\":{\"type\":\"enum\",\"name\":\"OSMType\",\"symbols\":[\"NODE\",\"WAY\",\"POLYGON\",\"RELATION\"]}},{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"node\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"ANode\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"x\",\"type\":\"double\"},{\"name\":\"y\",\"type\":\"double\"},{\"name\":\"fields\",\"type\":[\"null\",{\"type\":\"map\",\"values\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"avro.java.string\":\"String\"}]}]}]},{\"name\":\"way\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"AComplex\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"geometry\",\"type\":\"bytes\"},{\"name\":\"fields\",\"type\":[\"null\",{\"type\":\"map\",\"values\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"avro.java.string\":\"String\"}]}]}]},{\"name\":\"polygon\",\"type\":[\"null\",\"AComplex\"]},{\"name\":\"rel\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"ARelation\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"fields\",\"type\":[\"null\",{\"type\":\"map\",\"values\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"avro.java.string\":\"String\"}]},{\"name\":\"related\",\"type\":[\"null\",{\"type\":\"array\",\"items\":{\"type\":\"record\",\"name\":\"ARelated\",\"fields\":[{\"name\":\"relatedId\",\"type\":\"long\"},{\"name\":\"type\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"}},{\"name\":\"role\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"}}]}}]}]}]}]}"
--- End diff --

I think we'd be better off to externalize this into a .avsc file under 
src/test/resources


> Add MapRecord support inside avro union types
> -
>
> Key: NIFI-4441
> URL: https://issues.apache.org/jira/browse/NIFI-4441
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Patrice Freydiere
>
> Using an avro union type that contain maps in the definition lead to errors 
> in loading avro records.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4441) Add MapRecord support inside avro union types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185366#comment-16185366
 ] 

ASF GitHub Bot commented on NIFI-4441:
--

GitHub user frett27 opened a pull request:

https://github.com/apache/nifi/pull/2184

NIFI-4441 : add maprecord support for avro union types

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x ] Does your PR title start with NIFI- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.

- [x ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x ] Is your initial contribution a single, squashed commit?

### For code changes:
- [x ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ x] Have you written or updated unit tests to verify your changes?
- [ x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ x] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ x] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ x] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/frett27/nifi nifi-4441

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2184.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2184


commit d72a1c7d219d1cc12b0e61cb9e0644a6555dcf3c
Author: Patrice Freydiere 
Date:   2017-09-29T05:08:08Z

NIFI 4441 : add maprecord support for avro union types




> Add MapRecord support inside avro union types
> -
>
> Key: NIFI-4441
> URL: https://issues.apache.org/jira/browse/NIFI-4441
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Patrice Freydiere
>
> Using an avro union type that contain maps in the definition lead to errors 
> in loading avro records.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)