[jira] [Commented] (AVRO-2058) ReflectData#isNonStringMap returns true for Utf8 keys
[
https://issues.apache.org/jira/browse/AVRO-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217691#comment-16217691
]
Sam Schlegel commented on AVRO-2058:
What needs to be done to move this along?
> ReflectData#isNonStringMap returns true for Utf8 keys
> -
>
> Key: AVRO-2058
> URL: https://issues.apache.org/jira/browse/AVRO-2058
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.2
>Reporter: Sam Schlegel
>Assignee: Sam Schlegel
>Priority: Critical
> Attachments: AVRO-2058.patch
>
>
> Since {{Utf8}} does not have an {{Stringable}} notation, and is not in
> {{SpecificData#stringableClasses}}, {{ReflectData#isNonStringMap}} returns
> true. This also causes {{ReflectData#isArray}} to return true for maps with
> Utf8 keys, and thus {{GenericData#resolveUnion}} fails as well. This
> ultimately causes {{ReflectData#write}} to fail for schemas that contain a
> union that contains a map, where the data uses Utf8 for strings.
> This following test case reproduces the issue:
> {code:java}
> @Test public void testUnionWithMapWithUtf8Keys() {
> Schema s = new Schema.Parser().parse
> ("[\"null\", {\"type\":\"map\",\"values\":\"float\"}]");
> GenericData data = ReflectData.get();
> HashMap map = new HashMap();
> map.put(new Utf8("foo"), 1.0f);
> assertEquals(1, data.resolveUnion(s, map));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AVRO-2058) ReflectData#isNonStringMap returns true for Utf8 keys
[
https://issues.apache.org/jira/browse/AVRO-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Schlegel updated AVRO-2058:
---
Status: Patch Available (was: Open)
> ReflectData#isNonStringMap returns true for Utf8 keys
> -
>
> Key: AVRO-2058
> URL: https://issues.apache.org/jira/browse/AVRO-2058
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.2
>Reporter: Sam Schlegel
>Priority: Critical
> Attachments: AVRO-2058.patch
>
>
> Since {{Utf8}} does not have an {{Stringable}} notation, and is not in
> {{SpecificData#stringableClasses}}, {{ReflectData#isNonStringMap}} returns
> true. This also causes {{ReflectData#isArray}} to return true for maps with
> Utf8 keys, and thus {{GenericData#resolveUnion}} fails as well. This
> ultimately causes {{ReflectData#write}} to fail for schemas that contain a
> union that contains a map, where the data uses Utf8 for strings.
> This following test case reproduces the issue:
> {code:java}
> @Test public void testUnionWithMapWithUtf8Keys() {
> Schema s = new Schema.Parser().parse
> ("[\"null\", {\"type\":\"map\",\"values\":\"float\"}]");
> GenericData data = ReflectData.get();
> HashMap map = new HashMap();
> map.put(new Utf8("foo"), 1.0f);
> assertEquals(1, data.resolveUnion(s, map));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AVRO-2058) ReflectData#isNonStringMap returns true for Utf8 keys
[
https://issues.apache.org/jira/browse/AVRO-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Schlegel updated AVRO-2058:
---
Attachment: AVRO-2058.patch
> ReflectData#isNonStringMap returns true for Utf8 keys
> -
>
> Key: AVRO-2058
> URL: https://issues.apache.org/jira/browse/AVRO-2058
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.2
>Reporter: Sam Schlegel
>Priority: Critical
> Attachments: AVRO-2058.patch
>
>
> Since {{Utf8}} does not have an {{Stringable}} notation, and is not in
> {{SpecificData#stringableClasses}}, {{ReflectData#isNonStringMap}} returns
> true. This also causes {{ReflectData#isArray}} to return true for maps with
> Utf8 keys, and thus {{GenericData#resolveUnion}} fails as well. This
> ultimately causes {{ReflectData#write}} to fail for schemas that contain a
> union that contains a map, where the data uses Utf8 for strings.
> This following test case reproduces the issue:
> {code:java}
> @Test public void testUnionWithMapWithUtf8Keys() {
> Schema s = new Schema.Parser().parse
> ("[\"null\", {\"type\":\"map\",\"values\":\"float\"}]");
> GenericData data = ReflectData.get();
> HashMap map = new HashMap();
> map.put(new Utf8("foo"), 1.0f);
> assertEquals(1, data.resolveUnion(s, map));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AVRO-2058) ReflectData#isNonStringMap returns true for Utf8 keys
[
https://issues.apache.org/jira/browse/AVRO-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Schlegel updated AVRO-2058:
---
Description:
Since {{Utf8}} does not have an {{Stringable}} notation, and is not in
{{SpecificData#stringableClasses}}, {{ReflectData#isNonStringMap}} returns
true. This also causes {{ReflectData#isArray}} to return true for maps with
Utf8 keys, and thus {{GenericData#resolveUnion}} fails as well. This ultimately
causes {{ReflectData#write}} to fail for schemas that contain a union that
contains a map, where the data uses Utf8 for strings.
This following test case reproduces the issue:
{code:java}
@Test public void testUnionWithMapWithUtf8Keys() {
Schema s = new Schema.Parser().parse
("[\"null\", {\"type\":\"map\",\"values\":\"float\"}]");
GenericData data = ReflectData.get();
HashMap map = new HashMap();
map.put(new Utf8("foo"), 1.0f);
assertEquals(1, data.resolveUnion(s, map));
}
{code}
was:Since {{Utf8}} does not have an {{Stringable}} notation, and is not in
{{SpecificData#stringableClasses}}, {{ReflectData#isNonStringMap}} returns
true. This also causes {{ReflectData#isArray}} to return true for maps with
Utf8 keys, and thus {{GenericData#resolveUnion}} fails as well. This ultimately
causes {{ReflectData#write}} to fail for schemas that contain a union that
contains a map, where the data uses Utf8 for strings.
> ReflectData#isNonStringMap returns true for Utf8 keys
> -
>
> Key: AVRO-2058
> URL: https://issues.apache.org/jira/browse/AVRO-2058
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.2
>Reporter: Sam Schlegel
>Priority: Critical
>
> Since {{Utf8}} does not have an {{Stringable}} notation, and is not in
> {{SpecificData#stringableClasses}}, {{ReflectData#isNonStringMap}} returns
> true. This also causes {{ReflectData#isArray}} to return true for maps with
> Utf8 keys, and thus {{GenericData#resolveUnion}} fails as well. This
> ultimately causes {{ReflectData#write}} to fail for schemas that contain a
> union that contains a map, where the data uses Utf8 for strings.
> This following test case reproduces the issue:
> {code:java}
> @Test public void testUnionWithMapWithUtf8Keys() {
> Schema s = new Schema.Parser().parse
> ("[\"null\", {\"type\":\"map\",\"values\":\"float\"}]");
> GenericData data = ReflectData.get();
> HashMap map = new HashMap();
> map.put(new Utf8("foo"), 1.0f);
> assertEquals(1, data.resolveUnion(s, map));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-1855) Avro-mapred not evaluating map schema correctly when values are expected to be strings
[
https://issues.apache.org/jira/browse/AVRO-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102608#comment-16102608
]
Sam Schlegel commented on AVRO-1855:
Upon further inspection I believe might be due to AVRO-2058
> Avro-mapred not evaluating map schema correctly when values are expected to
> be strings
> --
>
> Key: AVRO-1855
> URL: https://issues.apache.org/jira/browse/AVRO-1855
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.0
>Reporter: Mikko Kupsu
>Priority: Critical
> Attachments: 20160530_AVRO-1855.patch
>
>
> When reading bunch of Avro file and concatenating them using avro-mapred,
> there is an issue with following schema definition line:
> {code}
> {"name": "headers", "type": ["null", {"type": "map", "values": "string"}]},
> {code}
> Below exceptions are thrown:
> {code}
> Caused by: org.apache.avro.UnresolvedUnionException: Not in union
> ["null",{"type":"map","values":"string"}]: {range=bytes=91553252-91557347,
> accept=*/*, response_status_code=206, host=108.175.39.172}
> at
> org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:709)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:150)
> at
> org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:153)
> at
> org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:182)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:150)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:302)
> {code}
> I've fixed this in my own [GitHub
> fork|https://github.com/mikkokupsu/avro/tree/hotfix/20160530/avro-schema-map-string-problem]
> and I've attached the patch too.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (AVRO-1855) Avro-mapred not evaluating map schema correctly when values are expected to be strings
[
https://issues.apache.org/jira/browse/AVRO-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102608#comment-16102608
]
Sam Schlegel edited comment on AVRO-1855 at 7/27/17 2:11 AM:
-
Upon further inspection I believe might be due to AVRO-2058.
was (Author: samschlegel):
Upon further inspection I believe might be due to AVRO-2058
> Avro-mapred not evaluating map schema correctly when values are expected to
> be strings
> --
>
> Key: AVRO-1855
> URL: https://issues.apache.org/jira/browse/AVRO-1855
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.0
>Reporter: Mikko Kupsu
>Priority: Critical
> Attachments: 20160530_AVRO-1855.patch
>
>
> When reading bunch of Avro file and concatenating them using avro-mapred,
> there is an issue with following schema definition line:
> {code}
> {"name": "headers", "type": ["null", {"type": "map", "values": "string"}]},
> {code}
> Below exceptions are thrown:
> {code}
> Caused by: org.apache.avro.UnresolvedUnionException: Not in union
> ["null",{"type":"map","values":"string"}]: {range=bytes=91553252-91557347,
> accept=*/*, response_status_code=206, host=108.175.39.172}
> at
> org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:709)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:150)
> at
> org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:153)
> at
> org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:182)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:150)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:302)
> {code}
> I've fixed this in my own [GitHub
> fork|https://github.com/mikkokupsu/avro/tree/hotfix/20160530/avro-schema-map-string-problem]
> and I've attached the patch too.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AVRO-2058) ReflectData#isNonStringMap returns true for Utf8 keys
[
https://issues.apache.org/jira/browse/AVRO-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Schlegel updated AVRO-2058:
---
Description: Since {{Utf8}} does not have an {{Stringable}} notation, and
is not in {{SpecificData#stringableClasses}}, {{ReflectData#isNonStringMap}}
returns true. This also causes {{ReflectData#isArray}} to return true for maps
with Utf8 keys, and thus {{GenericData#resolveUnion}} fails as well. This
ultimately causes {{ReflectData#write}} to fail for schemas that contain a
union that contains a map, where the data uses Utf8 for strings. (was: Since
{{org.apache.avro.util.Utf8}} does not have an
{{org.apache.reflect.Stringable}} notation, and is not in
{{org.apache.avro.specific.SpecificData#stringableClasses}},
{{ReflectData#isNonStringMap}} returns true. This also causes
{{ReflectData#isArray}} to return true for maps with Utf8 keys, and thus
{{GenericData#resolveUnion}} fails as well. This ultimately causes
{{ReflectData#write}} to fail for schemas that contain a union that contains a
map, where the data uses Utf8 for strings.)
Summary: ReflectData#isNonStringMap returns true for Utf8 keys (was:
ReflectData#isNonStringMap returns true for org.apache.avro.util.Utf8 keys)
> ReflectData#isNonStringMap returns true for Utf8 keys
> -
>
> Key: AVRO-2058
> URL: https://issues.apache.org/jira/browse/AVRO-2058
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.2
>Reporter: Sam Schlegel
>Priority: Critical
>
> Since {{Utf8}} does not have an {{Stringable}} notation, and is not in
> {{SpecificData#stringableClasses}}, {{ReflectData#isNonStringMap}} returns
> true. This also causes {{ReflectData#isArray}} to return true for maps with
> Utf8 keys, and thus {{GenericData#resolveUnion}} fails as well. This
> ultimately causes {{ReflectData#write}} to fail for schemas that contain a
> union that contains a map, where the data uses Utf8 for strings.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (AVRO-2058) ReflectData#isNonStringMap returns true for org.apache.avro.util.Utf8 keys
Sam Schlegel created AVRO-2058:
--
Summary: ReflectData#isNonStringMap returns true for
org.apache.avro.util.Utf8 keys
Key: AVRO-2058
URL: https://issues.apache.org/jira/browse/AVRO-2058
Project: Avro
Issue Type: Bug
Components: java
Affects Versions: 1.8.2
Reporter: Sam Schlegel
Priority: Critical
Since {{org.apache.avro.util.Utf8}} does not have an
{{org.apache.reflect.Stringable}} notation, and is not in
{{org.apache.avro.specific.SpecificData#stringableClasses}},
{{ReflectData#isNonStringMap}} returns true. This also causes
{{ReflectData#isArray}} to return true for maps with Utf8 keys, and thus
{{GenericData#resolveUnion}} fails as well. This ultimately causes
{{ReflectData#write}} to fail for schemas that contain a union that contains a
map, where the data uses Utf8 for strings.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-1855) Avro-mapred not evaluating map schema correctly when values are expected to be strings
[
https://issues.apache.org/jira/browse/AVRO-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102544#comment-16102544
]
Sam Schlegel commented on AVRO-1855:
I believe this related to AVRO-966 and is caused by the {{isArray}} check
coming before {{isMap}} in
{{org.apache.avro.generic.GenericData#getSchemaName}}. I'm not sure why, but
{{isArray(datum)}} is returning true when {{datum}} is a `java.util.HashMap`,
even though {{java.util.HashMap}} is not an instance of
{{java.util.Collection}} and {{datum instanceof Collection}} returns false when
evaluated in the debugger.
> Avro-mapred not evaluating map schema correctly when values are expected to
> be strings
> --
>
> Key: AVRO-1855
> URL: https://issues.apache.org/jira/browse/AVRO-1855
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.0
>Reporter: Mikko Kupsu
>Priority: Critical
> Attachments: 20160530_AVRO-1855.patch
>
>
> When reading bunch of Avro file and concatenating them using avro-mapred,
> there is an issue with following schema definition line:
> {code}
> {"name": "headers", "type": ["null", {"type": "map", "values": "string"}]},
> {code}
> Below exceptions are thrown:
> {code}
> Caused by: org.apache.avro.UnresolvedUnionException: Not in union
> ["null",{"type":"map","values":"string"}]: {range=bytes=91553252-91557347,
> accept=*/*, response_status_code=206, host=108.175.39.172}
> at
> org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:709)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:150)
> at
> org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:153)
> at
> org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:182)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:150)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:302)
> {code}
> I've fixed this in my own [GitHub
> fork|https://github.com/mikkokupsu/avro/tree/hotfix/20160530/avro-schema-map-string-problem]
> and I've attached the patch too.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
