[jira] [Commented] (AVRO-2805) Reflection-based schema not loading types
[
https://issues.apache.org/jira/browse/AVRO-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17087023#comment-17087023
]
Andy Le commented on AVRO-2805:
---
[~pcless] Yep. Be strong!
> Reflection-based schema not loading types
> --
>
> Key: AVRO-2805
> URL: https://issues.apache.org/jira/browse/AVRO-2805
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.1
>Reporter: Pedro Cardoso Silva
>Priority: Critical
>
> Avro reflection is unable to generate a schema for the following definition:
> {code:java}
> public class Definition {
> public Map
> }
> public enum Type {
> A,
> B,
> C
> }
> {code}
> {code:java}
> // Test code
> Schema schema = ReflectData.get().getSchema(Definition.class)
> {code}
> Fails with:
> Undefined name: "FieldType"
> org.apache.avro.SchemaParseException: Undefined name: "Type"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2808) Java: ReflectData incorrectly handles hidden fields
[
https://issues.apache.org/jira/browse/AVRO-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17087022#comment-17087022
]
Andy Le commented on AVRO-2808:
---
I think we can ignore fields `this$0`. Should I make a PR against this issue?
> Java: ReflectData incorrectly handles hidden fields
> ---
>
> Key: AVRO-2808
> URL: https://issues.apache.org/jira/browse/AVRO-2808
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.2
>Reporter: Andy Le
>Priority: Major
>
> Hi guys,
> I've got the following test:
> {code:java}
> public ReflectTest{
> public class Definition {
> public Map tokens;
> }
> public enum Type {
> A,
> B,
> C
> }
> @Test
> public void testAvro2805() {
> Schema schema = ReflectData.get().getSchema(Definition.class);
> final String schemaString = schema.toString(true);
> System.out.println(schemaString);
> }
> }
> {code}
> When I ran the test, an exception is raised:
> {noformat}
> org.apache.avro.SchemaParseException: Illegal character in: this$0
> at org.apache.avro.Schema.validateName(Schema.java:1530)
> at org.apache.avro.Schema.access$400(Schema.java:87)
> at org.apache.avro.Schema$Field.(Schema.java:518)
> at org.apache.avro.Schema$Field.(Schema.java:557)
> at
> org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:683)
> at
> org.apache.avro.specific.SpecificData$3.computeValue(SpecificData.java:335)
> at
> org.apache.avro.specific.SpecificData$3.computeValue(SpecificData.java:332)
> {noformat}
> From the log, you may see that: we should not handle Java hidden field
> `this$0` which is used to hold reference to outer class instances.
> This issue is somehow related to [this
> one|https://issues.apache.org/jira/browse/AVRO-2805]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2805) Reflection-based schema not loading types
[
https://issues.apache.org/jira/browse/AVRO-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086921#comment-17086921
]
Pedro Cardoso Silva commented on AVRO-2805:
---
I can not reproduce the issue now :D
For curiosity, this is the full test code (the only difference to my original
case was that the enum and class being serialized are in separate files):
{code:java}
import io.confluent.kafka.schemaregistry.client.MockSchemaRegistryClient
import io.confluent.kafka.schemaregistry.client.SchemaRegistryClient
import io.confluent.kafka.serializers.KafkaAvroDeserializer
import io.confluent.kafka.serializers.KafkaAvroDeserializerConfig
import io.confluent.kafka.serializers.KafkaAvroSerializer
import org.apache.avro.reflect.ReflectData
import org.junit.jupiter.api.Assertions.assertEquals
import org.junit.jupiter.api.Test
import java.util.HashMap
class ReflectionAvroSerDeTest {
private val schemaRegistry: SchemaRegistryClient =
MockSchemaRegistryClient()
private val reflectionAvroDeserializer: KafkaAvroDeserializer
private val reflectionAvroSerializer: KafkaAvroSerializer
private val topic = "test";
init {
val reflectionProps = HashMap()
// Intentionally invalid schema registry URL to satisfy the config
class's requirement that
// it be set.
// Intentionally invalid schema registry URL to satisfy the config
class's requirement that
// it be set.
reflectionProps[KafkaAvroDeserializerConfig.SCHEMA_REGISTRY_URL_CONFIG]
= "bogus"
reflectionProps[KafkaAvroDeserializerConfig.SCHEMA_REFLECTION_CONFIG] =
"true"
reflectionAvroSerializer = KafkaAvroSerializer(schemaRegistry,
reflectionProps)
reflectionAvroDeserializer = KafkaAvroDeserializer(schemaRegistry,
reflectionProps)
}
@Test
fun definitionTest() {
val definition = Definition(mapOf("a" to Type.A))
val avroSchema = ReflectData.get().getSchema(Definition::class.java)
val bytes = reflectionAvroSerializer.serialize(topic, definition)
val obj = reflectionAvroDeserializer.deserialize(topic, bytes,
avroSchema)
assertEquals(
Definition::class.java,
obj.javaClass,
"Returned object should be a Definition type"
)
assertEquals(definition, obj)
}
class Definition(val tokens: Map = mapOf()) {
}
enum class Type {
A, B, C
}
}
{code}
I am now wondering whether the original issue was not related to Avro but to
Kafka's Avro Serialization logic.
Either way, thank you for your time, Andy.
Have a great weekend.
> Reflection-based schema not loading types
> --
>
> Key: AVRO-2805
> URL: https://issues.apache.org/jira/browse/AVRO-2805
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.1
>Reporter: Pedro Cardoso Silva
>Priority: Critical
>
> Avro reflection is unable to generate a schema for the following definition:
> {code:java}
> public class Definition {
> public Map
> }
> public enum Type {
> A,
> B,
> C
> }
> {code}
> {code:java}
> // Test code
> Schema schema = ReflectData.get().getSchema(Definition.class)
> {code}
> Fails with:
> Undefined name: "FieldType"
> org.apache.avro.SchemaParseException: Undefined name: "Type"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2775) JacksonUtils: exception when calling toJsonNode()
[
https://issues.apache.org/jira/browse/AVRO-2775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086919#comment-17086919
]
Andy Le commented on AVRO-2775:
---
[~cutting] any other feedback?
> JacksonUtils: exception when calling toJsonNode()
> --
>
> Key: AVRO-2775
> URL: https://issues.apache.org/jira/browse/AVRO-2775
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.2
>Reporter: Andy Le
>Priority: Major
>
> I've got a simple test as followed
> {code:java}
> public class TestJacksonUtils {
> public static class Age{
> public int value = 9;
> }
> @Test
> public void testToJson(){
> Map kv = new HashMap<>();
> kv.put("age", 9);
> JsonNode node1 = JacksonUtils.toJsonNode(kv); // -> This is OK
> Object obj = new Age();
> JsonNode node2 = JacksonUtils.toJsonNode(obj); // -> This will trigger an
> exception
> }
> }
> {code}
> When I ran the test:
> {noformat}
> org.apache.avro.AvroRuntimeException: Unknown datum class: class
> org.apache.avro.util.internal.TestJacksonUtils$Age
> at
> org.apache.avro.util.internal.JacksonUtils.toJson(JacksonUtils.java:87)
> at
> org.apache.avro.util.internal.JacksonUtils.toJsonNode(JacksonUtils.java:48)
> at
> org.apache.avro.util.internal.TestJacksonUtils.testToJson(TestJacksonUtils.java:20)
> {noformat}
> I've read the code & tests for JacksonUtils. Instead of raising exceptions at
> [line
> #87|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/util/internal/JacksonUtils.java#L87],
> I see we can auto convert objects into maps, every thing's gonna fine.
> My question is:
> - Is raising exception acceptable?
> - Any other way to have `toJsonNode` for general objects?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (AVRO-2808) Java: ReflectData incorrectly handles hidden fields
Andy Le created AVRO-2808:
-
Summary: Java: ReflectData incorrectly handles hidden fields
Key: AVRO-2808
URL: https://issues.apache.org/jira/browse/AVRO-2808
Project: Apache Avro
Issue Type: Bug
Components: java
Reporter: Andy Le
Hi guys,
I've got the following test:
{code:java}
public ReflectTest{
public class Definition {
public Map tokens;
}
public enum Type {
A,
B,
C
}
@Test
public void testAvro2805() {
Schema schema = ReflectData.get().getSchema(Definition.class);
final String schemaString = schema.toString(true);
System.out.println(schemaString);
}
}
{code}
When I ran the test, an exception is raised:
{noformat}
org.apache.avro.SchemaParseException: Illegal character in: this$0
at org.apache.avro.Schema.validateName(Schema.java:1530)
at org.apache.avro.Schema.access$400(Schema.java:87)
at org.apache.avro.Schema$Field.(Schema.java:518)
at org.apache.avro.Schema$Field.(Schema.java:557)
at
org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:683)
at
org.apache.avro.specific.SpecificData$3.computeValue(SpecificData.java:335)
at
org.apache.avro.specific.SpecificData$3.computeValue(SpecificData.java:332)
{noformat}
>From the log, you may see that: we should not handle Java hidden field
>`this$0` which is used to hold reference to outer class instances.
This issue is somehow related to [this
one|https://issues.apache.org/jira/browse/AVRO-2805]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AVRO-2808) Java: ReflectData incorrectly handles hidden fields
[
https://issues.apache.org/jira/browse/AVRO-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andy Le updated AVRO-2808:
--
Affects Version/s: 1.9.2
> Java: ReflectData incorrectly handles hidden fields
> ---
>
> Key: AVRO-2808
> URL: https://issues.apache.org/jira/browse/AVRO-2808
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.2
>Reporter: Andy Le
>Priority: Major
>
> Hi guys,
> I've got the following test:
> {code:java}
> public ReflectTest{
> public class Definition {
> public Map tokens;
> }
> public enum Type {
> A,
> B,
> C
> }
> @Test
> public void testAvro2805() {
> Schema schema = ReflectData.get().getSchema(Definition.class);
> final String schemaString = schema.toString(true);
> System.out.println(schemaString);
> }
> }
> {code}
> When I ran the test, an exception is raised:
> {noformat}
> org.apache.avro.SchemaParseException: Illegal character in: this$0
> at org.apache.avro.Schema.validateName(Schema.java:1530)
> at org.apache.avro.Schema.access$400(Schema.java:87)
> at org.apache.avro.Schema$Field.(Schema.java:518)
> at org.apache.avro.Schema$Field.(Schema.java:557)
> at
> org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:683)
> at
> org.apache.avro.specific.SpecificData$3.computeValue(SpecificData.java:335)
> at
> org.apache.avro.specific.SpecificData$3.computeValue(SpecificData.java:332)
> {noformat}
> From the log, you may see that: we should not handle Java hidden field
> `this$0` which is used to hold reference to outer class instances.
> This issue is somehow related to [this
> one|https://issues.apache.org/jira/browse/AVRO-2805]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2808) Java: ReflectData incorrectly handles hidden fields
[
https://issues.apache.org/jira/browse/AVRO-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086918#comment-17086918
]
Andy Le commented on AVRO-2808:
---
[~cutting] [~sekikn] PATL
> Java: ReflectData incorrectly handles hidden fields
> ---
>
> Key: AVRO-2808
> URL: https://issues.apache.org/jira/browse/AVRO-2808
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.2
>Reporter: Andy Le
>Priority: Major
>
> Hi guys,
> I've got the following test:
> {code:java}
> public ReflectTest{
> public class Definition {
> public Map tokens;
> }
> public enum Type {
> A,
> B,
> C
> }
> @Test
> public void testAvro2805() {
> Schema schema = ReflectData.get().getSchema(Definition.class);
> final String schemaString = schema.toString(true);
> System.out.println(schemaString);
> }
> }
> {code}
> When I ran the test, an exception is raised:
> {noformat}
> org.apache.avro.SchemaParseException: Illegal character in: this$0
> at org.apache.avro.Schema.validateName(Schema.java:1530)
> at org.apache.avro.Schema.access$400(Schema.java:87)
> at org.apache.avro.Schema$Field.(Schema.java:518)
> at org.apache.avro.Schema$Field.(Schema.java:557)
> at
> org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:683)
> at
> org.apache.avro.specific.SpecificData$3.computeValue(SpecificData.java:335)
> at
> org.apache.avro.specific.SpecificData$3.computeValue(SpecificData.java:332)
> {noformat}
> From the log, you may see that: we should not handle Java hidden field
> `this$0` which is used to hold reference to outer class instances.
> This issue is somehow related to [this
> one|https://issues.apache.org/jira/browse/AVRO-2805]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2805) Reflection-based schema not loading types
[
https://issues.apache.org/jira/browse/AVRO-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086916#comment-17086916
]
Andy Le commented on AVRO-2805:
---
[~pcless] I'm just a contributor :-P Can't close your issue.
But feel free to show your Kotlin snippet. I think there's nothing specific to
Kotlin here.
> Reflection-based schema not loading types
> --
>
> Key: AVRO-2805
> URL: https://issues.apache.org/jira/browse/AVRO-2805
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.1
>Reporter: Pedro Cardoso Silva
>Priority: Critical
>
> Avro reflection is unable to generate a schema for the following definition:
> {code:java}
> public class Definition {
> public Map
> }
> public enum Type {
> A,
> B,
> C
> }
> {code}
> {code:java}
> // Test code
> Schema schema = ReflectData.get().getSchema(Definition.class)
> {code}
> Fails with:
> Undefined name: "FieldType"
> org.apache.avro.SchemaParseException: Undefined name: "Type"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2805) Reflection-based schema not loading types
[
https://issues.apache.org/jira/browse/AVRO-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086913#comment-17086913
]
Pedro Cardoso Silva commented on AVRO-2805:
---
The original code is in Kotlin as a data class & enum class, it seems I can not
reproduce the issue in Java sample test.
I will investigate further, you may close this issue in the meantime.
> Reflection-based schema not loading types
> --
>
> Key: AVRO-2805
> URL: https://issues.apache.org/jira/browse/AVRO-2805
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.1
>Reporter: Pedro Cardoso Silva
>Priority: Critical
>
> Avro reflection is unable to generate a schema for the following definition:
> {code:java}
> public class Definition {
> public Map
> }
> public enum Type {
> A,
> B,
> C
> }
> {code}
> {code:java}
> // Test code
> Schema schema = ReflectData.get().getSchema(Definition.class)
> {code}
> Fails with:
> Undefined name: "FieldType"
> org.apache.avro.SchemaParseException: Undefined name: "Type"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AVRO-2805) Reflection-based schema not loading types
[
https://issues.apache.org/jira/browse/AVRO-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pedro Cardoso Silva updated AVRO-2805:
--
Description:
Avro reflection is unable to generate a schema for the following definition:
{code:java}
public class Definition {
public Map
}
public enum Type {
A,
B,
C
}
{code}
{code:java}
// Test code
Schema schema = ReflectData.get().getSchema(Definition.class)
{code}
Fails with:
Undefined name: "FieldType"
org.apache.avro.SchemaParseException: Undefined name: "Type"
was:
Avro reflection is unable to generate a schema for the following definition:
{code:java}
public class Definition {
pubilc Map
}
public enum Type {
A,
B,
C
}
{code}
{code:java}
// Test code
Schema schema = ReflectData.get().getSchema(Definition.class)
{code}
Fails with:
Undefined name: "FieldType"
org.apache.avro.SchemaParseException: Undefined name: "Type"
> Reflection-based schema not loading types
> --
>
> Key: AVRO-2805
> URL: https://issues.apache.org/jira/browse/AVRO-2805
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.1
>Reporter: Pedro Cardoso Silva
>Priority: Critical
>
> Avro reflection is unable to generate a schema for the following definition:
> {code:java}
> public class Definition {
> public Map
> }
> public enum Type {
> A,
> B,
> C
> }
> {code}
> {code:java}
> // Test code
> Schema schema = ReflectData.get().getSchema(Definition.class)
> {code}
> Fails with:
> Undefined name: "FieldType"
> org.apache.avro.SchemaParseException: Undefined name: "Type"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2805) Reflection-based schema not loading types
[
https://issues.apache.org/jira/browse/AVRO-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086869#comment-17086869
]
Andy Le commented on AVRO-2805:
---
[~pcless] Your example code contain errors. I decided to wrap your logic inside
a test
{code:java}
public static class Definition {
public Map tokens;
}
public enum Type {
A,
B,
C
}
@Test
public void testAvro2805() {
Schema schema = ReflectData.get().getSchema(Definition.class);
final String schemaString = schema.toString(true);
System.out.println(schemaString);
}
{code}
I can NOT see your exception above. I've got this one instead:
{noformat}
org.apache.avro.SchemaParseException: Illegal character in: this$0
at org.apache.avro.Schema.validateName(Schema.java:1530)
at org.apache.avro.Schema.access$400(Schema.java:87)
at org.apache.avro.Schema$Field.(Schema.java:518)
at org.apache.avro.Schema$Field.(Schema.java:557)
at
org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:683)
{noformat}
Please kindly verify your issue again and let me know.
> Reflection-based schema not loading types
> --
>
> Key: AVRO-2805
> URL: https://issues.apache.org/jira/browse/AVRO-2805
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.1
>Reporter: Pedro Cardoso Silva
>Priority: Critical
>
> Avro reflection is unable to generate a schema for the following definition:
> {code:java}
> public class Definition {
> pubilc Map
> }
> public enum Type {
> A,
> B,
> C
> }
> {code}
> {code:java}
> // Test code
> Schema schema = ReflectData.get().getSchema(Definition.class)
> {code}
> Fails with:
> Undefined name: "FieldType"
> org.apache.avro.SchemaParseException: Undefined name: "Type"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
