[jira] [Commented] (AVRO-2723) Avro Java: Obtaining default values for POJO objects with ReflectData
[
https://issues.apache.org/jira/browse/AVRO-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17054340#comment-17054340
]
Andy Le commented on AVRO-2723:
---
[~rskraba] [~sekikn] I follow [this
guideline|https://cwiki.apache.org/confluence/display/AVRO/How+To+Contribute]
to contribute code. But when I issue commmand
{code:bash}
$ mvn javadoc:aggregate
{code}
I see various errors in other classes (not ReflectData). What should I do? Make
another PR to improve the documentation, right?
!Screen Shot 2020-03-08 at 16.13.29.png!
> Avro Java: Obtaining default values for POJO objects with ReflectData
> -
>
> Key: AVRO-2723
> URL: https://issues.apache.org/jira/browse/AVRO-2723
> Project: Apache Avro
> Issue Type: New Feature
> Components: java
>Affects Versions: 1.9.1
>Reporter: Andy Le
>Priority: Critical
> Attachments: Screen Shot 2020-03-08 at 16.13.29.png
>
>
> Hi guys,
>
> I've got a simple app using Avro Reflection:
>
> {code:java}
> public class App {
> public static void main(String[] args) {
> testReflection();
> }
> static class User {
> public String first = "Andy";
> public String last = "Le";
> }
> static void testReflection(){
> // get the reflected schema for packets
> Schema schema = ReflectData.AllowNull.get().getSchema(User.class);
> System.out.println(schema.toString(true));
> }
> {code}
> The output on console will be:
> {noformat}
> {
> "type" : "record",
> "name" : "User",
> "namespace" : "App",
> "fields" : [ {
> "name" : "first",
> "type" : [ "null", "string" ],
> "default" : null
> }, {
> "name" : "last",
> "type" : [ "null", "string" ],
> "default" : null
> } ]
> }
> {noformat}
>
> As you can see, there's no default values for fields. Would you please tell
> me how to obtain such values?
> Thank you.
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2723) Avro Java: Obtaining default values for POJO objects with ReflectData
[
https://issues.apache.org/jira/browse/AVRO-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17054334#comment-17054334
]
Andy Le commented on AVRO-2723:
---
[~rskraba] [~sekikn] Currently we have class `ReflectData.AllowNulls` declared
[at line
#175|https://github.com/apache/avro/blob/791ec6017b03e3cc79fbf546dd9d79d20d476b4e/lang/java/avro/src/main/java/org/apache/avro/reflect/ReflectData.java#L75].
Regarding this issue, should I add implementation for
`ReflectData.AllowDefaults` (please see class DefaultReflector above) ?
> Avro Java: Obtaining default values for POJO objects with ReflectData
> -
>
> Key: AVRO-2723
> URL: https://issues.apache.org/jira/browse/AVRO-2723
> Project: Apache Avro
> Issue Type: New Feature
> Components: java
>Affects Versions: 1.9.1
>Reporter: Andy Le
>Priority: Critical
>
> Hi guys,
>
> I've got a simple app using Avro Reflection:
>
> {code:java}
> public class App {
> public static void main(String[] args) {
> testReflection();
> }
> static class User {
> public String first = "Andy";
> public String last = "Le";
> }
> static void testReflection(){
> // get the reflected schema for packets
> Schema schema = ReflectData.AllowNull.get().getSchema(User.class);
> System.out.println(schema.toString(true));
> }
> {code}
> The output on console will be:
> {noformat}
> {
> "type" : "record",
> "name" : "User",
> "namespace" : "App",
> "fields" : [ {
> "name" : "first",
> "type" : [ "null", "string" ],
> "default" : null
> }, {
> "name" : "last",
> "type" : [ "null", "string" ],
> "default" : null
> } ]
> }
> {noformat}
>
> As you can see, there's no default values for fields. Would you please tell
> me how to obtain such values?
> Thank you.
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2723) Avro Java: Obtaining default values for POJO objects with ReflectData
[
https://issues.apache.org/jira/browse/AVRO-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17054327#comment-17054327
]
Andy Le commented on AVRO-2723:
---
[~sekikn] Thank you. I'll use the suggested name
> Avro Java: Obtaining default values for POJO objects with ReflectData
> -
>
> Key: AVRO-2723
> URL: https://issues.apache.org/jira/browse/AVRO-2723
> Project: Apache Avro
> Issue Type: New Feature
> Components: java
>Affects Versions: 1.9.1
>Reporter: Andy Le
>Priority: Critical
>
> Hi guys,
>
> I've got a simple app using Avro Reflection:
>
> {code:java}
> public class App {
> public static void main(String[] args) {
> testReflection();
> }
> static class User {
> public String first = "Andy";
> public String last = "Le";
> }
> static void testReflection(){
> // get the reflected schema for packets
> Schema schema = ReflectData.AllowNull.get().getSchema(User.class);
> System.out.println(schema.toString(true));
> }
> {code}
> The output on console will be:
> {noformat}
> {
> "type" : "record",
> "name" : "User",
> "namespace" : "App",
> "fields" : [ {
> "name" : "first",
> "type" : [ "null", "string" ],
> "default" : null
> }, {
> "name" : "last",
> "type" : [ "null", "string" ],
> "default" : null
> } ]
> }
> {noformat}
>
> As you can see, there's no default values for fields. Would you please tell
> me how to obtain such values?
> Thank you.
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2723) Avro Java: Obtaining default values for POJO objects with ReflectData
[
https://issues.apache.org/jira/browse/AVRO-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17052619#comment-17052619
]
Kengo Seki commented on AVRO-2723:
--
[~anhldbk],
bq. But if without refactoring ReflectData, where we will put this function?
I think Ryan is not against refactoring ReflectData, but just suggested a
better method name (createSchemaDefaultValue rather than getDefaultValue).
(Correct me if I misunderstood [~rskraba])
So I think you can submit a PR. Your proposal sounds reasonable to me.
BTW,
bq. will trigger an exception:
You have to put double quotes into the default value, as Ryan showed above:
{code}
package com.example;
import org.apache.avro.Schema;
import org.apache.avro.reflect.AvroDefault;
import org.apache.avro.reflect.ReflectData;
public class App {
public static void main(String[] args) {
Schema schema = ReflectData.get().getSchema(User.class);
System.out.println(schema.toString());
}
}
class User {
@AvroDefault("\"Andy\"")
public String first;
}
{code}
{code}
{"type":"record","name":"User","namespace":"com.example","fields":[{"name":"first","type":"string","default":"Andy"}]}
{code}
> Avro Java: Obtaining default values for POJO objects with ReflectData
> -
>
> Key: AVRO-2723
> URL: https://issues.apache.org/jira/browse/AVRO-2723
> Project: Apache Avro
> Issue Type: New Feature
> Components: java
>Affects Versions: 1.9.1
>Reporter: Andy Le
>Priority: Critical
>
> Hi guys,
>
> I've got a simple app using Avro Reflection:
>
> {code:java}
> public class App {
> public static void main(String[] args) {
> testReflection();
> }
> static class User {
> public String first = "Andy";
> public String last = "Le";
> }
> static void testReflection(){
> // get the reflected schema for packets
> Schema schema = ReflectData.AllowNull.get().getSchema(User.class);
> System.out.println(schema.toString(true));
> }
> {code}
> The output on console will be:
> {noformat}
> {
> "type" : "record",
> "name" : "User",
> "namespace" : "App",
> "fields" : [ {
> "name" : "first",
> "type" : [ "null", "string" ],
> "default" : null
> }, {
> "name" : "last",
> "type" : [ "null", "string" ],
> "default" : null
> } ]
> }
> {noformat}
>
> As you can see, there's no default values for fields. Would you please tell
> me how to obtain such values?
> Thank you.
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2723) Avro Java: Obtaining default values for POJO objects with ReflectData
[
https://issues.apache.org/jira/browse/AVRO-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048534#comment-17048534
]
Andy Le commented on AVRO-2723:
---
[~rskraba] Hi. Would you please tell me how to resolve this issue? Any
recommendation? I'm thinking of creating a PR to address this issue. But I
think we need a proper discussion first.
> Avro Java: Obtaining default values for POJO objects with ReflectData
> -
>
> Key: AVRO-2723
> URL: https://issues.apache.org/jira/browse/AVRO-2723
> Project: Apache Avro
> Issue Type: New Feature
> Components: java
>Affects Versions: 1.9.1
>Reporter: Andy Le
>Priority: Critical
>
> Hi guys,
>
> I've got a simple app using Avro Reflection:
>
> {code:java}
> public class App {
> public static void main(String[] args) {
> testReflection();
> }
> static class User {
> public String first = "Andy";
> public String last = "Le";
> }
> static void testReflection(){
> // get the reflected schema for packets
> Schema schema = ReflectData.AllowNull.get().getSchema(User.class);
> System.out.println(schema.toString(true));
> }
> {code}
> The output on console will be:
> {noformat}
> {
> "type" : "record",
> "name" : "User",
> "namespace" : "App",
> "fields" : [ {
> "name" : "first",
> "type" : [ "null", "string" ],
> "default" : null
> }, {
> "name" : "last",
> "type" : [ "null", "string" ],
> "default" : null
> } ]
> }
> {noformat}
>
> As you can see, there's no default values for fields. Would you please tell
> me how to obtain such values?
> Thank you.
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2723) Avro Java: Obtaining default values for POJO objects with ReflectData
[
https://issues.apache.org/jira/browse/AVRO-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032796#comment-17032796
]
Anh Le commented on AVRO-2723:
--
[~rskraba] I'm really glad to receive your thoughtful comments.
> However, I'm pretty sure you *can* apply it to *{{Integer}}* and complex
> classes.
Correct. I can use *{{Integer}}* but I can't use *String*.
The code below:
{code:java}
class User {
@AvroDefault("Andy")
public String first;
}
Schema schema = ReflectData.get().getSchema(User.class);
{code}
will trigger an exception:
{noformat}
Exception in thread "main" org.apache.avro.AvroRuntimeException:
java.lang.RuntimeException: com.fasterxml.jackson.core.JsonParseException:
Unrecognized token 'Andy': was expecting ('true', 'false' or 'null')
at [Source: (String)"Andy"; line: 1, column: 9]{noformat}
Anything wrong?
> I'd suggest picking something more explicit. *{{createSchemaDefaultValue()}}*
> perhaps?
Well pick. But if without refactoring ReflectData, where we will put this
function?
[~cutting] Do you think we need to refactor ReflectData? I really need it to
expose overridable functions for extracting default values through
instantiation.
> Avro Java: Obtaining default values for POJO objects with ReflectData
> -
>
> Key: AVRO-2723
> URL: https://issues.apache.org/jira/browse/AVRO-2723
> Project: Apache Avro
> Issue Type: Wish
> Components: java
>Affects Versions: 1.9.1
>Reporter: Anh Le
>Priority: Critical
>
> Hi guys,
>
> I've got a simple app using Avro Reflection:
>
> {code:java}
> public class App {
> public static void main(String[] args) {
> testReflection();
> }
> static class User {
> public String first = "Andy";
> public String last = "Le";
> }
> static void testReflection(){
> // get the reflected schema for packets
> Schema schema = ReflectData.AllowNull.get().getSchema(User.class);
> System.out.println(schema.toString(true));
> }
> {code}
> The output on console will be:
> {noformat}
> {
> "type" : "record",
> "name" : "User",
> "namespace" : "App",
> "fields" : [ {
> "name" : "first",
> "type" : [ "null", "string" ],
> "default" : null
> }, {
> "name" : "last",
> "type" : [ "null", "string" ],
> "default" : null
> } ]
> }
> {noformat}
>
> As you can see, there's no default values for fields. Would you please tell
> me how to obtain such values?
> Thank you.
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2723) Avro Java: Obtaining default values for POJO objects with ReflectData
[
https://issues.apache.org/jira/browse/AVRO-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032362#comment-17032362
]
Ryan Skraba commented on AVRO-2723:
---
OK! Thanks for the explanation, I think I understand better.
In my experience, schemas generated through ReflectData have been usually
"short-lived" and infrequently use schema evolution (where the default
attribute is useful). I have no objection to improving how schemas are
created from POJO's via reflection, however!
I suspect that the *{{AvroDefault}}* solution was chosen precisely because it
permits generating the whole schema without having to instantiate an instance.
However, I'm pretty sure you *can* apply it to *{{Integer}}* and complex
classes.
For example, the following works:
{code:java}
static class DefaultTest {
@AvroDefault("1")
Integer foo;
}
static class ComplexDefaultTest {
@AvroDefault("2")
Integer bar;
@AvroDefault("{\"foo\":3}}")
DefaultTest inner;
}
{code}
A couple of things to consider:
1. there's already a *{{getDefaultValue(Field)}}*, which does something
slightly different than the method of the same name you propose -- I'd suggest
picking something more explicit. *{{createSchemaDefaultValue()}}* perhaps?
2. as you point out *{{ReflectData.AllowNull}}* (used in your example) always
generates nullable fields with null-first unions, which can only have
*{{null}}* as a default. You might want to switch the order of types in the
union if a non-null default is discovered.
I think the only existing solution today (without refactoring ReflectData)
would be to generated schema, then "massage" it manually to modify the defaults
that it gave you before saving it.
> Avro Java: Obtaining default values for POJO objects with ReflectData
> -
>
> Key: AVRO-2723
> URL: https://issues.apache.org/jira/browse/AVRO-2723
> Project: Apache Avro
> Issue Type: Wish
> Components: java
>Affects Versions: 1.9.1
>Reporter: Anh Le
>Priority: Critical
>
> Hi guys,
>
> I've got a simple app using Avro Reflection:
>
> {code:java}
> public class App {
> public static void main(String[] args) {
> testReflection();
> }
> static class User {
> public String first = "Andy";
> public String last = "Le";
> }
> static void testReflection(){
> // get the reflected schema for packets
> Schema schema = ReflectData.AllowNull.get().getSchema(User.class);
> System.out.println(schema.toString(true));
> }
> {code}
> The output on console will be:
> {noformat}
> {
> "type" : "record",
> "name" : "User",
> "namespace" : "App",
> "fields" : [ {
> "name" : "first",
> "type" : [ "null", "string" ],
> "default" : null
> }, {
> "name" : "last",
> "type" : [ "null", "string" ],
> "default" : null
> } ]
> }
> {noformat}
>
> As you can see, there's no default values for fields. Would you please tell
> me how to obtain such values?
> Thank you.
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2723) Avro Java: Obtaining default values for POJO objects with ReflectData
[
https://issues.apache.org/jira/browse/AVRO-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032099#comment-17032099
]
Anh Le commented on AVRO-2723:
--
[~rskraba]
h4. Usecase
My use case is:
* I want to develop a standalone server for registering/storing Avro schemas,
including default values
* Applications (Java first) can freely register schemas. They can easily do
that by having a `middleware` library to inspect POJO objects into Avro Schema
h4. Feedbacks
> If you have a POJO instance and you want to get the default for a field, you
> can use plain Java: *{{new User().first}}*
I know. But from the server's view, it just see Avro schema, right?
> ... you can add an *{{@AvroDefault}}* annotation to the field [like this unit
> test|https://github.com/apache/avro/blob/70260919426f89825ca148f5ee815f3b2cf4764d/lang/java/avro/src/test/java/org/apache/avro/reflect/TestReflect.java#L1262].
> ...
I see the code. It works!
{code:java}
class User {
@AvroDefault("40")
public int age;
}
{code}
But I think there are several drawbacks for this approach
1. It's a less natural way compared to the following:
{code:java}
class User {
public int age = 40;
}
{code}
2. Fields can't be non-primitive objects (I
For example, this will yield exception
{code:java}
class User {
@AvroDefault(value = "40")
public Integer age;
}
{code}
This also result in #1 if we have a complicated class.
h4. Solution
I've hacked into Avro code base and see a solution
- Refactor: ReflectData.createSchema() (lines
[#L617-L627|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/reflect/ReflectData.java#L617-L627]
to have an overridable function named as `getDefaultValue()`
- I then can derive ReflectData class to provide the default values:
{code:java}
class DefaultReflector extends ReflectData {
private static final ReflectData INSTANCE = new DefaultReflector();
/** Return the singleton instance. */
public static ReflectData get() {
return INSTANCE;
}
@Override
protected Object getDefaultValue(Type type, Schema fieldSchema, Field field) {
try {
String className = ((Class) type).getName();
Object val = Class.forName(className).newInstance();
return field.get(val);
} catch (Exception e) {
e.printStackTrace();
return super.getDefaultValue(type, fieldSchema, field);
}
}
@Override
protected Schema createFieldSchema(Field field, Map names) {
Schema schema = super.createFieldSchema(field, names);
if (field.getType().isPrimitive()) {
// for primitive values, such as int, a null will result in a
// NullPointerException at read time
return schema;
}
return makeNullable(schema);
}
}
{code}
h4. Question
- Is there any solution for my use case?
- Should I make a PR to refactor ReflectData to have `getDefaultValue`
overridable in derived classes ?
> Avro Java: Obtaining default values for POJO objects with ReflectData
> -
>
> Key: AVRO-2723
> URL: https://issues.apache.org/jira/browse/AVRO-2723
> Project: Apache Avro
> Issue Type: Wish
> Components: java
>Affects Versions: 1.9.1
>Reporter: Anh Le
>Priority: Critical
>
> Hi guys,
>
> I've got a simple app using Avro Reflection:
>
> {code:java}
> public class App {
> public static void main(String[] args) {
> testReflection();
> }
> static class User {
> public String first = "Andy";
> public String last = "Le";
> }
> static void testReflection(){
> // get the reflected schema for packets
> Schema schema = ReflectData.AllowNull.get().getSchema(User.class);
> System.out.println(schema.toString(true));
> }
> {code}
> The output on console will be:
> {noformat}
> {
> "type" : "record",
> "name" : "User",
> "namespace" : "App",
> "fields" : [ {
> "name" : "first",
> "type" : [ "null", "string" ],
> "default" : null
> }, {
> "name" : "last",
> "type" : [ "null", "string" ],
> "default" : null
> } ]
> }
> {noformat}
>
> As you can see, there's no default values for fields. Would you please tell
> me how to obtain such values?
> Thank you.
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2723) Avro Java: Obtaining default values for POJO objects with ReflectData
[
https://issues.apache.org/jira/browse/AVRO-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031652#comment-17031652
]
Ryan Skraba commented on AVRO-2723:
---
Hello! I'm not sure I understand your use case!
Typically when you use reflection in Java to serialize/deserialize POJO
objects, you have enough information about the interesting POJO right at your
hands. There's no particular need to look for the default in the generated
schema, if you have access to a *{{User}}* instance, right?
If you have a POJO instance and you want to get the default for a field, you
can use plain Java: *{{new User().first}}*
If you have control over the POJO code and want to set the default in the
generated schema, you can add an *{{@AvroDefault}}* annotation to the field
[like this unit
test|https://github.com/apache/avro/blob/70260919426f89825ca148f5ee815f3b2cf4764d/lang/java/avro/src/test/java/org/apache/avro/reflect/TestReflect.java#L1262].
Is this what you are looking for?
> Avro Java: Obtaining default values for POJO objects with ReflectData
> -
>
> Key: AVRO-2723
> URL: https://issues.apache.org/jira/browse/AVRO-2723
> Project: Apache Avro
> Issue Type: Wish
> Components: java
>Affects Versions: 1.9.1
>Reporter: Anh Le
>Priority: Critical
>
> Hi guys,
>
> I've got a simple app using Avro Reflection:
>
> {code:java}
> public class App {
> public static void main(String[] args) {
> testReflection();
> }
> static class User {
> public String first = "Andy";
> public String last = "Le";
> }
> static void testReflection(){
> // get the reflected schema for packets
> Schema schema = ReflectData.AllowNull.get().getSchema(User.class);
> System.out.println(schema.toString(true));
> }
> {code}
> The output on console will be:
> {noformat}
> {
> "type" : "record",
> "name" : "User",
> "namespace" : "App",
> "fields" : [ {
> "name" : "first",
> "type" : [ "null", "string" ],
> "default" : null
> }, {
> "name" : "last",
> "type" : [ "null", "string" ],
> "default" : null
> } ]
> }
> {noformat}
>
> As you can see, there's no default values for fields. Would you please tell
> me how to obtain such values?
> Thank you.
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
