Re: stand alone avro-tools prints log4j warning

2016-10-12 Thread radai 
reading the doc page (http://logging.apache.org/log4j/1.2/faq.html#noconfig),
there's a log4j.defaultInitOverride property. have you tried adding
-Dlog4j.defaultInitOverride=true to your command line ?

On Wed, Oct 12, 2016 at 11:29 AM, Isaac Levy 
wrote:

> Is there any way to silence these?
>
> $ java -jar avro-tools-1.8.1.jar tojson foo
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> more info.
>
> Looks like avro-tools used to include org/jira/browse/AVRO-758> log4j config.
>
>
> -Isaac Levy
> OS X el cap, java 1.8.0_92
>

[jira] [Updated] (AVRO-1887) set up precommit checks via Apache Yetus

2016-10-12 Thread Suraj Acharya (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AVRO-1887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suraj Acharya updated AVRO-1887:

Attachment: AVRO-1887-1.patch

Updated some incorrect comment.

Also tagging [~aw] to do some code review since he helped with some yetus 
implementation.

> set up precommit checks via Apache Yetus
> 
>
> Key: AVRO-1887
> URL: https://issues.apache.org/jira/browse/AVRO-1887
> Project: Avro
>  Issue Type: Task
>  Components: build, community
>Reporter: Sean Busbey
>Assignee: Suraj Acharya
>Priority: Blocker
> Attachments: AVRO-1887-0.patch, AVRO-1887-1.patch
>
>
> The lack of automated checks for things like doing a full build is negatively 
> impacting my ability to do timely reviews. We should leverage the work done 
> by other communities and get precommit checks going with Apache Yetus ([ref 
> Apache Yetus 
> Precommit|http://yetus.apache.org/documentation/0.3.0/precommit-basic/])
> Yetus has built in support for using Docker as a containerized build 
> environment ([ref 
> docs|https://yetus.apache.org/documentation/0.3.0/precommit-advanced/#Docker_Support]),
>  but I don't know if we can use the one we have directly or if we'll need to 
> do some refactoring.
> Implementing this will ultimately require a committer (so that they can make 
> changes to builds.apache.org), but a non-cmmitter can get the bulk of the 
> work done by managing to do local yetus checks with the docker feature active.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

stand alone avro-tools prints log4j warning

2016-10-12 Thread Isaac Levy 
Is there any way to silence these?

$ java -jar avro-tools-1.8.1.jar tojson foo
log4j:WARN No appenders could be found for logger 
(org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
info.

Looks like avro-tools used to 
include log4j config.


-Isaac Levy
OS X el cap, java 1.8.0_92

[jira] [Comment Edited] (AVRO-1933) SchemaCompatibility class could be more user-friendly about incompatibilities

2016-10-12 Thread Anders Sundelin (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568638#comment-15568638
 ] 

Anders Sundelin edited comment on AVRO-1933 at 10/12/16 2:56 PM:
-

Patch set adding more error details to SchemaCompatibility.
Also includes part of AVRO-1931 (though not all tests therein)
This patch is not compatible with changes done in AVRO-1931 - if you come that 
way, please use the other patch attached to this issue.
Contact me if there is any issues.



was (Author: epkanol):
Patch set adding more error details to SchemaCompatibility.
Also includes part of AVRO-1931 (though not all tests therein)
Suggest to apply AVRO-1931 first, then this patch.
Contact me if there is any issues.


> SchemaCompatibility class could be more user-friendly about incompatibilities
> -
>
> Key: AVRO-1933
> URL: https://issues.apache.org/jira/browse/AVRO-1933
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.1
> Environment: Any Java env
>Reporter: Anders Sundelin
>Priority: Minor
> Fix For: 1.9.0
>
> Attachments: AVRO-1933-compatible-with-AVRO-1931.patch, 
> AVRO-1933.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Today, the class SchemaCompatibility reports incompatibilities with quite 
> little detail. The whole reader and the whole writer schema is listed, and no 
> particular detail about what was incompatible.
> The attached patch fixes this, introducing a new enum 
> (SchemaIncompatibilityType), and more specific sub-schemas that were 
> incompatible.
> The old, overall picture, is still there - the new compatibility state is 
> encapsulated in the SchemaCompatibilityDetails class.
> Lots of test cases have been added, and there has been refactoring done in 
> the TestSchemaCompatibility and other test classes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1927) If a default value is set, Avro allows null values in non-nullable fields.

2016-10-12 Thread Andreas Maier (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568879#comment-15568879
 ] 

Andreas Maier commented on AVRO-1927:
-

So I had a look at the generated Java code. And strangely enough, this code 
only throws an exception, if no default value is set:
{code}
   protected void validate(Field field, Object value) {
if(!isValidValue(field, value)) {
if(field.defaultValue() == null) {   // why this check?
throw new AvroRuntimeException("Field " + field + " does not 
accept null values");
}
  
}
}
{code}
I don't understand why Avro checks, if {{if(field.defaultValue() == null)}} 
before throwing an exception. In my opinion it should always throw an exception 
if the field value is invalid. 

> If a default value is set, Avro allows null values in non-nullable fields.
> --
>
> Key: AVRO-1927
> URL: https://issues.apache.org/jira/browse/AVRO-1927
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.1
>Reporter: Andreas Maier
>  Labels: newbie
>
> With an avro schema like
> {code}
> {
>   "name": "myfield",
>   "type": "string",
>   "default": ""
> }
> {code}
> the following code should throw an exception
> {code}
> MyObject myObject = MyObject.newBuilder().setMyfield(null).build();
> {code}
> But instead the value of myfield is set to null, which causes an exception 
> later when serializing myObject, because null is not a valid value for 
> myfield. 
> I believe in this case setMyfield(null) should throw an exception, 
> independent of the value of default. 
> See also
> https://stackoverflow.com/questions/38509279/generated-avro-builder-set-null-doesnt-overwrite-with-default



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1927) If a default value is set, Avro allows null values in non-nullable fields.

2016-10-12 Thread Andreas Maier (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568867#comment-15568867
 ] 

Andreas Maier commented on AVRO-1927:
-

Sorry for the long delay. I was on vacation.
I tried the code you suggested. You are right, it does throw a 
NullPointerException when I try to write the object:
{code}
java.lang.NullPointerException: null of string of de.am.MyObject

at 
org.apache.avro.generic.GenericDatumWriter.npe(GenericDatumWriter.java:132)
at 
org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:126)
at 
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
at 
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
at de.am.MyObjectTest.myObjectTest2(MyObjectTest.java:27)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at 
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:119)
at 
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
at 
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234)
at 
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Caused by: java.lang.NullPointerException
at 
org.apache.avro.specific.SpecificDatumWriter.writeString(SpecificDatumWriter.java:67)
at 
org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:115)
at 
org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
at 
org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
at 
org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
... 30 more
{code}

> If a default value is set, Avro allows null values in non-nullable fields.
> --
>
> Key: AVRO-1927
> URL: https://issues.apache.org/jira/browse/AVRO-1927
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.1
>Reporter: Andreas Maier
>  Labels: newbie
>
> With an avro schema like
> {code}
> {
>   "name": "myfield",
>   "type": "string",
>   "default": ""
> }
> {code}
> the following code should throw an exception
> {code}
> MyObject myObject = MyObject.newBuilder().setMyfield(null).build();
> {code}
> But instead the value of myfield is set to null, which causes an exception 
> later when serializing myObject, because null is not a valid value for 
> myfield. 
> I believe in this case setMyfield(null) should throw an exception, 
> independent of the value of default. 
> See also
> https://stackoverflow.com/questions/38509279/generated-avro-builder-set-null-doesnt-overwrite-with-default



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1933) SchemaCompatibility class could be more user-friendly about incompatibilities

2016-10-12 Thread Anders Sundelin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AVRO-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anders Sundelin updated AVRO-1933:
--
Attachment: AVRO-1933-compatible-with-AVRO-1931.patch

Patch set compatible with AVRO-1931 patch
One of these patches apply, depending on your starting point.


> SchemaCompatibility class could be more user-friendly about incompatibilities
> -
>
> Key: AVRO-1933
> URL: https://issues.apache.org/jira/browse/AVRO-1933
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.1
> Environment: Any Java env
>Reporter: Anders Sundelin
>Priority: Minor
> Fix For: 1.9.0
>
> Attachments: AVRO-1933-compatible-with-AVRO-1931.patch, 
> AVRO-1933.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Today, the class SchemaCompatibility reports incompatibilities with quite 
> little detail. The whole reader and the whole writer schema is listed, and no 
> particular detail about what was incompatible.
> The attached patch fixes this, introducing a new enum 
> (SchemaIncompatibilityType), and more specific sub-schemas that were 
> incompatible.
> The old, overall picture, is still there - the new compatibility state is 
> encapsulated in the SchemaCompatibilityDetails class.
> Lots of test cases have been added, and there has been refactoring done in 
> the TestSchemaCompatibility and other test classes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1933) SchemaCompatibility class could be more user-friendly about incompatibilities

2016-10-12 Thread Anders Sundelin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AVRO-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anders Sundelin updated AVRO-1933:
--
Attachment: AVRO-1933.patch

Patch set adding more error details to SchemaCompatibility.
Also includes part of AVRO-1931 (though not all tests therein)
Suggest to apply AVRO-1931 first, then this patch.
Contact me if there is any issues.


> SchemaCompatibility class could be more user-friendly about incompatibilities
> -
>
> Key: AVRO-1933
> URL: https://issues.apache.org/jira/browse/AVRO-1933
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.1
> Environment: Any Java env
>Reporter: Anders Sundelin
>Priority: Minor
> Fix For: 1.9.0
>
> Attachments: AVRO-1933.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Today, the class SchemaCompatibility reports incompatibilities with quite 
> little detail. The whole reader and the whole writer schema is listed, and no 
> particular detail about what was incompatible.
> The attached patch fixes this, introducing a new enum 
> (SchemaIncompatibilityType), and more specific sub-schemas that were 
> incompatible.
> The old, overall picture, is still there - the new compatibility state is 
> encapsulated in the SchemaCompatibilityDetails class.
> Lots of test cases have been added, and there has been refactoring done in 
> the TestSchemaCompatibility and other test classes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (AVRO-1933) SchemaCompatibility class could be more user-friendly about incompatibilities

2016-10-12 Thread Anders Sundelin (JIRA) 
Anders Sundelin created AVRO-1933:
-

 Summary: SchemaCompatibility class could be more user-friendly 
about incompatibilities
 Key: AVRO-1933
 URL: https://issues.apache.org/jira/browse/AVRO-1933
 Project: Avro
  Issue Type: Improvement
  Components: java
Affects Versions: 1.8.1
 Environment: Any Java env
Reporter: Anders Sundelin
Priority: Minor
 Fix For: 1.9.0


Today, the class SchemaCompatibility reports incompatibilities with quite 
little detail. The whole reader and the whole writer schema is listed, and no 
particular detail about what was incompatible.

The attached patch fixes this, introducing a new enum 
(SchemaIncompatibilityType), and more specific sub-schemas that were 
incompatible.
The old, overall picture, is still there - the new compatibility state is 
encapsulated in the SchemaCompatibilityDetails class.
Lots of test cases have been added, and there has been refactoring done in the 
TestSchemaCompatibility and other test classes.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1340) use default to allow old readers to specify default enum value when encountering new enum symbols

2016-10-12 Thread Zoltan Ivanfi (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568443#comment-15568443
 ] 

Zoltan Ivanfi commented on AVRO-1340:
-

I like the mechanism suggested by Matthieu, but I think declaring it as 
"aliases" is very misleading. Although creating an alias is one possible use 
case for this mechanism, specifying a fallback is a more typical one. Maybe 
name them fallback values instead?

> use default to allow old readers to specify default enum value when 
> encountering new enum symbols
> -
>
> Key: AVRO-1340
> URL: https://issues.apache.org/jira/browse/AVRO-1340
> Project: Avro
>  Issue Type: Improvement
>  Components: spec
> Environment: N/A
>Reporter: Jim Donofrio
>Priority: Minor
>
> The schema resolution page says:
> > if both are enums:
> > if the writer's symbol is not present in the reader's enum, then an
> error is signalled.
> This makes it difficult to use enum's because you can never add a enum value 
> and keep old reader's compatible. Why not use the default option to refer to 
> one of enum values so that when a old reader encounters a enum ordinal it 
> does not recognize, it can default to the optional schema provided one. If 
> the old schema does not provide a default then the older reader can continue 
> to fail as it does today.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1605) Remove Jackson classes from public API

2016-10-12 Thread Gabor Szadovszky (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567956#comment-15567956
 ] 

Gabor Szadovszky commented on AVRO-1605:


Thanks for the response, Ryan. Sure, it is reasonable. I'll try to work on it 
this week.

> Remove Jackson classes from public API
> --
>
> Key: AVRO-1605
> URL: https://issues.apache.org/jira/browse/AVRO-1605
> Project: Avro
>  Issue Type: Sub-task
>  Components: java
>Affects Versions: 1.7.8
>Reporter: Tom White
>Assignee: Gabor Szadovszky
> Fix For: 1.9.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)