[jira] [Commented] (AVRO-1886) Improve error reporting for schema validation

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816120#comment-15816120
 ] 

Sean Busbey commented on AVRO-1886:
---

BTW, thanks for the review [~sldblog]. Any chance I could get you to look at 
some other Ruby patches?

> Improve error reporting for schema validation
> -
>
> Key: AVRO-1886
> URL: https://issues.apache.org/jira/browse/AVRO-1886
> Project: Avro
>  Issue Type: Improvement
>  Components: ruby
>Reporter: Miroslav Csonka
>Assignee: Miroslav Csonka
> Fix For: 1.9.0, 1.8.2
>
>
> In Funding Circle we use avro schema to validate some financial data and 
> we've found ourselves wondering what is invalid. We've noticed that avro 
> [does already 
> validation|https://github.com/apache/avro/blob/master/lang/ruby/lib/avro/schema.rb#L94],
>  but does not report what fields are invalid, so we've decided to change it 
> and we use Avro as a validation library.
> h5. What error messages to expect?
> We communicate errors for a given payload in format "at   
> ".
> For  we've decided to use syntax similar to json path:
> * . refers to root
> * \[0\] first index in array
> * .age would match 42 from {"age": 42}
> Error within object *person* and his optional array *houses* on *second item* 
> with key *number_of_rooms*:
> {quote}
> at .person.houses\[1\].number_of_rooms expected type long, got string 
> with value "not valid at all"
> {quote}
> Example of multiple errors for a single payload:
> {quote}
> at .\[0\] expected type int, got null
> at .\[1\] expected type int, got string with value \"e\"
> {quote}
> More examples can be seen at 
> https://github.com/apache/avro/pull/111/files#diff-830ac80276854fceefa3f92c346d165f



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1886) Improve error reporting for schema validation

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816117#comment-15816117
 ] 

Sean Busbey commented on AVRO-1886:
---

Code looks good to me.

I'd like to get one thing improved, but it's fine as a follow-on.

{code}
 +   when :fixed
 +  if datum.is_a? String
 +message = "expected fixed with size #{expected_schema.size}, got 
\"#{datum}\" with size #{datum.size}"
 +result.add_error(path, message) unless datum.bytesize == 
expected_schema.size
 +  else
 +result.add_error(path, "expected fixed with size 
#{expected_schema.size}, got #{actual_value_message(datum)}")
 +  end
{code}

the message here should use {{datum.bytesize}} instead of {{datum.size}}.

Presuming no one else pipes up, I'll go ahead and push this change and file the 
follow-on later this evening.

> Improve error reporting for schema validation
> -
>
> Key: AVRO-1886
> URL: https://issues.apache.org/jira/browse/AVRO-1886
> Project: Avro
>  Issue Type: Improvement
>  Components: ruby
>Reporter: Miroslav Csonka
>Assignee: Miroslav Csonka
> Fix For: 1.9.0, 1.8.2
>
>
> In Funding Circle we use avro schema to validate some financial data and 
> we've found ourselves wondering what is invalid. We've noticed that avro 
> [does already 
> validation|https://github.com/apache/avro/blob/master/lang/ruby/lib/avro/schema.rb#L94],
>  but does not report what fields are invalid, so we've decided to change it 
> and we use Avro as a validation library.
> h5. What error messages to expect?
> We communicate errors for a given payload in format "at   
> ".
> For  we've decided to use syntax similar to json path:
> * . refers to root
> * \[0\] first index in array
> * .age would match 42 from {"age": 42}
> Error within object *person* and his optional array *houses* on *second item* 
> with key *number_of_rooms*:
> {quote}
> at .person.houses\[1\].number_of_rooms expected type long, got string 
> with value "not valid at all"
> {quote}
> Example of multiple errors for a single payload:
> {quote}
> at .\[0\] expected type int, got null
> at .\[1\] expected type int, got string with value \"e\"
> {quote}
> More examples can be seen at 
> https://github.com/apache/avro/pull/111/files#diff-830ac80276854fceefa3f92c346d165f



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: DISCUSSION : Create a Base Docker Image

[Sean Busbey]

This sounds great! If no one on the PMC objects, I can file the INFRA
ticket to get us access to publish to docker hub.

On Sat, Nov 26, 2016 at 5:20 PM, suraj acharya  wrote:
> Hi guys,
> Since there was no objections raised to my earlier email about Docker
> Image, I have gone ahead and created one. I have uploaded it currently in
> my repo. https://hub.docker.com/r/surajacharya/avro-build-test/tags/.
> The change needed for the build.sh script is as follows :
> sacharya@sacharya-MBP:~/repos/avro$ git diff
> diff --git a/build.sh b/build.sh
> index 42df09a..fb28889 100755
> --- a/build.sh
> +++ b/build.sh
> @@ -195,7 +195,6 @@ do
>;;
>
>  docker)
> -  docker build -t avro-build share/docker
>if [ "$(uname -s)" == "Linux" ]; then
>  USER_NAME=${SUDO_USER:=$USER}
>  USER_ID=$(id -u $USER_NAME)
> @@ -206,7 +205,7 @@ do
>  GROUP_ID=50
>fi
>docker build -t avro-build-${USER_NAME} - < -FROM avro-build
> +FROM docker.io/surajacharya/avro-build-test
>  RUN groupadd -g ${GROUP_ID} ${USER_NAME} || true
>  RUN useradd -g ${GROUP_ID} -u ${USER_ID} -k /root -m ${USER_NAME} || true
>  ENV HOME /home/${USER_NAME}
>
> This will prevent make the world everytime a new docker image is to be
> created.
> We can always update any package we wish and push a new image. This will
> help in making the docker images faster.
>
> If there are no objections, then we will need to a new repo, the PMC can
> create it, load the image there.
>
> Please let me know if you have any questions.
>
> -Suraj Acharya
>
> On Tue, Nov 1, 2016 at 4:35 PM, suraj acharya  wrote:
>
>> Hi,
>> I wanted to discuss whether it will be a good idea to create a base docker
>> image and then check that out every time we wish to use docker.
>> The main reason behind the request is that the initial startup time to
>> build a docker time will go down significantly ( from the present 20 ish
>> minutes). This will help when we have a pre-commit bot since the startup
>> time will reduce significantly since the overall startup time is low.
>> I know the docker machine is cached for a period of time, however, our
>> docker image itself doesn't change a lot, but building a new one takes a
>> large amount of time due to the diverse nature of the project.
>>
>> I suggest that we use the base debian image we use, install the
>> dependencies we install at every run https://github.com/apache/
>> avro/blob/master/share/docker/Dockerfile#L28-L38.
>> We push this to a central repository, something like docker hub. We don't
>> have any source code present so we don't need to worry about having
>> unreleased code in the open. We can still have the apt-update and apt-get
>> update  step present which will allow us to update to the latest
>> version of any package we might have missed.
>> On a six month/year routine we can update the docker image if there are a
>> lot of updates on the image.
>>
>> We will need some committer who can actually push the final image to the
>> docker-hub.
>>
>> I can help in creating the image and providing the same.
>>
>> Your comments are appreciated.
>>
>>
>> -Suraj Acharya
>>



-- 
busbey

Re: [DISCUSS] Community engagement

[Sean Busbey]

On Tue, Nov 15, 2016 at 3:45 PM, Doug Cutting  wrote:
> On Mon, Nov 14, 2016 at 8:40 PM, Ryan Blue  wrote:
>> we don't have enough votes for a release
>
> I don't think that's true.  You might not have gotten enough votes
> within a few days.  I too have had that problem for several releases.
> But when that happened I would send personal messages to a few PMC
> members asking them if they had time to please review a release.  Some
> PMC members get busy with other things and fall behind on Avro emails.
> A few reminders never failed to rouse enough reviews & votes.
>

In HBase we've largely switched away from release votes that have a
fixed deadline due to the need to corral PMC votes.

Instead, we now state the minimum voting period (almost always 72hrs
per ASF policy) and set a courtesy notice of when the RM would like to
close the vote. Then we just keep pinging private@ or individuals
until we get enough votes for a result, or find something to start
getting -1s.

We could try something like that for a bit to see if "flog the bushes
for PMC participation" is enough to keep us on a steady stream of
releases.

It would also probably help to make explicit to the community that
showing up to test and vote on RCs is "acting like a PMC" in the eyes
of some number of current PMC members (it is for me, for example).

-- 
busbey

[jira] [Updated] (AVRO-1885) Release 1.8.2

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1885:
--
Assignee: Ryan Blue  (was: Sean Busbey)

assigning to Ryan since there's an RC up now.

thanks Ryan! I'll schedule time to vote on this RC. If it doesn't pass for some 
reason, happy to take on the next one.

> Release 1.8.2
> -
>
> Key: AVRO-1885
> URL: https://issues.apache.org/jira/browse/AVRO-1885
> Project: Avro
>  Issue Type: Task
>  Components: community
>Affects Versions: 1.8.2
>    Reporter: Sean Busbey
>Assignee: Ryan Blue
> Fix For: 1.8.2
>
>
> Please link to any issues that should be considered blockers for the 1.8.2 
> release.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1887) set up precommit checks via Apache Yetus

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626140#comment-15626140
 ] 

Sean Busbey commented on AVRO-1887:
---

looks good. I'll play with this a bit before committing.

One small change, could you locate this in {{share/pre_commit}} instead of the 
top-level {{pre_commit}}?

That will put it near the other help-run-the-project stuff we have for e.g. 
Docker and git.

Have you gotten to try out using our Dockerfile to ensure the yetus run has all 
of our needed dependencies?

> set up precommit checks via Apache Yetus
> 
>
> Key: AVRO-1887
> URL: https://issues.apache.org/jira/browse/AVRO-1887
> Project: Avro
>  Issue Type: Task
>  Components: build, community
>    Reporter: Sean Busbey
>Assignee: Suraj Acharya
>Priority: Blocker
> Attachments: AVRO-1887-0.patch, AVRO-1887-1.patch
>
>
> The lack of automated checks for things like doing a full build is negatively 
> impacting my ability to do timely reviews. We should leverage the work done 
> by other communities and get precommit checks going with Apache Yetus ([ref 
> Apache Yetus 
> Precommit|http://yetus.apache.org/documentation/0.3.0/precommit-basic/])
> Yetus has built in support for using Docker as a containerized build 
> environment ([ref 
> docs|https://yetus.apache.org/documentation/0.3.0/precommit-advanced/#Docker_Support]),
>  but I don't know if we can use the one we have directly or if we'll need to 
> do some refactoring.
> Implementing this will ultimately require a committer (so that they can make 
> changes to builds.apache.org), but a non-cmmitter can get the bulk of the 
> work done by managing to do local yetus checks with the docker feature active.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: backporting a fix to avro 1.7

[Sean Busbey]

(For others, the jira under discussion is AVRO-1901
https://issues.apache.org/jira/browse/AVRO-1901)

On Tue, Oct 4, 2016 at 8:28 AM, Sean Busbey <bus...@cloudera.com> wrote:
> yes, it looks like the jira landed in 1.9, though there isn't a fix
> version set. I don't see anything in the patch that would prevent a
> backport to 1.7.
>
> On Mon, Oct 3, 2016 at 2:28 PM, radai <radai.rosenbl...@gmail.com> wrote:
>> Hi,
>>
>> I've recently encountered a bug in avro code generation, for which I
>> submitted a fix (
>> https://github.com/apache/avro/commit/be33922c03bd229b885fb9d664b55ce308f6ffaf
>> ).
>>
>> The issue was discovered while trying to migrate a sizable codebase from
>> avro 1.4 to 1.7
>>
>> Would it be possible to back-port this fix to 1.7? bumping 1.7 to latest
>> would be a lot easier than trying to upgrade 1.4 and now 1.7 code to 1.9
>> (which is where this will will end up in, when its released?)
>>
>> I can just open a PR porting the fix to the 1.7 branch (maybe 1.8 branch as
>> well, for consistency), if thats the procedure.
>>
>> Thanks in advance,
>>
>>
>> Radai.
>
>
>
> --
> busbey



-- 
busbey

Re: backporting a fix to avro 1.7

[Sean Busbey]

yes, it looks like the jira landed in 1.9, though there isn't a fix
version set. I don't see anything in the patch that would prevent a
backport to 1.7.

On Mon, Oct 3, 2016 at 2:28 PM, radai  wrote:
> Hi,
>
> I've recently encountered a bug in avro code generation, for which I
> submitted a fix (
> https://github.com/apache/avro/commit/be33922c03bd229b885fb9d664b55ce308f6ffaf
> ).
>
> The issue was discovered while trying to migrate a sizable codebase from
> avro 1.4 to 1.7
>
> Would it be possible to back-port this fix to 1.7? bumping 1.7 to latest
> would be a lot easier than trying to upgrade 1.4 and now 1.7 code to 1.9
> (which is where this will will end up in, when its released?)
>
> I can just open a PR porting the fix to the 1.7 branch (maybe 1.8 branch as
> well, for consistency), if thats the procedure.
>
> Thanks in advance,
>
>
> Radai.



-- 
busbey

[jira] [Commented] (AVRO-1582) Json serialization of nullable fileds and fields with default values improvement.

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509865#comment-15509865
 ] 

Sean Busbey commented on AVRO-1582:
---

I've increased this from 'minor' to 'major' priority, but I believe the current 
blocker on it getting accepted is that we need a contributor that's willing to 
finish things up.

> Json serialization of nullable fileds and fields with default values 
> improvement.
> -
>
> Key: AVRO-1582
> URL: https://issues.apache.org/jira/browse/AVRO-1582
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Zoltan Farkas
> Attachments: AVRO-1582-PATCH
>
>
> Currently serializing a nullable field of type union like:
> "type" : ["null","some type"]
> when serialized as JSON results in:  
> "field":{"some type":"value"}
> when it could be:
> "field":"value"
> Also fields that equal the the default value can be omitted from the 
> serialized data. This is possible because the reader will have the writer's 
> schema and can infer the field values. This reduces the size of the json 
> messages.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1582) Json serialization of nullable fileds and fields with default values improvement.

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1582:
--
Priority: Major  (was: Minor)

> Json serialization of nullable fileds and fields with default values 
> improvement.
> -
>
> Key: AVRO-1582
> URL: https://issues.apache.org/jira/browse/AVRO-1582
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Zoltan Farkas
> Attachments: AVRO-1582-PATCH
>
>
> Currently serializing a nullable field of type union like:
> "type" : ["null","some type"]
> when serialized as JSON results in:  
> "field":{"some type":"value"}
> when it could be:
> "field":"value"
> Also fields that equal the the default value can be omitted from the 
> serialized data. This is possible because the reader will have the writer's 
> schema and can infer the field values. This reduces the size of the json 
> messages.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1810) GenericDatumWriter broken with Enum

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504036#comment-15504036
 ] 

Sean Busbey commented on AVRO-1810:
---

That does seem broken. Could you share a code snippet that results in the 
stacktrace?

> GenericDatumWriter broken with Enum
> ---
>
> Key: AVRO-1810
> URL: https://issues.apache.org/jira/browse/AVRO-1810
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Ryon Day
>Priority: Blocker
>
> {panel:title=Description|titleBGColor=#3FA|bgColor=#DDD}
> Using the GenericDatumWriter with either Generic OR SpecificRecord will break 
> if an Enum is present.
> {panel}
> {panel:title=Steps To Reproduce|titleBGColor=#8DB|bgColor=#DDD}
> I have been tracking Avro decoding oddities for a while.
> The tests for this issue can be found 
> [here|https://github.com/ryonday/avroDecodingHelp/blob/master/src/test/java/com/ryonday/test/Avro180EnumFail.java]
> {panel}
> {panel:title=Notes|titleBGColor=#3AF|bgColor=#DDD}
> Due to the debacle that is the Avro "UTF8" object, we have been avoiding it 
> by using the following scheme:
> * Write incoming records to a byte array using the GenericDatumWriter
> * Read back the byte array to our compiled Java domain objects using a 
> SpecificDatumWriter
> This worked great with Avro 1.7.7, and this is a binary-incompatable breaking 
> change with 1.8.0.
> This would appear to be caused by an addition in the 
> {{GenericDatumWriter:163-164}}:
> {code}
>   if (!data.isEnum(datum))
>   throw new AvroTypeException("Not an enum: "+datum);
> {code}
> {panel}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Pre-commit Check Avro

[Sean Busbey]

gt; chance to make precommit smarter about multiple build tools in a single
>> project. (which would be a pretty hard undertaking, but something we should
>> probably do)
>>
>>
>> > On Sep 10, 2016, at 6:32 PM, suraj acharya <suraj@gmail.com> wrote:
>> >
>> > Yes.
>> > The ./build.sh test
>> > <https://github.com/apache/avro/blob/master/build.sh#L40> command helps
>> run
>> > all the tests from the top level.
>> > However, I am not sure if we need to run all the tests when a change is
>> > made?
>> > I was thinking of running language specific tests for changes in the
>> > specific directory.
>> > For example for changes in python will trigger ant test
>> > <https://github.com/apache/avro/blob/master/build.sh#L43>.
>>
>
> -Suraj Acharya
>
> On Fri, Aug 19, 2016 at 9:54 AM, Sean Busbey <bus...@cloudera.com> wrote:
>
>> If we have files that fail to include the required ASF licensing
>> headers, you're correct that's important to fix.
>>
>> I think either working on it before or after would be fine. Ideally,
>> Yetus Precommit should properly let us know when a patch is fixing
>> those kinds of problems, so it would be a good validation that things
>> are working.
>>
>> On Thu, Aug 18, 2016 at 8:18 PM, suraj acharya <suraj@gmail.com>
>> wrote:
>> > Okay. I will look at the docker command and the image.
>> >
>> > One other question I have is that the asflicense is an important aspect
>> of
>> > yetus. However, many java files have that missing. And whenever a patch
>> > touches that file it returns a -1. Do you think I should first fix all
>> the
>> > licenses and then continue with this?
>> >
>> > S
>> >
>> > -Suraj Acharya
>> >
>> > On Thu, Aug 18, 2016 at 12:37 PM, Sean Busbey <bus...@cloudera.com>
>> wrote:
>> >
>> >> is the 3-5 minutes doing the tests across all of the language
>> >> libraries or just the java ones?
>> >>
>> >> docker will definitely be needed, due to the number of different
>> >> system dependencies needed to build the different languages.
>> >> I was hoping we could reuse the Docker image that is currently
>> >> used for the "./build.sh docker" command.
>> >>
>> >>
>> >> On Wed, Aug 17, 2016 at 6:41 PM, suraj acharya <suraj@gmail.com>
>> >> wrote:
>> >> > Hi,
>> >> > I am a new guy and I have been working on setting up pre-commit check
>> for
>> >> > Avro : AVRO-1887 <https://issues.apache.org/jira/browse/AVRO-1887>.
>> >> >
>> >> > I was looking at the unit tests being run currently. It takes
>> somewhere
>> >> > around 3-5 minutes depending on the network and the repo's present.
>> >> > I was thinking that since the time is so low why not run all the tests
>> >> for
>> >> > every commit. Also, the tests being lightweight I was thinking that
>> there
>> >> > is no need for docker to be needed.
>> >> >
>> >> > I would like to gather your opinions on this topic before going
>> further
>> >> > down this path.
>> >> >
>> >> > Thanks
>> >> >
>> >> > -Suraj Acharya
>> >>
>> >>
>> >>
>> >> --
>> >> busbey
>> >>
>>
>>
>>
>> --
>> busbey
>>



-- 
busbey

[jira] [Commented] (AVRO-1908) IPC TestSpecificCompiler build is broken

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15470542#comment-15470542
 ] 

Sean Busbey commented on AVRO-1908:
---

+1

> IPC TestSpecificCompiler build is broken
> 
>
> Key: AVRO-1908
> URL: https://issues.apache.org/jira/browse/AVRO-1908
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Reporter: Ryan Blue
>
> AVRO-1884 changed {{SpecificCompiler.makePath}} to {{private 
> SpecificCompiler#makePath}}, which broke building the tests in the IPC 
> package.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-607) SpecificData.getSchema not thread-safe

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15463382#comment-15463382
 ] 

Sean Busbey commented on AVRO-607:
--

I marked the helpwanted task as done. Thanks!

> SpecificData.getSchema not thread-safe
> --
>
> Key: AVRO-607
> URL: https://issues.apache.org/jira/browse/AVRO-607
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.3.3, 1.8.1
>Reporter: Stephen Tu
>Assignee: Andrius Druzinis-Vitkus
>Priority: Blocker
>  Labels: newbie, patch
> Fix For: 1.8.2
>
> Attachments: 
> 0001-AVRO-607-Changed-SpecificData.getSchema-to-use-a-thr.patch, 
> AVRO-607.patch, AVRO-607.patch
>
>
> SpecificData.getSchema uses a WeakHashMap to cache schemas, but WeakHashMap 
> is not thread-safe, and the method itself is not synchronized. Seems like 
> this could lead to the data structure getting corrupted. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1704) Standardized format for encoding messages with Avro

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15463379#comment-15463379
 ] 

Sean Busbey commented on AVRO-1704:
---

+1 on AVRO-1704.4.patch

> Standardized format for encoding messages with Avro
> ---
>
> Key: AVRO-1704
> URL: https://issues.apache.org/jira/browse/AVRO-1704
> Project: Avro
>  Issue Type: Improvement
>  Components: java, spec
>Reporter: Daniel Schierbeck
>Assignee: Niels Basjes
> Fix For: 1.9.0, 1.8.3
>
> Attachments: AVRO-1704-2016-05-03-Unfinished.patch, 
> AVRO-1704-20160410.patch, AVRO-1704.3.patch, AVRO-1704.4.patch
>
>
> I'm currently using the Datafile format for encoding messages that are 
> written to Kafka and Cassandra. This seems rather wasteful:
> 1. I only encode a single record at a time, so there's no need for sync 
> markers and other metadata related to multi-record files.
> 2. The entire schema is inlined every time.
> However, the Datafile format is the only one that has been standardized, 
> meaning that I can read and write data with minimal effort across the various 
> languages in use in my organization. If there was a standardized format for 
> encoding single values that was optimized for out-of-band schema transfer, I 
> would much rather use that.
> I think the necessary pieces of the format would be:
> 1. A format version number.
> 2. A schema fingerprint type identifier, i.e. Rabin, MD5, SHA256, etc.
> 3. The actual schema fingerprint (according to the type.)
> 4. Optional metadata map.
> 5. The encoded datum.
> The language libraries would implement a MessageWriter that would encode 
> datums in this format, as well as a MessageReader that, given a SchemaStore, 
> would be able to decode datums. The reader would decode the fingerprint and 
> ask its SchemaStore to return the corresponding writer's schema.
> The idea is that SchemaStore would be an abstract interface that allowed 
> library users to inject custom backends. A simple, file system based one 
> could be provided out of the box.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1895) DeepCopy does not work with logical types

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455460#comment-15455460
 ] 

Sean Busbey commented on AVRO-1895:
---

I reverted and repushed with the correct JIRA key on master.

I didn't see commits on branch-1.8, so I pushed the corrected ones there.

> DeepCopy does not work with logical types
> -
>
> Key: AVRO-1895
> URL: https://issues.apache.org/jira/browse/AVRO-1895
> Project: Avro
>  Issue Type: Improvement
>  Components: logical types
>Affects Versions: 1.8.1
>Reporter: Taras Bobrovytsky
>Assignee: Doug Cutting
>Priority: Critical
> Fix For: 1.9.0, 1.8.2
>
> Attachments: AVRO-1895.patch, AVRO-1895.patch, AVRO-1895.patch, 
> AVRO-1895.patch
>
>
> AvroSchema is taken from a compiled avsc file which contains a decimal field.
> {code}
> AvroSchema.Builder builder = AvroSchema.newBuilder();
> BigDecimal bd = new BigDecimal(new BigInteger("155"), 3);
> campaignBuilder.setDecimalField(bd);
> AvroSchema source = builder.build();
> //This line causes an exception
> AvroSchema.Builder builder1 = AvroSchema.newBuilder(source);
> {code}
> Exception:
> {code}
> InvocationTargetException: java.math.BigDecimal cannot be cast to 
> java.nio.ByteBuffer
> {code}
> The same failure happens with GenericData as well:
> {code}
> GenericRecord copy = GenericData.get().deepCopy(AvroSchema.getClassSchema(), 
> source);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (AVRO-1907) Add logging to logical type handling in java library

[Sean Busbey (JIRA)]

Sean Busbey created AVRO-1907:
-

 Summary: Add logging to logical type handling in java library
 Key: AVRO-1907
 URL: https://issues.apache.org/jira/browse/AVRO-1907
 Project: Avro
  Issue Type: Improvement
  Components: java
Affects Versions: 1.8.0
Reporter: Sean Busbey


Right now we don't have any logging while handling logical type information in 
a Schema. In particular, we use {{LogicalTypes.fromSchemaIgnoreInvalid}} which 
means when folks have a problem in their schema logical types just disappear 
with no debugging information.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1895) DeepCopy does not work with logical types

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15453696#comment-15453696
 ] 

Sean Busbey commented on AVRO-1895:
---

{code}+Conversion conversion = getConversionByClass(value.getClass());{code}

should be {{getConversionByClass(value.getClass(), logicalType)}}, I think?

> DeepCopy does not work with logical types
> -
>
> Key: AVRO-1895
> URL: https://issues.apache.org/jira/browse/AVRO-1895
> Project: Avro
>  Issue Type: Improvement
>  Components: logical types
>Affects Versions: 1.8.1
>Reporter: Taras Bobrovytsky
>Assignee: Doug Cutting
>Priority: Critical
> Attachments: AVRO-1895.patch, AVRO-1895.patch, AVRO-1895.patch, 
> AVRO-1895.patch
>
>
> AvroSchema is taken from a compiled avsc file which contains a decimal field.
> {code}
> AvroSchema.Builder builder = AvroSchema.newBuilder();
> BigDecimal bd = new BigDecimal(new BigInteger("155"), 3);
> campaignBuilder.setDecimalField(bd);
> AvroSchema source = builder.build();
> //This line causes an exception
> AvroSchema.Builder builder1 = AvroSchema.newBuilder(source);
> {code}
> Exception:
> {code}
> InvocationTargetException: java.math.BigDecimal cannot be cast to 
> java.nio.ByteBuffer
> {code}
> The same failure happens with GenericData as well:
> {code}
> GenericRecord copy = GenericData.get().deepCopy(AvroSchema.getClassSchema(), 
> source);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1895) DeepCopy does not work with logical types

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15453480#comment-15453480
 ] 

Sean Busbey commented on AVRO-1895:
---

My concern is that reading through GenericData it looks like I could register 
multiple conversions for a given logical type that differ only in their 
expected converted type. Looking up with {{getConversion(LogicalType)}} will 
only succeed in getting the last such conversion registered for the 
logicalType, whereas {{getConversionByClass (Class, LogicalType)}} would 
work for the last registered conversion for each logicalType/converted-type 
pair. This would be useful, for example, if you were trying to coalesce data 
from multiple libraries that themselves used different converted types for a 
given logical type.

> DeepCopy does not work with logical types
> -
>
> Key: AVRO-1895
> URL: https://issues.apache.org/jira/browse/AVRO-1895
> Project: Avro
>  Issue Type: Improvement
>  Components: logical types
>Affects Versions: 1.8.1
>Reporter: Taras Bobrovytsky
>Assignee: Doug Cutting
>Priority: Critical
> Attachments: AVRO-1895.patch, AVRO-1895.patch, AVRO-1895.patch
>
>
> AvroSchema is taken from a compiled avsc file which contains a decimal field.
> {code}
> AvroSchema.Builder builder = AvroSchema.newBuilder();
> BigDecimal bd = new BigDecimal(new BigInteger("155"), 3);
> campaignBuilder.setDecimalField(bd);
> AvroSchema source = builder.build();
> //This line causes an exception
> AvroSchema.Builder builder1 = AvroSchema.newBuilder(source);
> {code}
> Exception:
> {code}
> InvocationTargetException: java.math.BigDecimal cannot be cast to 
> java.nio.ByteBuffer
> {code}
> The same failure happens with GenericData as well:
> {code}
> GenericRecord copy = GenericData.get().deepCopy(AvroSchema.getClassSchema(), 
> source);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1895) DeepCopy does not work with logical types

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15453256#comment-15453256
 ] 

Sean Busbey commented on AVRO-1895:
---

Just occurred to me: Do we need to check that T is the type expected by the 
conversion ({{getConvertedType}})? Should we be using 
{{getConversionByClass(Class, LogicalType)}} instead of 
{{getConversion(LogicalType)}}?

> DeepCopy does not work with logical types
> -
>
> Key: AVRO-1895
> URL: https://issues.apache.org/jira/browse/AVRO-1895
> Project: Avro
>  Issue Type: Improvement
>  Components: logical types
>Affects Versions: 1.8.1
>Reporter: Taras Bobrovytsky
>Assignee: Doug Cutting
>Priority: Critical
> Attachments: AVRO-1895.patch, AVRO-1895.patch, AVRO-1895.patch
>
>
> AvroSchema is taken from a compiled avsc file which contains a decimal field.
> {code}
> AvroSchema.Builder builder = AvroSchema.newBuilder();
> BigDecimal bd = new BigDecimal(new BigInteger("155"), 3);
> campaignBuilder.setDecimalField(bd);
> AvroSchema source = builder.build();
> //This line causes an exception
> AvroSchema.Builder builder1 = AvroSchema.newBuilder(source);
> {code}
> Exception:
> {code}
> InvocationTargetException: java.math.BigDecimal cannot be cast to 
> java.nio.ByteBuffer
> {code}
> The same failure happens with GenericData as well:
> {code}
> GenericRecord copy = GenericData.get().deepCopy(AvroSchema.getClassSchema(), 
> source);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (AVRO-1905) Backward and forward compatible

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey resolved AVRO-1905.
---
Resolution: Information Provided
  Assignee: Sean Busbey

When using Apache Avro as a serialization format, you must always know what 
schema was used to write the data and present that schema to the library as a 
part of reading.

In your examples above, it appears you are attempting to read data written with 
_old_schema_ using only _new_schema_, which will not work. Avro will need to 
know both _old_schemea_ and _new_schema_.

Please use the user@avro mailing list for help with getting started using Avro: 
http://avro.apache.org/mailing_lists.html#Users. It is easiest to do so by 
first subscribing to the user list and then posting your question.

> Backward and forward compatible
> ---
>
> Key: AVRO-1905
> URL: https://issues.apache.org/jira/browse/AVRO-1905
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Reporter: Amod Kumar Pandey
>    Assignee: Sean Busbey
>
> I am under the understanding that Avro is both backward and forward 
> compatible (for certain schema changes). But as per my test it is neither 
> backward nor forward compatible.
> Maven project with avro
> {code}
> {"namespace": "example.avro",
>  "type": "record",
>  "name": "user",
>  "fields": [
>  {"name": "name", "type": "string"},
>  {"name": "favorite_number",  "type": "int"}
>  ]
> }
> {code}
> Producer
> {code}
> public class Producer {
>   public static void main(String[] args) throws IOException {
> try (ByteArrayOutputStream outputStream = new ByteArrayOutputStream()) {
>   user u1 = 
> user.newBuilder().setFavoriteNumber(1).setName("Amod").build();
>   writeBinaryEncodedAvro(u1, outputStream);
>   user u2 = 
> user.newBuilder().setFavoriteNumber(2).setName("Pandey").build();
>   writeBinaryEncodedAvro(u2, outputStream);
>   System.out.println(Arrays.toString(outputStream.toByteArray()));
> }
>   }
>   static void writeBinaryEncodedAvro(SpecificRecord specificRecord, 
> OutputStream os) throws IOException {
> BinaryEncoder binaryEncoder = EncoderFactory.get().binaryEncoder(os, 
> null);
> @SuppressWarnings("unchecked")
> DatumWriter datumWriter =
> new SpecificDatumWriter((Class) 
> specificRecord.getClass());
> datumWriter.write(specificRecord, binaryEncoder);
> binaryEncoder.flush();
>   }
> }
> {code}
> Consumer 
> {code}
> public class Consumer {
>   public static void main(String[] args) throws IOException {
> byte[] data = {8, 65, 109, 111, 100, 2, 10, 103, 114, 101, 101, 110};
> try (ByteArrayInputStream inputStream = new ByteArrayInputStream(data)) {
>   System.out.println(fromBinaryMulti(inputStream));
> }
>   }
>   static user fromBinary(InputStream is) throws IOException {
> BinaryDecoder binaryDecoder = DecoderFactory.get().binaryDecoder(is, 
> null);
> DatumReader datumReader = new SpecificDatumReader(user.class);
> return datumReader.read(null, binaryDecoder);
>   }
>   static List fromBinaryMulti(InputStream is) throws IOException {
> List users = new ArrayList();
> BinaryDecoder binaryDecoder = DecoderFactory.get().binaryDecoder(is, 
> null);
> while (!binaryDecoder.isEnd()) {
>   DatumReader datumReader = new 
> SpecificDatumReader(user.class);
>   users.add(datumReader.read(null, binaryDecoder));
> }
> return users;
>   }
> }
> {code}
> I changed the schema to 
> {code}
> {"namespace": "example.avro",
>  "type": "record",
>  "name": "user",
>  "fields": [
>  {"name": "name", "type": "string"},
>  {"name": "favorite_number",  "type": "int"},
>  {"name": "favorite_color", "type": "string", "default": "green"}
>  ]
> }
> {code}
> The following does not work.
> Consume code using new schema generated code cannot consume byte array 
> generated by old schema.
> Consume code using old schema generated code cannot consume byte array 
> generated with new schema.
> Is there a problem in the way I am trying to understand the forward or 
> backward compatibility.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1905) Backward and forward compatible

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1905:
--
Description: 
I am under the understanding that Avro is both backward and forward compatible 
(for certain schema changes). But as per my test it is neither backward nor 
forward compatible.

Maven project with avro
{code}
{"namespace": "example.avro",
 "type": "record",
 "name": "user",
 "fields": [
 {"name": "name", "type": "string"},
 {"name": "favorite_number",  "type": "int"}
 ]
}
{code}
Producer

{code}
public class Producer {
  public static void main(String[] args) throws IOException {
try (ByteArrayOutputStream outputStream = new ByteArrayOutputStream()) {
  user u1 = user.newBuilder().setFavoriteNumber(1).setName("Amod").build();
  writeBinaryEncodedAvro(u1, outputStream);
  user u2 = 
user.newBuilder().setFavoriteNumber(2).setName("Pandey").build();
  writeBinaryEncodedAvro(u2, outputStream);
  System.out.println(Arrays.toString(outputStream.toByteArray()));
}
  }

  static void writeBinaryEncodedAvro(SpecificRecord specificRecord, 
OutputStream os) throws IOException {
BinaryEncoder binaryEncoder = EncoderFactory.get().binaryEncoder(os, null);
@SuppressWarnings("unchecked")
DatumWriter datumWriter =
new SpecificDatumWriter((Class) 
specificRecord.getClass());
datumWriter.write(specificRecord, binaryEncoder);
binaryEncoder.flush();
  }

}
{code}

Consumer 

{code}
public class Consumer {
  public static void main(String[] args) throws IOException {
byte[] data = {8, 65, 109, 111, 100, 2, 10, 103, 114, 101, 101, 110};
try (ByteArrayInputStream inputStream = new ByteArrayInputStream(data)) {
  System.out.println(fromBinaryMulti(inputStream));
}
  }

  static user fromBinary(InputStream is) throws IOException {
BinaryDecoder binaryDecoder = DecoderFactory.get().binaryDecoder(is, null);
DatumReader datumReader = new SpecificDatumReader(user.class);
return datumReader.read(null, binaryDecoder);
  }

  static List fromBinaryMulti(InputStream is) throws IOException {
List users = new ArrayList();
BinaryDecoder binaryDecoder = DecoderFactory.get().binaryDecoder(is, null);
while (!binaryDecoder.isEnd()) {
  DatumReader datumReader = new SpecificDatumReader(user.class);
  users.add(datumReader.read(null, binaryDecoder));
}
return users;
  }

}
{code}

I changed the schema to 

{code}
{"namespace": "example.avro",
 "type": "record",
 "name": "user",
 "fields": [
 {"name": "name", "type": "string"},
 {"name": "favorite_number",  "type": "int"},
 {"name": "favorite_color", "type": "string", "default": "green"}
 ]
}
{code}

The following does not work.

Consume code using new schema generated code cannot consume byte array 
generated by old schema.

Consume code using old schema generated code cannot consume byte array 
generated with new schema.

Is there a problem in the way I am trying to understand the forward or backward 
compatibility.

  was:
I am under the understanding that Avro is both backward and forward compatible 
(for certain schema changes). But as per my test it is neither backward nor 
forward compatible.

Maven project with avro

{"namespace": "example.avro",
 "type": "record",
 "name": "user",
 "fields": [
 {"name": "name", "type": "string"},
 {"name": "favorite_number",  "type": "int"}
 ]
}

Producer

public class Producer {
  public static void main(String[] args) throws IOException {
try (ByteArrayOutputStream outputStream = new ByteArrayOutputStream()) {
  user u1 = user.newBuilder().setFavoriteNumber(1).setName("Amod").build();
  writeBinaryEncodedAvro(u1, outputStream);
  user u2 = 
user.newBuilder().setFavoriteNumber(2).setName("Pandey").build();
  writeBinaryEncodedAvro(u2, outputStream);
  System.out.println(Arrays.toString(outputStream.toByteArray()));
}
  }

  static void writeBinaryEncodedAvro(SpecificRecord specificRecord, 
OutputStream os) throws IOException {
BinaryEncoder binaryEncoder = EncoderFactory.get().binaryEncoder(os, null);
@SuppressWarnings("unchecked")
DatumWriter datumWriter =
new SpecificDatumWriter((Class) 
specificRecord.getClass());
datumWriter.write(specificRecord, binaryEncoder);
binaryEncoder.flush();
  }

}

Co

[jira] [Commented] (AVRO-1895) DeepCopy does not work with logical types

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15452355#comment-15452355
 ] 

Sean Busbey commented on AVRO-1895:
---

patch looks good [~cutting]. a few points of feedback:

* could we update the javadoc for deepcopy to call out that logical types may 
go through a round trip of conversion? Just to make sure folks are aware in the 
case of expensive conversions.
* could we update the test to include a logical type nested in a complex type 
deeper than just a containing record? looking at the patch, I believe it should 
work properly but I'd rather avoid having that large of a gap.
* could we include a test that we maintain expected behavior for a field with a 
logical type when there are no conversions present?

> DeepCopy does not work with logical types
> -
>
> Key: AVRO-1895
> URL: https://issues.apache.org/jira/browse/AVRO-1895
> Project: Avro
>  Issue Type: Improvement
>  Components: logical types
>Affects Versions: 1.8.1
>Reporter: Taras Bobrovytsky
>Assignee: Doug Cutting
>Priority: Critical
> Attachments: AVRO-1895.patch, AVRO-1895.patch
>
>
> AvroSchema is taken from a compiled avsc file which contains a decimal field.
> {code}
> AvroSchema.Builder builder = AvroSchema.newBuilder();
> BigDecimal bd = new BigDecimal(new BigInteger("155"), 3);
> campaignBuilder.setDecimalField(bd);
> AvroSchema source = builder.build();
> //This line causes an exception
> AvroSchema.Builder builder1 = AvroSchema.newBuilder(source);
> {code}
> Exception:
> {code}
> InvocationTargetException: java.math.BigDecimal cannot be cast to 
> java.nio.ByteBuffer
> {code}
> The same failure happens with GenericData as well:
> {code}
> GenericRecord copy = GenericData.get().deepCopy(AvroSchema.getClassSchema(), 
> source);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1895) DeepCopy does not work with logical types

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15452334#comment-15452334
 ] 

Sean Busbey commented on AVRO-1895:
---

{quote}
bq. The workaround in AVRO-1891 may not help here
We should find out.
{quote}

I confirmed that the workaround in AVRO-1891 (adding conversions to the 
GenericData instance) doesn't help, fyi.

> DeepCopy does not work with logical types
> -
>
> Key: AVRO-1895
> URL: https://issues.apache.org/jira/browse/AVRO-1895
> Project: Avro
>  Issue Type: Improvement
>  Components: logical types
>Affects Versions: 1.8.1
>Reporter: Taras Bobrovytsky
>Assignee: Doug Cutting
>Priority: Critical
> Attachments: AVRO-1895.patch, AVRO-1895.patch
>
>
> AvroSchema is taken from a compiled avsc file which contains a decimal field.
> {code}
> AvroSchema.Builder builder = AvroSchema.newBuilder();
> BigDecimal bd = new BigDecimal(new BigInteger("155"), 3);
> campaignBuilder.setDecimalField(bd);
> AvroSchema source = builder.build();
> //This line causes an exception
> AvroSchema.Builder builder1 = AvroSchema.newBuilder(source);
> {code}
> Exception:
> {code}
> InvocationTargetException: java.math.BigDecimal cannot be cast to 
> java.nio.ByteBuffer
> {code}
> The same failure happens with GenericData as well:
> {code}
> GenericRecord copy = GenericData.get().deepCopy(AvroSchema.getClassSchema(), 
> source);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1874) py3 avro module import upsets logging level in host application

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1874:
--
Priority: Critical  (was: Major)

> py3 avro module import upsets logging level in host application
> ---
>
> Key: AVRO-1874
> URL: https://issues.apache.org/jira/browse/AVRO-1874
> Project: Avro
>  Issue Type: Bug
>  Components: python
>Affects Versions: 1.8.1
> Environment: Mac OSX El Capitan, Macbook Pro,
> Anaconda Python v. 3.5.1
> Avro installed from source of Avro1.8.1/lang/py3
> (apache package "avro-src-1.8.1.tar.gz")
> using "sudo python setup.py install"
>Reporter: Torgeir Børresen
>Assignee: Torgeir Børresen
>Priority: Critical
> Fix For: 1.8.2
>
>
> When importing "avro.datafile" the logging level of the host application gets 
> overriden.
> In the simple example provided here: 
> https://github.com/torgebo/avro-1.8.1-logging-break
> the logging level is wrongfully set to "logging.WARNING" during execution 
> instead of "logging.INFO".
> The issue seems to be resolved by using module level loggers in the pattern of
> logger = logging.getLogger(__name__)
> and replacing current calls to the logger named "logging" as this logger 
> "logger" instead. This approach is described here: 
> https://docs.python.org/3/howto/logging.html#logging-advanced-tutorial
> When setting logger across all avro source files, it is observed that the 
> application sets the logging level faithfully.
> 
> This issue was not observed with python version 2, although the recommended 
> way to resolve module level logging as described in the logging python docs 
> seems to be the same (ie. using the logging.getLogger method to access the 
> logger handle).
> 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1874) py3 avro module import upsets logging level in host application

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1874:
--
Assignee: Torgeir Børresen
  Status: Patch Available  (was: Open)

Hi Torgeir!

I've moved this into patch available so that it'll show up for folks who are 
looking to do reviews of changes. Sorry no one noticed it was ready earlier.

The best way to get things reviewed faster is to find other waiting "Patch 
Available" changes and help review them. Every set of eyes makes it faster for 
committers to check things out and merge them in.

> py3 avro module import upsets logging level in host application
> ---
>
> Key: AVRO-1874
> URL: https://issues.apache.org/jira/browse/AVRO-1874
> Project: Avro
>  Issue Type: Bug
>  Components: python
>Affects Versions: 1.8.1
> Environment: Mac OSX El Capitan, Macbook Pro,
> Anaconda Python v. 3.5.1
> Avro installed from source of Avro1.8.1/lang/py3
> (apache package "avro-src-1.8.1.tar.gz")
> using "sudo python setup.py install"
>Reporter: Torgeir Børresen
>Assignee: Torgeir Børresen
> Fix For: 1.8.2
>
>
> When importing "avro.datafile" the logging level of the host application gets 
> overriden.
> In the simple example provided here: 
> https://github.com/torgebo/avro-1.8.1-logging-break
> the logging level is wrongfully set to "logging.WARNING" during execution 
> instead of "logging.INFO".
> The issue seems to be resolved by using module level loggers in the pattern of
> logger = logging.getLogger(__name__)
> and replacing current calls to the logger named "logging" as this logger 
> "logger" instead. This approach is described here: 
> https://docs.python.org/3/howto/logging.html#logging-advanced-tutorial
> When setting logger across all avro source files, it is observed that the 
> application sets the logging level faithfully.
> 
> This issue was not observed with python version 2, although the recommended 
> way to resolve module level logging as described in the logging python docs 
> seems to be the same (ie. using the logging.getLogger method to access the 
> logger handle).
> 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1895) DeepCopy does not work with logical types

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1895:
--
Priority: Critical  (was: Major)

> DeepCopy does not work with logical types
> -
>
> Key: AVRO-1895
> URL: https://issues.apache.org/jira/browse/AVRO-1895
> Project: Avro
>  Issue Type: Improvement
>  Components: logical types
>Affects Versions: 1.8.1
>Reporter: Taras Bobrovytsky
>Priority: Critical
>
> AvroSchema is taken from a compiled avsc file which contains a decimal field.
> {code}
> AvroSchema.Builder builder = AvroSchema.newBuilder();
> BigDecimal bd = new BigDecimal(new BigInteger("155"), 3);
> campaignBuilder.setDecimalField(bd);
> AvroSchema source = builder.build();
> //This line causes an exception
> AvroSchema.Builder builder1 = AvroSchema.newBuilder(source);
> {code}
> Exception:
> {code}
> InvocationTargetException: java.math.BigDecimal cannot be cast to 
> java.nio.ByteBuffer
> {code}
> The same failure happens with GenericData as well:
> {code}
> GenericRecord copy = GenericData.get().deepCopy(AvroSchema.getClassSchema(), 
> source);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1897) clean checkout fails when following BUILD.txt instructions for test

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15449015#comment-15449015
 ] 

Sean Busbey commented on AVRO-1897:
---

if this gets us through python working, I'm happy to handle c++ in an 
additional ticket.

is python similarly impacted on clean checkout / repository if one does 
{{./build.sh dist}} instead of {{./build.sh test}}?

> clean checkout fails when following BUILD.txt instructions for test
> ---
>
> Key: AVRO-1897
> URL: https://issues.apache.org/jira/browse/AVRO-1897
> Project: Avro
>  Issue Type: Bug
>  Components: build, python
>Affects Versions: 1.8.1
>    Reporter: Sean Busbey
>Assignee: Suraj Acharya
>Priority: Blocker
> Fix For: 1.8.2
>
> Attachments: AVRO-1897.patch
>
>
> Clean checkout of branch-1.8 run in docker mode fails {{./build.sh test}} on 
> the python module because it can't find a copy of the avro-tools jar.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1886) Improve error reporting for schema validation

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436759#comment-15436759
 ] 

Sean Busbey commented on AVRO-1886:
---

Sorry for the lag on reviews. I'm going to try to block out some time this 
weekend to do a pass through things in patch-available status.

The most impactful thing folks can do to speed up reviews of their patches is 
to review the patches of other folks, since that makes it easier for a commiter 
to work on things if another set of eyes has already done a round of feedback.

> Improve error reporting for schema validation
> -
>
> Key: AVRO-1886
> URL: https://issues.apache.org/jira/browse/AVRO-1886
> Project: Avro
>  Issue Type: Improvement
>  Components: ruby
>Reporter: Miroslav Csonka
>Assignee: Miroslav Csonka
> Fix For: 1.9.0, 1.8.2
>
>
> In Funding Circle we use avro schema to validate some financial data and 
> we've found ourselves wondering what is invalid. We've noticed that avro 
> [does already 
> validation|https://github.com/apache/avro/blob/master/lang/ruby/lib/avro/schema.rb#L94],
>  but does not report what fields are invalid, so we've decided to change it 
> and we use Avro as a validation library.
> h5. What error messages to expect?
> We communicate errors for a given payload in format "at   
> ".
> For  we've decided to use syntax similar to json path:
> * . refers to root
> * \[0\] first index in array
> * .age would match 42 from {"age": 42}
> Error within object *person* and his optional array *houses* on *second item* 
> with key *number_of_rooms*:
> {quote}
> at .person.houses\[1\].number_of_rooms expected type long, got string 
> with value "not valid at all"
> {quote}
> Example of multiple errors for a single payload:
> {quote}
> at .\[0\] expected type int, got null
> at .\[1\] expected type int, got string with value \"e\"
> {quote}
> More examples can be seen at 
> https://github.com/apache/avro/pull/111/files#diff-830ac80276854fceefa3f92c346d165f



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1900) build.sh test fails for dev-tools.jar

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433348#comment-15433348
 ] 

Sean Busbey commented on AVRO-1900:
---

I think we also need to change how {{./build.sh dist}} works, but I haven't 
confirmed yet.

> build.sh test fails for dev-tools.jar
> -
>
> Key: AVRO-1900
> URL: https://issues.apache.org/jira/browse/AVRO-1900
> Project: Avro
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Suraj Acharya
>Assignee: Suraj Acharya
>  Labels: build
> Attachments: AVRO-1900.patch
>
>
> When i ran {{./build.sh test}} in the docker container I was getting an error 
> mentioning dev-tools.jar was not present.
> When I looked further into it I realized that {{build.sh}} never actually 
> builds dev-tools.jar.
> I added a line in the test option to first {{mvn install}} on the dev-tools 
> folder. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1900) build.sh test fails for dev-tools.jar

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1900:
--
Assignee: Suraj Acharya

> build.sh test fails for dev-tools.jar
> -
>
> Key: AVRO-1900
> URL: https://issues.apache.org/jira/browse/AVRO-1900
> Project: Avro
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Suraj Acharya
>Assignee: Suraj Acharya
>  Labels: build
> Attachments: AVRO-1900.patch
>
>
> When i ran {{./build.sh test}} in the docker container I was getting an error 
> mentioning dev-tools.jar was not present.
> When I looked further into it I realized that {{build.sh}} never actually 
> builds dev-tools.jar.
> I added a line in the test option to first {{mvn install}} on the dev-tools 
> folder. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Pre-commit Check Avro

[Sean Busbey]

If we have files that fail to include the required ASF licensing
headers, you're correct that's important to fix.

I think either working on it before or after would be fine. Ideally,
Yetus Precommit should properly let us know when a patch is fixing
those kinds of problems, so it would be a good validation that things
are working.

On Thu, Aug 18, 2016 at 8:18 PM, suraj acharya <suraj@gmail.com> wrote:
> Okay. I will look at the docker command and the image.
>
> One other question I have is that the asflicense is an important aspect of
> yetus. However, many java files have that missing. And whenever a patch
> touches that file it returns a -1. Do you think I should first fix all the
> licenses and then continue with this?
>
> S
>
> -Suraj Acharya
>
> On Thu, Aug 18, 2016 at 12:37 PM, Sean Busbey <bus...@cloudera.com> wrote:
>
>> is the 3-5 minutes doing the tests across all of the language
>> libraries or just the java ones?
>>
>> docker will definitely be needed, due to the number of different
>> system dependencies needed to build the different languages.
>> I was hoping we could reuse the Docker image that is currently
>> used for the "./build.sh docker" command.
>>
>>
>> On Wed, Aug 17, 2016 at 6:41 PM, suraj acharya <suraj@gmail.com>
>> wrote:
>> > Hi,
>> > I am a new guy and I have been working on setting up pre-commit check for
>> > Avro : AVRO-1887 <https://issues.apache.org/jira/browse/AVRO-1887>.
>> >
>> > I was looking at the unit tests being run currently. It takes somewhere
>> > around 3-5 minutes depending on the network and the repo's present.
>> > I was thinking that since the time is so low why not run all the tests
>> for
>> > every commit. Also, the tests being lightweight I was thinking that there
>> > is no need for docker to be needed.
>> >
>> > I would like to gather your opinions on this topic before going further
>> > down this path.
>> >
>> > Thanks
>> >
>> > -Suraj Acharya
>>
>>
>>
>> --
>> busbey
>>



-- 
busbey

Re: Pre-commit Check Avro

[Sean Busbey]

is the 3-5 minutes doing the tests across all of the language
libraries or just the java ones?

docker will definitely be needed, due to the number of different
system dependencies needed to build the different languages.
I was hoping we could reuse the Docker image that is currently
used for the "./build.sh docker" command.


On Wed, Aug 17, 2016 at 6:41 PM, suraj acharya  wrote:
> Hi,
> I am a new guy and I have been working on setting up pre-commit check for
> Avro : AVRO-1887 .
>
> I was looking at the unit tests being run currently. It takes somewhere
> around 3-5 minutes depending on the network and the repo's present.
> I was thinking that since the time is so low why not run all the tests for
> every commit. Also, the tests being lightweight I was thinking that there
> is no need for docker to be needed.
>
> I would like to gather your opinions on this topic before going further
> down this path.
>
> Thanks
>
> -Suraj Acharya



-- 
busbey

[jira] [Commented] (AVRO-1897) clean checkout fails when following BUILD.txt instructions for test

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411938#comment-15411938
 ] 

Sean Busbey commented on AVRO-1897:
---

we either need to

1) update the BUILD.txt instructions to manually do a local install of the 
avro-tools jar
2) update the build.sh script for python to do a local install of the 
avro-tools jar (possibly in a dedicated local repository to avoid polluting the 
default local repository used by maven).

> clean checkout fails when following BUILD.txt instructions for test
> ---
>
> Key: AVRO-1897
> URL: https://issues.apache.org/jira/browse/AVRO-1897
> Project: Avro
>  Issue Type: Bug
>  Components: build, python
>Affects Versions: 1.8.1
>    Reporter: Sean Busbey
>Priority: Blocker
> Fix For: 1.8.2
>
>
> Clean checkout of branch-1.8 run in docker mode fails {{./build.sh test}} on 
> the python module because it can't find a copy of the avro-tools jar.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (AVRO-1897) clean checkout fails when following BUILD.txt instructions for test

[Sean Busbey (JIRA)]

Sean Busbey created AVRO-1897:
-

 Summary: clean checkout fails when following BUILD.txt 
instructions for test
 Key: AVRO-1897
 URL: https://issues.apache.org/jira/browse/AVRO-1897
 Project: Avro
  Issue Type: Bug
  Components: build, python
Affects Versions: 1.8.1
Reporter: Sean Busbey
Priority: Blocker
 Fix For: 1.8.2


Clean checkout of branch-1.8 run in docker mode fails {{./build.sh test}} on 
the python module because it can't find a copy of the avro-tools jar.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411928#comment-15411928
 ] 

Sean Busbey commented on AVRO-1891:
---

yes, we should expect those to work.

> Generated Java code fails with union containing logical type
> 
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
>  Issue Type: Bug
>  Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
>
> Example schema:
> {code}
> {
>   "type": "record",
>   "name": "RecordV1",
>   "namespace": "org.brasslock.event",
>   "fields": [
> { "name": "first", "type": ["null", {"type": "long", 
> "logicalType":"timestamp-millis"}]}
>   ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new 
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new 
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder = 
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
>  org.apache.avro.AvroRuntimeException: Unknown datum type 
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at 
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at 
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at 
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at 
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
> at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTe

[jira] [Updated] (AVRO-1894) GenericData does not add Logical Type conversions by default

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1894:
--
Component/s: logical types
 java

> GenericData does not add Logical Type conversions by default
> 
>
> Key: AVRO-1894
> URL: https://issues.apache.org/jira/browse/AVRO-1894
> Project: Avro
>  Issue Type: Improvement
>  Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Taras Bobrovytsky
>
> In order to support Decimal fields, the following line needs to be added:
> {code}
> new GenericData().addLogicalTypeConversion(new 
> Conversions.DecimalConversion());
> {code}
> It would be more convenient if logical type conversions were added by default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1894) GenericData does not add Logical Type conversions by default

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408436#comment-15408436
 ] 

Sean Busbey commented on AVRO-1894:
---

Or a subclass of GenericData that included them?

> GenericData does not add Logical Type conversions by default
> 
>
> Key: AVRO-1894
> URL: https://issues.apache.org/jira/browse/AVRO-1894
> Project: Avro
>  Issue Type: Improvement
>Affects Versions: 1.8.1
>Reporter: Taras Bobrovytsky
>
> In order to support Decimal fields, the following line needs to be added:
> {code}
> new GenericData().addLogicalTypeConversion(new 
> Conversions.DecimalConversion());
> {code}
> It would be more convenient if logical type conversions were added by default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1894) GenericData does not add Logical Type conversions by default

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408435#comment-15408435
 ] 

Sean Busbey commented on AVRO-1894:
---

how about better convenience methods (or Builders?) for creating the 
GenericData with the conversions?

> GenericData does not add Logical Type conversions by default
> 
>
> Key: AVRO-1894
> URL: https://issues.apache.org/jira/browse/AVRO-1894
> Project: Avro
>  Issue Type: Improvement
>Affects Versions: 1.8.1
>Reporter: Taras Bobrovytsky
>
> In order to support Decimal fields, the following line needs to be added:
> {code}
> new GenericData().addLogicalTypeConversion(new 
> Conversions.DecimalConversion());
> {code}
> It would be more convenient if logical type conversions were added by default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: verifying the build instructions

[Sean Busbey]

I pulled from the git repo last night to run through things again, but
I didn't see any revert.

On Wed, Aug 3, 2016 at 2:36 PM, Sean Busbey <bus...@cloudera.com> wrote:
> sweet. I'll pull down an updated repo tonight and restart the run.
>
> On Wed, Aug 3, 2016 at 11:41 AM, Ryan Blue <rb...@netflix.com.invalid> wrote:
>> Sorry about this, I ran into it the last time and forgot to fix the problem.
>>
>> The addition of the dev-tools Java module (AVRO-1838) to run checkstyle
>> isn't compatible with our current script. I think the easiest thing to do
>> is to revert the changes that added the module and instead patch the
>> original checkstyle rules. To get the 1.8.1 release out, I just reverted
>> the dev-tools module since it only affected the build.
>>
>> rb
>>
>> On Wed, Aug 3, 2016 at 6:55 AM, Sean Busbey <bus...@cloudera.com> wrote:
>>
>>> This weekend I tried getting my ASF release machine to do a full
>>> build, in prep for a 1.8 RC. I followed the instructions in BUILD.txt
>>> to use docker to handle the dependencies.
>>>
>>> Once I was in the docker shell, I tried both `./build.sh test` and
>>> `./build.sh dist` and both failed while trying to build the java
>>> language libraries due to an unsatisfied dependency on
>>> org.apache.avro:dev-tools.
>>>
>>> Anyone know if this is more likely to be an error in my setup vs a gap
>>> in the build script? My intuition says gap in the build script, but I
>>> haven't had time to dig in at all.
>>>
>>> --
>>> busbey
>>>
>>
>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>
>
>
> --
> busbey



-- 
busbey

Re: verifying the build instructions

[Sean Busbey]

sweet. I'll pull down an updated repo tonight and restart the run.

On Wed, Aug 3, 2016 at 11:41 AM, Ryan Blue <rb...@netflix.com.invalid> wrote:
> Sorry about this, I ran into it the last time and forgot to fix the problem.
>
> The addition of the dev-tools Java module (AVRO-1838) to run checkstyle
> isn't compatible with our current script. I think the easiest thing to do
> is to revert the changes that added the module and instead patch the
> original checkstyle rules. To get the 1.8.1 release out, I just reverted
> the dev-tools module since it only affected the build.
>
> rb
>
> On Wed, Aug 3, 2016 at 6:55 AM, Sean Busbey <bus...@cloudera.com> wrote:
>
>> This weekend I tried getting my ASF release machine to do a full
>> build, in prep for a 1.8 RC. I followed the instructions in BUILD.txt
>> to use docker to handle the dependencies.
>>
>> Once I was in the docker shell, I tried both `./build.sh test` and
>> `./build.sh dist` and both failed while trying to build the java
>> language libraries due to an unsatisfied dependency on
>> org.apache.avro:dev-tools.
>>
>> Anyone know if this is more likely to be an error in my setup vs a gap
>> in the build script? My intuition says gap in the build script, but I
>> haven't had time to dig in at all.
>>
>> --
>> busbey
>>
>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix



-- 
busbey

verifying the build instructions

[Sean Busbey]

This weekend I tried getting my ASF release machine to do a full
build, in prep for a 1.8 RC. I followed the instructions in BUILD.txt
to use docker to handle the dependencies.

Once I was in the docker shell, I tried both `./build.sh test` and
`./build.sh dist` and both failed while trying to build the java
language libraries due to an unsatisfied dependency on
org.apache.avro:dev-tools.

Anyone know if this is more likely to be an error in my setup vs a gap
in the build script? My intuition says gap in the build script, but I
haven't had time to dig in at all.

-- 
busbey

[jira] [Updated] (AVRO-1891) Generated Java code fails with union containing logical type

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1891:
--
Component/s: logical types

> Generated Java code fails with union containing logical type
> 
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
>  Issue Type: Bug
>  Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
>
> Example schema:
> {code}
> {
>   "type": "record",
>   "name": "RecordV1",
>   "namespace": "org.brasslock.event",
>   "fields": [
> { "name": "first", "type": ["null", {"type": "long", 
> "logicalType":"timestamp-millis"}]}
>   ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new 
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new 
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder = 
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
>  org.apache.avro.AvroRuntimeException: Unknown datum type 
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at 
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at 
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at 
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at 
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
> at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
> at 
> com.in

[jira] [Updated] (AVRO-1891) Generated Java code fails with union containing logical type

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1891:
--
Fix Version/s: 1.8.3

> Generated Java code fails with union containing logical type
> 
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
>  Issue Type: Bug
>  Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
> Fix For: 1.8.3
>
>
> Example schema:
> {code}
> {
>   "type": "record",
>   "name": "RecordV1",
>   "namespace": "org.brasslock.event",
>   "fields": [
> { "name": "first", "type": ["null", {"type": "long", 
> "logicalType":"timestamp-millis"}]}
>   ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new 
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new 
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder = 
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
>  org.apache.avro.AvroRuntimeException: Unknown datum type 
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at 
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at 
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at 
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at 
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
> at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
> at 
> com.intellij.rt.execution

[jira] [Updated] (AVRO-1891) Generated Java code fails with union containing logical type

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1891:
--
Priority: Blocker  (was: Major)

> Generated Java code fails with union containing logical type
> 
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
>  Issue Type: Bug
>  Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
>
> Example schema:
> {code}
> {
>   "type": "record",
>   "name": "RecordV1",
>   "namespace": "org.brasslock.event",
>   "fields": [
> { "name": "first", "type": ["null", {"type": "long", 
> "logicalType":"timestamp-millis"}]}
>   ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new 
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new 
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder = 
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
>  org.apache.avro.AvroRuntimeException: Unknown datum type 
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at 
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at 
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at 
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at 
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
> at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
> at 
> com.in

[jira] [Updated] (AVRO-1704) Standardized format for encoding messages with Avro

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1704:
--
Component/s: spec
 java

> Standardized format for encoding messages with Avro
> ---
>
> Key: AVRO-1704
> URL: https://issues.apache.org/jira/browse/AVRO-1704
> Project: Avro
>  Issue Type: Improvement
>  Components: java, spec
>Reporter: Daniel Schierbeck
>Assignee: Niels Basjes
> Fix For: 1.9.0, 1.8.3
>
> Attachments: AVRO-1704-2016-05-03-Unfinished.patch, 
> AVRO-1704-20160410.patch, AVRO-1704.3.patch, AVRO-1704.4.patch
>
>
> I'm currently using the Datafile format for encoding messages that are 
> written to Kafka and Cassandra. This seems rather wasteful:
> 1. I only encode a single record at a time, so there's no need for sync 
> markers and other metadata related to multi-record files.
> 2. The entire schema is inlined every time.
> However, the Datafile format is the only one that has been standardized, 
> meaning that I can read and write data with minimal effort across the various 
> languages in use in my organization. If there was a standardized format for 
> encoding single values that was optimized for out-of-band schema transfer, I 
> would much rather use that.
> I think the necessary pieces of the format would be:
> 1. A format version number.
> 2. A schema fingerprint type identifier, i.e. Rabin, MD5, SHA256, etc.
> 3. The actual schema fingerprint (according to the type.)
> 4. Optional metadata map.
> 5. The encoded datum.
> The language libraries would implement a MessageWriter that would encode 
> datums in this format, as well as a MessageReader that, given a SchemaStore, 
> would be able to decode datums. The reader would decode the fingerprint and 
> ask its SchemaStore to return the corresponding writer's schema.
> The idea is that SchemaStore would be an abstract interface that allowed 
> library users to inject custom backends. A simple, file system based one 
> could be provided out of the box.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1888) Java: Single-record encoding marker bytes check is incorrect

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1888:
--
Priority: Blocker  (was: Major)

> Java: Single-record encoding marker bytes check is incorrect
> 
>
> Key: AVRO-1888
> URL: https://issues.apache.org/jira/browse/AVRO-1888
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Reporter: Ryan Blue
>Assignee: Ryan Blue
>Priority: Blocker
> Fix For: 1.8.2
>
> Attachments: AVRO-1888.1.patch
>
>
> It looks like the check for correct marker bytes is incorrect.
> The check should validate both marker/version bytes match what is expected, 
> but is instead this:
> {code}
> if (! (BinaryMessageEncoder.V1_HEADER[0] == header[0])
> && BinaryMessageEncoder.V1_HEADER[1] == header[1]) { . . . }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1889) Upgrade maven-shade plugin

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1889:
--
Fix Version/s: 1.8.2
   1.9.0

> Upgrade maven-shade plugin
> --
>
> Key: AVRO-1889
> URL: https://issues.apache.org/jira/browse/AVRO-1889
> Project: Avro
>  Issue Type: Bug
>  Components: build
>Reporter: Sachin Goyal
>Assignee: Sachin Goyal
> Fix For: 1.9.0, 1.8.2
>
>
> Latest version of maven-shade plugin is 2.4.3 while Avro is using 1.7.1
> This should be upgraded as it fails to build with Java 1.8, mvn=3.3.9 on some 
> Macs



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1889) Upgrade maven-shade plugin

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1889:
--
Component/s: build

> Upgrade maven-shade plugin
> --
>
> Key: AVRO-1889
> URL: https://issues.apache.org/jira/browse/AVRO-1889
> Project: Avro
>  Issue Type: Bug
>  Components: build
>Reporter: Sachin Goyal
>Assignee: Sachin Goyal
> Fix For: 1.9.0, 1.8.2
>
>
> Latest version of maven-shade plugin is 2.4.3 while Avro is using 1.7.1
> This should be upgraded as it fails to build with Java 1.8, mvn=3.3.9 on some 
> Macs



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1870) Avro compiler generate JAVA enum interfaces but writeEnum expects GenericEnumSymbol

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392097#comment-15392097
 ] 

Sean Busbey commented on AVRO-1870:
---

I think historically Specific API objects happened to work with the 
GenericDatumWriter, even though we don't claim that they will. If you're going 
to process both Specific and Generic API objects, you should use 
SpecificDatumWriter. All the better if you can differentiate between Generic 
and Specific and use the appropriate writer with each.

> Avro compiler generate JAVA enum interfaces but writeEnum expects 
> GenericEnumSymbol
> ---
>
> Key: AVRO-1870
> URL: https://issues.apache.org/jira/browse/AVRO-1870
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.1
>Reporter: Ali Hashemi
>
> Avro compiler generate an enum JAVA class from an enum Avro type 
> and this class is used in the Java class of the records pointing to the enum 
> type. However, org.apache.avro.generic.GenericDatumWriter.writeEnum expects a 
> GenericEnumSymbol rather than Java Enum!
> This is problem because the class interfaces for Avro messages expect Java 
> Enum and not GenericEnumSymbol!
> Consider the following schema:
> TrafficLights.avsc:
> {"namespace": "com.comp.message",
>  "type": "record",
>  "name": "TrafficLights",
>  "fields": [
> {"name": "id",  "type": "string"},
> {"name": "signal",  "type": "Signals" }
>  ]
> }
> Signals.avsc:
> {
>"namespace": "com.comp.message",
>"type": "enum",
>"name":"Signals",
>"symbols" : ["GREEN", "YELLOW", "RED"]
> }
> Using maven-avro-plugin to import "Signals" as an external schema, Avro 
> compiler creates the following constructor for TrafficLights class:
> public TrafficLights(java.lang.CharSequence id, com.comp.message.Signals 
> signal){...}
> and an Enum class fot the Signals:
> package com.comp.message;
> @SuppressWarnings("all")
> @org.apache.avro.specific.AvroGenerated
> public enum Signals {
>   GREEN, YELLOW, RED  ;
>   public static final org.apache.avro.Schema SCHEMA$ = new 
> org.apache.avro.Schema.Parser().parse("{\"type\":\"enum\",\"name\":\"Signals\",\"namespace\":\"com.comp.message\",\"symbols\":[\"GREEN\",\"YELLOW\",\"RED\"]}");
>   public static org.apache.avro.Schema getClassSchema() { return SCHEMA$; }
> }
> However, when I try to serialize a TrafficLights message, 
> GenericDatumWriter.writeEnum throws an exeception:
> TrafficLights  mainTrafficLight = new TrafficLights  ("123",  Signals.GREEN);
> byte[] msg = new AvroMsgManager( mainTrafficLight .getSchema().toString() 
> ).serialize (mainTrafficLight))
> org.apache.avro.AvroTypeException: Not an enum: GREEN
> at 
> org.apache.avro.generic.GenericDatumWriter.writeEnum(GenericDatumWriter.java:164)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:106)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:153)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Java Mvn Compile error

[Sean Busbey]

it's odd that maven is pointing at a JRE and not a JDK, but that
doesn't appear to be the error.

I'll need to find a windows box I can try to reproduce this on.

On Fri, Jul 22, 2016 at 12:18 PM, Theodore Ravindranath
<theoma...@gmail.com> wrote:
> Here is the output of mvn -version ...
>
> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> 2015-11-10T22:11:47+05:30)
> Maven home: F:\PNR\apache-maven-3.3.9
> Java version: 1.7.0_65, vendor: Oracle Corporation
> Java home: C:\Program Files\Java\jdk1.7.0_65\jre
> Default locale: en_US, platform encoding: Cp1252
> OS name: "windows 8.1", version: "6.3", arch: "amd64", family: "windows"
>
>
>
> On Fri, Jul 22, 2016 at 9:49 PM Sean Busbey <bus...@cloudera.com> wrote:
>
>> What's the output of "mvn --version"
>>
>> On Thu, Jul 21, 2016 at 10:55 PM, Theodore Ravindranath
>> <theoma...@gmail.com> wrote:
>> > Hi All,
>> >
>> > I cloned avro source from git, and went through the getting started
>> > instructions here:
>> > https://cwiki.apache.org/confluence/display/AVRO/How+To+Contribute
>> > I'm hoping to work on this:
>> >
>> https://helpwanted.apache.org/task.html?7016995edb84589a099db564394845047da3e1c8
>> >
>> > The maven compile failed. Below are the details of the failure. Please
>> > suggest how to proceed to get it working. Thanks!
>> >
>> > I got this 'INFO' when running mvn compile inside lang/java folder:
>> >
>> > *skip non existing resourceDirectory
>> > C:\Users\Theodore\git\avro\lang\java\maven-plugin\src\main\resources*
>> >
>> > And ended with this failure:
>> > *[INFO] Building Apache Avro IPC 1.9.0-SNAPSHOT*
>> > *[INFO]
>> > *
>> > *[INFO]
>> > *
>> > *[INFO] Reactor Summary:*
>> > *[INFO]*
>> > *[INFO] Apache Avro Java ... SUCCESS [
>> > 55.235 s]*
>> > *[INFO] Apache Avro Guava Dependencies . SUCCESS
>> [02:13
>> > min]*
>> > *[INFO] Apache Avro  SUCCESS [
>> > 44.749 s]*
>> > *[INFO] Apache Avro Compiler ... SUCCESS
>> [01:36
>> > min]*
>> > *[INFO] Apache Avro Maven Plugin ... SUCCESS
>> [02:54
>> > min]*
>> > *[INFO] Apache Avro IPC  FAILURE [
>> >  0.009 s]*
>> > *[INFO] Trevni Java  SKIPPED*
>> > *[INFO] Trevni Java Core ... SKIPPED*
>> > *[INFO] Apache Avro Mapred API . SKIPPED*
>> > *[INFO] Trevni Java Avro ... SKIPPED*
>> > *[INFO] Trevni Specification ... SKIPPED*
>> > *[INFO] Apache Avro Tools .. SKIPPED*
>> > *[INFO] Apache Avro Protobuf Compatibility . SKIPPED*
>> > *[INFO] Apache Avro Thrift Compatibility ... SKIPPED*
>> > *[INFO] Apache Avro Maven Archetypes ... SKIPPED*
>> > *[INFO] Apache Avro Maven Service Archetype  SKIPPED*
>> > *[INFO]
>> > *
>> > *[INFO] BUILD FAILURE*
>> > *[INFO]
>> > *
>> > *[INFO] Total time: 11:27 min*
>> > *[INFO] Finished at: 2016-07-22T00:31:55+05:30*
>> > *[INFO] Final Memory: 33M/107M*
>> > *[INFO] **[INFO] Building Apache Avro IPC 1.9.0-SNAPSHOT*
>> > *[INFO]
>> > *
>> > *[INFO]
>> > *
>> > *[INFO] Reactor Summary:*
>> > *[INFO]*
>> > *[INFO] Apache Avro Java ... SUCCESS [
>> > 55.235 s]*
>> > *[INFO] Apache Avro Guava Dependencies . SUCCESS
>> [02:13
>> > min]*
>> > *[INFO] Apache Avro  SUCCESS [
>> > 44.749 s]*
>> > *[INFO] Apache Avro Compiler ... SUCCESS
>> [01:36
>> > min]*
>> > *[INFO] Apache Avro Mav

[jira] [Commented] (AVRO-1704) Standardized format for encoding messages with Avro

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15391974#comment-15391974
 ] 

Sean Busbey commented on AVRO-1704:
---

FWIW, I belatedly agree with Doug's statement.

Do we have our compatibility promises documented somewhere? I feel like I have 
a good sense of them, but I don't know if that's just because I've been in the 
community for several years.

> Standardized format for encoding messages with Avro
> ---
>
> Key: AVRO-1704
> URL: https://issues.apache.org/jira/browse/AVRO-1704
> Project: Avro
>  Issue Type: Improvement
>Reporter: Daniel Schierbeck
>Assignee: Niels Basjes
> Fix For: 1.9.0, 1.8.3
>
> Attachments: AVRO-1704-2016-05-03-Unfinished.patch, 
> AVRO-1704-20160410.patch, AVRO-1704.3.patch, AVRO-1704.4.patch
>
>
> I'm currently using the Datafile format for encoding messages that are 
> written to Kafka and Cassandra. This seems rather wasteful:
> 1. I only encode a single record at a time, so there's no need for sync 
> markers and other metadata related to multi-record files.
> 2. The entire schema is inlined every time.
> However, the Datafile format is the only one that has been standardized, 
> meaning that I can read and write data with minimal effort across the various 
> languages in use in my organization. If there was a standardized format for 
> encoding single values that was optimized for out-of-band schema transfer, I 
> would much rather use that.
> I think the necessary pieces of the format would be:
> 1. A format version number.
> 2. A schema fingerprint type identifier, i.e. Rabin, MD5, SHA256, etc.
> 3. The actual schema fingerprint (according to the type.)
> 4. Optional metadata map.
> 5. The encoded datum.
> The language libraries would implement a MessageWriter that would encode 
> datums in this format, as well as a MessageReader that, given a SchemaStore, 
> would be able to decode datums. The reader would decode the fingerprint and 
> ask its SchemaStore to return the corresponding writer's schema.
> The idea is that SchemaStore would be an abstract interface that allowed 
> library users to inject custom backends. A simple, file system based one 
> could be provided out of the box.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1886) Improve error reporting for schema validation

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1886:
--
Status: Patch Available  (was: Open)

> Improve error reporting for schema validation
> -
>
> Key: AVRO-1886
> URL: https://issues.apache.org/jira/browse/AVRO-1886
> Project: Avro
>  Issue Type: Improvement
>  Components: ruby
>Reporter: Miroslav Csonka
>Assignee: Miroslav Csonka
> Fix For: 1.9.0, 1.8.2
>
>
> In Funding Circle we use avro schema to validate some financial data and 
> we've found ourselves wondering what is invalid. We've noticed that avro 
> [does already 
> validation|https://github.com/apache/avro/blob/master/lang/ruby/lib/avro/schema.rb#L94],
>  but does not report what fields are invalid.
>  At this point, we can clearly communicate validation scenarios and avro can 
> be even used as a validation library.
> Little preview of what we've done
> {quote}
> at .person.houses\[1\].number_of_rooms expected type long, got string 
> with value "not valid at all"
> {quote}
> At this point, our fork is quite outdated so it will take us some time to 
> update it and submit a patch. This is just a question if you would be 
> interested in receiving such a patch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1886) Improve error reporting for schema validation

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1886:
--
Assignee: Miroslav Csonka

> Improve error reporting for schema validation
> -
>
> Key: AVRO-1886
> URL: https://issues.apache.org/jira/browse/AVRO-1886
> Project: Avro
>  Issue Type: Improvement
>  Components: ruby
>Reporter: Miroslav Csonka
>Assignee: Miroslav Csonka
> Fix For: 1.9.0, 1.8.2
>
>
> In Funding Circle we use avro schema to validate some financial data and 
> we've found ourselves wondering what is invalid. We've noticed that avro 
> [does already 
> validation|https://github.com/apache/avro/blob/master/lang/ruby/lib/avro/schema.rb#L94],
>  but does not report what fields are invalid.
>  At this point, we can clearly communicate validation scenarios and avro can 
> be even used as a validation library.
> Little preview of what we've done
> {quote}
> at .person.houses\[1\].number_of_rooms expected type long, got string 
> with value "not valid at all"
> {quote}
> At this point, our fork is quite outdated so it will take us some time to 
> update it and submit a patch. This is just a question if you would be 
> interested in receiving such a patch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1886) Improve error reporting for schema validation

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1886:
--
Fix Version/s: 1.8.2
   1.9.0

> Improve error reporting for schema validation
> -
>
> Key: AVRO-1886
> URL: https://issues.apache.org/jira/browse/AVRO-1886
> Project: Avro
>  Issue Type: Improvement
>  Components: ruby
>Reporter: Miroslav Csonka
> Fix For: 1.9.0, 1.8.2
>
>
> In Funding Circle we use avro schema to validate some financial data and 
> we've found ourselves wondering what is invalid. We've noticed that avro 
> [does already 
> validation|https://github.com/apache/avro/blob/master/lang/ruby/lib/avro/schema.rb#L94],
>  but does not report what fields are invalid.
>  At this point, we can clearly communicate validation scenarios and avro can 
> be even used as a validation library.
> Little preview of what we've done
> {quote}
> at .person.houses\[1\].number_of_rooms expected type long, got string 
> with value "not valid at all"
> {quote}
> At this point, our fork is quite outdated so it will take us some time to 
> update it and submit a patch. This is just a question if you would be 
> interested in receiving such a patch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Java Mvn Compile error

[Sean Busbey]

What's the output of "mvn --version"

On Thu, Jul 21, 2016 at 10:55 PM, Theodore Ravindranath
 wrote:
> Hi All,
>
> I cloned avro source from git, and went through the getting started
> instructions here:
> https://cwiki.apache.org/confluence/display/AVRO/How+To+Contribute
> I'm hoping to work on this:
> https://helpwanted.apache.org/task.html?7016995edb84589a099db564394845047da3e1c8
>
> The maven compile failed. Below are the details of the failure. Please
> suggest how to proceed to get it working. Thanks!
>
> I got this 'INFO' when running mvn compile inside lang/java folder:
>
> *skip non existing resourceDirectory
> C:\Users\Theodore\git\avro\lang\java\maven-plugin\src\main\resources*
>
> And ended with this failure:
> *[INFO] Building Apache Avro IPC 1.9.0-SNAPSHOT*
> *[INFO]
> *
> *[INFO]
> *
> *[INFO] Reactor Summary:*
> *[INFO]*
> *[INFO] Apache Avro Java ... SUCCESS [
> 55.235 s]*
> *[INFO] Apache Avro Guava Dependencies . SUCCESS [02:13
> min]*
> *[INFO] Apache Avro  SUCCESS [
> 44.749 s]*
> *[INFO] Apache Avro Compiler ... SUCCESS [01:36
> min]*
> *[INFO] Apache Avro Maven Plugin ... SUCCESS [02:54
> min]*
> *[INFO] Apache Avro IPC  FAILURE [
>  0.009 s]*
> *[INFO] Trevni Java  SKIPPED*
> *[INFO] Trevni Java Core ... SKIPPED*
> *[INFO] Apache Avro Mapred API . SKIPPED*
> *[INFO] Trevni Java Avro ... SKIPPED*
> *[INFO] Trevni Specification ... SKIPPED*
> *[INFO] Apache Avro Tools .. SKIPPED*
> *[INFO] Apache Avro Protobuf Compatibility . SKIPPED*
> *[INFO] Apache Avro Thrift Compatibility ... SKIPPED*
> *[INFO] Apache Avro Maven Archetypes ... SKIPPED*
> *[INFO] Apache Avro Maven Service Archetype  SKIPPED*
> *[INFO]
> *
> *[INFO] BUILD FAILURE*
> *[INFO]
> *
> *[INFO] Total time: 11:27 min*
> *[INFO] Finished at: 2016-07-22T00:31:55+05:30*
> *[INFO] Final Memory: 33M/107M*
> *[INFO] **[INFO] Building Apache Avro IPC 1.9.0-SNAPSHOT*
> *[INFO]
> *
> *[INFO]
> *
> *[INFO] Reactor Summary:*
> *[INFO]*
> *[INFO] Apache Avro Java ... SUCCESS [
> 55.235 s]*
> *[INFO] Apache Avro Guava Dependencies . SUCCESS [02:13
> min]*
> *[INFO] Apache Avro  SUCCESS [
> 44.749 s]*
> *[INFO] Apache Avro Compiler ... SUCCESS [01:36
> min]*
> *[INFO] Apache Avro Maven Plugin ... SUCCESS [02:54
> min]*
> *[INFO] Apache Avro IPC  FAILURE [
>  0.009 s]*
> *[INFO] Trevni Java  SKIPPED*
> *[INFO] Trevni Java Core ... SKIPPED*
> *[INFO] Apache Avro Mapred API . SKIPPED*
> *[INFO] Trevni Java Avro ... SKIPPED*
> *[INFO] Trevni Specification ... SKIPPED*
> *[INFO] Apache Avro Tools .. SKIPPED*
> *[INFO] Apache Avro Protobuf Compatibility . SKIPPED*
> *[INFO] Apache Avro Thrift Compatibility ... SKIPPED*
> *[INFO] Apache Avro Maven Archetypes ... SKIPPED*
> *[INFO] Apache Avro Maven Service Archetype  SKIPPED*
> *[INFO]
> *
> *[INFO] BUILD FAILURE*
> *[INFO]
> *
> *[INFO] Total time: 11:27 min*
> *[INFO] Finished at: 2016-07-22T00:31:55+05:30*
> *[INFO] Final Memory: 33M/107M*
> *[INFO]
> *
> *[ERROR] Failed to parse plugin descriptor for
> org.apache.avro:avro-maven-plugin:1.9.0-SNAPSHOT
> (C:\Users\Theodore\git\avro\lang\java\maven-plugin\target\classes): No
> plugin descriptor found at META-INF/maven/plugin.xml -> [Help 1]*
> *[ERROR]***
> *[ERROR] Failed to parse plugin descriptor for
> org.apache.avro:avro-maven-plugin:1.9.0-SNAPSHOT
> (C:\Users\Theodore\git\avro\lang\java\maven-plugin\target\classes): No
> plugin descriptor found 

[jira] [Commented] (AVRO-607) SpecificData.getSchema not thread-safe

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388425#comment-15388425
 ] 

Sean Busbey commented on AVRO-607:
--

{code}
+import com.google.common.cache.CacheBuilder;
+import com.google.common.cache.CacheLoader;
+import com.google.common.cache.LoadingCache;
{code}

If these are in the shaded module, shouldn't they be relocated from the 
com.google namespace?

> SpecificData.getSchema not thread-safe
> --
>
> Key: AVRO-607
> URL: https://issues.apache.org/jira/browse/AVRO-607
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.3.3, 1.8.1
>Reporter: Stephen Tu
>Assignee: Andrius Druzinis-Vitkus
>Priority: Blocker
>  Labels: newbie, patch
> Fix For: 1.8.2
>
> Attachments: AVRO-607.patch, AVRO-607.patch
>
>
> SpecificData.getSchema uses a WeakHashMap to cache schemas, but WeakHashMap 
> is not thread-safe, and the method itself is not synchronized. Seems like 
> this could lead to the data structure getting corrupted. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-607) SpecificData.getSchema not thread-safe

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388423#comment-15388423
 ] 

Sean Busbey commented on AVRO-607:
--

thanks for the patch. could you regenerate it using the {{git format-patch}} 
command? that way I'll know how you want yourself attributed.

> SpecificData.getSchema not thread-safe
> --
>
> Key: AVRO-607
> URL: https://issues.apache.org/jira/browse/AVRO-607
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.3.3, 1.8.1
>Reporter: Stephen Tu
>Assignee: Andrius Druzinis-Vitkus
>Priority: Blocker
>  Labels: newbie, patch
> Fix For: 1.8.2
>
> Attachments: AVRO-607.patch, AVRO-607.patch
>
>
> SpecificData.getSchema uses a WeakHashMap to cache schemas, but WeakHashMap 
> is not thread-safe, and the method itself is not synchronized. Seems like 
> this could lead to the data structure getting corrupted. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1877:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

thanks [~Yibing]!

> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.9.0, 1.8.2
>Reporter: Yibing Shi
>Assignee: Yibing Shi
>Priority: Blocker
> Fix For: 1.9.0, 1.8.2
>
> Attachments: AVRO-1877.1.patch, AVRO-1877.2.patch, AVRO-1877.3.patch
>
>
> AVRO-1847 accidentally removes the logical type conversion logic in method 
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in 
> specific compiler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387931#comment-15387931
 ] 

Sean Busbey commented on AVRO-1877:
---

(nit for [~Yibing], please use `git format-patch` in the future to make your 
patch files)

> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.9.0, 1.8.2
>Reporter: Yibing Shi
>Assignee: Yibing Shi
>Priority: Blocker
> Fix For: 1.9.0, 1.8.2
>
> Attachments: AVRO-1877.1.patch, AVRO-1877.2.patch, AVRO-1877.3.patch
>
>
> AVRO-1847 accidentally removes the logical type conversion logic in method 
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in 
> specific compiler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (AVRO-1887) set up precommit checks via Apache Yetus

[Sean Busbey (JIRA)]

Sean Busbey created AVRO-1887:
-

 Summary: set up precommit checks via Apache Yetus
 Key: AVRO-1887
 URL: https://issues.apache.org/jira/browse/AVRO-1887
 Project: Avro
  Issue Type: Task
  Components: build, community
Reporter: Sean Busbey
Priority: Blocker


The lack of automated checks for things like doing a full build is negatively 
impacting my ability to do timely reviews. We should leverage the work done by 
other communities and get precommit checks going with Apache Yetus ([ref Apache 
Yetus Precommit|http://yetus.apache.org/documentation/0.3.0/precommit-basic/])

Yetus has built in support for using Docker as a containerized build 
environment ([ref 
docs|https://yetus.apache.org/documentation/0.3.0/precommit-advanced/#Docker_Support]),
 but I don't know if we can use the one we have directly or if we'll need to do 
some refactoring.

Implementing this will ultimately require a committer (so that they can make 
changes to builds.apache.org), but a non-cmmitter can get the bulk of the work 
done by managing to do local yetus checks with the docker feature active.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1887) set up precommit checks via Apache Yetus

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1887:
--
Labels:   (was: beginner)

> set up precommit checks via Apache Yetus
> 
>
> Key: AVRO-1887
> URL: https://issues.apache.org/jira/browse/AVRO-1887
> Project: Avro
>  Issue Type: Task
>  Components: build, community
>    Reporter: Sean Busbey
>Priority: Blocker
>
> The lack of automated checks for things like doing a full build is negatively 
> impacting my ability to do timely reviews. We should leverage the work done 
> by other communities and get precommit checks going with Apache Yetus ([ref 
> Apache Yetus 
> Precommit|http://yetus.apache.org/documentation/0.3.0/precommit-basic/])
> Yetus has built in support for using Docker as a containerized build 
> environment ([ref 
> docs|https://yetus.apache.org/documentation/0.3.0/precommit-advanced/#Docker_Support]),
>  but I don't know if we can use the one we have directly or if we'll need to 
> do some refactoring.
> Implementing this will ultimately require a committer (so that they can make 
> changes to builds.apache.org), but a non-cmmitter can get the bulk of the 
> work done by managing to do local yetus checks with the docker feature active.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387884#comment-15387884
 ] 

Sean Busbey commented on AVRO-1877:
---

confirmed it does impact branch-1.8

> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.9.0, 1.8.2
>Reporter: Yibing Shi
>Assignee: Yibing Shi
>Priority: Blocker
> Fix For: 1.9.0, 1.8.2
>
> Attachments: AVRO-1877.1.patch, AVRO-1877.2.patch, AVRO-1877.3.patch
>
>
> AVRO-1847 accidentally removes the logical type conversion logic in method 
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in 
> specific compiler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1877:
--
Affects Version/s: 1.8.2

> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.9.0, 1.8.2
>Reporter: Yibing Shi
>Assignee: Yibing Shi
>Priority: Blocker
> Fix For: 1.9.0, 1.8.2
>
> Attachments: AVRO-1877.1.patch, AVRO-1877.2.patch, AVRO-1877.3.patch
>
>
> AVRO-1847 accidentally removes the logical type conversion logic in method 
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in 
> specific compiler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1877:
--
Fix Version/s: 1.8.2

> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.9.0, 1.8.2
>Reporter: Yibing Shi
>Assignee: Yibing Shi
>Priority: Blocker
> Fix For: 1.9.0, 1.8.2
>
> Attachments: AVRO-1877.1.patch, AVRO-1877.2.patch, AVRO-1877.3.patch
>
>
> AVRO-1847 accidentally removes the logical type conversion logic in method 
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in 
> specific compiler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1877:
--
Assignee: Yibing Shi

> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.9.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
>Priority: Blocker
> Fix For: 1.9.0
>
> Attachments: AVRO-1877.1.patch, AVRO-1877.2.patch, AVRO-1877.3.patch
>
>
> AVRO-1847 accidentally removes the logical type conversion logic in method 
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in 
> specific compiler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1877:
--
Fix Version/s: 1.9.0

> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.9.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
>Priority: Blocker
> Fix For: 1.9.0
>
> Attachments: AVRO-1877.1.patch, AVRO-1877.2.patch, AVRO-1877.3.patch
>
>
> AVRO-1847 accidentally removes the logical type conversion logic in method 
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in 
> specific compiler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387739#comment-15387739
 ] 

Sean Busbey commented on AVRO-1877:
---

While I review, can someone tell me if this impacts branch-1.8?

> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.9.0
>Reporter: Yibing Shi
>Priority: Blocker
> Attachments: AVRO-1877.1.patch, AVRO-1877.2.patch, AVRO-1877.3.patch
>
>
> AVRO-1847 accidentally removes the logical type conversion logic in method 
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in 
> specific compiler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-607) SpecificData.getSchema not thread-safe

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384468#comment-15384468
 ] 

Sean Busbey commented on AVRO-607:
--

Hi [~Andrius Driu]! I've assigned the jira to you now, so you ought to be able 
to attach a patch. Please either do that or create a pull request (and 
reference this jira there).

> SpecificData.getSchema not thread-safe
> --
>
> Key: AVRO-607
> URL: https://issues.apache.org/jira/browse/AVRO-607
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.3.3, 1.8.1
>Reporter: Stephen Tu
>Assignee: Andrius Druzinis-Vitkus
>Priority: Blocker
> Fix For: 1.8.2
>
> Attachments: AVRO-607.patch
>
>
> SpecificData.getSchema uses a WeakHashMap to cache schemas, but WeakHashMap 
> is not thread-safe, and the method itself is not synchronized. Seems like 
> this could lead to the data structure getting corrupted. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-607) SpecificData.getSchema not thread-safe

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-607:
-
Assignee: Andrius Druzinis-Vitkus

> SpecificData.getSchema not thread-safe
> --
>
> Key: AVRO-607
> URL: https://issues.apache.org/jira/browse/AVRO-607
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.3.3, 1.8.1
>Reporter: Stephen Tu
>Assignee: Andrius Druzinis-Vitkus
>Priority: Blocker
> Fix For: 1.8.2
>
> Attachments: AVRO-607.patch
>
>
> SpecificData.getSchema uses a WeakHashMap to cache schemas, but WeakHashMap 
> is not thread-safe, and the method itself is not synchronized. Seems like 
> this could lead to the data structure getting corrupted. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1886) Improve error reporting for schema validation

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384383#comment-15384383
 ] 

Sean Busbey commented on AVRO-1886:
---

yeah, that'd be great.

> Improve error reporting for schema validation
> -
>
> Key: AVRO-1886
> URL: https://issues.apache.org/jira/browse/AVRO-1886
> Project: Avro
>  Issue Type: Improvement
>  Components: ruby
>Reporter: Miroslav Csonka
>Priority: Minor
>
> In Funding Circle we use avro schema to validate some financial data and 
> we've found ourselves wondering what is invalid. We've noticed that avro 
> [does already 
> validation|https://github.com/apache/avro/blob/master/lang/ruby/lib/avro/schema.rb#L94],
>  but does not report what fields are invalid.
>  At this point, we can clearly communicate validation scenarios and avro can 
> be even used as a validation library.
> Little preview of what we've done
> {quote}
> at .person.houses\[1\].number_of_rooms expected type long, got string 
> with value "not valid at all"
> {quote}
> At this point, our fork is quite outdated so it will take us some time to 
> update it and submit a patch. This is just a question if you would be 
> interested in receiving such a patch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1885) Release 1.8.2

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384126#comment-15384126
 ] 

Sean Busbey commented on AVRO-1885:
---

added things currently targeting 1.8.2 that are critical, blocker, or have a 
patch available.

> Release 1.8.2
> -
>
> Key: AVRO-1885
> URL: https://issues.apache.org/jira/browse/AVRO-1885
> Project: Avro
>  Issue Type: Task
>  Components: community
>Affects Versions: 1.8.2
>    Reporter: Sean Busbey
>Assignee: Sean Busbey
> Fix For: 1.8.2
>
>
> Please link to any issues that should be considered blockers for the 1.8.2 
> release.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] 1.8.2 RCs on or about july 7th?

[Sean Busbey]

Okay, life got away from me a bit obviously. :)

I'm going to aim for an RC on July 26th. I have to finish working out
having a machine I own that can run docker successfully in order to
get everything built.

I filed AVRO-1885 to track the release's progress[2].

[2]: https://issues.apache.org/jira/browse/AVRO-1885

On Tue, Jul 5, 2016 at 11:45 AM, Ryan Blue <rb...@netflix.com.invalid> wrote:
> +1 for a RC. Thanks, Sean!
>
> Could you open an issue for a 1.8.2 release if you haven't already, like
> AVRO-1833 [1]? That way we can see what needs to be reviewed easily.
>
> rb
>
>
> [1]: https://issues.apache.org/jira/browse/AVRO-1833
>
> On Tue, Jun 28, 2016 at 9:22 AM, Sean Busbey <bus...@apache.org> wrote:
>
>> Hi Folks!
>>
>> Our last release was about 5 weeks ago and we've had issues close on
>> branch-1.8 since then. I'd like to start doing RCs by the end of next week
>> in hopes that we can stay under 2 months between releases.
>>
>> I'm going to dedicate some time at the end of this week / start of next to
>> do a pass of getting available patches applied. I'd appreciate it if some
>> other committers could do so as well.
>>
>> For contributors generally if you have a patch nearly ready or with
>> pending feedback, it'd be nice if you could make a push as well but please
>> don't worry if things don't make it into 1.8.2. I'm going to start pushing
>> to get us on a monthly release cadence, so there should be plenty of
>> additional chances for your work to get into a release.
>>
>> -
>> busbey
>>
>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix



-- 
busbey

[jira] [Created] (AVRO-1885) Release 1.8.2

[Sean Busbey (JIRA)]

Sean Busbey created AVRO-1885:
-

 Summary: Release 1.8.2
 Key: AVRO-1885
 URL: https://issues.apache.org/jira/browse/AVRO-1885
 Project: Avro
  Issue Type: Task
  Components: community
Affects Versions: 1.8.2
Reporter: Sean Busbey
Assignee: Sean Busbey
 Fix For: 1.8.2


Please link to any issues that should be considered blockers for the 1.8.2 
release.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1879) Make conversions field transient in compiled SpecificRecord

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384025#comment-15384025
 ] 

Sean Busbey commented on AVRO-1879:
---

[~chunfy.tseng] please post questions to the [user@avro mailing 
list|http://mail-archives.apache.org/mod_mbox/avro-user/] and please do not 
post your question multiple times.

> Make conversions field transient in compiled SpecificRecord
> ---
>
> Key: AVRO-1879
> URL: https://issues.apache.org/jira/browse/AVRO-1879
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.1
>Reporter: Michael Wong
>Assignee: Michael Wong
> Fix For: 1.9.0, 1.8.2
>
> Attachments: conversions-transient.patch
>
>
> Add a transient modifier to the conversions field as such
> {code}
> private final transient org.apache.avro.Conversion[] conversions
> {code}
> This will allow Serializers and type inference systems like Flink's 
> TypeExtractor to not consider the {{conversions}} field and not try to 
> serialize it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1882) ConcurrentHashMap with non-string keys fails in Java 1.8

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384022#comment-15384022
 ] 

Sean Busbey commented on AVRO-1882:
---

[~chunfy.tseng] please post questions to the [user@avro mailing 
list|http://mail-archives.apache.org/mod_mbox/avro-user/].

> ConcurrentHashMap with non-string keys fails in Java 1.8
> 
>
> Key: AVRO-1882
> URL: https://issues.apache.org/jira/browse/AVRO-1882
> Project: Avro
>  Issue Type: Bug
>Affects Versions: 1.8.1
>Reporter: Sachin Goyal
> Fix For: 1.8.2
>
> Attachments: TestNonStringConcurrentMap.java
>
>
> Support for ConcurrentHashMaps with non-string keys seems to be broken when 
> 1.8 version of Java is used because the newer ConcurrentHashMap uses the 
> names "key" and "val" instead of "key" and "values" for its Map.Entry class.
> [HashEntry in 
> 1.7|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/util/concurrent/ConcurrentHashMap.java#218]
> [MapEntry in 
> 1.8|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8u40-b25/java/util/concurrent/ConcurrentHashMap.java?av=h#3468]
> Hence avro-code that assumes the presence of key/value breaks.
> ([ReflectData.java:L434-L443|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/reflect/ReflectData.java#L434-L443])
> Run the attached test to see the same.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1882) ConcurrentHashMap with non-string keys fails in Java 1.8

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1882:
--
Fix Version/s: 1.8.2

> ConcurrentHashMap with non-string keys fails in Java 1.8
> 
>
> Key: AVRO-1882
> URL: https://issues.apache.org/jira/browse/AVRO-1882
> Project: Avro
>  Issue Type: Bug
>Affects Versions: 1.8.1
>Reporter: Sachin Goyal
> Fix For: 1.8.2
>
> Attachments: TestNonStringConcurrentMap.java
>
>
> Support for ConcurrentHashMaps with non-string keys seems to be broken when 
> 1.8 version of Java is used because the newer ConcurrentHashMap uses the 
> names "key" and "val" instead of "key" and "values" for its Map.Entry class.
> [HashEntry in 
> 1.7|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/util/concurrent/ConcurrentHashMap.java#218]
> [MapEntry in 
> 1.8|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8u40-b25/java/util/concurrent/ConcurrentHashMap.java?av=h#3468]
> Hence avro-code that assumes the presence of key/value breaks.
> ([ReflectData.java:L434-L443|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/reflect/ReflectData.java#L434-L443])
> Run the attached test to see the same.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1870) Avro compiler generate JAVA enum interfaces but writeEnum expects GenericEnumSymbol

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382256#comment-15382256
 ] 

Sean Busbey commented on AVRO-1870:
---

Since the TrafficLights object being serialized is from the Specific API, it 
should be serialized using the SpecificDatumWriter, as Matt suggests. I don't 
think we generally guarantee that Specific objects will be serializable with 
GenericDatumWriter, though I think it is confusing that we generate classes 
that can't be.

Is the AvroMsgManager's serialization method supposed to work with both 
generated Specific API classes and Object instances returned from the Generic 
API?

> Avro compiler generate JAVA enum interfaces but writeEnum expects 
> GenericEnumSymbol
> ---
>
> Key: AVRO-1870
> URL: https://issues.apache.org/jira/browse/AVRO-1870
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.1
>Reporter: Ali Hashemi
>
> Avro compiler generate an enum JAVA class from an enum Avro type 
> and this class is used in the Java class of the records pointing to the enum 
> type. However, org.apache.avro.generic.GenericDatumWriter.writeEnum expects a 
> GenericEnumSymbol rather than Java Enum!
> This is problem because the class interfaces for Avro messages expect Java 
> Enum and not GenericEnumSymbol!
> Consider the following schema:
> TrafficLights.avsc:
> {"namespace": "com.comp.message",
>  "type": "record",
>  "name": "TrafficLights",
>  "fields": [
> {"name": "id",  "type": "string"},
> {"name": "signal",  "type": "Signals" }
>  ]
> }
> Signals.avsc:
> {
>"namespace": "com.comp.message",
>"type": "enum",
>"name":"Signals",
>"symbols" : ["GREEN", "YELLOW", "RED"]
> }
> Using maven-avro-plugin to import "Signals" as an external schema, Avro 
> compiler creates the following constructor for TrafficLights class:
> public TrafficLights(java.lang.CharSequence id, com.comp.message.Signals 
> signal){...}
> and an Enum class fot the Signals:
> package com.comp.message;
> @SuppressWarnings("all")
> @org.apache.avro.specific.AvroGenerated
> public enum Signals {
>   GREEN, YELLOW, RED  ;
>   public static final org.apache.avro.Schema SCHEMA$ = new 
> org.apache.avro.Schema.Parser().parse("{\"type\":\"enum\",\"name\":\"Signals\",\"namespace\":\"com.comp.message\",\"symbols\":[\"GREEN\",\"YELLOW\",\"RED\"]}");
>   public static org.apache.avro.Schema getClassSchema() { return SCHEMA$; }
> }
> However, when I try to serialize a TrafficLights message, 
> GenericDatumWriter.writeEnum throws an exeception:
> TrafficLights  mainTrafficLight = new TrafficLights  ("123",  Signals.GREEN);
> byte[] msg = new AvroMsgManager( mainTrafficLight .getSchema().toString() 
> ).serialize (mainTrafficLight))
> org.apache.avro.AvroTypeException: Not an enum: GREEN
> at 
> org.apache.avro.generic.GenericDatumWriter.writeEnum(GenericDatumWriter.java:164)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:106)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:153)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (AVRO-1871) Avro compiler generate JAVA enum interfaces but writeEnum expects GenericEnumSymbol

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey resolved AVRO-1871.
---
Resolution: Duplicate

> Avro compiler generate JAVA enum interfaces but writeEnum expects 
> GenericEnumSymbol
> ---
>
> Key: AVRO-1871
> URL: https://issues.apache.org/jira/browse/AVRO-1871
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.1
>Reporter: Ali Hashemi
>
> Avro compiler generate an enum JAVA class from an enum Avro type 
> and this class is used in the Java class of the records pointing to the enum 
> type. However, org.apache.avro.generic.GenericDatumWriter.writeEnum expects a 
> GenericEnumSymbol rather than Java Enum!
> This is problem because the class interfaces for Avro messages expect Java 
> Enum and not GenericEnumSymbol!
> Consider the following schema:
> TrafficLights.avsc:
> {"namespace": "com.comp.message",
>  "type": "record",
>  "name": "TrafficLights",
>  "fields": [
> {"name": "id",  "type": "string"},
> {"name": "signal",  "type": "Signals" }
>  ]
> }
> Signals.avsc:
> {
>"namespace": "com.comp.message",
>"type": "enum",
>"name":"Signals",
>"symbols" : ["GREEN", "YELLOW", "RED"]
> }
> Using maven-avro-plugin to import "Signals" as an external schema, Avro 
> compiler creates the following constructor for TrafficLights class:
> public TrafficLights(java.lang.CharSequence id, com.comp.message.Signals 
> signal){...}
> and an Enum class fot the Signals:
> package com.comp.message;
> @SuppressWarnings("all")
> @org.apache.avro.specific.AvroGenerated
> public enum Signals {
>   GREEN, YELLOW, RED  ;
>   public static final org.apache.avro.Schema SCHEMA$ = new 
> org.apache.avro.Schema.Parser().parse("{\"type\":\"enum\",\"name\":\"Signals\",\"namespace\":\"com.comp.message\",\"symbols\":[\"GREEN\",\"YELLOW\",\"RED\"]}");
>   public static org.apache.avro.Schema getClassSchema() { return SCHEMA$; }
> }
> However, when I try to serialize a TrafficLights message, 
> GenericDatumWriter.writeEnum throws an exeception:
> TrafficLights  mainTrafficLight = new TrafficLights  ("123",  Signals.GREEN);
> byte[] msg = new AvroMsgManager( mainTrafficLight .getSchema().toString() 
> ).serialize (mainTrafficLight))
> org.apache.avro.AvroTypeException: Not an enum: GREEN
> at 
> org.apache.avro.generic.GenericDatumWriter.writeEnum(GenericDatumWriter.java:164)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:106)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:153)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-607) SpecificData.getSchema not thread-safe

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-607:
-
Priority: Blocker  (was: Major)

> SpecificData.getSchema not thread-safe
> --
>
> Key: AVRO-607
> URL: https://issues.apache.org/jira/browse/AVRO-607
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.3.3, 1.8.1
>Reporter: Stephen Tu
>Priority: Blocker
> Fix For: 1.8.2
>
> Attachments: AVRO-607.patch
>
>
> SpecificData.getSchema uses a WeakHashMap to cache schemas, but WeakHashMap 
> is not thread-safe, and the method itself is not synchronized. Seems like 
> this could lead to the data structure getting corrupted. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-607) SpecificData.getSchema not thread-safe

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382210#comment-15382210
 ] 

Sean Busbey commented on AVRO-607:
--

No worries. It helps to know the relative demand for outstanding issues. :)

> SpecificData.getSchema not thread-safe
> --
>
> Key: AVRO-607
> URL: https://issues.apache.org/jira/browse/AVRO-607
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.3.3, 1.8.1
>Reporter: Stephen Tu
> Fix For: 1.8.2
>
> Attachments: AVRO-607.patch
>
>
> SpecificData.getSchema uses a WeakHashMap to cache schemas, but WeakHashMap 
> is not thread-safe, and the method itself is not synchronized. Seems like 
> this could lead to the data structure getting corrupted. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1884) Add set source file suffix function for generate non-java file #90

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1884:
--
Fix Version/s: 1.8.2
   1.9.0

> Add set source file suffix function for generate non-java file #90
> --
>
> Key: AVRO-1884
> URL: https://issues.apache.org/jira/browse/AVRO-1884
> Project: Avro
>  Issue Type: Wish
>  Components: java
>Reporter: shijinkui
>Assignee: shijinkui
> Fix For: 1.9.0, 1.8.2
>
>
> support generate non-java source file,  example:
> compiler.setSuffix(".scala")
> compiler.setTemplateDir(templatePath)
> compiler.compileToDestination(file, new File("src/main/scala/"))



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1884) Add set source file suffix function for generate non-java file #90

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1884:
--
Status: Patch Available  (was: Open)

> Add set source file suffix function for generate non-java file #90
> --
>
> Key: AVRO-1884
> URL: https://issues.apache.org/jira/browse/AVRO-1884
> Project: Avro
>  Issue Type: Wish
>  Components: java
>Reporter: shijinkui
>Assignee: shijinkui
>
> support generate non-java source file,  example:
> compiler.setSuffix(".scala")
> compiler.setTemplateDir(templatePath)
> compiler.compileToDestination(file, new File("src/main/scala/"))



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1884) Add set source file suffix function for generate non-java file #90

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1884:
--
Assignee: shijinkui

> Add set source file suffix function for generate non-java file #90
> --
>
> Key: AVRO-1884
> URL: https://issues.apache.org/jira/browse/AVRO-1884
> Project: Avro
>  Issue Type: Wish
>  Components: java
>Reporter: shijinkui
>Assignee: shijinkui
>
> support generate non-java source file,  example:
> compiler.setSuffix(".scala")
> compiler.setTemplateDir(templatePath)
> compiler.compileToDestination(file, new File("src/main/scala/"))



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1879) Make conversions field transient in compiled SpecificRecord

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1879:
--
Assignee: Michael Wong

> Make conversions field transient in compiled SpecificRecord
> ---
>
> Key: AVRO-1879
> URL: https://issues.apache.org/jira/browse/AVRO-1879
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.1
>Reporter: Michael Wong
>Assignee: Michael Wong
> Fix For: 1.9.0, 1.8.2
>
> Attachments: conversions-transient.patch
>
>
> Add a transient modifier to the conversions field as such
> {code}
> private final transient org.apache.avro.Conversion[] conversions
> {code}
> This will allow Serializers and type inference systems like Flink's 
> TypeExtractor to not consider the {{conversions}} field and not try to 
> serialize it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1879) Make conversions field transient in compiled SpecificRecord

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1879:
--
Fix Version/s: 1.8.2
   1.9.0

> Make conversions field transient in compiled SpecificRecord
> ---
>
> Key: AVRO-1879
> URL: https://issues.apache.org/jira/browse/AVRO-1879
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.1
>Reporter: Michael Wong
> Fix For: 1.9.0, 1.8.2
>
> Attachments: conversions-transient.patch
>
>
> Add a transient modifier to the conversions field as such
> {code}
> private final transient org.apache.avro.Conversion[] conversions
> {code}
> This will allow Serializers and type inference systems like Flink's 
> TypeExtractor to not consider the {{conversions}} field and not try to 
> serialize it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Unable to build clean checkout of avro from master

[Sean Busbey]

hurm.

When I do the same steps is finishes fine:

https://paste.apache.org/iWLJ

What if you try building from the top level rather than lang/java/avro?

-Sean

On Fri, Jul 8, 2016 at 4:23 PM, S G <sg.online.em...@gmail.com> wrote:
> mvn --version is as follows:
>
> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> 2015-11-10T08:41:47-08:00)
> Maven home: ~/work/software/maven
> Java version: 1.8.0_71, vendor: Oracle Corporation
> Java home:
> /Library/Java/JavaVirtualMachines/jdk1.8.0_71.jdk/Contents/Home/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "mac os x", version: "10.10.5", arch: "x86_64", family: "mac"
>
>
>
> On Fri, Jul 8, 2016 at 1:02 PM, Sean Busbey <bus...@cloudera.com> wrote:
>
>> Could you copy/paste the output of running "mvn --version" ?
>>
>> On Fri, Jul 8, 2016 at 2:34 PM, S G <sg.online.em...@gmail.com> wrote:
>> > Hi,
>> >
>> > I am unable to build a clean checkout of avro from master.
>> >
>> > Is it a known issue or I am doing something wrong?
>> >
>> >
>> > cd lang/java/avro
>> >
>> > mvn clean install
>> >
>> > [INFO] --- maven-bundle-plugin:2.5.3:bundle (default-bundle) @ avro
>> ---
>> > [WARNING] Bundle org.apache.avro:avro:bundle:1.9.0-SNAPSHOT : Split
>> > package, multiple jars provide the same package:org/apache/avro
>> > Use Import/Export Package directive
>> > -split-package:=(merge-first|merge-last|error|first) to get rid of this
>> > warning
>> > Package found in   [Jar:., Jar:avro-guava-dependencies]
>> > Class path [Jar:., Jar:plexus-utils,
>> > Jar:avro-guava-dependencies, Jar:findbugs-annotations,
>> > Jar:jackson-core-asl, Jar:jackson-mapper-asl, Jar:paranamer,
>> > Jar:snappy-java, Jar:commons-compress, Jar:xz, Jar:joda-time,
>> > Jar:slf4j-api, Jar:slf4j-simple]
>> > [INFO]
>> > [INFO] --- maven-site-plugin:3.0:attach-descriptor
>> (attach-descriptor)
>> > @ avro ---
>> > [INFO]
>> > [INFO] --- maven-jar-plugin:2.5:test-jar (default) @ avro ---
>> > [INFO] Building jar:
>> >
>> ~/work/git/opensource/avro.upstream/avro/lang/java/avro/target/avro-1.9.0-SNAPSHOT-tests.jar
>> > [INFO]
>> > [INFO] --- maven-shade-plugin:1.7.1:shade (default) @ avro ---
>> > [WARNING] Error injecting:
>> org.apache.maven.plugins.shade.mojo.ShadeMojo
>> > java.lang.NoClassDefFoundError:
>> > org/codehaus/plexus/util/xml/XmlStreamWriter
>> > at java.lang.Class.getDeclaredConstructors0(Native Method)
>> > at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
>> > at java.lang.Class.getDeclaredConstructors(Class.java:2020)
>> > at
>> >
>> com.google.inject.spi.InjectionPoint.forConstructorOf(InjectionPoint.java:245)
>> > at
>> >
>> com.google.inject.internal.ConstructorBindingImpl.create(ConstructorBindingImpl.java:99)
>> > at
>> >
>> com.google.inject.internal.InjectorImpl.createUninitializedBinding(InjectorImpl.java:658)
>> > at
>> >
>> com.google.inject.internal.InjectorImpl.createJustInTimeBinding(InjectorImpl.java:882)
>> > at
>> >
>> com.google.inject.internal.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:805)
>> > at
>> >
>> com.google.inject.internal.InjectorImpl.getJustInTimeBinding(InjectorImpl.java:282)
>> > at
>> >
>> com.google.inject.internal.InjectorImpl.getBindingOrThrow(InjectorImpl.java:214)
>> > at
>> >
>> com.google.inject.internal.InjectorImpl.getProviderOrThrow(InjectorImpl.java:1006)
>> > at
>> >
>> com.google.inject.internal.InjectorImpl.getProvider(InjectorImpl.java:1038)
>> > at
>> >
>> com.google.inject.internal.InjectorImpl.getProvider(InjectorImpl.java:1001)
>> > at
>> >
>> com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1051)
>> > at
>> >
>> org.eclipse.sisu.space.AbstractDeferredClass.get(AbstractDeferredClass.java:48)
>> > at
>> >
>> com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
>> > at
>> >
>> com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53)
>> > at
>> >
&

Re: Unable to build clean checkout of avro from master

[Sean Busbey]

Could you copy/paste the output of running "mvn --version" ?

On Fri, Jul 8, 2016 at 2:34 PM, S G  wrote:
> Hi,
>
> I am unable to build a clean checkout of avro from master.
>
> Is it a known issue or I am doing something wrong?
>
>
> cd lang/java/avro
>
> mvn clean install
>
> [INFO] --- maven-bundle-plugin:2.5.3:bundle (default-bundle) @ avro ---
> [WARNING] Bundle org.apache.avro:avro:bundle:1.9.0-SNAPSHOT : Split
> package, multiple jars provide the same package:org/apache/avro
> Use Import/Export Package directive
> -split-package:=(merge-first|merge-last|error|first) to get rid of this
> warning
> Package found in   [Jar:., Jar:avro-guava-dependencies]
> Class path [Jar:., Jar:plexus-utils,
> Jar:avro-guava-dependencies, Jar:findbugs-annotations,
> Jar:jackson-core-asl, Jar:jackson-mapper-asl, Jar:paranamer,
> Jar:snappy-java, Jar:commons-compress, Jar:xz, Jar:joda-time,
> Jar:slf4j-api, Jar:slf4j-simple]
> [INFO]
> [INFO] --- maven-site-plugin:3.0:attach-descriptor (attach-descriptor)
> @ avro ---
> [INFO]
> [INFO] --- maven-jar-plugin:2.5:test-jar (default) @ avro ---
> [INFO] Building jar:
> ~/work/git/opensource/avro.upstream/avro/lang/java/avro/target/avro-1.9.0-SNAPSHOT-tests.jar
> [INFO]
> [INFO] --- maven-shade-plugin:1.7.1:shade (default) @ avro ---
> [WARNING] Error injecting: org.apache.maven.plugins.shade.mojo.ShadeMojo
> java.lang.NoClassDefFoundError:
> org/codehaus/plexus/util/xml/XmlStreamWriter
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
> at java.lang.Class.getDeclaredConstructors(Class.java:2020)
> at
> com.google.inject.spi.InjectionPoint.forConstructorOf(InjectionPoint.java:245)
> at
> com.google.inject.internal.ConstructorBindingImpl.create(ConstructorBindingImpl.java:99)
> at
> com.google.inject.internal.InjectorImpl.createUninitializedBinding(InjectorImpl.java:658)
> at
> com.google.inject.internal.InjectorImpl.createJustInTimeBinding(InjectorImpl.java:882)
> at
> com.google.inject.internal.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:805)
> at
> com.google.inject.internal.InjectorImpl.getJustInTimeBinding(InjectorImpl.java:282)
> at
> com.google.inject.internal.InjectorImpl.getBindingOrThrow(InjectorImpl.java:214)
> at
> com.google.inject.internal.InjectorImpl.getProviderOrThrow(InjectorImpl.java:1006)
> at
> com.google.inject.internal.InjectorImpl.getProvider(InjectorImpl.java:1038)
> at
> com.google.inject.internal.InjectorImpl.getProvider(InjectorImpl.java:1001)
> at
> com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1051)
> at
> org.eclipse.sisu.space.AbstractDeferredClass.get(AbstractDeferredClass.java:48)
> at
> com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
> at
> com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53)
> at
> com.google.inject.internal.ProviderInternalFactory$1.call(ProviderInternalFactory.java:65)
> at
> com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:115)
> at
> org.eclipse.sisu.bean.BeanScheduler$Activator.onProvision(BeanScheduler.java:176)
> at
> com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:126)
> at
> com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:68)
> at
> com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:63)
> at
> com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45)
> at
> com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016)
> at
> com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
> at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012)
> at org.eclipse.sisu.inject.Guice4$1.get(Guice4.java:162)
> at org.eclipse.sisu.inject.LazyBeanEntry.getValue(LazyBeanEntry.java:81)
> at
> org.eclipse.sisu.plexus.LazyPlexusBean.getValue(LazyPlexusBean.java:51)
> at
> org.codehaus.plexus.DefaultPlexusContainer.lookup(DefaultPlexusContainer.java:263)
> at
> org.codehaus.plexus.DefaultPlexusContainer.lookup(DefaultPlexusContainer.java:255)
> at
> org.apache.maven.plugin.internal.DefaultMavenPluginManager.getConfiguredMojo(DefaultMavenPluginManager.java:517)
> at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:121)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
> at
> 

[jira] [Commented] (AVRO-1875) ability encode/decode byte arrays in Base64

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15362853#comment-15362853
 ] 

Sean Busbey commented on AVRO-1875:
---

{quote}
The reason why I want to have Base64 - as of now byte arrays in JSON are 
encoded in Latin-1 encoding, which couldn't be displayed by UTF-8. According to 
JSON specification (https://tools.ietf.org/html/rfc7159#section-8.1) JSON 
should be encoded in UTF-8, 16 or 32, otherwise it is not valid, which causes a 
lot of problems when you try parse and work with them with Json libs. 
{quote}

This sounds like a bug, separate from the issue of having a base64 encoding. We 
shouldn't be encoding things counter to the JSON spec. I don't think we can 
ensure an arbitrary array of bytes is necessarily valid UTF-8, so I should 
probably read up on what we're doing right now (which also gets back to my 
prior question about if we have a spec for the JSON encoding).

{quote}
If you write as it was earlier:
{code}
JsonDecoder jsonDecoder = DecoderFactory.get().jsonDecoder(schema, 
recordInString);
{code}
you will get old version of JsonDecoder, which encodes in Latin-1.
To get Base64 with this patch you need write this:
{code}
boolean base64 = true;
JsonDecoder jsonDecoder = DecoderFactory.get().jsonDecoder(schema, 
recordInString, base64);
{code}
This boolean variable is a member of JsonEncoder and JsonDecoder class and has 
default value "false", which means using Latin-1.
{quote}

Right, I get how the mechanics of the change work when a developer is picking 
what gets used on each side of the serialization. What I'm not clear on is how 
given just serialized data one determines which decoding should be used 
(preferably where "one" is not a human but instead some application or library).

> ability encode/decode byte arrays in Base64
> ---
>
> Key: AVRO-1875
> URL: https://issues.apache.org/jira/browse/AVRO-1875
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: Paul Dudenkov
> Fix For: 1.9.0
>
>
> Hi, 
> I would like to add ability encode/decode byte arrays in Base64 in classes 
> JsonEncoder JsonDecoder.
> For this purpose I will add new constructors and overloaded factory methods.
> Old code will work as before.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1856) Update ConcatTool to support appending to an input file instead of creating a new output file

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1856:
--
Fix Version/s: 1.8.2
   1.9.0

> Update ConcatTool to support appending to an input file instead of creating a 
> new output file
> -
>
> Key: AVRO-1856
> URL: https://issues.apache.org/jira/browse/AVRO-1856
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.7.7, 1.8.1
>Reporter: Mike Hurley
> Fix For: 1.9.0, 1.8.2
>
> Attachments: AVRO-1856-ConcatTool-AppendToInput.diff
>
>
> It would be nice to have ConcatTool be able to append other input files into 
> one of the input files instead of creating a new output file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1875) ability encode/decode byte arrays in Base64

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15362607#comment-15362607
 ] 

Sean Busbey commented on AVRO-1875:
---

Do we have any kind of spec for the JSON serialization? We should make sure 
this use of base64 is documented somewhere. Also, given the current patch I'm 
not sure how I would tell programmatically wether I need the base64 version or 
not for a given set of data. any ideas on how we can correct that? 

> ability encode/decode byte arrays in Base64
> ---
>
> Key: AVRO-1875
> URL: https://issues.apache.org/jira/browse/AVRO-1875
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: Paul Dudenkov
> Fix For: 1.9.0
>
>
> Hi, 
> I would like to add ability encode/decode byte arrays in Base64 in classes 
> JsonEncoder JsonDecoder.
> For this purpose I will add new constructors and overloaded factory methods.
> Old code will work as before.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1875) ability encode/decode byte arrays in Base64

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1875:
--
Fix Version/s: 1.9.0

> ability encode/decode byte arrays in Base64
> ---
>
> Key: AVRO-1875
> URL: https://issues.apache.org/jira/browse/AVRO-1875
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: Paul Dudenkov
> Fix For: 1.9.0
>
>
> Hi, 
> I would like to add ability encode/decode byte arrays in Base64 in classes 
> JsonEncoder JsonDecoder.
> For this purpose I will add new constructors and overloaded factory methods.
> Old code will work as before.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (AVRO-1876) update TestConcatTool.testAppend to use MiniDFSCluster

[Sean Busbey (JIRA)]

Sean Busbey created AVRO-1876:
-

 Summary: update TestConcatTool.testAppend to use MiniDFSCluster
 Key: AVRO-1876
 URL: https://issues.apache.org/jira/browse/AVRO-1876
 Project: Avro
  Issue Type: Improvement
  Components: java
Affects Versions: 1.8.2
Reporter: Sean Busbey


Currently, the test added in AVRO-1856 for checking append correctness is 
limited by the use of hte LocalFileSystem implementation. It should be updated 
to rely on an instance of MiniDFSCluster so we can get the same semantics we'd 
have writing to HDFS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1843) Clarify importance of writer's schema in documentation

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1843:
--
Priority: Critical  (was: Minor)

> Clarify importance of writer's schema in documentation
> --
>
> Key: AVRO-1843
> URL: https://issues.apache.org/jira/browse/AVRO-1843
> Project: Avro
>  Issue Type: Improvement
>  Components: doc
>Reporter: Shannon Carey
>Priority: Critical
> Fix For: 1.7.8, 1.9.0, 1.8.2
>
>
> I'll be submitting a PR with some improvements to the Java Getting Started 
> page as well as the Specification which make it clearer that Avro must read 
> all data with the writer's schema before converting it into the reader's 
> schema and why, and explaining that's why the schema should be available next 
> to serialized data. Currently, it's arguably too easy to misinterpret Avro as 
> only requiring a single, reader's schema in order to read data while still 
> following the resolution rules which make Avro seem similar to JSON 
> (resolution by field name). For example, the Java API examples only appear to 
> involve one schema, hiding the fact that it reads in the writer's schema 
> implicitly. Also, the ability to serialize to JSON (where field names and 
> some type info is present) makes this misconception easy to believe.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1843) Clarify importance of writer's schema in documentation

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1843:
--
Fix Version/s: 1.8.2
   1.9.0
   1.7.8

> Clarify importance of writer's schema in documentation
> --
>
> Key: AVRO-1843
> URL: https://issues.apache.org/jira/browse/AVRO-1843
> Project: Avro
>  Issue Type: Improvement
>  Components: doc
>Reporter: Shannon Carey
>Priority: Minor
> Fix For: 1.7.8, 1.9.0, 1.8.2
>
>
> I'll be submitting a PR with some improvements to the Java Getting Started 
> page as well as the Specification which make it clearer that Avro must read 
> all data with the writer's schema before converting it into the reader's 
> schema and why, and explaining that's why the schema should be available next 
> to serialized data. Currently, it's arguably too easy to misinterpret Avro as 
> only requiring a single, reader's schema in order to read data while still 
> following the resolution rules which make Avro seem similar to JSON 
> (resolution by field name). For example, the Java API examples only appear to 
> involve one schema, hiding the fact that it reads in the writer's schema 
> implicitly. Also, the ability to serialize to JSON (where field names and 
> some type info is present) makes this misconception easy to believe.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1719) Avro fails to build against Boost 1.59.0

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15362480#comment-15362480
 ] 

Sean Busbey commented on AVRO-1719:
---

I've just escalated this to blocker for 1.8.2. It would be helpful if one of 
the C++ maintainers can confirm that I'm correct that this is safe to include 
in maintenance releases.

> Avro fails to build against Boost 1.59.0
> 
>
> Key: AVRO-1719
> URL: https://issues.apache.org/jira/browse/AVRO-1719
> Project: Avro
>  Issue Type: Bug
>  Components: build
>Affects Versions: 1.7.7
>Reporter: Tim Smith
>Priority: Blocker
> Fix For: 1.8.2
>
>
> Avro fails to build on OS X with Boost 1.59.0, dying on errors about 
> undeclared BOOST_ identifiers.
> Build logs are here: 
> https://gist.github.com/anonymous/03736608223d42f45ab1#file-02-make-L180
> Homebrew is tracking packages which fail to build against Boost 1.59.0 at 
> https://github.com/Homebrew/homebrew/pull/42960.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1704) Standardized format for encoding messages with Avro

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1704:
--
Fix Version/s: (was: 1.7.8)

> Standardized format for encoding messages with Avro
> ---
>
> Key: AVRO-1704
> URL: https://issues.apache.org/jira/browse/AVRO-1704
> Project: Avro
>  Issue Type: Improvement
>Reporter: Daniel Schierbeck
>Assignee: Niels Basjes
> Fix For: 1.9.0, 1.8.3
>
> Attachments: AVRO-1704-2016-05-03-Unfinished.patch, 
> AVRO-1704-20160410.patch
>
>
> I'm currently using the Datafile format for encoding messages that are 
> written to Kafka and Cassandra. This seems rather wasteful:
> 1. I only encode a single record at a time, so there's no need for sync 
> markers and other metadata related to multi-record files.
> 2. The entire schema is inlined every time.
> However, the Datafile format is the only one that has been standardized, 
> meaning that I can read and write data with minimal effort across the various 
> languages in use in my organization. If there was a standardized format for 
> encoding single values that was optimized for out-of-band schema transfer, I 
> would much rather use that.
> I think the necessary pieces of the format would be:
> 1. A format version number.
> 2. A schema fingerprint type identifier, i.e. Rabin, MD5, SHA256, etc.
> 3. The actual schema fingerprint (according to the type.)
> 4. Optional metadata map.
> 5. The encoded datum.
> The language libraries would implement a MessageWriter that would encode 
> datums in this format, as well as a MessageReader that, given a SchemaStore, 
> would be able to decode datums. The reader would decode the fingerprint and 
> ask its SchemaStore to return the corresponding writer's schema.
> The idea is that SchemaStore would be an abstract interface that allowed 
> library users to inject custom backends. A simple, file system based one 
> could be provided out of the box.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1704) Standardized format for encoding messages with Avro

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1704:
--
Fix Version/s: 1.7.8

> Standardized format for encoding messages with Avro
> ---
>
> Key: AVRO-1704
> URL: https://issues.apache.org/jira/browse/AVRO-1704
> Project: Avro
>  Issue Type: Improvement
>Reporter: Daniel Schierbeck
>Assignee: Niels Basjes
> Fix For: 1.9.0, 1.8.3
>
> Attachments: AVRO-1704-2016-05-03-Unfinished.patch, 
> AVRO-1704-20160410.patch
>
>
> I'm currently using the Datafile format for encoding messages that are 
> written to Kafka and Cassandra. This seems rather wasteful:
> 1. I only encode a single record at a time, so there's no need for sync 
> markers and other metadata related to multi-record files.
> 2. The entire schema is inlined every time.
> However, the Datafile format is the only one that has been standardized, 
> meaning that I can read and write data with minimal effort across the various 
> languages in use in my organization. If there was a standardized format for 
> encoding single values that was optimized for out-of-band schema transfer, I 
> would much rather use that.
> I think the necessary pieces of the format would be:
> 1. A format version number.
> 2. A schema fingerprint type identifier, i.e. Rabin, MD5, SHA256, etc.
> 3. The actual schema fingerprint (according to the type.)
> 4. Optional metadata map.
> 5. The encoded datum.
> The language libraries would implement a MessageWriter that would encode 
> datums in this format, as well as a MessageReader that, given a SchemaStore, 
> would be able to decode datums. The reader would decode the fingerprint and 
> ask its SchemaStore to return the corresponding writer's schema.
> The idea is that SchemaStore would be an abstract interface that allowed 
> library users to inject custom backends. A simple, file system based one 
> could be provided out of the box.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1704) Standardized format for encoding messages with Avro

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1704:
--
Fix Version/s: 1.8.3

> Standardized format for encoding messages with Avro
> ---
>
> Key: AVRO-1704
> URL: https://issues.apache.org/jira/browse/AVRO-1704
> Project: Avro
>  Issue Type: Improvement
>Reporter: Daniel Schierbeck
>Assignee: Niels Basjes
> Fix For: 1.9.0, 1.8.3
>
> Attachments: AVRO-1704-2016-05-03-Unfinished.patch, 
> AVRO-1704-20160410.patch
>
>
> I'm currently using the Datafile format for encoding messages that are 
> written to Kafka and Cassandra. This seems rather wasteful:
> 1. I only encode a single record at a time, so there's no need for sync 
> markers and other metadata related to multi-record files.
> 2. The entire schema is inlined every time.
> However, the Datafile format is the only one that has been standardized, 
> meaning that I can read and write data with minimal effort across the various 
> languages in use in my organization. If there was a standardized format for 
> encoding single values that was optimized for out-of-band schema transfer, I 
> would much rather use that.
> I think the necessary pieces of the format would be:
> 1. A format version number.
> 2. A schema fingerprint type identifier, i.e. Rabin, MD5, SHA256, etc.
> 3. The actual schema fingerprint (according to the type.)
> 4. Optional metadata map.
> 5. The encoded datum.
> The language libraries would implement a MessageWriter that would encode 
> datums in this format, as well as a MessageReader that, given a SchemaStore, 
> would be able to decode datums. The reader would decode the fingerprint and 
> ask its SchemaStore to return the corresponding writer's schema.
> The idea is that SchemaStore would be an abstract interface that allowed 
> library users to inject custom backends. A simple, file system based one 
> could be provided out of the box.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1704) Standardized format for encoding messages with Avro

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated AVRO-1704:
--
Fix Version/s: 1.9.0

> Standardized format for encoding messages with Avro
> ---
>
> Key: AVRO-1704
> URL: https://issues.apache.org/jira/browse/AVRO-1704
> Project: Avro
>  Issue Type: Improvement
>Reporter: Daniel Schierbeck
>Assignee: Niels Basjes
> Fix For: 1.9.0
>
> Attachments: AVRO-1704-2016-05-03-Unfinished.patch, 
> AVRO-1704-20160410.patch
>
>
> I'm currently using the Datafile format for encoding messages that are 
> written to Kafka and Cassandra. This seems rather wasteful:
> 1. I only encode a single record at a time, so there's no need for sync 
> markers and other metadata related to multi-record files.
> 2. The entire schema is inlined every time.
> However, the Datafile format is the only one that has been standardized, 
> meaning that I can read and write data with minimal effort across the various 
> languages in use in my organization. If there was a standardized format for 
> encoding single values that was optimized for out-of-band schema transfer, I 
> would much rather use that.
> I think the necessary pieces of the format would be:
> 1. A format version number.
> 2. A schema fingerprint type identifier, i.e. Rabin, MD5, SHA256, etc.
> 3. The actual schema fingerprint (according to the type.)
> 4. Optional metadata map.
> 5. The encoded datum.
> The language libraries would implement a MessageWriter that would encode 
> datums in this format, as well as a MessageReader that, given a SchemaStore, 
> would be able to decode datums. The reader would decode the fingerprint and 
> ask its SchemaStore to return the corresponding writer's schema.
> The idea is that SchemaStore would be an abstract interface that allowed 
> library users to inject custom backends. A simple, file system based one 
> could be provided out of the box.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)