[jira] [Updated] (AVRO-1973) Upgrade netty to the latest 3.x release
[
https://issues.apache.org/jira/browse/AVRO-1973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated AVRO-1973:
---
Release Note: Netty was upgraded from 3.5.13.Final to 3.10.6.Final. After
3.6.0 Netty does not interrupt its worker threads therefore,
NettyServer.close() would hang until all the ongoing responder calls finish
their job. Normally it means that all the ongoing client requests will be
responded before the server stops. It also means that if any of the responder
calls are in dead lock NettyServer.close() will never return.
Added release note about the identified problem might occur due to the Netty
upgrade.
I've failed to check the Netty issue tracker whether any other breaking change
introduced.
> Upgrade netty to the latest 3.x release
> ---
>
> Key: AVRO-1973
> URL: https://issues.apache.org/jira/browse/AVRO-1973
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.8.1
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
>
> Netty version {{3.5.13}} is pretty old. Shall be updated to the latest 3.x
> release: {{3.10.6}}.
> (Not suggesting to upgrade to 4.x as it requires much more effort due to API
> incompatibilities.)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (AVRO-1937) C++ generator for recursive structure crashes
[
https://issues.apache.org/jira/browse/AVRO-1937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thiruvalluvan M. G. updated AVRO-1937:
--
Status: Patch Available (was: Open)
Here is a patch that fixes the problem. It handles recursive structures through
arrays, maps and unions well.
> C++ generator for recursive structure crashes
> -
>
> Key: AVRO-1937
> URL: https://issues.apache.org/jira/browse/AVRO-1937
> Project: Avro
> Issue Type: Bug
> Components: c++
>Affects Versions: 1.8.1
>Reporter: Jared Grubb
>
> I was testing out a Avro specification for a tree type and the generator
> crashed. I saw that there was a prior bug (AVRO-1667) that fixed some things
> in the Java parser.
> The same test-case on that bug will cause avrogencpp to segfault.
> $ cat tree.avro
> {
> "type":"record",
> "name":"SampleNode",
> "namespace":"org.spf4j.ssdump2.avro",
> "fields":[
> {"name":"count","type":"int","default":0},
> {"name":"subNodes","type":
> {"type":"array","items":{
> "type":"record","name":"SamplePair",
> "fields":[
>{"name":"method","type":
>{"type":"record","name":"Method",
>"fields":[
>
> {"name":"declaringClass","type":{"type":"string","avro.java.string":"String"}},
>
> {"name":"methodName","type":{"type":"string","avro.java.string":"String"}}
>]}},
>{"name":"node","type":"SampleNode"}
> ]
> }}
> }
> ]
> }
> $ avrogencpp -i tree.avro -o avro.cpp
> [ segfault ]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (AVRO-1937) C++ generator for recursive structure crashes
[
https://issues.apache.org/jira/browse/AVRO-1937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thiruvalluvan M. G. updated AVRO-1937:
--
Attachment: AVRO-1937.patch
> C++ generator for recursive structure crashes
> -
>
> Key: AVRO-1937
> URL: https://issues.apache.org/jira/browse/AVRO-1937
> Project: Avro
> Issue Type: Bug
> Components: c++
>Affects Versions: 1.8.1
>Reporter: Jared Grubb
> Attachments: AVRO-1937.patch
>
>
> I was testing out a Avro specification for a tree type and the generator
> crashed. I saw that there was a prior bug (AVRO-1667) that fixed some things
> in the Java parser.
> The same test-case on that bug will cause avrogencpp to segfault.
> $ cat tree.avro
> {
> "type":"record",
> "name":"SampleNode",
> "namespace":"org.spf4j.ssdump2.avro",
> "fields":[
> {"name":"count","type":"int","default":0},
> {"name":"subNodes","type":
> {"type":"array","items":{
> "type":"record","name":"SamplePair",
> "fields":[
>{"name":"method","type":
>{"type":"record","name":"Method",
>"fields":[
>
> {"name":"declaringClass","type":{"type":"string","avro.java.string":"String"}},
>
> {"name":"methodName","type":{"type":"string","avro.java.string":"String"}}
>]}},
>{"name":"node","type":"SampleNode"}
> ]
> }}
> }
> ]
> }
> $ avrogencpp -i tree.avro -o avro.cpp
> [ segfault ]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (AVRO-1886) Improve error reporting for schema validation
[
https://issues.apache.org/jira/browse/AVRO-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15865660#comment-15865660
]
David Lantos commented on AVRO-1886:
It does seem to be 2.1 related:
On 2.3.3 I get 0 failures:
{code}
Finished in 0.526283 seconds.
83 tests, 945 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0
notifications
100% passed
157.71 tests/s, 1795.61 assertions/s
{code}
On 2.4.0 I get three failures:
{code}
83 tests, 939 assertions, 3 failures, 0 errors, 0 pendings, 0 omissions, 0
notifications
96.3855% passed
164.37 tests/s, 1859.58 assertions/s
expected 'at . expected type boolean, got int with value 1' to be in '["at .
expected type boolean, got long with value 1"]'.
expected 'at .fruits expected type array, got int with value 1' to be in
'["at .fruits expected type array, got long with value 1"]'.
expected 'at . expected type null, got int with value 1' to be in '["at .
expected type null, got long with value 1"]'.
{code}
They all seem to be related to {{1}} being {{long}}, not {{int}} as the spec
expects -- but that is expected due to [Fixnum-Integer changes in
2.4|https://blog.heroku.com/ruby-2-4-features-hashes-integers-rounding#fixnum-by-ruby].
> Improve error reporting for schema validation
> -
>
> Key: AVRO-1886
> URL: https://issues.apache.org/jira/browse/AVRO-1886
> Project: Avro
> Issue Type: Improvement
> Components: ruby
>Reporter: Miroslav Csonka
>Assignee: Miroslav Csonka
> Fix For: 1.9.0, 1.8.2
>
>
> In Funding Circle we use avro schema to validate some financial data and
> we've found ourselves wondering what is invalid. We've noticed that avro
> [does already
> validation|https://github.com/apache/avro/blob/master/lang/ruby/lib/avro/schema.rb#L94],
> but does not report what fields are invalid, so we've decided to change it
> and we use Avro as a validation library.
> h5. What error messages to expect?
> We communicate errors for a given payload in format "at
> ".
> For we've decided to use syntax similar to json path:
> * . refers to root
> * \[0\] first index in array
> * .age would match 42 from {"age": 42}
> Error within object *person* and his optional array *houses* on *second item*
> with key *number_of_rooms*:
> {quote}
> at .person.houses\[1\].number_of_rooms expected type long, got string
> with value "not valid at all"
> {quote}
> Example of multiple errors for a single payload:
> {quote}
> at .\[0\] expected type int, got null
> at .\[1\] expected type int, got string with value \"e\"
> {quote}
> More examples can be seen at
> https://github.com/apache/avro/pull/111/files#diff-830ac80276854fceefa3f92c346d165f
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (AVRO-1935) C++ avro not reading snappy compressed file properly
[ https://issues.apache.org/jira/browse/AVRO-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. resolved AVRO-1935. --- Resolution: Duplicate The same problem is being addressed by AVRO-1748. > C++ avro not reading snappy compressed file properly > - > > Key: AVRO-1935 > URL: https://issues.apache.org/jira/browse/AVRO-1935 > Project: Avro > Issue Type: Bug > Components: c++ >Affects Versions: 1.8.0 > Environment: Linux >Reporter: razi > > I am trying to parse a snappy compressed avro file and I am getting the > following error: > Unknown Codec in data file: AVRO_CODEC_KEY = snappy > Is there support for snappy compression in Avro c++ ? > Patch ? > Thanks. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (AVRO-1886) Improve error reporting for schema validation
[
https://issues.apache.org/jira/browse/AVRO-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15865660#comment-15865660
]
David Lantos edited comment on AVRO-1886 at 2/14/17 11:58 AM:
--
It does seem to be 2.1 related:
On 2.3.3 I get 0 failures:
{code}
83 tests, 945 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0
notifications
100% passed
{code}
On 2.4.0 I get three failures:
{code}
83 tests, 939 assertions, 3 failures, 0 errors, 0 pendings, 0 omissions, 0
notifications
96.3855% passed
expected 'at . expected type boolean, got int with value 1' to be in '["at .
expected type boolean, got long with value 1"]'.
expected 'at .fruits expected type array, got int with value 1' to be in
'["at .fruits expected type array, got long with value 1"]'.
expected 'at . expected type null, got int with value 1' to be in '["at .
expected type null, got long with value 1"]'.
{code}
They all seem to be related to {{1}} being {{long}}, not {{int}} as the spec
expects -- but that is expected due to [Fixnum-Integer changes in
2.4|https://blog.heroku.com/ruby-2-4-features-hashes-integers-rounding#fixnum-by-ruby].
was (Author: sldblog):
It does seem to be 2.1 related:
On 2.3.3 I get 0 failures:
{code}
Finished in 0.526283 seconds.
83 tests, 945 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0
notifications
100% passed
157.71 tests/s, 1795.61 assertions/s
{code}
On 2.4.0 I get three failures:
{code}
83 tests, 939 assertions, 3 failures, 0 errors, 0 pendings, 0 omissions, 0
notifications
96.3855% passed
164.37 tests/s, 1859.58 assertions/s
expected 'at . expected type boolean, got int with value 1' to be in '["at .
expected type boolean, got long with value 1"]'.
expected 'at .fruits expected type array, got int with value 1' to be in
'["at .fruits expected type array, got long with value 1"]'.
expected 'at . expected type null, got int with value 1' to be in '["at .
expected type null, got long with value 1"]'.
{code}
They all seem to be related to {{1}} being {{long}}, not {{int}} as the spec
expects -- but that is expected due to [Fixnum-Integer changes in
2.4|https://blog.heroku.com/ruby-2-4-features-hashes-integers-rounding#fixnum-by-ruby].
> Improve error reporting for schema validation
> -
>
> Key: AVRO-1886
> URL: https://issues.apache.org/jira/browse/AVRO-1886
> Project: Avro
> Issue Type: Improvement
> Components: ruby
>Reporter: Miroslav Csonka
>Assignee: Miroslav Csonka
> Fix For: 1.9.0, 1.8.2
>
>
> In Funding Circle we use avro schema to validate some financial data and
> we've found ourselves wondering what is invalid. We've noticed that avro
> [does already
> validation|https://github.com/apache/avro/blob/master/lang/ruby/lib/avro/schema.rb#L94],
> but does not report what fields are invalid, so we've decided to change it
> and we use Avro as a validation library.
> h5. What error messages to expect?
> We communicate errors for a given payload in format "at
> ".
> For we've decided to use syntax similar to json path:
> * . refers to root
> * \[0\] first index in array
> * .age would match 42 from {"age": 42}
> Error within object *person* and his optional array *houses* on *second item*
> with key *number_of_rooms*:
> {quote}
> at .person.houses\[1\].number_of_rooms expected type long, got string
> with value "not valid at all"
> {quote}
> Example of multiple errors for a single payload:
> {quote}
> at .\[0\] expected type int, got null
> at .\[1\] expected type int, got string with value \"e\"
> {quote}
> More examples can be seen at
> https://github.com/apache/avro/pull/111/files#diff-830ac80276854fceefa3f92c346d165f
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (AVRO-1748) Add Snappy Compression to C++ DataFile
[ https://issues.apache.org/jira/browse/AVRO-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvalluvan M. G. updated AVRO-1748: -- Assignee: J. Langley Status: Patch Available (was: Open) > Add Snappy Compression to C++ DataFile > -- > > Key: AVRO-1748 > URL: https://issues.apache.org/jira/browse/AVRO-1748 > Project: Avro > Issue Type: New Feature > Components: c++ >Affects Versions: 1.7.7 >Reporter: J. Langley >Assignee: J. Langley > Fix For: 1.9.0 > > Original Estimate: 40h > Remaining Estimate: 40h > > The C++ component of the Avro project should support Snappy compression as > the Java, C, and Python components do. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (AVRO-1886) Improve error reporting for schema validation
[
https://issues.apache.org/jira/browse/AVRO-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15865660#comment-15865660
]
David Lantos edited comment on AVRO-1886 at 2/14/17 12:10 PM:
--
It does seem to be 2.1 related:
On 2.2.6, 0 fails (although I had to run with {{bundle exec rake test}}):
{code}
83 tests, 945 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0
notifications
100% passed
{code}
On 2.3.3 I get 0 failures:
{code}
83 tests, 945 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0
notifications
100% passed
{code}
On 2.4.0 I get three failures:
{code}
83 tests, 939 assertions, 3 failures, 0 errors, 0 pendings, 0 omissions, 0
notifications
96.3855% passed
expected 'at . expected type boolean, got int with value 1' to be in '["at .
expected type boolean, got long with value 1"]'.
expected 'at .fruits expected type array, got int with value 1' to be in
'["at .fruits expected type array, got long with value 1"]'.
expected 'at . expected type null, got int with value 1' to be in '["at .
expected type null, got long with value 1"]'.
{code}
They all seem to be related to {{1}} being {{long}}, not {{int}} as the spec
expects -- but that is expected due to [Fixnum-Integer changes in
2.4|https://blog.heroku.com/ruby-2-4-features-hashes-integers-rounding#fixnum-by-ruby].
was (Author: sldblog):
It does seem to be 2.1 related:
On 2.3.3 I get 0 failures:
{code}
83 tests, 945 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0
notifications
100% passed
{code}
On 2.4.0 I get three failures:
{code}
83 tests, 939 assertions, 3 failures, 0 errors, 0 pendings, 0 omissions, 0
notifications
96.3855% passed
expected 'at . expected type boolean, got int with value 1' to be in '["at .
expected type boolean, got long with value 1"]'.
expected 'at .fruits expected type array, got int with value 1' to be in
'["at .fruits expected type array, got long with value 1"]'.
expected 'at . expected type null, got int with value 1' to be in '["at .
expected type null, got long with value 1"]'.
{code}
They all seem to be related to {{1}} being {{long}}, not {{int}} as the spec
expects -- but that is expected due to [Fixnum-Integer changes in
2.4|https://blog.heroku.com/ruby-2-4-features-hashes-integers-rounding#fixnum-by-ruby].
> Improve error reporting for schema validation
> -
>
> Key: AVRO-1886
> URL: https://issues.apache.org/jira/browse/AVRO-1886
> Project: Avro
> Issue Type: Improvement
> Components: ruby
>Reporter: Miroslav Csonka
>Assignee: Miroslav Csonka
> Fix For: 1.9.0, 1.8.2
>
>
> In Funding Circle we use avro schema to validate some financial data and
> we've found ourselves wondering what is invalid. We've noticed that avro
> [does already
> validation|https://github.com/apache/avro/blob/master/lang/ruby/lib/avro/schema.rb#L94],
> but does not report what fields are invalid, so we've decided to change it
> and we use Avro as a validation library.
> h5. What error messages to expect?
> We communicate errors for a given payload in format "at
> ".
> For we've decided to use syntax similar to json path:
> * . refers to root
> * \[0\] first index in array
> * .age would match 42 from {"age": 42}
> Error within object *person* and his optional array *houses* on *second item*
> with key *number_of_rooms*:
> {quote}
> at .person.houses\[1\].number_of_rooms expected type long, got string
> with value "not valid at all"
> {quote}
> Example of multiple errors for a single payload:
> {quote}
> at .\[0\] expected type int, got null
> at .\[1\] expected type int, got string with value \"e\"
> {quote}
> More examples can be seen at
> https://github.com/apache/avro/pull/111/files#diff-830ac80276854fceefa3f92c346d165f
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (AVRO-1886) Improve error reporting for schema validation
[
https://issues.apache.org/jira/browse/AVRO-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15865691#comment-15865691
]
David Lantos commented on AVRO-1886:
I installed 2.1.10 and can't reproduce it:
{code}
$ bundle exec ruby -v
ruby 2.1.10p492 (2016-04-01 revision 54464) [x86_64-darwin15.0]
$ bundle exec rake test 2>&1 | grep -v "warning:"
/Users/davidlantos/.rbenv/versions/2.1.10/bin/ruby -w -I"lib:ext:bin:test"
-I"/Users/davidlantos/.rbenv/versions/2.1.10/lib/ruby/gems/2.1.0/gems/rake-12.0.0/lib"
"/Users/davidlantos/.rbenv/versions/2.1.10/lib/ruby/gems/2.1.0/gems/rake-12.0.0/lib/rake/rake_test_loader.rb"
"test/test_datafile.rb" "test/test_fingerprints.rb" "test/test_help.rb"
"test/test_io.rb" "test/test_protocol.rb" "test/test_schema.rb"
"test/test_schema_normalization.rb" "test/test_schema_validator.rb"
"test/test_socket_transport.rb"
Loaded suite
/Users/davidlantos/.rbenv/versions/2.1.10/lib/ruby/gems/2.1.0/gems/rake-12.0.0/lib/rake/rake_test_loader
..
Finished in 0.427299 seconds.
--
83 tests, 945 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0
notifications
100% passed
--
194.24 tests/s, 2211.57 assertions/s
{code}
> Improve error reporting for schema validation
> -
>
> Key: AVRO-1886
> URL: https://issues.apache.org/jira/browse/AVRO-1886
> Project: Avro
> Issue Type: Improvement
> Components: ruby
>Reporter: Miroslav Csonka
>Assignee: Miroslav Csonka
> Fix For: 1.9.0, 1.8.2
>
>
> In Funding Circle we use avro schema to validate some financial data and
> we've found ourselves wondering what is invalid. We've noticed that avro
> [does already
> validation|https://github.com/apache/avro/blob/master/lang/ruby/lib/avro/schema.rb#L94],
> but does not report what fields are invalid, so we've decided to change it
> and we use Avro as a validation library.
> h5. What error messages to expect?
> We communicate errors for a given payload in format "at
> ".
> For we've decided to use syntax similar to json path:
> * . refers to root
> * \[0\] first index in array
> * .age would match 42 from {"age": 42}
> Error within object *person* and his optional array *houses* on *second item*
> with key *number_of_rooms*:
> {quote}
> at .person.houses\[1\].number_of_rooms expected type long, got string
> with value "not valid at all"
> {quote}
> Example of multiple errors for a single payload:
> {quote}
> at .\[0\] expected type int, got null
> at .\[1\] expected type int, got string with value \"e\"
> {quote}
> More examples can be seen at
> https://github.com/apache/avro/pull/111/files#diff-830ac80276854fceefa3f92c346d165f
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (AVRO-1886) Improve error reporting for schema validation
[
https://issues.apache.org/jira/browse/AVRO-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15865842#comment-15865842
]
Tim Perkins commented on AVRO-1886:
---
There's an issue, https://issues.apache.org/jira/browse/AVRO-1996, and patch
available for the ruby 2.4 failures.
> Improve error reporting for schema validation
> -
>
> Key: AVRO-1886
> URL: https://issues.apache.org/jira/browse/AVRO-1886
> Project: Avro
> Issue Type: Improvement
> Components: ruby
>Reporter: Miroslav Csonka
>Assignee: Miroslav Csonka
> Fix For: 1.9.0, 1.8.2
>
>
> In Funding Circle we use avro schema to validate some financial data and
> we've found ourselves wondering what is invalid. We've noticed that avro
> [does already
> validation|https://github.com/apache/avro/blob/master/lang/ruby/lib/avro/schema.rb#L94],
> but does not report what fields are invalid, so we've decided to change it
> and we use Avro as a validation library.
> h5. What error messages to expect?
> We communicate errors for a given payload in format "at
> ".
> For we've decided to use syntax similar to json path:
> * . refers to root
> * \[0\] first index in array
> * .age would match 42 from {"age": 42}
> Error within object *person* and his optional array *houses* on *second item*
> with key *number_of_rooms*:
> {quote}
> at .person.houses\[1\].number_of_rooms expected type long, got string
> with value "not valid at all"
> {quote}
> Example of multiple errors for a single payload:
> {quote}
> at .\[0\] expected type int, got null
> at .\[1\] expected type int, got string with value \"e\"
> {quote}
> More examples can be seen at
> https://github.com/apache/avro/pull/111/files#diff-830ac80276854fceefa3f92c346d165f
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (AVRO-1886) Improve error reporting for schema validation
[
https://issues.apache.org/jira/browse/AVRO-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15865896#comment-15865896
]
Tim Perkins commented on AVRO-1886:
---
The problem seems to be dependent on the version of test-unit used. [~sacharya]
used test-unit 2.1.0.5 (July 2010) which displays the issue:
{code}
$ ruby --version
ruby 2.1.5p273 (2014-11-13 revision 48405) [x86_64-darwin15.0]
$ bundle show test-unit
/Users/tim/.rvm/rubies/ruby-2.1.5/lib/ruby/gems/2.1.0/gems/test-unit-2.1.5.0
$ RUBYOPT="-W0" bundle exec rake test
/Users/tim/.rvm/rubies/ruby-2.1.5/bin/ruby -w -I"lib:ext:bin:test"
-I"/Users/tim/.rvm/gems/ruby-2.1.5/gems/rake-12.0.0/lib"
"/Users/tim/.rvm/gems/ruby-2.1.5/gems/rake-12.0.0/lib/rake/rake_test_loader.rb"
"test/test_datafile.rb" "test/test_fingerprints.rb" "test/test_help.rb"
"test/test_io.rb" "test/test_protocol.rb" "test/test_schema.rb"
"test/test_schema_normalization.rb" "test/test_schema_validator.rb"
"test/test_socket_transport.rb"
# individual error output omitted
Finished tests in 0.366456s, 226.4938 tests/s, 2546.0083 assertions/s.
83 tests, 933 assertions, 11 failures, 0 errors, 0 skips
ruby -v: ruby 2.1.5p273 (2014-11-13 revision 48405) [x86_64-darwin15.0]
rake aborted!
{code}
The latest version of test-unit 3.2.3 (Nov 2016) does not:
{code}
$ ruby --version
ruby 2.1.5p273 (2014-11-13 revision 48405) [x86_64-darwin15.0]
$ bundle show test-unit
/Users/tim/.rvm/gems/ruby-2.1.5/gems/test-unit-3.2.3
$ RUBYOPT="-W0" bundle exec rake test
/Users/tim/.rvm/rubies/ruby-2.1.5/bin/ruby -w -I"lib:ext:bin:test"
-I"/Users/tim/.rvm/gems/ruby-2.1.5/gems/rake-12.0.0/lib"
"/Users/tim/.rvm/gems/ruby-2.1.5/gems/rake-12.0.0/lib/rake/rake_test_loader.rb"
"test/test_datafile.rb" "test/test_fingerprints.rb" "test/test_help.rb"
"test/test_io.rb" "test/test_protocol.rb" "test/test_schema.rb"
"test/test_schema_normalization.rb" "test/test_schema_validator.rb"
"test/test_socket_transport.rb"
Loaded suite
/Users/tim/.rvm/gems/ruby-2.1.5/gems/rake-12.0.0/lib/rake/rake_test_loader
Started
...
Finished in 0.450729 seconds.
---
83 tests, 945 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0
notifications
100% passed
---
184.15 tests/s, 2096.60 assertions/s
{code}
> Improve error reporting for schema validation
> -
>
> Key: AVRO-1886
> URL: https://issues.apache.org/jira/browse/AVRO-1886
> Project: Avro
> Issue Type: Improvement
> Components: ruby
>Reporter: Miroslav Csonka
>Assignee: Miroslav Csonka
> Fix For: 1.9.0, 1.8.2
>
>
> In Funding Circle we use avro schema to validate some financial data and
> we've found ourselves wondering what is invalid. We've noticed that avro
> [does already
> validation|https://github.com/apache/avro/blob/master/lang/ruby/lib/avro/schema.rb#L94],
> but does not report what fields are invalid, so we've decided to change it
> and we use Avro as a validation library.
> h5. What error messages to expect?
> We communicate errors for a given payload in format "at
> ".
> For we've decided to use syntax similar to json path:
> * . refers to root
> * \[0\] first index in array
> * .age would match 42 from {"age": 42}
> Error within object *person* and his optional array *houses* on *second item*
> with key *number_of_rooms*:
> {quote}
> at .person.houses\[1\].number_of_rooms expected type long, got string
> with value "not valid at all"
> {quote}
> Example of multiple errors for a single payload:
> {quote}
> at .\[0\] expected type int, got null
> at .\[1\] expected type int, got string with value \"e\"
> {quote}
> More examples can be seen at
> https://github.com/apache/avro/pull/111/files#diff-830ac80276854fceefa3f92c346d165f
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
org.apache.avro.SchemaParseException: Illegal initial character:
HI,
I am using Avro 1.8.1 and trying to parse the following schema_2. Can you
please help?
val schema: Schema = new Schema.Parser().parse(StrSchema)
I am getting
Exception in thread "main" org.apache.avro.SchemaParseException: Illegal
initial character:
{"type":"record","name":"WS_MESSAGES","fields":[{"name":"message_id","type":"string"},{"name":"action","type":"string"},{"name":"status","type":"string"},{"name":"request","type":"string"},{"name":"response","type":"string"}]}
at org.apache.avro.Schema.validateName(Schema.java:1079)
at org.apache.avro.Schema.access$200(Schema.java:79)
at org.apache.avro.Schema$Name.(Schema.java:436)
at org.apache.avro.Schema$Names.get(Schema.java:1043)
at org.apache.avro.Schema.parse(Schema.java:1160)
at org.apache.avro.Schema$Parser.parse(Schema.java:965)
at org.apache.avro.Schema$Parser.parse(Schema.java:953)
at main.scala.SampleStreaming$.main(SampleStreaming.scala:42)
at main.scala.SampleStreaming.main(SampleStreaming.scala)
Schema_2 :
"{\"type\":\"record\",\"name\":\"WS_MESSAGES\",\"fields\":[{\"name\":\"message_id\",\"type\":\"string\"},{\"name\":\"action\",\"type\":\"string\"},{\"name\":\"status\",\"type\":\"string\"},{\"name\":\"request\",\"type\":\"string\"},{\"name\":\"response\",\"type\":\"string\"}]}"
I initially was trying to parse the following schema_1 but got different
error , then manipulated the string to make the schema_2 above.
Exception in thread "main" org.apache.avro.SchemaParseException: No type:
{"schema":"{\"type\":\"record\",\"name\":\"WS_MESSAGES\",\"fields\":[{\"name\":\"message_id\",\"type\":\"string\"},{\"name\":\"action\",\"type\":\"string\"},{\"name\":\"status\",\"type\":\"string\"},{\"name\":\"request\",\"type\":\"string\"},{\"name\":\"response\",\"type\":\"string\"}],\"connect.name\":\"WS_MESSAGES\"}"}
at org.apache.avro.Schema.getRequiredText(Schema.java:1305)
at org.apache.avro.Schema.parse(Schema.java:1166)
Schema_1:
{"schema":"{\"type\":\"record\",\"name\":\"WS_MESSAGES\",\"fields\":[{\"name\":\"message_id\",\"type\":\"string\"},{\"name\":\"action\",\"type\":\"string\"},{\"name\":\"status\",\"type\":\"string\"},{\"name\":\"request\",\"type\":\"string\"},{\"name\":\"response\",\"type\":\"string\"}],\"connect.name\":\"WS_MESSAGES\"}"}
Thanks,
Amritendu
--
View this message in context:
http://apache-avro.679487.n3.nabble.com/org-apache-avro-SchemaParseException-Illegal-initial-character-tp4036786.html
Sent from the Avro - Developers mailing list archive at Nabble.com.
[jira] [Assigned] (AVRO-1996) incompatibility with ruby 2.4
[ https://issues.apache.org/jira/browse/AVRO-1996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suraj Acharya reassigned AVRO-1996: --- Assignee: Tim Perkins > incompatibility with ruby 2.4 > - > > Key: AVRO-1996 > URL: https://issues.apache.org/jira/browse/AVRO-1996 > Project: Avro > Issue Type: Bug > Components: ruby >Reporter: Tim Perkins >Assignee: Tim Perkins >Priority: Minor > > On master, there are failing tests with ruby 2.4. > The problem is due to the unification of Fixnum and Bignum types as Integer > in ruby 2.4. > The only loss of functionality is minor: in schema validation messages the > datum is reported as long instead of int. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AVRO-1996) incompatibility with ruby 2.4
[ https://issues.apache.org/jira/browse/AVRO-1996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suraj Acharya updated AVRO-1996: Labels: test (was: ) > incompatibility with ruby 2.4 > - > > Key: AVRO-1996 > URL: https://issues.apache.org/jira/browse/AVRO-1996 > Project: Avro > Issue Type: Bug > Components: ruby >Reporter: Tim Perkins >Assignee: Tim Perkins >Priority: Minor > Labels: test > Fix For: 1.9.0 > > > On master, there are failing tests with ruby 2.4. > The problem is due to the unification of Fixnum and Bignum types as Integer > in ruby 2.4. > The only loss of functionality is minor: in schema validation messages the > datum is reported as long instead of int. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AVRO-1996) incompatibility with ruby 2.4
[ https://issues.apache.org/jira/browse/AVRO-1996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suraj Acharya updated AVRO-1996: Fix Version/s: 1.9.0 Status: Patch Available (was: Open) > incompatibility with ruby 2.4 > - > > Key: AVRO-1996 > URL: https://issues.apache.org/jira/browse/AVRO-1996 > Project: Avro > Issue Type: Bug > Components: ruby >Reporter: Tim Perkins >Assignee: Tim Perkins >Priority: Minor > Fix For: 1.9.0 > > > On master, there are failing tests with ruby 2.4. > The problem is due to the unification of Fixnum and Bignum types as Integer > in ruby 2.4. > The only loss of functionality is minor: in schema validation messages the > datum is reported as long instead of int. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] avro pull request #191: AVRO-1996: compatibility with Ruby 2.4
Github user asfgit closed the pull request at: https://github.com/apache/avro/pull/191 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (AVRO-1996) incompatibility with ruby 2.4
[ https://issues.apache.org/jira/browse/AVRO-1996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suraj Acharya updated AVRO-1996: Resolution: Fixed Status: Resolved (was: Patch Available) Thanks [~tjwp] > incompatibility with ruby 2.4 > - > > Key: AVRO-1996 > URL: https://issues.apache.org/jira/browse/AVRO-1996 > Project: Avro > Issue Type: Bug > Components: ruby >Reporter: Tim Perkins >Assignee: Tim Perkins >Priority: Minor > Labels: test > Fix For: 1.9.0 > > > On master, there are failing tests with ruby 2.4. > The problem is due to the unification of Fixnum and Bignum types as Integer > in ruby 2.4. > The only loss of functionality is minor: in schema validation messages the > datum is reported as long instead of int. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AVRO-1996) incompatibility with ruby 2.4
[ https://issues.apache.org/jira/browse/AVRO-1996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15866947#comment-15866947 ] ASF GitHub Bot commented on AVRO-1996: -- Github user asfgit closed the pull request at: https://github.com/apache/avro/pull/191 > incompatibility with ruby 2.4 > - > > Key: AVRO-1996 > URL: https://issues.apache.org/jira/browse/AVRO-1996 > Project: Avro > Issue Type: Bug > Components: ruby >Reporter: Tim Perkins >Assignee: Tim Perkins >Priority: Minor > Labels: test > Fix For: 1.9.0 > > > On master, there are failing tests with ruby 2.4. > The problem is due to the unification of Fixnum and Bignum types as Integer > in ruby 2.4. > The only loss of functionality is minor: in schema validation messages the > datum is reported as long instead of int. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AVRO-1996) incompatibility with ruby 2.4
[ https://issues.apache.org/jira/browse/AVRO-1996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15866945#comment-15866945 ] ASF subversion and git services commented on AVRO-1996: --- Commit e9fd99ffa53e95ddbb1d136ddd6fabbfd2961f7f in avro's branch refs/heads/master from [~timperkins] [ https://git-wip-us.apache.org/repos/asf?p=avro.git;h=e9fd99f ] AVRO-1996: compatibility with Ruby 2.4 Fixed failing tests for ruby. Signed-off-by: Suraj Acharya Signed-off-by: Sean Busbey This closes #191 > incompatibility with ruby 2.4 > - > > Key: AVRO-1996 > URL: https://issues.apache.org/jira/browse/AVRO-1996 > Project: Avro > Issue Type: Bug > Components: ruby >Reporter: Tim Perkins >Assignee: Tim Perkins >Priority: Minor > Labels: test > Fix For: 1.9.0 > > > On master, there are failing tests with ruby 2.4. > The problem is due to the unification of Fixnum and Bignum types as Integer > in ruby 2.4. > The only loss of functionality is minor: in schema validation messages the > datum is reported as long instead of int. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AVRO-1989) Ruby schema validation for fixed types should use bytesize in error message
[
https://issues.apache.org/jira/browse/AVRO-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15866957#comment-15866957
]
Suraj Acharya commented on AVRO-1989:
-
I added the changes in AVRO-1996 and the tests are now passing.
+1 to the patch.
> Ruby schema validation for fixed types should use bytesize in error message
> ---
>
> Key: AVRO-1989
> URL: https://issues.apache.org/jira/browse/AVRO-1989
> Project: Avro
> Issue Type: Bug
> Components: ruby
>Affects Versions: 1.9.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
> Fix For: 1.9.0
>
> Attachments: AVRO-1989.0.patch
>
>
> From AVRO-1886:
> I'd like to get one thing improved, but it's fine as a follow-on.
> {code}
> + when :fixed
> + if datum.is_a? String
> +message = "expected fixed with size #{expected_schema.size},
> got \"#{datum}\" with size #{datum.size}"
> +result.add_error(path, message) unless datum.bytesize ==
> expected_schema.size
> + else
> +result.add_error(path, "expected fixed with size
> #{expected_schema.size}, got #{actual_value_message(datum)}")
> + end
> {code}
> the message here should use datum.bytesize instead of datum.size.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] avro pull request #196: Avro 1887
GitHub user spacharya opened a pull request: https://github.com/apache/avro/pull/196 Avro 1887 I have made the setup to use yetus. A quick setup of yetus can be done using this script : https://gist.github.com/spacharya/c1013fda8fe5d864fd028e8e5c478b81 You will need to have docker present on that machine. `/bin/bash ../yetus/precommit/test-patch.sh --plugins=all,-compile,-javadoc --user-plugins=share/pre_commit/ --build-tool=build https://github.com/apache/avro/pull/191 --docker --dockerFile=share/docker/Dockerfile --dirty-workspace` Is a sample deployment. I would appreciate if people can look at the changes and test it and let me know if there are any improvements. You can merge this pull request into a Git repository by running: $ git pull https://github.com/spacharya/avro AVRO-1887 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/avro/pull/196.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #196 commit 4dbf3dc7303b379b742fc9dd5c5d3b742297a085 Author: sacharya Date: 2016-10-10T21:56:40Z AVRO-1887 Setup precommit checks via Apache Yetus Signed-off-by: sacharya commit 5cedf25884649cf26a93bfd296bf2e5b867ac87b Author: sacharya Date: 2017-02-06T00:28:04Z Update Docket to read from the new docker image commit a12ce6ace51b53fe876a7d4a592c577bed7447c8 Author: sacharya Date: 2017-02-06T00:41:57Z removing unwanted installs --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (AVRO-1887) set up precommit checks via Apache Yetus
[ https://issues.apache.org/jira/browse/AVRO-1887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suraj Acharya updated AVRO-1887: Status: Patch Available (was: In Progress) > set up precommit checks via Apache Yetus > > > Key: AVRO-1887 > URL: https://issues.apache.org/jira/browse/AVRO-1887 > Project: Avro > Issue Type: Task > Components: build, community >Reporter: Sean Busbey >Assignee: Suraj Acharya >Priority: Blocker > Attachments: AVRO-1887-0.patch, AVRO-1887-1.patch, AVRO-1887-2.patch > > > The lack of automated checks for things like doing a full build is negatively > impacting my ability to do timely reviews. We should leverage the work done > by other communities and get precommit checks going with Apache Yetus ([ref > Apache Yetus > Precommit|http://yetus.apache.org/documentation/0.3.0/precommit-basic/]) > Yetus has built in support for using Docker as a containerized build > environment ([ref > docs|https://yetus.apache.org/documentation/0.3.0/precommit-advanced/#Docker_Support]), > but I don't know if we can use the one we have directly or if we'll need to > do some refactoring. > Implementing this will ultimately require a committer (so that they can make > changes to builds.apache.org), but a non-cmmitter can get the bulk of the > work done by managing to do local yetus checks with the docker feature active. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AVRO-1887) set up precommit checks via Apache Yetus
[ https://issues.apache.org/jira/browse/AVRO-1887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15867146#comment-15867146 ] Suraj Acharya commented on AVRO-1887: - Created a pull request here : https://github.com/apache/avro/pull/196 I have added some comments in the pull request. Please test it and let me know your comments. > set up precommit checks via Apache Yetus > > > Key: AVRO-1887 > URL: https://issues.apache.org/jira/browse/AVRO-1887 > Project: Avro > Issue Type: Task > Components: build, community >Reporter: Sean Busbey >Assignee: Suraj Acharya >Priority: Blocker > Attachments: AVRO-1887-0.patch, AVRO-1887-1.patch, AVRO-1887-2.patch > > > The lack of automated checks for things like doing a full build is negatively > impacting my ability to do timely reviews. We should leverage the work done > by other communities and get precommit checks going with Apache Yetus ([ref > Apache Yetus > Precommit|http://yetus.apache.org/documentation/0.3.0/precommit-basic/]) > Yetus has built in support for using Docker as a containerized build > environment ([ref > docs|https://yetus.apache.org/documentation/0.3.0/precommit-advanced/#Docker_Support]), > but I don't know if we can use the one we have directly or if we'll need to > do some refactoring. > Implementing this will ultimately require a committer (so that they can make > changes to builds.apache.org), but a non-cmmitter can get the bulk of the > work done by managing to do local yetus checks with the docker feature active. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
