[jira] [Commented] (AVRO-1559) Drop support for Ruby 1.8
[ https://issues.apache.org/jira/browse/AVRO-1559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110588#comment-15110588 ] Willem van Bergen commented on AVRO-1559: - If we're going to drop support for Ruby versions anyway, we should drop support for 1.9 as well because it is also EOL, and will no longer receive security fixes. > Drop support for Ruby 1.8 > - > > Key: AVRO-1559 > URL: https://issues.apache.org/jira/browse/AVRO-1559 > Project: Avro > Issue Type: Wish >Affects Versions: 1.7.7 > Reporter: Willem van Bergen > Assignee: Willem van Bergen > Fix For: 1.8.0 > > Attachments: AVRO-1559.patch > > > - Ruby 1.8 is EOL, and is even security issues aren't addressed anymore. > - It is also getting hard to set up Ruby 1.8 to run the tests (e.g. on a > recent OSX, it won't compile without manual fiddling). > - Handling character encodings in Ruby 1.9 is very different than Ruby 1.8. > Supporting both at the same time adds a lot of overhead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1559) Drop support for Ruby 1.8
[ https://issues.apache.org/jira/browse/AVRO-1559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14616730#comment-14616730 ] Willem van Bergen commented on AVRO-1559: - Ypu may want to bump the minimum Ruby requirement to 2.0, because Ruby 1.9 is also EOL now. Drop support for Ruby 1.8 - Key: AVRO-1559 URL: https://issues.apache.org/jira/browse/AVRO-1559 Project: Avro Issue Type: Wish Affects Versions: 1.7.7 Reporter: Willem van Bergen Assignee: Willem van Bergen Fix For: 1.8.0 Attachments: AVRO-1559.patch - Ruby 1.8 is EOL, and is even security issues aren't addressed anymore. - It is also getting hard to set up Ruby 1.8 to run the tests (e.g. on a recent OSX, it won't compile without manual fiddling). - Handling character encodings in Ruby 1.9 is very different than Ruby 1.8. Supporting both at the same time adds a lot of overhead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1559) Drop support for Ruby 1.8
[ https://issues.apache.org/jira/browse/AVRO-1559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14616740#comment-14616740 ] Willem van Bergen commented on AVRO-1559: - No code changes necessary for that; just updating the documentation is fine. Drop support for Ruby 1.8 - Key: AVRO-1559 URL: https://issues.apache.org/jira/browse/AVRO-1559 Project: Avro Issue Type: Wish Affects Versions: 1.7.7 Reporter: Willem van Bergen Assignee: Willem van Bergen Fix For: 1.8.0 Attachments: AVRO-1559.patch - Ruby 1.8 is EOL, and is even security issues aren't addressed anymore. - It is also getting hard to set up Ruby 1.8 to run the tests (e.g. on a recent OSX, it won't compile without manual fiddling). - Handling character encodings in Ruby 1.9 is very different than Ruby 1.8. Supporting both at the same time adds a lot of overhead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AVRO-1559) Drop support for Ruby 1.8
[ https://issues.apache.org/jira/browse/AVRO-1559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Willem van Bergen updated AVRO-1559: Fix Version/s: 1.8.0 Status: Patch Available (was: Open) This patch removes the checks for `bytesize` and `force_encoding`; on Ruby 1.9+ they will always exist. Drop support for Ruby 1.8 - Key: AVRO-1559 URL: https://issues.apache.org/jira/browse/AVRO-1559 Project: Avro Issue Type: Wish Affects Versions: 1.7.7 Reporter: Willem van Bergen Fix For: 1.8.0 - Ruby 1.8 is EOL, and is even security issues aren't addressed anymore. - It is also getting hard to set up Ruby 1.8 to run the tests (e.g. on a recent OSX, it won't compile without manual fiddling). - Handling character encodings in Ruby 1.9 is very different than Ruby 1.8. Supporting both at the same time adds a lot of overhead. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (AVRO-1559) Drop support for Ruby 1.8
[ https://issues.apache.org/jira/browse/AVRO-1559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Willem van Bergen updated AVRO-1559: Attachment: AVRO-1559.patch Drop support for Ruby 1.8 - Key: AVRO-1559 URL: https://issues.apache.org/jira/browse/AVRO-1559 Project: Avro Issue Type: Wish Affects Versions: 1.7.7 Reporter: Willem van Bergen Fix For: 1.8.0 Attachments: AVRO-1559.patch - Ruby 1.8 is EOL, and is even security issues aren't addressed anymore. - It is also getting hard to set up Ruby 1.8 to run the tests (e.g. on a recent OSX, it won't compile without manual fiddling). - Handling character encodings in Ruby 1.9 is very different than Ruby 1.8. Supporting both at the same time adds a lot of overhead. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Release Avro 1.8.0 soon?
We also previously discussed dropping support for Ruby 1.8, as it is EOL.
I created a patch that removes some of the hacks needed to support string
encodings in Ruby 1.8 and 1.9+ simultaneously:
https://issues.apache.org/jira/browse/AVRO-1559
Willem
On Aug 4, 2014, at 6:31 PM, Doug Cutting cutt...@apache.org wrote:
This introduces a minor incompatibility, so needs to go into 1.8.0, not 1.7.8.
Perhaps we should identify other changes that also introduce minor
incompatibilities and push out a 1.8.0 release soon? I don't want to
open the gates to major incompatibilities, but a few whose
incompatibilities that are high value and are relatively easy to
diagnose might be reasonable.
Other obvious candidates might be:
- AVRO-1334 (update java dependencies)
- AVRO-1550 (update protobuf dependency)
- AVRO-1514 (update perl dependencies)
What do others think?
Doug
On Mon, Aug 4, 2014 at 2:50 PM, Sean Busbey (JIRA) j...@apache.org wrote:
[
https://issues.apache.org/jira/browse/AVRO-997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated AVRO-997:
-
Status: Patch Available (was: In Progress)
Union of enum and null cannot be serialized
---
Key: AVRO-997
URL: https://issues.apache.org/jira/browse/AVRO-997
Project: Avro
Issue Type: Bug
Affects Versions: 1.5.1
Reporter: Aaron Kimball
Assignee: Sean Busbey
Fix For: 1.8.0
Attachments: AVRO-997.patch, AVRO-997.patch, AVRO-997.patch,
AVRO-997.permissive-generic-api.patch
I have a schema like:
{code}
[
{
type: enum,
name: Gender,
symbols: [M, F]
},
{
type : record,
name : Foo,
fields : [
{ type : [Gender, null], name : gender },
...
]
}
]
{code}
I build a record like {{Foo foo = new Foo(); foo.gender = Gender.M;}}
When I go to serialize this, I get:
{code}Not in union
[{type:enum,name:Gender,symbols:[M,F]},null]: M
at
org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:482)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:70)
at
org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:57)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (AVRO-1516) Unit test failure in Ruby 2.0 and above
[
https://issues.apache.org/jira/browse/AVRO-1516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14050045#comment-14050045
]
Willem van Bergen commented on AVRO-1516:
-
The problem is `union[double, int]` types. Integers get encoded as doubles,
because that's the first matching type. When reading it back, it will be read
as a double instead of an int. In more recent version of Ruby, the equality
operator will return false when comparing the int vs. the double.
The fix requires changing the `schema#validate` method, something like this:
https://github.com/wvanbergen/tros/commit/e5941173c37553b417663ae0ef4e6b4d9265c65b
I can work on a patch when I get back from vacation, at the end of this month.
Unit test failure in Ruby 2.0 and above
---
Key: AVRO-1516
URL: https://issues.apache.org/jira/browse/AVRO-1516
Project: Avro
Issue Type: Test
Components: ruby
Affects Versions: 1.7.6
Reporter: Martin Kleppmann
The following unit test fails when run with Ruby 2.0 and above:
{noformat}
$ bundle exec rake test
/Users/mkleppma/.rubies/ruby-2.0.0-p195/bin/ruby -Ilib:ext:bin:test
-I/Users/mkleppma/.gem/ruby/2.0.0/gems/rake-10.3.1/lib
/Users/mkleppma/.gem/ruby/2.0.0/gems/rake-10.3.1/lib/rake/rake_test_loader.rb
test/test_datafile.rb test/test_help.rb test/test_io.rb
test/test_protocol.rb test/test_schema.rb test/test_socket_transport.rb
Run options:
# Running tests:
[30/41] TestIO#test_union = 0.00 s
1) Failure:
test_union(TestIO)
[/Users/mkleppma/Applications/avro/lang/ruby/test/test_io.rb:339]:
-3372032630846393039 expected but was
-3.372032630846393e+18.
Finished tests in 0.346139s, 118.4495 tests/s, 2207.2058 assertions/s.
41 tests, 764 assertions, 1 failures, 0 errors, 0 skips
ruby -v: ruby 2.0.0p195 (2013-05-14 revision 40734) [x86_64-darwin12.3.0]
rake aborted!
Command failed with status (1): [ruby -Ilib:ext:bin:test
-I/Users/mkleppma/.gem/ruby/2.0.0/gems/rake-10.3.1/lib
/Users/mkleppma/.gem/ruby/2.0.0/gems/rake-10.3.1/lib/rake/rake_test_loader.rb
test/test_datafile.rb test/test_help.rb test/test_io.rb
test/test_protocol.rb test/test_schema.rb test/test_socket_transport.rb
]
/Users/mkleppma/.gem/ruby/2.0.0/gems/echoe-4.6.5/lib/echoe.rb:749:in `block
in define_tasks'
Tasks: TOP = test_inner
(See full trace by running task with --trace)
{noformat}
Brief investigation suggests that this isn't a bug in Avro, but just a badly
written test. The test is comparing -3372032630846393039 and
-3372032630846393000.0, which Ruby 1.9 and below consider to be equal, but
Ruby 2.0 and above consider to be non-equal.
Our tests shouldn't be relying on such edge cases of type coercion.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (AVRO-1499) Ruby 2+ Writes Invalid avro files using the avro gem
[
https://issues.apache.org/jira/browse/AVRO-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14050073#comment-14050073
]
Willem van Bergen commented on AVRO-1499:
-
W.r.t. monkey patching String. I agree with you in general, but in this case I
think it is preferable over doing a `respond_to` in the write method.
- It basically backports a method in a way that is completely compatible with
Ruby 1.9+, and it only does so if it's not available.
- This makes it a lot easier to drop 1.8 support later. Just one file of
backports to delete, instead of having to go through the entire source code to
find occurrences.
- Performance: only one respond_to check when the library is loaded, instead of
a check on every write.
I have no real strong feelings about it, so feel free to ignore this :)
Ruby 2+ Writes Invalid avro files using the avro gem
Key: AVRO-1499
URL: https://issues.apache.org/jira/browse/AVRO-1499
Project: Avro
Issue Type: Bug
Components: ruby
Affects Versions: 1.7.5
Reporter: Michael Ries
Assignee: Martin Kleppmann
Labels: ruby
Fix For: 1.7.7
Attachments: AVRO-1499-2.patch, AVRO-1499-3.patch, AVRO-1499.patch
The rubygem writes corrupted avro files under ruby 2.0.0 and ruby 2.1.1. It
appears to work correctly under jruby-1.7.10 and ruby 1.9.3.
Here is a reproducible:
```ruby
require 'avro'
data = [
{guid=144045de-eb44-dd1b-d9af-6c8b5d41a96e,
user_guid=0cd41235-5c14-eae9-00ed-c6eb11dd9119, name=My Awesome
Bank, created_at=1390617818, updated_at=1398180288, deleted_at=nil},
{guid=51e06057-14d2-7527-81fa-b07dba0a263b,
user_guid=0cd41235-5c14-eae9-00ed-c6eb11dd9119, name=Student Loans
R' Us, created_at=1386178342, updated_at=1398180286,
deleted_at=nil},
{guid=b4d1d99f-4351-d0e7-221c-a3fae08716bc,
user_guid=0cd41235-5c14-eae9-00ed-c6eb11dd9119, name=My Awesome
Bank, created_at=1390617026, updated_at=1398180288, deleted_at=nil},
{guid=084638fa-a78d-bbdd-e075-7c9c957a9b46,
user_guid=0cd41235-5c14-eae9-00ed-c6eb11dd9119, name=My Awesome
Bank, created_at=1390617138, updated_at=1398180288, deleted_at=nil},
{guid=79287c76-4e8f-0a21-7569-a2bcdc2b2f4d,
user_guid=0cd41235-5c14-eae9-00ed-c6eb11dd9119, name=My Awesome
Bank, created_at=1390617135, updated_at=1398180288, deleted_at=nil},
{guid=3bcc26b2-7d3b-6c4d-cb27-4eb1574b3c20,
user_guid=0cd41235-5c14-eae9-00ed-c6eb11dd9119, name=Cayman Islands
Bank, created_at=1386902345, updated_at=1398180288, deleted_at=nil},
{guid=75e1e56c-7611-4030-d002-afa2af70e5a1,
user_guid=0cd41235-5c14-eae9-00ed-c6eb11dd9119, name=My Awesome
Bank, created_at=1390617427, updated_at=1398180288, deleted_at=nil},
]
member_schema = -SCHEMA
{namespace: md.data_logs,
type: record,
name: Member,
fields: [
{name: guid, type: string},
{name: user_guid, type: string},
{name: name, type: [string,null]},
{name: created_at, type:long},
{name: updated_at, type:long},
{name: deleted_at, type:[long,null]}
]
}
SCHEMA
filepath = ./members.avro
File.unlink(filepath) if File.exists?(filepath)
Avro::DataFile.open(filepath, w, member_schema) do |dw|
data.each do |entry|
dw entry
end
end
entries = []
Avro::DataFile.open(filepath, r) do |reader|
reader.each do |entry|
entries entry
end
end
puts Here is the data I wrote into the file:
data.each{|e| p e }
print \n\n\n\n
puts Here is the data I read from the file:
entries.each{|e| p e }
```
Under ruby 2+ it fails with the message undefined method 'unpack' for
nil:NilClass (NoMethodError). I have also tested that the rubygem can
correctly read avro files written by the java client, but the java client
fails to read files written by the ruby client, so the issue is definitely in
how the rubygem is trying to write the binary file.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (AVRO-1536) Remove monkeypatching of Enumerable
[
https://issues.apache.org/jira/browse/AVRO-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14050222#comment-14050222
]
Willem van Bergen commented on AVRO-1536:
-
Looks good!
Remove monkeypatching of Enumerable
---
Key: AVRO-1536
URL: https://issues.apache.org/jira/browse/AVRO-1536
Project: Avro
Issue Type: Improvement
Components: ruby
Affects Versions: 1.7.6
Reporter: Martin Kleppmann
Assignee: Martin Kleppmann
Fix For: 1.7.7
Attachments: AVRO-1536.patch
The Avro Ruby gem adds a method {{collect_hash}} to the core module
{{Enumerable}}. It's bad form for a library to extend core modules like this,
and it's also unnecessary (stdlib methods can do the job perfectly well).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (AVRO-1499) Ruby 2+ Writes Invalid avro files using the avro gem
[
https://issues.apache.org/jira/browse/AVRO-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14047936#comment-14047936
]
Willem van Bergen commented on AVRO-1499:
-
I think it fails because the library uses `size` instead of `bytesize`. In Ruby
1.9+, size returns the number of characters, not the number of bytes in a
string. Which means that in a unicode string, the length of a string that gets
written is too short.
I will attach a patch that aliases `bytesize` to `size` in Ruby 1.8, and uses
bytesize.
Ruby 2+ Writes Invalid avro files using the avro gem
Key: AVRO-1499
URL: https://issues.apache.org/jira/browse/AVRO-1499
Project: Avro
Issue Type: Bug
Components: ruby
Affects Versions: 1.7.5
Reporter: Michael Ries
Assignee: Martin Kleppmann
Labels: ruby
Fix For: 1.7.7
Attachments: AVRO-1499.patch
The rubygem writes corrupted avro files under ruby 2.0.0 and ruby 2.1.1. It
appears to work correctly under jruby-1.7.10 and ruby 1.9.3.
Here is a reproducible:
```ruby
require 'avro'
data = [
{guid=144045de-eb44-dd1b-d9af-6c8b5d41a96e,
user_guid=0cd41235-5c14-eae9-00ed-c6eb11dd9119, name=My Awesome
Bank, created_at=1390617818, updated_at=1398180288, deleted_at=nil},
{guid=51e06057-14d2-7527-81fa-b07dba0a263b,
user_guid=0cd41235-5c14-eae9-00ed-c6eb11dd9119, name=Student Loans
R' Us, created_at=1386178342, updated_at=1398180286,
deleted_at=nil},
{guid=b4d1d99f-4351-d0e7-221c-a3fae08716bc,
user_guid=0cd41235-5c14-eae9-00ed-c6eb11dd9119, name=My Awesome
Bank, created_at=1390617026, updated_at=1398180288, deleted_at=nil},
{guid=084638fa-a78d-bbdd-e075-7c9c957a9b46,
user_guid=0cd41235-5c14-eae9-00ed-c6eb11dd9119, name=My Awesome
Bank, created_at=1390617138, updated_at=1398180288, deleted_at=nil},
{guid=79287c76-4e8f-0a21-7569-a2bcdc2b2f4d,
user_guid=0cd41235-5c14-eae9-00ed-c6eb11dd9119, name=My Awesome
Bank, created_at=1390617135, updated_at=1398180288, deleted_at=nil},
{guid=3bcc26b2-7d3b-6c4d-cb27-4eb1574b3c20,
user_guid=0cd41235-5c14-eae9-00ed-c6eb11dd9119, name=Cayman Islands
Bank, created_at=1386902345, updated_at=1398180288, deleted_at=nil},
{guid=75e1e56c-7611-4030-d002-afa2af70e5a1,
user_guid=0cd41235-5c14-eae9-00ed-c6eb11dd9119, name=My Awesome
Bank, created_at=1390617427, updated_at=1398180288, deleted_at=nil},
]
member_schema = -SCHEMA
{namespace: md.data_logs,
type: record,
name: Member,
fields: [
{name: guid, type: string},
{name: user_guid, type: string},
{name: name, type: [string,null]},
{name: created_at, type:long},
{name: updated_at, type:long},
{name: deleted_at, type:[long,null]}
]
}
SCHEMA
filepath = ./members.avro
File.unlink(filepath) if File.exists?(filepath)
Avro::DataFile.open(filepath, w, member_schema) do |dw|
data.each do |entry|
dw entry
end
end
entries = []
Avro::DataFile.open(filepath, r) do |reader|
reader.each do |entry|
entries entry
end
end
puts Here is the data I wrote into the file:
data.each{|e| p e }
print \n\n\n\n
puts Here is the data I read from the file:
entries.each{|e| p e }
```
Under ruby 2+ it fails with the message undefined method 'unpack' for
nil:NilClass (NoMethodError). I have also tested that the rubygem can
correctly read avro files written by the java client, but the java client
fails to read files written by the ruby client, so the issue is definitely in
how the rubygem is trying to write the binary file.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (AVRO-1499) Ruby 2+ Writes Invalid avro files using the avro gem
[
https://issues.apache.org/jira/browse/AVRO-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Willem van Bergen updated AVRO-1499:
Attachment: AVRO-1499-2.patch
- Ruby 1.8 compatibility changes are implemented in lib/avro/compat.rb so it's
easy to remove them later.
- Tested with Ruby 1.8.7, 1.9.3, 2.0.0, and 2.1.1.
- Not sure if I should include Manifest changes
Note: in Ruby 2.0+ there is still a failing test that is unrelated to this
change. I have a fix for that as well.
Ruby 2+ Writes Invalid avro files using the avro gem
Key: AVRO-1499
URL: https://issues.apache.org/jira/browse/AVRO-1499
Project: Avro
Issue Type: Bug
Components: ruby
Affects Versions: 1.7.5
Reporter: Michael Ries
Assignee: Martin Kleppmann
Labels: ruby
Fix For: 1.7.7
Attachments: AVRO-1499-2.patch, AVRO-1499.patch
The rubygem writes corrupted avro files under ruby 2.0.0 and ruby 2.1.1. It
appears to work correctly under jruby-1.7.10 and ruby 1.9.3.
Here is a reproducible:
```ruby
require 'avro'
data = [
{guid=144045de-eb44-dd1b-d9af-6c8b5d41a96e,
user_guid=0cd41235-5c14-eae9-00ed-c6eb11dd9119, name=My Awesome
Bank, created_at=1390617818, updated_at=1398180288, deleted_at=nil},
{guid=51e06057-14d2-7527-81fa-b07dba0a263b,
user_guid=0cd41235-5c14-eae9-00ed-c6eb11dd9119, name=Student Loans
R' Us, created_at=1386178342, updated_at=1398180286,
deleted_at=nil},
{guid=b4d1d99f-4351-d0e7-221c-a3fae08716bc,
user_guid=0cd41235-5c14-eae9-00ed-c6eb11dd9119, name=My Awesome
Bank, created_at=1390617026, updated_at=1398180288, deleted_at=nil},
{guid=084638fa-a78d-bbdd-e075-7c9c957a9b46,
user_guid=0cd41235-5c14-eae9-00ed-c6eb11dd9119, name=My Awesome
Bank, created_at=1390617138, updated_at=1398180288, deleted_at=nil},
{guid=79287c76-4e8f-0a21-7569-a2bcdc2b2f4d,
user_guid=0cd41235-5c14-eae9-00ed-c6eb11dd9119, name=My Awesome
Bank, created_at=1390617135, updated_at=1398180288, deleted_at=nil},
{guid=3bcc26b2-7d3b-6c4d-cb27-4eb1574b3c20,
user_guid=0cd41235-5c14-eae9-00ed-c6eb11dd9119, name=Cayman Islands
Bank, created_at=1386902345, updated_at=1398180288, deleted_at=nil},
{guid=75e1e56c-7611-4030-d002-afa2af70e5a1,
user_guid=0cd41235-5c14-eae9-00ed-c6eb11dd9119, name=My Awesome
Bank, created_at=1390617427, updated_at=1398180288, deleted_at=nil},
]
member_schema = -SCHEMA
{namespace: md.data_logs,
type: record,
name: Member,
fields: [
{name: guid, type: string},
{name: user_guid, type: string},
{name: name, type: [string,null]},
{name: created_at, type:long},
{name: updated_at, type:long},
{name: deleted_at, type:[long,null]}
]
}
SCHEMA
filepath = ./members.avro
File.unlink(filepath) if File.exists?(filepath)
Avro::DataFile.open(filepath, w, member_schema) do |dw|
data.each do |entry|
dw entry
end
end
entries = []
Avro::DataFile.open(filepath, r) do |reader|
reader.each do |entry|
entries entry
end
end
puts Here is the data I wrote into the file:
data.each{|e| p e }
print \n\n\n\n
puts Here is the data I read from the file:
entries.each{|e| p e }
```
Under ruby 2+ it fails with the message undefined method 'unpack' for
nil:NilClass (NoMethodError). I have also tested that the rubygem can
correctly read avro files written by the java client, but the java client
fails to read files written by the ruby client, so the issue is definitely in
how the rubygem is trying to write the binary file.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Re: Ruby gem fork - contribute back?
I have attached another patch to AVRO-1499 that solves a big unicode encoding problem. Because we are using string.size instead of string.bytesize, we end up writing the number of characters instead of the number of bytes in Ruby 1.9+. I also took the liberty of removing the collect_hash mixin. If someone could review it, that would be great. I had to set up a VM to test this against Ruby 1.8, because I can’t install Ruby 1.8 on my OSX work station anymore.** There is still a failing test on Ruby 2.0+, due to encoding of union[double, long] types. I have fixed that in my fork as well, but that one is a bit more involved. A CI server that runs against different the different language versions would be a great addition to this project, to make verifying changes less of a pain. Willem ** This s the main reason why I suggest dropping support for Ruby 1.8. Yes, it is possible to build the gem in a way to be compatible with Ruby 1.8 and up, but it is really deterring people from becoming contributors. On Jun 26, 2014, at 8:01 AM, Martin Kleppmann mar...@kleppmann.com wrote: IMHO it's not a big problem to support many versions of Ruby in a single gem. I would much rather have one gem that works in all widespread versions of Ruby (1.8.7, 1.9.3, 2.0, 2.1) than the complexity of multiple branches or forks. On the whole, Avro should work in modern versions of Ruby; if anything is broken, we should be able to fix it without breaking compatibility with 1.8. For example, this patch I posted a while ago fixes a major encoding issue under Ruby 2.0:https://issues.apache.org/jira/browse/AVRO-1499 -- I'd love someone to give it a review please, so that we can get it committed. Continuing to support Ruby 1.8 means we can't use Ruby 1.9 language features (e.g. the compact syntax for hashes with symbol keys) which is a shame, but not a major problem IMHO. All the unicode handling can be dynamically enabled/disabled at runtime depending on whether the Ruby VM supports it. Willem, it would be great to get your changes merged upstream. I'm happy to help you get them committed. Is there something in particular that is forcing you to drop 1.8 support? As long as Avro works in modern versions of Ruby, and supporting Ruby 1.8.7 isn't overly burdensome, I think it would be ok to drop Ruby 1.8.7 support in Avro 1.8.0, but keep it until then. Martin On 26 Jun 2014, at 03:46, Sean Busbey bus...@cloudera.com wrote: Personally, I'd rather see #2. I think it's very hard to know what the current use of Ruby 1.8 is. Support from the MRI community only ended ~1 year ago[1]. JRuby still supports running in 1.8 mode. They'll be dropping it in their next major release, but there isn't a schedule for that yet and they expect the current major release line to continue for some time after that[2]. Additionally, Heroku won't be ending support until the end of this month[3]. Even after that, it's not clear to me that they won't allow users to keep using it. As mentioned previously I'm a JRuby-in-1.9-mode user and I usually just work with the Java libraries directly. So this won't directly impact me, but I agree that it sucks when upgrades break things. So I don't feel like #1 is an option. We could also investigate maintaining a single gem that just had two implementations with in it, with the active one determined by the Ruby version. [1]: https://www.ruby-lang.org/en/news/2013/06/30/we-retire-1-8-7/ [2]: https://groups.google.com/d/msg/jruby-users/qmLpZ7qDwZo/J_iYViplcq4J [3]: https://devcenter.heroku.com/articles/ruby-support#ruby-versions -Sean On Wed, Jun 25, 2014 at 6:41 PM, Doug Cutting cutt...@apache.org wrote: On Wed, Jun 25, 2014 at 2:16 AM, Willem van Bergen wil...@vanbergen.org wrote: It's possible to have 2 different gems, but this is not very common in the Ruby world. Because Ruby 1.8 is not maintained anymore, not even for security issues, most people have moved on to newer versions. I can see a couple of options: 1. Assume that no one actually uses Ruby 1.8 anymore, and upgrade to 1.9 in Avro 1.7.7. A change that doesn't break anyone isn't incompatible. 2. Assume some folks still use Ruby 1.8 and add a ruby1.9 fork in Avro 1.7.7. Ruby users who upgrade to Avro 1.7.7 would need to opt-in to the Ruby 1.9 version. 3. Wait until we release 1.8.0 to upgrade Avro to support Ruby 1.9. (3) seems like a bad option unless we're confident we're going to release a 1.8.0 soon, which I am not. Folks hate getting broken by upgrades. Avro is a dependency of a lot of Java applications. Having an incompatible release makes it hard for one component to upgrade without forcing all to upgrade. Either you end up with a broken stack or with one that can never upgrade. Which of (1) or (2) seems more palatable to Ruby folks? Are there other options? Doug -- Sean
Ruby gem fork - contribute back?
Hi, For a Ruby project, I am using AVRO schemas to validate Ruby objects. Because I ran into some issues with the official avro gem, so I forked it: https://github.com/wvanbergen/tros. (The name probably only makes sense to Dutch people :) ### Changes - Fixed a round trip encoding issue for union(double, int) types. Integers were being encoded as floats, and read back as float. In Ruby versions 2.0 and later, a float == bigint equality check will return false. This caused a test to fail. - Fix UTF-8 support for Ruby 1.9+, and JRuby. The original code was written for Ruby 1.8, and there's some big changes to how to properly do this in Ruby 1.9+ and JRuby. - Remove monkey patching of Enumerable Monkey patching builtin objects is frowned upon, especially in libraries. Fixing it was easy: https://github.com/wvanbergen/tros/commit/c81d6189277111008ebb05239af91d286dd01061 - Dropped dependency of yajl-ruby and/or multi_json. The yajl-ruby dependency was causing compatibility issues with the rest of my application, and there's no released version yet with working multi_json (1.7.6 cannot be installed because multi_json is misspelled multi-json). Instead of fixing that, I decided to simply use Ruby's built in support for JSON. For libraries, the less external dependencies the better. I also did some heavy refactoring to make the Ruby codebase work outside of the context of the greater Avro project, and applied some best practices of the Ruby ecosystem. Finally, I set up CI (https://travis-ci.org/wvanbergen/tros) that checks the gem on multiple Ruby versions. ### Contributing back? I would like to contribute back my changes if you are interested. However, maintaining Ruby 1.8 support will make this very hard. Ruby 1.8 doesn't come with built in JSON support, and it's unicode handling is severely broken. It is also no longer maintained: https://www.ruby-lang.org/en/news/2013/12/17/maintenance-of-1-8-7-and-1-9-2/ Is it acceptable to drop support for Ruby 1.8? If so, I can work with you to get my changes back into the main codebase. Cheers, Willem van Bergen
Re: Ruby gem fork - contribute back?
I forked off trunk 2 days ago. It's possible to have 2 different gems, but this is not very common in the Ruby world. Because Ruby 1.8 is not maintained anymore, not even for security issues, most people have moved on to newer versions. This is in contrast with Python 2, which is still maintained and heavily used. My preference would be to document that the last release of avro that supports Ruby 1.8 is 1.7.5. (Version 1.7.6 won't install because of the multi_json issue). Maintaining 1.8 compatibility will be harder and harder over time and hold back development. E.g. it is already hard to even install a Ruby 1.8 version on a recent OSX due to compiler changes. Cheers, Willem On Jun 25, 2014, at 5:06 AM, Sean Busbey bus...@cloudera.com wrote: how far back did you fork? could we have a Ruby 1.8 gem and a Ruby 1.9+ gem? we have python and python 3 support broken out, for example. On Wed, Jun 25, 2014 at 3:51 AM, Willem van Bergen wil...@vanbergen.org wrote: Hi, For a Ruby project, I am using AVRO schemas to validate Ruby objects. Because I ran into some issues with the official avro gem, so I forked it: https://github.com/wvanbergen/tros. (The name probably only makes sense to Dutch people :) ### Changes - Fixed a round trip encoding issue for union(double, int) types. Integers were being encoded as floats, and read back as float. In Ruby versions 2.0 and later, a float == bigint equality check will return false. This caused a test to fail. - Fix UTF-8 support for Ruby 1.9+, and JRuby. The original code was written for Ruby 1.8, and there's some big changes to how to properly do this in Ruby 1.9+ and JRuby. - Remove monkey patching of Enumerable Monkey patching builtin objects is frowned upon, especially in libraries. Fixing it was easy: https://github.com/wvanbergen/tros/commit/c81d6189277111008ebb05239af91d286dd01061 - Dropped dependency of yajl-ruby and/or multi_json. The yajl-ruby dependency was causing compatibility issues with the rest of my application, and there's no released version yet with working multi_json (1.7.6 cannot be installed because multi_json is misspelled multi-json). Instead of fixing that, I decided to simply use Ruby's built in support for JSON. For libraries, the less external dependencies the better. I also did some heavy refactoring to make the Ruby codebase work outside of the context of the greater Avro project, and applied some best practices of the Ruby ecosystem. Finally, I set up CI ( https://travis-ci.org/wvanbergen/tros) that checks the gem on multiple Ruby versions. ### Contributing back? I would like to contribute back my changes if you are interested. However, maintaining Ruby 1.8 support will make this very hard. Ruby 1.8 doesn't come with built in JSON support, and it's unicode handling is severely broken. It is also no longer maintained: https://www.ruby-lang.org/en/news/2013/12/17/maintenance-of-1-8-7-and-1-9-2/ Is it acceptable to drop support for Ruby 1.8? If so, I can work with you to get my changes back into the main codebase. Cheers, Willem van Bergen -- Sean
Re: Ruby gem fork - contribute back?
Dropping support for Ruby 1.8 in the Avro 1.8.x series sounds like a plan. Is there already a branch for the 1.8 series? Until that time happens, I can maintain my fork for people requiring unicode UTF support on Ruby 1.9+. I know the multi_json issue is fixed in trunk. However, due to the project's structure, it's very hard to use a non-release version inside a project. Because the project doesn't include a gemspec file, you cannot make Bundler use the latest trunk version. (In my case, I use avro inside of another gem. Gem can only depend on released versions of other gems, so I had to fork release it.) Willem On Jun 25, 2014, at 5:33 AM, Sean Busbey bus...@cloudera.com wrote: IIRC, the multijson issue is fixed in the current snapshot. I dunno, I certainly stopped using Ruby 1.8 several years ago. The issue is that Avro has a strong history of favoring compatibility. It would be surprising for us to drop Ruby 1.8 support while still in the Avro 1.7 line. We could plan to only support Ruby 1.9+ in Avro 1.8 and take a contribution that targeted that, maybe? -- Sean On Jun 25, 2014 4:16 AM, Willem van Bergen wil...@vanbergen.org wrote: I forked off trunk 2 days ago. It's possible to have 2 different gems, but this is not very common in the Ruby world. Because Ruby 1.8 is not maintained anymore, not even for security issues, most people have moved on to newer versions. This is in contrast with Python 2, which is still maintained and heavily used. My preference would be to document that the last release of avro that supports Ruby 1.8 is 1.7.5. (Version 1.7.6 won't install because of the multi_json issue). Maintaining 1.8 compatibility will be harder and harder over time and hold back development. E.g. it is already hard to even install a Ruby 1.8 version on a recent OSX due to compiler changes. Cheers, Willem On Jun 25, 2014, at 5:06 AM, Sean Busbey bus...@cloudera.com wrote: how far back did you fork? could we have a Ruby 1.8 gem and a Ruby 1.9+ gem? we have python and python 3 support broken out, for example. On Wed, Jun 25, 2014 at 3:51 AM, Willem van Bergen wil...@vanbergen.org wrote: Hi, For a Ruby project, I am using AVRO schemas to validate Ruby objects. Because I ran into some issues with the official avro gem, so I forked it: https://github.com/wvanbergen/tros. (The name probably only makes sense to Dutch people :) ### Changes - Fixed a round trip encoding issue for union(double, int) types. Integers were being encoded as floats, and read back as float. In Ruby versions 2.0 and later, a float == bigint equality check will return false. This caused a test to fail. - Fix UTF-8 support for Ruby 1.9+, and JRuby. The original code was written for Ruby 1.8, and there's some big changes to how to properly do this in Ruby 1.9+ and JRuby. - Remove monkey patching of Enumerable Monkey patching builtin objects is frowned upon, especially in libraries. Fixing it was easy: https://github.com/wvanbergen/tros/commit/c81d6189277111008ebb05239af91d286dd01061 - Dropped dependency of yajl-ruby and/or multi_json. The yajl-ruby dependency was causing compatibility issues with the rest of my application, and there's no released version yet with working multi_json (1.7.6 cannot be installed because multi_json is misspelled multi-json). Instead of fixing that, I decided to simply use Ruby's built in support for JSON. For libraries, the less external dependencies the better. I also did some heavy refactoring to make the Ruby codebase work outside of the context of the greater Avro project, and applied some best practices of the Ruby ecosystem. Finally, I set up CI ( https://travis-ci.org/wvanbergen/tros) that checks the gem on multiple Ruby versions. ### Contributing back? I would like to contribute back my changes if you are interested. However, maintaining Ruby 1.8 support will make this very hard. Ruby 1.8 doesn't come with built in JSON support, and it's unicode handling is severely broken. It is also no longer maintained: https://www.ruby-lang.org/en/news/2013/12/17/maintenance-of-1-8-7-and-1-9-2/ Is it acceptable to drop support for Ruby 1.8? If so, I can work with you to get my changes back into the main codebase. Cheers, Willem van Bergen -- Sean
