[jira] [Comment Edited] (AVRO-2178) avro C++ api support of tail reading of a growing avro file
[ https://issues.apache.org/jira/browse/AVRO-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637782#comment-16637782 ] Thiruvalluvan M. G. edited comment on AVRO-2178 at 10/4/18 3:52 AM: A solution to your problem is to use [Unix named pipe|https://en.wikipedia.org/wiki/Named_pipe]. Instead of writing to a standard file, create a named pipe and then use that file to write and read. This would work because the reading named pipe blocks until more data is available or the writer explicitly closes the file (see [this documentation|https://www.systutorials.com/docs/linux/man/7-pipe/] for example) was (Author: thiru_mg): A solution to your problem is to use [Unix named pipe|[http://example.com|https://en.wikipedia.org/wiki/Named_pipe]]. Instead of writing to a standard file, create a named pipe and then use that file to write and read. This would work because the reading named pipe blocks until more data is available or the writer explicitly closes the file (see [this documentation|https://www.systutorials.com/docs/linux/man/7-pipe/] for example) > avro C++ api support of tail reading of a growing avro file > --- > > Key: AVRO-2178 > URL: https://issues.apache.org/jira/browse/AVRO-2178 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.8.2 >Reporter: peien >Priority: Major > > Two processes, one is writing to an avro data file, another wishes to read > the latest written data. > The problem with current C++ API is that when it reaches the EOF, an > exception will be thrown, and from the user perspective, I have no way to > retry or 'tail read' it again from the last good position. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2178) avro C++ api support of tail reading of a growing avro file
[ https://issues.apache.org/jira/browse/AVRO-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637782#comment-16637782 ] Thiruvalluvan M. G. commented on AVRO-2178: --- A solution to your problem is to use [Unix named pipe|[http://example.com|https://en.wikipedia.org/wiki/Named_pipe]]. Instead of writing to a standard file, create a named pipe and then use that file to write and read. This would work because the reading named pipe blocks until more data is available or the writer explicitly closes the file (see [this documentation|https://www.systutorials.com/docs/linux/man/7-pipe/] for example) > avro C++ api support of tail reading of a growing avro file > --- > > Key: AVRO-2178 > URL: https://issues.apache.org/jira/browse/AVRO-2178 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.8.2 >Reporter: peien >Priority: Major > > Two processes, one is writing to an avro data file, another wishes to read > the latest written data. > The problem with current C++ API is that when it reaches the EOF, an > exception will be thrown, and from the user perspective, I have no way to > retry or 'tail read' it again from the last good position. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2235) Regenerate TestRecordWithLogicalTypes
[ https://issues.apache.org/jira/browse/AVRO-2235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637698#comment-16637698 ] ASF GitHub Bot commented on AVRO-2235: -- kojiromike commented on issue #341: Fix AVRO-2235 URL: https://github.com/apache/avro/pull/341#issuecomment-426858213 Ref #342 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Regenerate TestRecordWithLogicalTypes > - > > Key: AVRO-2235 > URL: https://issues.apache.org/jira/browse/AVRO-2235 > Project: Avro > Issue Type: Bug > Components: logical types >Reporter: Raymie Stata >Assignee: Raymie Stata >Priority: Major > > TestRecordWithLogicalTypes.java is code that was generated by the specific > compiler and then moved into the testing code tree. It hasn't been changed > in a while, although the compiler is evolving. I tried to regenerate it and > found there is a problem with record_with_logical_types.avsc. I will fix the > schema file and then regenerate TestRecordWithLogicalTypes and check both in. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2226) UnionSchema deduction is too permissive
[
https://issues.apache.org/jira/browse/AVRO-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637734#comment-16637734
]
ASF GitHub Bot commented on AVRO-2226:
--
kojiromike commented on issue #331: [AVRO-2226] Fixes UnionSchema specificity
URL: https://github.com/apache/avro/pull/331#issuecomment-426865455
This looks good to me. I ran the test without the code change and it failed
as it should:
```
==
FAIL: testUnionSchemaSpecificity (avro.tests.test_io.TestIO)
--
Traceback (most recent call last):
File "/home/michaels/avro/lang/py3/avro/tests/test_io.py", line 371, in
testUnionSchemaSpecificity
self.assertEqual(datum_to_read, datum_read)
AssertionError: {'entity': {'foo': 'this is an instance of schema A'}} !=
{'entity': {'bar': None}}
- {'entity': {'foo': 'this is an instance of schema A'}}
+ {'entity': {'bar': None}}
```
then I added the code change back and verified the tests all pass.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> UnionSchema deduction is too permissive
> ---
>
> Key: AVRO-2226
> URL: https://issues.apache.org/jira/browse/AVRO-2226
> Project: Avro
> Issue Type: Bug
> Components: python
>Affects Versions: 1.8.2
>Reporter: Andrew Kelleher
>Priority: Major
> Attachments: AVRO-2226.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> When given a schema of the form
> {code:java}
> {
> "type" : "record",
> "name" : "A",
> "namespace" : "com.example",
> "fields" : [
> {
> "name" : "foo",
> "type" : ["string", "null"]
> }
> ]
> }
> {
> "type" : "record",
> "name" : "B",
> "namespace" : "com.example",
> "fields" : [
> {
> "name" : "bar",
> "type" : ["string", "null"]
> }
> ]
> }
> {
> "type" : "record",
> "name" : "AOrB",
> "namespace" : "com.example",
> "fields" : [
> {
> "name" : "entity",
> "type" : [
> "com.example.A",
> "com.example.B"
> ]
> }
> ]
> }
> {code}
> And a datum of the form
> {code}
> {'entity': {'foo': 'this is an instance of schema A'}}{code}
> Converting to a message, and then from a message chooses the incorrect
> `entity` schema:
> {code}
> {'entity': {'bar': None}}{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2145) Can't generate Javadoc on master
[
https://issues.apache.org/jira/browse/AVRO-2145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637720#comment-16637720
]
ASF GitHub Bot commented on AVRO-2145:
--
kojiromike commented on issue #334: [AVRO-2145] Cannot create Javadoc on JDK8+
URL: https://github.com/apache/avro/pull/334#issuecomment-426861821
I can confirm this passes all the java-related tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Can't generate Javadoc on master
>
>
> Key: AVRO-2145
> URL: https://issues.apache.org/jira/browse/AVRO-2145
> Project: Avro
> Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Nandor Kollar
>Priority: Major
>
> {{mvn javadoc:aggregate}} fails with a bunch of Javadoc warnings on master
> when building with JDK8.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2145) Can't generate Javadoc on master
[
https://issues.apache.org/jira/browse/AVRO-2145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637710#comment-16637710
]
ASF GitHub Bot commented on AVRO-2145:
--
kojiromike commented on issue #334: [AVRO-2145] Cannot create Javadoc on JDK8+
URL: https://github.com/apache/avro/pull/334#issuecomment-426860151
@cutting I'm happy to help review stuff when I can, but 157 files is a bit
much for manual review. Can I help here by confirming I have run the tests
locally?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Can't generate Javadoc on master
>
>
> Key: AVRO-2145
> URL: https://issues.apache.org/jira/browse/AVRO-2145
> Project: Avro
> Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Nandor Kollar
>Priority: Major
>
> {{mvn javadoc:aggregate}} fails with a bunch of Javadoc warnings on master
> when building with JDK8.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-1658) Add avroDoc on reflect
[
https://issues.apache.org/jira/browse/AVRO-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637656#comment-16637656
]
ASF GitHub Bot commented on AVRO-1658:
--
erhwenkuo commented on issue #240: AVRO-1658: Java: Add reflection annotation
@AvroDoc.
URL: https://github.com/apache/avro/pull/240#issuecomment-426847421
Is AVRO-1658 merged yet? Still can't see in 1.8 version.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add avroDoc on reflect
> --
>
> Key: AVRO-1658
> URL: https://issues.apache.org/jira/browse/AVRO-1658
> Project: Avro
> Issue Type: New Feature
> Components: java
>Affects Versions: 1.7.7
>Reporter: Zhaonan Sun
>Assignee: Zhaonan Sun
>Priority: Major
> Labels: reflection
> Attachments:
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch,
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch,
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch
>
>
> Looks like @AvroMeta can't add reserved fields, like @AvroMeta("doc", "some
> doc") will have exceptions.
> I would be greate if we have a @AvroDoc("some documentations") in
> org.apache.avro.reflect
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2235) Regenerate TestRecordWithLogicalTypes
[ https://issues.apache.org/jira/browse/AVRO-2235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637359#comment-16637359 ] Raymie Stata commented on AVRO-2235: I'm going to keep this open in case someone else wants to comment. But based on what Thiru says, I think I'm going to close this issue as "won't fix." > Regenerate TestRecordWithLogicalTypes > - > > Key: AVRO-2235 > URL: https://issues.apache.org/jira/browse/AVRO-2235 > Project: Avro > Issue Type: Bug > Components: logical types >Reporter: Raymie Stata >Assignee: Raymie Stata >Priority: Major > > TestRecordWithLogicalTypes.java is code that was generated by the specific > compiler and then moved into the testing code tree. It hasn't been changed > in a while, although the compiler is evolving. I tried to regenerate it and > found there is a problem with record_with_logical_types.avsc. I will fix the > schema file and then regenerate TestRecordWithLogicalTypes and check both in. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2229) Ability to test using a Docker image
[
https://issues.apache.org/jira/browse/AVRO-2229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637337#comment-16637337
]
ASF GitHub Bot commented on AVRO-2229:
--
kojiromike commented on issue #336: [AVRO-2229] Test using a Docker image
URL: https://github.com/apache/avro/pull/336#issuecomment-426742942
Bypassing the rat test and all the known-passing tests, I ran the rest of
the `./build.sh test` in docker as described above. The very last step failed:
```
+ /bin/bash share/test/interop/bin/test_rpc_interop.sh
TEST: share/test/interop/rpc/add/onePlusOne
18/10/03 18:16:25 INFO util.log: Logging initialized @337ms to
org.eclipse.jetty.util.log.Slf4jLog
18/10/03 18:16:25 INFO server.Server: jetty-9.4.z-SNAPSHOT
18/10/03 18:16:25 INFO handler.ContextHandler: Started
o.e.j.s.ServletContextHandler@441772e{/,null,AVAILABLE}
18/10/03 18:16:25 INFO server.AbstractConnector: Started
ServerConnector@79924b{HTTP/1.1,[http/1.1]}{0.0.0.0:46719}
18/10/03 18:16:25 INFO server.Server: Started @813ms
2
Closing server.
18/10/03 18:16:27 INFO server.AbstractConnector: Stopped
ServerConnector@79924b{HTTP/1.1,[http/1.1]}{0.0.0.0:0}
18/10/03 18:16:27 INFO handler.ContextHandler: Stopped
o.e.j.s.ServletContextHandler@441772e{/,null,UNAVAILABLE}
18/10/03 18:16:27 INFO util.log: Logging initialized @324ms to
org.eclipse.jetty.util.log.Slf4jLog
18/10/03 18:16:27 INFO server.Server: jetty-9.4.z-SNAPSHOT
18/10/03 18:16:28 INFO handler.ContextHandler: Started
o.e.j.s.ServletContextHandler@441772e{/,null,AVAILABLE}
18/10/03 18:16:28 INFO server.AbstractConnector: Started
ServerConnector@79924b{HTTP/1.1,[http/1.1]}{0.0.0.0:33359}
18/10/03 18:16:28 INFO server.Server: Started @420ms
2
Closing server.
18/10/03 18:16:29 INFO server.AbstractConnector: Stopped
ServerConnector@79924b{HTTP/1.1,[http/1.1]}{0.0.0.0:0}
18/10/03 18:16:29 INFO handler.ContextHandler: Stopped
o.e.j.s.ServletContextHandler@441772e{/,null,UNAVAILABLE}
18/10/03 18:16:30 INFO util.log: Logging initialized @323ms to
org.eclipse.jetty.util.log.Slf4jLog
18/10/03 18:16:30 INFO server.Server: jetty-9.4.z-SNAPSHOT
18/10/03 18:16:30 INFO handler.ContextHandler: Started
o.e.j.s.ServletContextHandler@441772e{/,null,AVAILABLE}
18/10/03 18:16:30 INFO server.AbstractConnector: Started
ServerConnector@79924b{HTTP/1.1,[http/1.1]}{0.0.0.0:34025}
18/10/03 18:16:30 INFO server.Server: Started @419ms
/home/michaels/avro/lang/ruby/lib/avro/schema_validator.rb:127:in
`validate_map': undefined method `keys' for nil:NilClass (NoMethodError)
from
/home/michaels/avro/lang/ruby/lib/avro/schema_validator.rb:103:in
`validate_recursive'
from
/home/michaels/avro/lang/ruby/lib/avro/schema_validator.rb:156:in `block in
validate_possible_types'
from
/home/michaels/avro/lang/ruby/lib/avro/schema_validator.rb:154:in `map'
from
/home/michaels/avro/lang/ruby/lib/avro/schema_validator.rb:154:in
`validate_possible_types'
from
/home/michaels/avro/lang/ruby/lib/avro/schema_validator.rb:141:in
`validate_union'
from
/home/michaels/avro/lang/ruby/lib/avro/schema_validator.rb:105:in
`validate_recursive'
from
/home/michaels/avro/lang/ruby/lib/avro/schema_validator.rb:110:in `block in
validate_recursive'
from
/home/michaels/avro/lang/ruby/lib/avro/schema_validator.rb:108:in `each'
from
/home/michaels/avro/lang/ruby/lib/avro/schema_validator.rb:108:in
`validate_recursive'
from
/home/michaels/avro/lang/ruby/lib/avro/schema_validator.rb:67:in `validate!'
from /home/michaels/avro/lang/ruby/lib/avro/schema.rb:97:in
`validate'
from /home/michaels/avro/lang/ruby/lib/avro/io.rb:503:in `write_data'
from /home/michaels/avro/lang/ruby/lib/avro/io.rb:499:in `write'
from /home/michaels/avro/lang/ruby/lib/avro/ipc.rb:136:in
`write_handshake_request'
from /home/michaels/avro/lang/ruby/lib/avro/ipc.rb:105:in `request'
from lang/ruby/test/tool.rb:69:in `send_message'
from lang/ruby/test/tool.rb:137:in `main'
from lang/ruby/test/tool.rb:143:in `'
```
I don't know if this is reflected in any other ticket.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Ability to test using a Docker image
>
>
> Key: AVRO-2229
> URL: https://issues.apache.org/jira/browse/AVRO-2229
> Project: Avro
> Issue Type: Improvement
>Affects Versions: 1.8.2
>Reporter: Fokko Driesprong
>
[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type
[
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637056#comment-16637056
]
ASF GitHub Bot commented on AVRO-1891:
--
kgalieva edited a comment on issue #329: Improved conversions handling +
pluggable conversions support [AVRO-1891, AVRO-2065]
URL: https://github.com/apache/avro/pull/329#issuecomment-426582878
@scatrin thank you for this patch! I pulled, rebased and tested it in
integration with #309
Everything works just fine. What do you think about rebasing and proceeding
with code review? Thanks again, this is a super useful fix of a common problem.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Generated Java code fails with union containing logical type
>
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
> Issue Type: Bug
> Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
> Attachments: AVRO-1891.patch, AVRO-1891.yshi.1.patch,
> AVRO-1891.yshi.2.patch, AVRO-1891.yshi.3.patch, AVRO-1891.yshi.4.patch
>
>
> Example schema:
> {code}
> {
> "type": "record",
> "name": "RecordV1",
> "namespace": "org.brasslock.event",
> "fields": [
> { "name": "first", "type": ["null", {"type": "long",
> "logicalType":"timestamp-millis"}]}
> ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder =
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
> org.apache.avro.AvroRuntimeException: Unknown datum type
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at
[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type
[
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636982#comment-16636982
]
ASF GitHub Bot commented on AVRO-1891:
--
ivangreene commented on issue #329: Improved conversions handling + pluggable
conversions support [AVRO-1891, AVRO-2065]
URL: https://github.com/apache/avro/pull/329#issuecomment-426641789
+1 to that. Would really like to see this in the next release of Avro as
we're currently using a backport of #118 onto 1.8.2 and want to get back onto
an official release ASAP
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Generated Java code fails with union containing logical type
>
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
> Issue Type: Bug
> Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
> Attachments: AVRO-1891.patch, AVRO-1891.yshi.1.patch,
> AVRO-1891.yshi.2.patch, AVRO-1891.yshi.3.patch, AVRO-1891.yshi.4.patch
>
>
> Example schema:
> {code}
> {
> "type": "record",
> "name": "RecordV1",
> "namespace": "org.brasslock.event",
> "fields": [
> { "name": "first", "type": ["null", {"type": "long",
> "logicalType":"timestamp-millis"}]}
> ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder =
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
> org.apache.avro.AvroRuntimeException: Unknown datum type
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at
[jira] [Commented] (AVRO-1902) C: namespace "" should mean namespace = NULL
[
https://issues.apache.org/jira/browse/AVRO-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636963#comment-16636963
]
ASF GitHub Bot commented on AVRO-1902:
--
c4urself commented on issue #158: AVRO-1902: C: Namespace "" means space = NULL.
URL: https://github.com/apache/avro/pull/158#issuecomment-426640874
@johnsgill3 @walshb -- any progress here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> C: namespace "" should mean namespace = NULL
>
>
> Key: AVRO-1902
> URL: https://issues.apache.org/jira/browse/AVRO-1902
> Project: Avro
> Issue Type: Bug
> Components: c
>Reporter: Ben Walsh
>Priority: Minor
> Attachments: AVRO-1902.patch
>
>
> Similar to AVRO-1295 but for the C library. Namespace "" should be the NULL
> namespace when the schema is parsed. For example this breaks:
> ```
> {"type": "record", "name": "R", "fields": [
> {"name": "s", "type": {"type": "record", "namespace": "x", "name": "Y",
> "fields": [
> {"name": "e", "type": {"type": "record", "namespace": "", "name": "Z",
> "fields": [
> {"name": "f", "type": "Z"}
> ]}}
> ]}},
> {"name": "t", "type": "Z"}
> ]}
> ```
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-1528) avro_schema_enum_get returns invalid pointer for unknown indices
[ https://issues.apache.org/jira/browse/AVRO-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636961#comment-16636961 ] Christian Verkerk commented on AVRO-1528: - Bump – we're using a fork of the patch that [~bw123] made and it would make life a lot easier if we didn't have to patch this everytime we build pyavroc > avro_schema_enum_get returns invalid pointer for unknown indices > > > Key: AVRO-1528 > URL: https://issues.apache.org/jira/browse/AVRO-1528 > Project: Avro > Issue Type: Bug > Components: c >Affects Versions: 1.7.6 >Reporter: Jeno I. Hajdu >Priority: Trivial > Attachments: AVRO-1528.patch > > > When avro_schema_enum_get() is called with invalid indices it returns an > invalid pointer instead of NULL, this leads to segfaults. Similar problem is > already handled in avro_schema_enum_get_by_name() when converting the other > way round, avro_schema_enum_get() could handle it likewise. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2235) Regenerate TestRecordWithLogicalTypes
[ https://issues.apache.org/jira/browse/AVRO-2235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636784#comment-16636784 ] Thiruvalluvan M. G. commented on AVRO-2235: --- The intent was to test the compatibility of the code generated just before and just after AVRO-1684. So that test need not be touched now. In fact, it should probably be deleted because the current generated code is not the same as either of the files here and hence it does not really test anything current. But if we want to test the compatibility between the code generated before AVRO-1684 and the *current* version, we should regenerate both the classes. We need to regenerate them because the schema used for generating those two classes has changed in the meantime. I personally don't think the pain is worth it. However, if it should be done, here is how it can be achieved: # Obtain a copy of old avro-tools jar from [https://mvnrepository.com/artifact/org.apache.avro/avro-tools/1.7.7.] Click on the `jar` button to download the jar. # From lang/java issue the command `java -jar avro-tools-1.7.7.jar compile schema avro/src/test/resources/record_with_logical_types.avsc tmp`. # Massage the generated code (`tmp/org/apache/avro/specific/TestRecordWithLogicalTypes.java`) a bit: `%s/org.apache.avro.specific.TestRecordWithout/TestRecordWithout/g` # `cp tmp/org/apache/avro/specific/TestRecordWithLogicalTypes.java avro/src/test/java/org/apache/avro/specific/TestRecordWithoutLogicalTypes.java` This is not perfect; but it gets something close to what got generated then. > Regenerate TestRecordWithLogicalTypes > - > > Key: AVRO-2235 > URL: https://issues.apache.org/jira/browse/AVRO-2235 > Project: Avro > Issue Type: Bug > Components: logical types >Reporter: Raymie Stata >Assignee: Raymie Stata >Priority: Major > > TestRecordWithLogicalTypes.java is code that was generated by the specific > compiler and then moved into the testing code tree. It hasn't been changed > in a while, although the compiler is evolving. I tried to regenerate it and > found there is a problem with record_with_logical_types.avsc. I will fix the > schema file and then regenerate TestRecordWithLogicalTypes and check both in. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type
[
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636742#comment-16636742
]
ASF GitHub Bot commented on AVRO-1891:
--
kgalieva commented on issue #329: Improved conversions handling + pluggable
conversions support [AVRO-1891, AVRO-2065]
URL: https://github.com/apache/avro/pull/329#issuecomment-426582878
@scatrin thank you for this patch! I pulled, rebased and tested it with
integration with #309
Everything works just fine. What do you think about rebasing and proceeding
with code review? Thanks again, this is a super useful fix of a common problem.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Generated Java code fails with union containing logical type
>
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
> Issue Type: Bug
> Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
> Attachments: AVRO-1891.patch, AVRO-1891.yshi.1.patch,
> AVRO-1891.yshi.2.patch, AVRO-1891.yshi.3.patch, AVRO-1891.yshi.4.patch
>
>
> Example schema:
> {code}
> {
> "type": "record",
> "name": "RecordV1",
> "namespace": "org.brasslock.event",
> "fields": [
> { "name": "first", "type": ["null", {"type": "long",
> "logicalType":"timestamp-millis"}]}
> ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder =
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
> org.apache.avro.AvroRuntimeException: Unknown datum type
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at
