[jira] [Created] (AVRO-1492) Upgrade snappy-java to 1.1.0 (or later)
Gabriel Reid created AVRO-1492: -- Summary: Upgrade snappy-java to 1.1.0 (or later) Key: AVRO-1492 URL: https://issues.apache.org/jira/browse/AVRO-1492 Project: Avro Issue Type: Improvement Reporter: Gabriel Reid Priority: Minor The version of snappy-java that is currently used in Avro (1.0.5) uses a method of loading native code that can cause issues in certain cases. A specific example of this is making use of Avro with snappy to write files from a Groovy script. Another example of the same situation can be seen [here on StackOverflow|http://stackoverflow.com/questions/16036179/snappy-java-exception-after-elastic-search-upgrade]. As of [version 1.1.0-m4|https://github.com/xerial/snappy-java/blob/develop/Milestone.md#snappy-java-110-m4-20-august-2013], snappy-java makes use of a new method of loading native code which no longer presents the issue when using snappy from Groovy. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (AVRO-1492) Upgrade snappy-java to 1.1.0 (or later)
[ https://issues.apache.org/jira/browse/AVRO-1492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabriel Reid updated AVRO-1492: --- Status: Patch Available (was: Open) Upgrade snappy-java to 1.1.0 (or later) --- Key: AVRO-1492 URL: https://issues.apache.org/jira/browse/AVRO-1492 Project: Avro Issue Type: Improvement Reporter: Gabriel Reid Priority: Minor Attachments: AVRO-1492.patch The version of snappy-java that is currently used in Avro (1.0.5) uses a method of loading native code that can cause issues in certain cases. A specific example of this is making use of Avro with snappy to write files from a Groovy script. Another example of the same situation can be seen [here on StackOverflow|http://stackoverflow.com/questions/16036179/snappy-java-exception-after-elastic-search-upgrade]. As of [version 1.1.0-m4|https://github.com/xerial/snappy-java/blob/develop/Milestone.md#snappy-java-110-m4-20-august-2013], snappy-java makes use of a new method of loading native code which no longer presents the issue when using snappy from Groovy. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (AVRO-1492) Upgrade snappy-java to 1.1.0 (or later)
[ https://issues.apache.org/jira/browse/AVRO-1492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabriel Reid updated AVRO-1492: --- Attachment: AVRO-1492.patch Here's the trivial patch to bump the version. I've successfully tested this by running the full automated test suite on both Mac OS 10 and Ubuntu 12.04, both with JDK 1.7. Upgrade snappy-java to 1.1.0 (or later) --- Key: AVRO-1492 URL: https://issues.apache.org/jira/browse/AVRO-1492 Project: Avro Issue Type: Improvement Reporter: Gabriel Reid Priority: Minor Attachments: AVRO-1492.patch The version of snappy-java that is currently used in Avro (1.0.5) uses a method of loading native code that can cause issues in certain cases. A specific example of this is making use of Avro with snappy to write files from a Groovy script. Another example of the same situation can be seen [here on StackOverflow|http://stackoverflow.com/questions/16036179/snappy-java-exception-after-elastic-search-upgrade]. As of [version 1.1.0-m4|https://github.com/xerial/snappy-java/blob/develop/Milestone.md#snappy-java-110-m4-20-august-2013], snappy-java makes use of a new method of loading native code which no longer presents the issue when using snappy from Groovy. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (AVRO-1465) Lack of information on error in GenericData#getSchemaName
[ https://issues.apache.org/jira/browse/AVRO-1465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabriel Reid updated AVRO-1465: --- Status: Patch Available (was: Open) Lack of information on error in GenericData#getSchemaName - Key: AVRO-1465 URL: https://issues.apache.org/jira/browse/AVRO-1465 Project: Avro Issue Type: Bug Components: java Reporter: Gabriel Reid Priority: Minor Attachments: AVRO-1465.patch When the type of a datum cannot be determined in GenericData#getSchemaName, an AvroRuntimeException is thrown (correctly), and the message of the exception contains the string representation of the datum. In the odd case that the datum is something that totally doesn't belong there, and it's toString representation doesn't clearly show what it is, it's very difficult to determine what the actual type of the error datum is. Including the class name of the unresolvable datum would make debugging situations like this much easier. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (AVRO-1473) Nested schema reference with empty namespace cannot be parsed
[
https://issues.apache.org/jira/browse/AVRO-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabriel Reid updated AVRO-1473:
---
Attachment: AVRO-1473.patch
Patch to resolve this issue. The approach taken is to attempt to retrieve the
name with the enclosing record's namespace, and if that fails, the name lookup
is attempted with the empty namespace.
Nested schema reference with empty namespace cannot be parsed
-
Key: AVRO-1473
URL: https://issues.apache.org/jira/browse/AVRO-1473
Project: Avro
Issue Type: Bug
Components: java
Reporter: Gabriel Reid
Attachments: AVRO-1473.patch
Related to AVRO-1295, a reference to a nested named schema with an empty
namespace cannot be parsed. An example of such a schema is as follows:
{code}
{
type : record,
name : Outer,
namespace : space,
fields : [ {
name : f1,
type : {
type : record,
name : Inner,
namespace : ,
fields : [ ]
}
}, {
name : f2,
type : Inner
} ]
}
{code}
Attempting to parse this results in the following exception:
org.apache.avro.SchemaParseException: Inner is not a defined name. The type
of the f2 field must be a defined name or a \{type: ...\} expression.
The issue seems to be that the lookup for the name Inner is done with the
namspace of the encoding record, and not with the empty namespace for this
case.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (AVRO-1465) Lack of information on error in GenericData#getSchemaName
Gabriel Reid created AVRO-1465: -- Summary: Lack of information on error in GenericData#getSchemaName Key: AVRO-1465 URL: https://issues.apache.org/jira/browse/AVRO-1465 Project: Avro Issue Type: Bug Components: java Reporter: Gabriel Reid Priority: Minor When the type of a datum cannot be determined in GenericData#getSchemaName, an AvroRuntimeException is thrown (correctly), and the message of the exception contains the string representation of the datum. In the odd case that the datum is something that totally doesn't belong there, and it's toString representation doesn't clearly show what it is, it's very difficult to determine what the actual type of the error datum is. Including the class name of the unresolvable datum would make debugging situations like this much easier. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (AVRO-1465) Lack of information on error in GenericData#getSchemaName
[ https://issues.apache.org/jira/browse/AVRO-1465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabriel Reid updated AVRO-1465: --- Attachment: AVRO-1465.patch Trivial patch to add class name information to the exception that is thrown Lack of information on error in GenericData#getSchemaName - Key: AVRO-1465 URL: https://issues.apache.org/jira/browse/AVRO-1465 Project: Avro Issue Type: Bug Components: java Reporter: Gabriel Reid Priority: Minor Attachments: AVRO-1465.patch When the type of a datum cannot be determined in GenericData#getSchemaName, an AvroRuntimeException is thrown (correctly), and the message of the exception contains the string representation of the datum. In the odd case that the datum is something that totally doesn't belong there, and it's toString representation doesn't clearly show what it is, it's very difficult to determine what the actual type of the error datum is. Including the class name of the unresolvable datum would make debugging situations like this much easier. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (AVRO-1150) TestSpecificCompiler leaks temporary directories
[ https://issues.apache.org/jira/browse/AVRO-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabriel Reid updated AVRO-1150: --- Attachment: AVRO-1150.patch Patch that moves to a single AvroTestUtil class, and removes all creation of temporary files and directories under tmp (or java.io.tmpdir). All temporary files and directories are now created under target/tmpfiles in a directory specific to the test class that is creating the file or directory. Run successfully against mvn verify. TestSpecificCompiler leaks temporary directories Key: AVRO-1150 URL: https://issues.apache.org/jira/browse/AVRO-1150 Project: Avro Issue Type: Bug Components: java Reporter: Gabriel Reid Priority: Minor Attachments: AVRO-1150.patch, AVRO-1150.patch The test class org.apache.avro.compiler.TestSpecificCompiler (in the compiler sub-project) creates a temporary directory, but only deletes the contents of the directory at the end of the test, and not the directory itself. This will cause a (small) issue on the long term when running in CI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1150) TestSpecificCompiler leaks temporary directories
[ https://issues.apache.org/jira/browse/AVRO-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13455663#comment-13455663 ] Gabriel Reid commented on AVRO-1150: There's a AvroTestUtil class that handles creation of temporary directories under target/ -- however, it's copied in the avro, ipc, and tools modules. I was thinking of making an avro-test module to contain a single copy of this class, and build it out a bit to include segregating temp files per test class. Any objections to this approach, particularly about the creation of a new module? TestSpecificCompiler leaks temporary directories Key: AVRO-1150 URL: https://issues.apache.org/jira/browse/AVRO-1150 Project: Avro Issue Type: Bug Components: java Reporter: Gabriel Reid Priority: Minor Attachments: AVRO-1150.patch The test class org.apache.avro.compiler.TestSpecificCompiler (in the compiler sub-project) creates a temporary directory, but only deletes the contents of the directory at the end of the test, and not the directory itself. This will cause a (small) issue on the long term when running in CI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1150) TestSpecificCompiler leaks temporary directories
[ https://issues.apache.org/jira/browse/AVRO-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13456044#comment-13456044 ] Gabriel Reid commented on AVRO-1150: Ok, thanks for the pointer on that, I'll go for that approach. TestSpecificCompiler leaks temporary directories Key: AVRO-1150 URL: https://issues.apache.org/jira/browse/AVRO-1150 Project: Avro Issue Type: Bug Components: java Reporter: Gabriel Reid Priority: Minor Attachments: AVRO-1150.patch The test class org.apache.avro.compiler.TestSpecificCompiler (in the compiler sub-project) creates a temporary directory, but only deletes the contents of the directory at the end of the test, and not the directory itself. This will cause a (small) issue on the long term when running in CI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1150) TestSpecificCompiler leaks temporary directories
[ https://issues.apache.org/jira/browse/AVRO-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13453792#comment-13453792 ] Gabriel Reid commented on AVRO-1150: That makes sense. This specific test isn't doing that though -- instead, it's doing a partial job of cleaning up after itself, and the files that it's making aren't under target, they're under the system's temp directory. For now this patch is just fixing the incomplete cleanup. If you think it's useful, I'd be happy to go through the tests and make it more consistent so that a temp directory under target is used, with cleanup of temporary files/directories only being done as part of the clean target. What do you think? TestSpecificCompiler leaks temporary directories Key: AVRO-1150 URL: https://issues.apache.org/jira/browse/AVRO-1150 Project: Avro Issue Type: Bug Components: java Reporter: Gabriel Reid Priority: Minor Attachments: AVRO-1150.patch The test class org.apache.avro.compiler.TestSpecificCompiler (in the compiler sub-project) creates a temporary directory, but only deletes the contents of the directory at the end of the test, and not the directory itself. This will cause a (small) issue on the long term when running in CI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (AVRO-1149) Add all-args constructor to generated class definitions
Gabriel Reid created AVRO-1149: -- Summary: Add all-args constructor to generated class definitions Key: AVRO-1149 URL: https://issues.apache.org/jira/browse/AVRO-1149 Project: Avro Issue Type: Improvement Components: java Reporter: Gabriel Reid Having an all-args constructor (in addition to the default no-args constructor) can make it a lot easier to work with generated Avro classes directly, as it allows setting up the state of the class in one call. The included builders are useful for this, but the use of the builder can be overly verbose and not very readable when a small number of parameters are involved. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1149) Add all-args constructor to generated class definitions
[ https://issues.apache.org/jira/browse/AVRO-1149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabriel Reid updated AVRO-1149: --- Attachment: AVRO-1149.patch Patch to add all-args constructor to generated java files. Passes all unit tests under lang/java. Add all-args constructor to generated class definitions --- Key: AVRO-1149 URL: https://issues.apache.org/jira/browse/AVRO-1149 Project: Avro Issue Type: Improvement Components: java Reporter: Gabriel Reid Attachments: AVRO-1149.patch Having an all-args constructor (in addition to the default no-args constructor) can make it a lot easier to work with generated Avro classes directly, as it allows setting up the state of the class in one call. The included builders are useful for this, but the use of the builder can be overly verbose and not very readable when a small number of parameters are involved. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (AVRO-1150) TestSpecificCompiler leaks temporary directories
Gabriel Reid created AVRO-1150: -- Summary: TestSpecificCompiler leaks temporary directories Key: AVRO-1150 URL: https://issues.apache.org/jira/browse/AVRO-1150 Project: Avro Issue Type: Bug Components: java Reporter: Gabriel Reid Priority: Minor Attachments: AVRO-1150.patch The test class org.apache.avro.compiler.TestSpecificCompiler (in the compiler sub-project) creates a temporary directory, but only deletes the contents of the directory at the end of the test, and not the directory itself. This will cause a (small) issue on the long term when running in CI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1046) ReflectDatumReader doesn't work with SpecificRecords containing an array of values
[ https://issues.apache.org/jira/browse/AVRO-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13274036#comment-13274036 ] Gabriel Reid commented on AVRO-1046: Yes, that's excellent, thanks! I was originally looking down the path of updating ReflectDatumReader, but clearly didn't realize how (relatively) easy it could be. BTW, sorry for the failing unit tests -- I thought I was in the clear as all of the tests in the avro sub-project were passing, so I didn't see that the ipc sub-project had failing tests (assuming it was there that you were getting failing tests?) ReflectDatumReader doesn't work with SpecificRecords containing an array of values -- Key: AVRO-1046 URL: https://issues.apache.org/jira/browse/AVRO-1046 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.6.2 Reporter: Gabriel Reid Assignee: Doug Cutting Priority: Minor Fix For: 1.7.0 Attachments: AVRO-1046.patch, AVRO-1046.patch When a ReflectDatumReader is used to read implementations of SpecificRecord, it fails if the SpecificRecord includes an (avro) array. This appears to be due to the fact that the newArray method in ReflectDatumReader works differently (based on a reflection-based schema) than the newArray method in GenericDatumReader (which is the base class of the DatumReaders). The included patch simply removes the delegation to SpecificData for the creation of a schema in the case of a SpecificRecord. My assumption is that the delegation to SpecificData is for performance reasons. The rationale in removing the delegation to SpecificData is that a) it doesn't work completely correctly, as demonstrated, and b) if a user is using ReflectDatumReader (instead of SpecificDatumReader) to read SpecificRecords, there may be other underlying reasons that reflection is specifically chosen, and so this intention should not be undermined by the delegation to SpecificData. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
