[jira] [Created] (AVRO-1243) Support all compression codecs
Ted Malaska created AVRO-1243: - Summary: Support all compression codecs Key: AVRO-1243 URL: https://issues.apache.org/jira/browse/AVRO-1243 Project: Avro Issue Type: Improvement Components: java Affects Versions: 1.7.3 Reporter: Ted Malaska Priority: Minor I may be reading this wrong but at this time org.apache.avro.file.CodecFactory only supports null, deflate, and snappy compression codecs. I would like to change the fromString method to use Class.forName(codec).newInstance(); after the codec was not found in the REGISTERED map but before the AvroRuntimeException is thrown. Here are some of my supporting thoughts 1. This should not interduce much slowness because it will only be called initialize. 2. This will allow for support for GZip, BZip2, and LZO with out adding more dependances to the maven pom file. 3. This will allow for a future Jiri I would like to do that would allow AvroOutputFormat to be able to use the following configs: mapred.output.compress and mapred.output.compression.codec -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1243) Support all compression codecs
[ https://issues.apache.org/jira/browse/AVRO-1243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573598#comment-13573598 ] Ted Malaska commented on AVRO-1243: --- Hmm. Now I understand why this is difficult. All the hadoop compression codec use the org.apache.hadoop.io.compress.CompressionCodec interface. And there are no maven dependancies in the Avro maven module for Hadoop, so this interface will not be there. But there are dependancies in the avro-mapred module for Hadoop. So I will try something that I think will work in terms of execution but because I'm new to Avro I'm not sure it will fit in with the Avro spirt. I will try to get my patch in today and I will be interested to see what the Avro community feedback will be. Support all compression codecs -- Key: AVRO-1243 URL: https://issues.apache.org/jira/browse/AVRO-1243 Project: Avro Issue Type: Improvement Components: java Affects Versions: 1.7.3 Reporter: Ted Malaska Priority: Minor I may be reading this wrong but at this time org.apache.avro.file.CodecFactory only supports null, deflate, and snappy compression codecs. I would like to change the fromString method to use Class.forName(codec).newInstance(); after the codec was not found in the REGISTERED map but before the AvroRuntimeException is thrown. Here are some of my supporting thoughts 1. This should not interduce much slowness because it will only be called initialize. 2. This will allow for support for GZip, BZip2, and LZO with out adding more dependances to the maven pom file. 3. This will allow for a future Jiri I would like to do that would allow AvroOutputFormat to be able to use the following configs: mapred.output.compress and mapred.output.compression.codec -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (AVRO-1244) Provide a SeekableInput implementation for FileSystem retrieved output streams
Harsh J created AVRO-1244:
-
Summary: Provide a SeekableInput implementation for FileSystem
retrieved output streams
Key: AVRO-1244
URL: https://issues.apache.org/jira/browse/AVRO-1244
Project: Avro
Issue Type: Improvement
Components: java
Reporter: Harsh J
Priority: Minor
To use the DFW#appendTo API, one needs to pass a SeekableInput interface
object. Avro provides a usable utility for files that can be represented by a
File object, but in the Hadoop land, HDFS and other FSes can't be represented
via a File object and need a longer route to implement this interface.
We can add a simple HadoopSeekableFSInput or so that can take Hadoop provided
objects and wrap it into a SeekableInput interface ready for passing to Avro.
I propose something of the following type:
{code}
public static class HadoopSeekableFSInput implements SeekableInput {
FSDataInputStream in;
long length;
public SeekableFSInput(FSDataInputStream in, long length) {
this.in = in;
this.length = length;
}
public void close() throws IOException {
in.close();
}
public void seek(long p) throws IOException {
in.seek(p);
}
public long tell() throws IOException {
return in.getPos();
}
public long length() throws IOException {
return length;
}
public int read(byte[] b, int off, int len) throws IOException {
return in.read(b, off, len);
}
}
{code}
The above can be constructed by users via a simple call such as {{new
HadoopSeekableFSInput(fs.open(filePath), fs.getFileStatus(filePath).getLen())}}.
Ideally this class should belong in the avro core module but that strictly does
not depend on Hadoop-Common today, and hence somewhere else may be more
suitable.
This lets users write Avro-append code such as
https://gist.github.com/QwertyManiac/4724582 more easily.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (AVRO-1244) Provide a SeekableInput implementation for FileSystem retrieved output streams
[
https://issues.apache.org/jira/browse/AVRO-1244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved AVRO-1244.
---
Resolution: Invalid
I missed {{FsInput}} in avro-mapred. It already implements this! :)
Provide a SeekableInput implementation for FileSystem retrieved output streams
--
Key: AVRO-1244
URL: https://issues.apache.org/jira/browse/AVRO-1244
Project: Avro
Issue Type: Improvement
Components: java
Reporter: Harsh J
Priority: Minor
To use the DFW#appendTo API, one needs to pass a SeekableInput interface
object. Avro provides a usable utility for files that can be represented by a
File object, but in the Hadoop land, HDFS and other FSes can't be represented
via a File object and need a longer route to implement this interface.
We can add a simple HadoopSeekableFSInput or so that can take Hadoop provided
objects and wrap it into a SeekableInput interface ready for passing to Avro.
I propose something of the following type:
{code}
public static class HadoopSeekableFSInput implements SeekableInput {
FSDataInputStream in;
long length;
public SeekableFSInput(FSDataInputStream in, long length) {
this.in = in;
this.length = length;
}
public void close() throws IOException {
in.close();
}
public void seek(long p) throws IOException {
in.seek(p);
}
public long tell() throws IOException {
return in.getPos();
}
public long length() throws IOException {
return length;
}
public int read(byte[] b, int off, int len) throws IOException {
return in.read(b, off, len);
}
}
{code}
The above can be constructed by users via a simple call such as {{new
HadoopSeekableFSInput(fs.open(filePath),
fs.getFileStatus(filePath).getLen())}}.
Ideally this class should belong in the avro core module but that strictly
does not depend on Hadoop-Common today, and hence somewhere else may be more
suitable.
This lets users write Avro-append code such as
https://gist.github.com/QwertyManiac/4724582 more easily.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1243) Support all compression codecs
[ https://issues.apache.org/jira/browse/AVRO-1243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573666#comment-13573666 ] Doug Cutting commented on AVRO-1243: You might add a HadoopCodec to the avro-mapred module that implements an Avro codec in terms of a Hadoop CompressionCodec... Support all compression codecs -- Key: AVRO-1243 URL: https://issues.apache.org/jira/browse/AVRO-1243 Project: Avro Issue Type: Improvement Components: java Affects Versions: 1.7.3 Reporter: Ted Malaska Priority: Minor I may be reading this wrong but at this time org.apache.avro.file.CodecFactory only supports null, deflate, and snappy compression codecs. I would like to change the fromString method to use Class.forName(codec).newInstance(); after the codec was not found in the REGISTERED map but before the AvroRuntimeException is thrown. Here are some of my supporting thoughts 1. This should not interduce much slowness because it will only be called initialize. 2. This will allow for support for GZip, BZip2, and LZO with out adding more dependances to the maven pom file. 3. This will allow for a future Jiri I would like to do that would allow AvroOutputFormat to be able to use the following configs: mapred.output.compress and mapred.output.compression.codec -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1242) AvroTrevniOutputFormat Metadata filter subString bug
[ https://issues.apache.org/jira/browse/AVRO-1242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Cutting updated AVRO-1242: --- Resolution: Fixed Status: Resolved (was: Patch Available) I committed this. Thanks, Ted! AvroTrevniOutputFormat Metadata filter subString bug Key: AVRO-1242 URL: https://issues.apache.org/jira/browse/AVRO-1242 Project: Avro Issue Type: Bug Affects Versions: 1.7.4 Reporter: Ted Malaska Assignee: Ted Malaska Fix For: 1.7.4 Attachments: AVRO-1234.patch.1, AVRO-1242.patch In AvroTrevniOutputFormat there seems to be a cut and post error in the metadata filtering. The result is not trevni.meta. metadatas will successfully get to the writer. if (e.getKey().startsWith(META_PREFIX)) meta.put(e.getKey().substring(AvroJob.TEXT_PREFIX.length()), e.getValue().getBytes(MetaData.UTF8)); META_PREFIX = trevni.meta. TEXT_PREFIX = avro.meta.text. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1237) Avro-C segfaults when union discriminant out of bounds
[ https://issues.apache.org/jira/browse/AVRO-1237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573693#comment-13573693 ] Doug Cutting commented on AVRO-1237: Can someone familiar with Avro C please review this? Also, is it possible to add a test case for this? Avro-C segfaults when union discriminant out of bounds -- Key: AVRO-1237 URL: https://issues.apache.org/jira/browse/AVRO-1237 Project: Avro Issue Type: Bug Components: c Environment: Avro-C 1.7.2 Ubuntu 12.04 x86_64 Reporter: Michael Cooper Attachments: 0001-Check-union-discriminant-bounds-in-both-directions.patch libavro will segfault when decrypting a specially crafted (or corrupted) avro file when the discriminant is out of bounds. There is already a check for 0, but there is no upper bounds check. I have attached a patch that checks the bounds. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1243) Support all compression codecs
[ https://issues.apache.org/jira/browse/AVRO-1243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573706#comment-13573706 ] Joey Echeverria commented on AVRO-1243: --- q. You might add a HadoopCodec to the avro-mapred module that implements an Avro codec in terms of a Hadoop CompressionCodec.. I like the idea of a Hadoop Codec, but should it be only available as part of avro-mapred or should it be it's own module? As I understand it, with the exception of null and deflate, compression codecs are an optional part of the Avro spec so there should already be ways of handling trying to load a codec that doesn't exist. Support all compression codecs -- Key: AVRO-1243 URL: https://issues.apache.org/jira/browse/AVRO-1243 Project: Avro Issue Type: Improvement Components: java Affects Versions: 1.7.3 Reporter: Ted Malaska Priority: Minor I may be reading this wrong but at this time org.apache.avro.file.CodecFactory only supports null, deflate, and snappy compression codecs. I would like to change the fromString method to use Class.forName(codec).newInstance(); after the codec was not found in the REGISTERED map but before the AvroRuntimeException is thrown. Here are some of my supporting thoughts 1. This should not interduce much slowness because it will only be called initialize. 2. This will allow for support for GZip, BZip2, and LZO with out adding more dependances to the maven pom file. 3. This will allow for a future Jiri I would like to do that would allow AvroOutputFormat to be able to use the following configs: mapred.output.compress and mapred.output.compression.codec -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1035) Add the possibility to append to existing avro files
[
https://issues.apache.org/jira/browse/AVRO-1035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573711#comment-13573711
]
Michael Malak commented on AVRO-1035:
-
ha...@cloudera.com has provided example code on how to accomplish HDFS Avro
append at https://gist.github.com/QwertyManiac/4724582
Add the possibility to append to existing avro files
--
Key: AVRO-1035
URL: https://issues.apache.org/jira/browse/AVRO-1035
Project: Avro
Issue Type: New Feature
Reporter: Vyacheslav Zholudev
Currently it is not possible to append to avro files that were written and
closed.
Here is a Scott Carey's reply on the mailing list:
{quote}
It is not possible without modifying DataFileWriter. Please open a JIRA
ticket.
It could not simply append to an OutputStream, since it must either:
* Seek to the start to validate the schemas match and find the sync
marker, or
* Trust that the schemas match and find the sync marker from the last block
DataFileWriter cannot refer to Hadoop classes such as FileSystem, but we
could add something to the mapred module that takes a Path and FileSystem
and returns
something that implemements an interface that DataFileWriter can append
to. This would be something that is both a
http://avro.apache.org/docs/1.6.2/api/java/org/apache/avro/file/SeekableInp
ut.html
and an OutputStream, or has both an InputStream from the start of the
existing file and an OutputStream at the end.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1243) Support all compression codecs
[ https://issues.apache.org/jira/browse/AVRO-1243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573718#comment-13573718 ] Ted Malaska commented on AVRO-1243: --- Wow this is great. Yes I'm thinking along the same lines. I'm so glad that you guys added your comments. So the one remaining piece that I need to figure out is how does Avro do it's uncompressing and how will this be impacted. Support all compression codecs -- Key: AVRO-1243 URL: https://issues.apache.org/jira/browse/AVRO-1243 Project: Avro Issue Type: Improvement Components: java Affects Versions: 1.7.3 Reporter: Ted Malaska Priority: Minor I may be reading this wrong but at this time org.apache.avro.file.CodecFactory only supports null, deflate, and snappy compression codecs. I would like to change the fromString method to use Class.forName(codec).newInstance(); after the codec was not found in the REGISTERED map but before the AvroRuntimeException is thrown. Here are some of my supporting thoughts 1. This should not interduce much slowness because it will only be called initialize. 2. This will allow for support for GZip, BZip2, and LZO with out adding more dependances to the maven pom file. 3. This will allow for a future Jiri I would like to do that would allow AvroOutputFormat to be able to use the following configs: mapred.output.compress and mapred.output.compression.codec -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1243) Support all compression codecs
[ https://issues.apache.org/jira/browse/AVRO-1243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573746#comment-13573746 ] Ted Malaska commented on AVRO-1243: --- OK I think I have it. The reader gets the Codec through DataFileStream.resolveCodec and that has access to all the meta data. I think I have everything I need to implement a patch. My first attempt will use the following parameters to read and write with a Hadoop codec when not running map/reduce avro.codec=reflectionCodec avro.reflection.codec.class=org.apache.avro.hadoop.file.HadoopCodec mapred.output.compression=true mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec When running map/reduce and going through the AvroOutputFormat only the following parameters will be needed: mapred.output.compression=true mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec So with this we get two things: 1. The normal Avro reader and writer can read files produced by AvroOutputFormat 2. AvroOutputFormat will behave the same of RCFiles and Sequence Files when it comes to compression Let me know what you think. I have to finish some work first then I will try to get this done through the weekend. Support all compression codecs -- Key: AVRO-1243 URL: https://issues.apache.org/jira/browse/AVRO-1243 Project: Avro Issue Type: Improvement Components: java Affects Versions: 1.7.3 Reporter: Ted Malaska Priority: Minor I may be reading this wrong but at this time org.apache.avro.file.CodecFactory only supports null, deflate, and snappy compression codecs. I would like to change the fromString method to use Class.forName(codec).newInstance(); after the codec was not found in the REGISTERED map but before the AvroRuntimeException is thrown. Here are some of my supporting thoughts 1. This should not interduce much slowness because it will only be called initialize. 2. This will allow for support for GZip, BZip2, and LZO with out adding more dependances to the maven pom file. 3. This will allow for a future Jiri I would like to do that would allow AvroOutputFormat to be able to use the following configs: mapred.output.compress and mapred.output.compression.codec -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1243) Avro support for all compression codecs
[ https://issues.apache.org/jira/browse/AVRO-1243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Malaska updated AVRO-1243: -- Summary: Avro support for all compression codecs (was: Support all compression codecs) Avro support for all compression codecs --- Key: AVRO-1243 URL: https://issues.apache.org/jira/browse/AVRO-1243 Project: Avro Issue Type: Improvement Components: java Affects Versions: 1.7.3 Reporter: Ted Malaska Priority: Minor I may be reading this wrong but at this time org.apache.avro.file.CodecFactory only supports null, deflate, and snappy compression codecs. I would like to change the fromString method to use Class.forName(codec).newInstance(); after the codec was not found in the REGISTERED map but before the AvroRuntimeException is thrown. Here are some of my supporting thoughts 1. This should not interduce much slowness because it will only be called initialize. 2. This will allow for support for GZip, BZip2, and LZO with out adding more dependances to the maven pom file. 3. This will allow for a future Jiri I would like to do that would allow AvroOutputFormat to be able to use the following configs: mapred.output.compress and mapred.output.compression.codec -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Build failed in Jenkins: AvroJava #344
See https://builds.apache.org/job/AvroJava/344/changes Changes: [cutting] AVRO-1242. Java: Fix AvroTrevniOutputFormat to correctly get file metadata from JobConf. Contributed by Ted Malaska. -- [...truncated 242 lines...] [INFO] [INFO] [INFO] Building Apache Avro Maven Plugin 1.7.4-SNAPSHOT [INFO] [INFO] [INFO] --- maven-plugin-plugin:2.9:helpmojo (generated-helpmojo) @ avro-maven-plugin --- [INFO] Using 'UTF-8' encoding to read mojo metadata. [INFO] Applying mojo extractor for language: java [INFO] Mojo extractor for language: java found 3 mojo descriptors. [INFO] Applying mojo extractor for language: bsh [INFO] Mojo extractor for language: bsh found 0 mojo descriptors. [INFO] [INFO] --- maven-plugin-plugin:2.9:descriptor (default-descriptor) @ avro-maven-plugin --- [INFO] Using 'UTF-8' encoding to read mojo metadata. [INFO] Applying mojo extractor for language: java [INFO] Mojo extractor for language: java found 4 mojo descriptors. [INFO] Applying mojo extractor for language: bsh [INFO] Mojo extractor for language: bsh found 0 mojo descriptors. [INFO] [INFO] --- maven-remote-resources-plugin:1.2.1:process (default) @ avro-maven-plugin --- [INFO] [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ avro-maven-plugin --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory https://builds.apache.org/job/AvroJava/ws/trunk/lang/java/maven-plugin/src/main/resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ avro-maven-plugin --- [INFO] Compiling 5 source files to https://builds.apache.org/job/AvroJava/ws/trunk/lang/java/maven-plugin/target/classes [INFO] [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ avro-maven-plugin --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 3 resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ avro-maven-plugin --- [INFO] Compiling 4 source files to https://builds.apache.org/job/AvroJava/ws/trunk/lang/java/maven-plugin/target/test-classes [INFO] [INFO] --- maven-surefire-plugin:2.12:test (default-test) @ avro-maven-plugin --- [INFO] Surefire report directory: https://builds.apache.org/job/AvroJava/ws/trunk/lang/java/maven-plugin/target/surefire-reports --- T E S T S --- --- T E S T S --- Running org.apache.avro.mojo.TestProtocolMojo Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.375 sec Running org.apache.avro.mojo.TestIDLProtocolMojo Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.197 sec Running org.apache.avro.mojo.TestSchemaMojo Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.338 sec Results : Tests run: 3, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] --- maven-checkstyle-plugin:2.8:check (checkstyle-check) @ avro-maven-plugin --- [INFO] Starting audit... Audit done. [INFO] [INFO] [INFO] Building Apache Avro IPC 1.7.4-SNAPSHOT [INFO] [INFO] [INFO] --- avro-maven-plugin:1.7.4-SNAPSHOT:schema (schemas) @ avro-ipc --- [INFO] [INFO] --- avro-maven-plugin:1.7.4-SNAPSHOT:protocol (schemas) @ avro-ipc --- [INFO] [INFO] --- avro-maven-plugin:1.7.4-SNAPSHOT:idl-protocol (schemas) @ avro-ipc --- [INFO] [INFO] --- maven-remote-resources-plugin:1.2.1:process (default) @ avro-ipc --- [INFO] [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ avro-ipc --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 6 resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ avro-ipc --- [INFO] Compiling 57 source files to https://builds.apache.org/job/AvroJava/ws/trunk/lang/java/ipc/target/classes [INFO] [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ avro-ipc --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 1 resource [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ avro-ipc --- [INFO] Compiling 81 source files to
[jira] [Commented] (AVRO-1242) AvroTrevniOutputFormat Metadata filter subString bug
[ https://issues.apache.org/jira/browse/AVRO-1242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573832#comment-13573832 ] Hudson commented on AVRO-1242: -- Integrated in AvroJava #344 (See [https://builds.apache.org/job/AvroJava/344/]) AVRO-1242. Java: Fix AvroTrevniOutputFormat to correctly get file metadata from JobConf. Contributed by Ted Malaska. (Revision 1443604) Result = FAILURE cutting : Files : * /avro/trunk/CHANGES.txt * /avro/trunk/lang/java/trevni/avro/src/main/java/org/apache/trevni/avro/AvroTrevniOutputFormat.java * /avro/trunk/lang/java/trevni/avro/src/test/java/org/apache/trevni/avro/TestMetadataFiltering.java AvroTrevniOutputFormat Metadata filter subString bug Key: AVRO-1242 URL: https://issues.apache.org/jira/browse/AVRO-1242 Project: Avro Issue Type: Bug Affects Versions: 1.7.4 Reporter: Ted Malaska Assignee: Ted Malaska Fix For: 1.7.4 Attachments: AVRO-1234.patch.1, AVRO-1242.patch In AvroTrevniOutputFormat there seems to be a cut and post error in the metadata filtering. The result is not trevni.meta. metadatas will successfully get to the writer. if (e.getKey().startsWith(META_PREFIX)) meta.put(e.getKey().substring(AvroJob.TEXT_PREFIX.length()), e.getValue().getBytes(MetaData.UTF8)); META_PREFIX = trevni.meta. TEXT_PREFIX = avro.meta.text. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1198) Malformed Avro data may cause confusing ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/AVRO-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573843#comment-13573843 ] Doug Cutting commented on AVRO-1198: Unless there are objections I'll commit this soon. Malformed Avro data may cause confusing ArrayIndexOutOfBoundsException -- Key: AVRO-1198 URL: https://issues.apache.org/jira/browse/AVRO-1198 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.7.2 Reporter: Mike Percy Assignee: Mike Percy Fix For: 1.7.4 Attachments: AVRO-1198-1.patch, AVRO-1198.patch I am currently debugging an issue where I am getting an ArrayIndexOutOfBoundsException from the decoder while reading some Avro data. Turns out that the integer indicating number of bytes to read next is negative. It would be better if a more helpful error message were provided. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1220) Dead lock
[ https://issues.apache.org/jira/browse/AVRO-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573848#comment-13573848 ] Doug Cutting commented on AVRO-1220: Stéphane, if you (or someone else) can confirm this fix then we can include it in Avro 1.7.4. Dead lock -- Key: AVRO-1220 URL: https://issues.apache.org/jira/browse/AVRO-1220 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.7.3 Environment: OSX, JDK6 Reporter: Stéphane Landelle Fix For: 1.7.4 Attachments: AVRO-1220.txt, AVRO-1220.txt I experience a dead lock when running multiple DataFileReader in concurrent threads. See test case here: https://github.com/slandelle/avro-test AvroBinaryEncodingTest randomly stalls about 50% of the time. A thread dump would show a dead lock in org.apache.avro.io.parsing.Symbol class initialization. IHMO, the problem is that the Symbol class has static final members that are instances of Symbol subclasses. I built a custom version of avro where all the constants (NULL, BOOLEAN, INT, etc) have been extracted into a dedicated class outside of Symbol hierarchy and the test case now runs fine. Cheers, Stéphane Landelle -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Jenkins build is back to normal : AvroJava #345
See https://builds.apache.org/job/AvroJava/345/
