[jira] [Commented] (AVRO-2806) Upgrade Netty to 4.x
[
https://issues.apache.org/jira/browse/AVRO-2806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17089468#comment-17089468
]
Gabor Szadovszky commented on AVRO-2806:
[~zeshuai007], as you've said there are many major changes between 3.x and 4.x.
I don't really remember what were the exact issues but I could not refactor our
code to use the 4.x. Please note, that I have no experience with Netty. My only
motivation are the CVEs reported for 3.x.
Seems that {{netty-codec-http2}} was introduced separately from {{netty}} in
AVRO-2279. I don't know why don't we use the same version or if it matters at
all.
> Upgrade Netty to 4.x
>
>
> Key: AVRO-2806
> URL: https://issues.apache.org/jira/browse/AVRO-2806
> Project: Apache Avro
> Issue Type: Task
> Components: java
>Affects Versions: 1.9.2
>Reporter: Gabor Szadovszky
>Priority: Major
>
> ipc-netty still uses the 3.x line which is ended almost 4 years ago. We have
> a couple of CVEs which won't be fixed in 3.x so it is important to migrate to
> the 4.x.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2806) Upgrade Netty to 4.x
[ https://issues.apache.org/jira/browse/AVRO-2806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17087701#comment-17087701 ] Gabor Szadovszky commented on AVRO-2806: [~zeshuai007], thanks for pointing to this version. I know that it requires significant efforts as it is a major version upgrade. A while ago I've tried to upgrade to a 4.x version but failed. I don't have any experience on Netty. What I know is 3.x is not maintained any more while has some CVEs reported. I hoped that someone might find my jira and fix it who have more experience. :) > Upgrade Netty to 4.x > > > Key: AVRO-2806 > URL: https://issues.apache.org/jira/browse/AVRO-2806 > Project: Apache Avro > Issue Type: Task > Components: java >Affects Versions: 1.9.2 >Reporter: Gabor Szadovszky >Priority: Major > > ipc-netty still uses the 3.x line which is ended almost 4 years ago. We have > a couple of CVEs which won't be fixed in 3.x so it is important to migrate to > the 4.x. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AVRO-2806) Upgrade Netty to 4.x
Gabor Szadovszky created AVRO-2806: -- Summary: Upgrade Netty to 4.x Key: AVRO-2806 URL: https://issues.apache.org/jira/browse/AVRO-2806 Project: Apache Avro Issue Type: Task Components: java Affects Versions: 1.9.2 Reporter: Gabor Szadovszky ipc-netty still uses the 3.x line which is ended almost 4 years ago. We have a couple of CVEs which won't be fixed in 3.x so it is important to migrate to the 4.x. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AVRO-2109) Reset buffers in case of IOException
[
https://issues.apache.org/jira/browse/AVRO-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated AVRO-2109:
---
Resolution: Fixed
Fix Version/s: 1.9.0
Status: Resolved (was: Patch Available)
> Reset buffers in case of IOException
>
>
> Key: AVRO-2109
> URL: https://issues.apache.org/jira/browse/AVRO-2109
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.8.2
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
> Fix For: 1.7.8, 1.9.0, 1.8.3
>
>
> In case of an {{IOException}} is thrown out from
> {{DataFileWriter.writeBlock}} the {{buffer}} and {{blockCount}} are not reset
> therefore duplicated data is written out when {{close}}/{{flush}}.
> This is actually a conceptual question whether we should reset the buffer or
> not in case of an exception. In case of an exception occurs during writing
> the file we shall expect that the file will be corrupt. So, the possible
> duplication of data shall not matter.
> In the other hand if the file is already corrupt why would we try to write
> anything again at file close?
> This issue comes from a Flume issue where the HDFS wait thread is interrupted
> because of a timeout during writing an Avro file. The actual block is
> properly written already but because of the {{IOException}} caused by the
> thread interrupt we invoke {{close()}} on the writer which writes the block
> again with some other stuff (maybe duplicated sync marker) that makes the
> file corrupt.
> [~busbey], [~nkollar], [~zi], any thoughts?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (AVRO-1605) Remove Jackson classes from public API
[ https://issues.apache.org/jira/browse/AVRO-1605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16281727#comment-16281727 ] Gabor Szadovszky edited comment on AVRO-1605 at 12/7/17 12:27 PM: -- This PR has been opened for a long time. [~rdblue] do you think it can be merged ? was (Author: dgesino): This PR has been opened for a long time. @Ryan Blue do you think it can be merged ? > Remove Jackson classes from public API > -- > > Key: AVRO-1605 > URL: https://issues.apache.org/jira/browse/AVRO-1605 > Project: Avro > Issue Type: Sub-task > Components: java >Affects Versions: 1.7.8 >Reporter: Tom White >Assignee: Gabor Szadovszky > Fix For: 1.9.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AVRO-2109) Reset buffers in case of IOException
[
https://issues.apache.org/jira/browse/AVRO-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated AVRO-2109:
---
Fix Version/s: 1.8.3
1.7.8
Status: Patch Available (was: Open)
> Reset buffers in case of IOException
>
>
> Key: AVRO-2109
> URL: https://issues.apache.org/jira/browse/AVRO-2109
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.8.2
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
> Fix For: 1.7.8, 1.8.3
>
>
> In case of an {{IOException}} is thrown out from
> {{DataFileWriter.writeBlock}} the {{buffer}} and {{blockCount}} are not reset
> therefore duplicated data is written out when {{close}}/{{flush}}.
> This is actually a conceptual question whether we should reset the buffer or
> not in case of an exception. In case of an exception occurs during writing
> the file we shall expect that the file will be corrupt. So, the possible
> duplication of data shall not matter.
> In the other hand if the file is already corrupt why would we try to write
> anything again at file close?
> This issue comes from a Flume issue where the HDFS wait thread is interrupted
> because of a timeout during writing an Avro file. The actual block is
> properly written already but because of the {{IOException}} caused by the
> thread interrupt we invoke {{close()}} on the writer which writes the block
> again with some other stuff (maybe duplicated sync marker) that makes the
> file corrupt.
> [~busbey], [~nkollar], [~zi], any thoughts?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (AVRO-2109) Reset buffers in case of IOException
[
https://issues.apache.org/jira/browse/AVRO-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky reassigned AVRO-2109:
--
Assignee: Gabor Szadovszky
> Reset buffers in case of IOException
>
>
> Key: AVRO-2109
> URL: https://issues.apache.org/jira/browse/AVRO-2109
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.8.2
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
>
> In case of an {{IOException}} is thrown out from
> {{DataFileWriter.writeBlock}} the {{buffer}} and {{blockCount}} are not reset
> therefore duplicated data is written out when {{close}}/{{flush}}.
> This is actually a conceptual question whether we should reset the buffer or
> not in case of an exception. In case of an exception occurs during writing
> the file we shall expect that the file will be corrupt. So, the possible
> duplication of data shall not matter.
> In the other hand if the file is already corrupt why would we try to write
> anything again at file close?
> This issue comes from a Flume issue where the HDFS wait thread is interrupted
> because of a timeout during writing an Avro file. The actual block is
> properly written already but because of the {{IOException}} caused by the
> thread interrupt we invoke {{close()}} on the writer which writes the block
> again with some other stuff (maybe duplicated sync marker) that makes the
> file corrupt.
> [~busbey], [~nkollar], [~zi], any thoughts?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (AVRO-2109) Reset buffers in case of IOException
[
https://issues.apache.org/jira/browse/AVRO-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky reassigned AVRO-2109:
--
Assignee: (was: Gabor Szadovszky)
> Reset buffers in case of IOException
>
>
> Key: AVRO-2109
> URL: https://issues.apache.org/jira/browse/AVRO-2109
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.8.2
>Reporter: Gabor Szadovszky
>
> In case of an {{IOException}} is thrown out from
> {{DataFileWriter.writeBlock}} the {{buffer}} and {{blockCount}} are not reset
> therefore duplicated data is written out when {{close}}/{{flush}}.
> This is actually a conceptual question whether we should reset the buffer or
> not in case of an exception. In case of an exception occurs during writing
> the file we shall expect that the file will be corrupt. So, the possible
> duplication of data shall not matter.
> In the other hand if the file is already corrupt why would we try to write
> anything again at file close?
> This issue comes from a Flume issue where the HDFS wait thread is interrupted
> because of a timeout during writing an Avro file. The actual block is
> properly written already but because of the {{IOException}} caused by the
> thread interrupt we invoke {{close()}} on the writer which writes the block
> again with some other stuff (maybe duplicated sync marker) that makes the
> file corrupt.
> [~busbey], [~nkollar], [~zi], any thoughts?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (AVRO-2109) Reset buffers in case of IOException
Gabor Szadovszky created AVRO-2109:
--
Summary: Reset buffers in case of IOException
Key: AVRO-2109
URL: https://issues.apache.org/jira/browse/AVRO-2109
Project: Avro
Issue Type: Improvement
Components: java
Affects Versions: 1.8.2
Reporter: Gabor Szadovszky
Assignee: Gabor Szadovszky
In case of an {{IOException}} is thrown out from {{DataFileWriter.writeBlock}}
the {{buffer}} and {[blockCount}} are not reset therefore duplicated data is
written out when {{close}}/{{flush}}.
This is actually a conceptual question whether we should reset the buffer or
not in case of an exception. In case of an exception occurs during writing the
file we shall expect that the file will be corrupt. So, the possible
duplication of data shall not matter.
In the other hand if the file is already corrupt why would we try to write
anything again at file close?
This issue comes from a Flume issue where the HDFS wait thread is interrupted
because of a timeout during writing an Avro file. The actual block is properly
written already but because of the {{IOException}} caused by the thread
interrupt we invoke {{close()}} on the writer which writes the block again with
some other stuff (maybe duplicated sync marker) that makes the file corrupt.
[~busbey], [~nkollar], [~zi], any thoughts?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AVRO-2055) Remove Magic Value From org.apache.avro.hadoop.io.AvroSequenceFile
[ https://issues.apache.org/jira/browse/AVRO-2055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky updated AVRO-2055: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.9.0 Status: Resolved (was: Patch Available) > Remove Magic Value From org.apache.avro.hadoop.io.AvroSequenceFile > -- > > Key: AVRO-2055 > URL: https://issues.apache.org/jira/browse/AVRO-2055 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.7.7, 1.8.2 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Fix For: 1.9.0 > > Attachments: AVRO-2055.1.patch > > > Remove magic string _io.file.buffer.size_ and _DEFAULT_BUFFER_SIZE_BYTES_ and > instead rely on the Hadoop libraries to provide this information. Will help > to keep Avro in sync with changes in Hadoop. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AVRO-2053) Remove Reference To Deprecated Property mapred.output.compression.type
[
https://issues.apache.org/jira/browse/AVRO-2053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated AVRO-2053:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 1.9.0
Status: Resolved (was: Patch Available)
> Remove Reference To Deprecated Property mapred.output.compression.type
> --
>
> Key: AVRO-2053
> URL: https://issues.apache.org/jira/browse/AVRO-2053
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.7.7, 1.8.2
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Fix For: 1.9.0
>
> Attachments: AVRO-2053.1.patch
>
>
> Avro utilizes
> [deprecated|https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/DeprecatedProperties.html]
> property _mapred.output.compression.type_. Update code to use the MRv2
> property and don't override default behaviors/settings. Use the appropriate
> facilities from {{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat}}
> and {{org.apache.hadoop.io.SequenceFile}}.
> {code:title=org.apache.avro.mapreduce.AvroSequenceFileOutputFormat}
> /** Configuration key for storing the type of compression for the target
> sequence file. */
> private static final String CONF_COMPRESSION_TYPE =
> "mapred.output.compression.type";
> /** The default compression type for the target sequence file. */
> private static final CompressionType DEFAULT_COMPRESSION_TYPE =
> CompressionType.RECORD;
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AVRO-2059) Remove support of Hadoop 1
[ https://issues.apache.org/jira/browse/AVRO-2059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky updated AVRO-2059: --- Resolution: Fixed Hadoop Flags: Incompatible change,Reviewed (was: Incompatible change) Status: Resolved (was: Patch Available) > Remove support of Hadoop 1 > -- > > Key: AVRO-2059 > URL: https://issues.apache.org/jira/browse/AVRO-2059 > Project: Avro > Issue Type: New Feature > Components: java >Affects Versions: 1.9.0 >Reporter: Gabor Szadovszky >Assignee: Gabor Szadovszky > Fix For: 1.9.0 > > > Remove support of Hadoop 1 in the next major release 1.9.0. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AVRO-2072) ResolvingGrammarGenerator doesn't implement schema resolution correctly for unions
[ https://issues.apache.org/jira/browse/AVRO-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky updated AVRO-2072: --- Hadoop Flags: Incompatible change Release Note: However, schema resolution is now working according to spec it is a backward incompatible change in the behaviour that int is promotable to float (in addition to long and double) and log to float (in addition to double) when used in unions. > ResolvingGrammarGenerator doesn't implement schema resolution correctly for > unions > -- > > Key: AVRO-2072 > URL: https://issues.apache.org/jira/browse/AVRO-2072 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.7.7, 1.8.1 >Reporter: Nandor Kollar >Assignee: Nandor Kollar > Attachments: 0001-AVRO-1931-Additional-test-cases.patch, > AVRO-2072_2.patch, AVRO-2072_3.patch, AVRO-2072.patch > > > According to > [specification|https://avro.apache.org/docs/current/spec.html#Schema+Resolution], > int and long is promotable to float, but when using SchemaValidator, a union > with a single int or long branch is not readable by an union with a float > branch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2072) ResolvingGrammarGenerator doesn't implement schema resolution correctly for unions
[ https://issues.apache.org/jira/browse/AVRO-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161149#comment-16161149 ] Gabor Szadovszky commented on AVRO-2072: +1 Will push it soon > ResolvingGrammarGenerator doesn't implement schema resolution correctly for > unions > -- > > Key: AVRO-2072 > URL: https://issues.apache.org/jira/browse/AVRO-2072 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.7.7, 1.8.1 >Reporter: Nandor Kollar >Assignee: Nandor Kollar > Attachments: 0001-AVRO-1931-Additional-test-cases.patch, > AVRO-2072_2.patch, AVRO-2072_3.patch, AVRO-2072.patch > > > According to > [specification|https://avro.apache.org/docs/current/spec.html#Schema+Resolution], > int and long is promotable to float, but when using SchemaValidator, a union > with a single int or long branch is not readable by an union with a float > branch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AVRO-1933) SchemaCompatibility class could be more user-friendly about incompatibilities
[ https://issues.apache.org/jira/browse/AVRO-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky updated AVRO-1933: --- Resolution: Fixed Status: Resolved (was: Patch Available) > SchemaCompatibility class could be more user-friendly about incompatibilities > - > > Key: AVRO-1933 > URL: https://issues.apache.org/jira/browse/AVRO-1933 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.7.7, 1.8.1 > Environment: Any Java env >Reporter: Anders Sundelin >Assignee: Anders Sundelin >Priority: Minor > Fix For: 1.9.0 > > Attachments: AVRO-1933-compatible-with-AVRO-1931.patch, > AVRO-1933.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > Today, the class SchemaCompatibility reports incompatibilities with quite > little detail. The whole reader and the whole writer schema is listed, and no > particular detail about what was incompatible. > The attached patch fixes this, introducing a new enum > (SchemaIncompatibilityType), and more specific sub-schemas that were > incompatible. > The old, overall picture, is still there - the new compatibility state is > encapsulated in the SchemaCompatibilityDetails class. > Lots of test cases have been added, and there has been refactoring done in > the TestSchemaCompatibility and other test classes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AVRO-1933) SchemaCompatibility class could be more user-friendly about incompatibilities
[ https://issues.apache.org/jira/browse/AVRO-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky updated AVRO-1933: --- Hadoop Flags: Incompatible change Release Note: SchemaCompatibility returns the more detailed object SchemaCompatibilityResult instead of the simple enum SchemaCompatibilityType. > SchemaCompatibility class could be more user-friendly about incompatibilities > - > > Key: AVRO-1933 > URL: https://issues.apache.org/jira/browse/AVRO-1933 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.7.7, 1.8.1 > Environment: Any Java env >Reporter: Anders Sundelin >Assignee: Anders Sundelin >Priority: Minor > Fix For: 1.9.0 > > Attachments: AVRO-1933-compatible-with-AVRO-1931.patch, > AVRO-1933.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > Today, the class SchemaCompatibility reports incompatibilities with quite > little detail. The whole reader and the whole writer schema is listed, and no > particular detail about what was incompatible. > The attached patch fixes this, introducing a new enum > (SchemaIncompatibilityType), and more specific sub-schemas that were > incompatible. > The old, overall picture, is still there - the new compatibility state is > encapsulated in the SchemaCompatibilityDetails class. > Lots of test cases have been added, and there has been refactoring done in > the TestSchemaCompatibility and other test classes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AVRO-2073) Update example projects to be compatible with the actual release
[
https://issues.apache.org/jira/browse/AVRO-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated AVRO-2073:
---
Labels: newbie (was: )
> Update example projects to be compatible with the actual release
>
>
> Key: AVRO-2073
> URL: https://issues.apache.org/jira/browse/AVRO-2073
> Project: Avro
> Issue Type: Bug
> Components: doc, java
>Affects Versions: 1.9.0
>Reporter: Gabor Szadovszky
>Priority: Minor
> Labels: newbie
>
> Update the example code and the related dependencies at {{doc/examples}} so
> that they are working properly with the actual Avro release. (E.g. currently
> outdated Avro and Hadoop versions are referenced in the dependencies.)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (AVRO-2073) Update example projects to be compatible with the actual release
Gabor Szadovszky created AVRO-2073:
--
Summary: Update example projects to be compatible with the actual
release
Key: AVRO-2073
URL: https://issues.apache.org/jira/browse/AVRO-2073
Project: Avro
Issue Type: Bug
Components: doc, java
Affects Versions: 1.9.0
Reporter: Gabor Szadovszky
Priority: Minor
Update the example code and the related dependencies at {{doc/examples}} so
that they are working properly with the actual Avro release. (E.g. currently
outdated Avro and Hadoop versions are referenced in the dependencies.)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AVRO-2061) Improve Invalid File Format Error Message
[
https://issues.apache.org/jira/browse/AVRO-2061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated AVRO-2061:
---
Resolution: Fixed
Fix Version/s: 1.9.0
Status: Resolved (was: Patch Available)
> Improve Invalid File Format Error Message
> -
>
> Key: AVRO-2061
> URL: https://issues.apache.org/jira/browse/AVRO-2061
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.9.0, 1.8.2
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Fix For: 1.9.0
>
> Attachments: AVRO-2061.1.patch, AVRO-2061.2.patch
>
>
> {code:title=org.apache.avro.file.DataFileStream}
> try {
> vin.readFixed(magic); // read magic
> } catch (IOException e) {
> throw new IOException("Not a data file.", e);
> }
> if (!Arrays.equals(DataFileConstants.MAGIC, magic))
> throw new IOException("Not a data file.");
> {code}
> Please consider improving the error message here. I just saw a MapReduce job
> fail with an IOException with the message "Not a data file." There was
> definitely data files in the input directory, however, they were not Avro
> files. It would have been much more helpful if it told me that.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AVRO-2061) Improve Invalid File Format Error Message
[
https://issues.apache.org/jira/browse/AVRO-2061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated AVRO-2061:
---
Hadoop Flags: Incompatible change
Release Note: Changed the message of the IOException thrown in case of a
non avro file is passed for parsing.
JIRA set to incompatible and added some release note (feel free to update).
+1
> Improve Invalid File Format Error Message
> -
>
> Key: AVRO-2061
> URL: https://issues.apache.org/jira/browse/AVRO-2061
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.9.0, 1.8.2
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: AVRO-2061.1.patch, AVRO-2061.2.patch
>
>
> {code:title=org.apache.avro.file.DataFileStream}
> try {
> vin.readFixed(magic); // read magic
> } catch (IOException e) {
> throw new IOException("Not a data file.", e);
> }
> if (!Arrays.equals(DataFileConstants.MAGIC, magic))
> throw new IOException("Not a data file.");
> {code}
> Please consider improving the error message here. I just saw a MapReduce job
> fail with an IOException with the message "Not a data file." There was
> definitely data files in the input directory, however, they were not Avro
> files. It would have been much more helpful if it told me that.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-2059) Remove support of Hadoop 1
[ https://issues.apache.org/jira/browse/AVRO-2059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16112331#comment-16112331 ] Gabor Szadovszky commented on AVRO-2059: Good point, [~cutting]. Thanks. I'm afraid, I don't have the required privileges to edit the wiki. After committing the changes I would add a note that from 1.9.0 we do not have to different versions, only one for hadoop2 and there is no related classifier anymore. > Remove support of Hadoop 1 > -- > > Key: AVRO-2059 > URL: https://issues.apache.org/jira/browse/AVRO-2059 > Project: Avro > Issue Type: New Feature > Components: java >Affects Versions: 1.9.0 >Reporter: Gabor Szadovszky >Assignee: Gabor Szadovszky > Fix For: 1.9.0 > > > Remove support of Hadoop 1 in the next major release 1.9.0. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AVRO-2059) Remove support of Hadoop 1
[ https://issues.apache.org/jira/browse/AVRO-2059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky updated AVRO-2059: --- Fix Version/s: 1.9.0 Status: Patch Available (was: Open) > Remove support of Hadoop 1 > -- > > Key: AVRO-2059 > URL: https://issues.apache.org/jira/browse/AVRO-2059 > Project: Avro > Issue Type: New Feature >Affects Versions: 1.9.0 >Reporter: Gabor Szadovszky >Assignee: Gabor Szadovszky > Fix For: 1.9.0 > > > Remove support of Hadoop 1 in the next major release 1.9.0. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AVRO-2059) Remove support of Hadoop 1
[ https://issues.apache.org/jira/browse/AVRO-2059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky updated AVRO-2059: --- Component/s: java > Remove support of Hadoop 1 > -- > > Key: AVRO-2059 > URL: https://issues.apache.org/jira/browse/AVRO-2059 > Project: Avro > Issue Type: New Feature > Components: java >Affects Versions: 1.9.0 >Reporter: Gabor Szadovszky >Assignee: Gabor Szadovszky > Fix For: 1.9.0 > > > Remove support of Hadoop 1 in the next major release 1.9.0. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (AVRO-2059) Remove support of Hadoop 1
[ https://issues.apache.org/jira/browse/AVRO-2059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky reassigned AVRO-2059: -- Assignee: Gabor Szadovszky > Remove support of Hadoop 1 > -- > > Key: AVRO-2059 > URL: https://issues.apache.org/jira/browse/AVRO-2059 > Project: Avro > Issue Type: New Feature >Affects Versions: 1.9.0 >Reporter: Gabor Szadovszky >Assignee: Gabor Szadovszky > > Remove support of Hadoop 1 in the next major release 1.9.0. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (AVRO-2058) ReflectData#isNonStringMap returns true for Utf8 keys
[
https://issues.apache.org/jira/browse/AVRO-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky reassigned AVRO-2058:
--
Assignee: Sam Schlegel
> ReflectData#isNonStringMap returns true for Utf8 keys
> -
>
> Key: AVRO-2058
> URL: https://issues.apache.org/jira/browse/AVRO-2058
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.2
>Reporter: Sam Schlegel
>Assignee: Sam Schlegel
>Priority: Critical
> Attachments: AVRO-2058.patch
>
>
> Since {{Utf8}} does not have an {{Stringable}} notation, and is not in
> {{SpecificData#stringableClasses}}, {{ReflectData#isNonStringMap}} returns
> true. This also causes {{ReflectData#isArray}} to return true for maps with
> Utf8 keys, and thus {{GenericData#resolveUnion}} fails as well. This
> ultimately causes {{ReflectData#write}} to fail for schemas that contain a
> union that contains a map, where the data uses Utf8 for strings.
> This following test case reproduces the issue:
> {code:java}
> @Test public void testUnionWithMapWithUtf8Keys() {
> Schema s = new Schema.Parser().parse
> ("[\"null\", {\"type\":\"map\",\"values\":\"float\"}]");
> GenericData data = ReflectData.get();
> HashMap map = new HashMap();
> map.put(new Utf8("foo"), 1.0f);
> assertEquals(1, data.resolveUnion(s, map));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (AVRO-2059) Remove support of Hadoop 1
Gabor Szadovszky created AVRO-2059: -- Summary: Remove support of Hadoop 1 Key: AVRO-2059 URL: https://issues.apache.org/jira/browse/AVRO-2059 Project: Avro Issue Type: New Feature Affects Versions: 1.9.0 Reporter: Gabor Szadovszky Remove support of Hadoop 1 in the next major release 1.9.0. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AVRO-2054) Use StringBuilder instead of StringBuffer
[
https://issues.apache.org/jira/browse/AVRO-2054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated AVRO-2054:
---
Resolution: Fixed
Fix Version/s: 1.9.0
Status: Resolved (was: Patch Available)
> Use StringBuilder instead of StringBuffer
> -
>
> Key: AVRO-2054
> URL: https://issues.apache.org/jira/browse/AVRO-2054
> Project: Avro
> Issue Type: Improvement
>Affects Versions: 1.7.7, 1.8.2
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Fix For: 1.9.0
>
> Attachments: AVRO-2054.1.patch, AVRO-2054.2.patch
>
>
> Use the un-synchronized StringBuilder instead of StringBuffer. Use _char_
> values instead of Strings.
> {code:title=org.apache.trevni.MetaData}
> @Override public String toString() {
> StringBuffer buffer = new StringBuffer();
> buffer.append("{ ");
> for (Map.Entry e : entrySet()) {
> buffer.append(e.getKey());
> buffer.append("=");
> try {
> buffer.append(new String(e.getValue(), "ISO-8859-1"));
> } catch (java.io.UnsupportedEncodingException error) {
> throw new TrevniRuntimeException(error);
> }
> buffer.append(" ");
> }
> buffer.append("}");
> return buffer.toString();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AVRO-2056) DirectBinaryEncoder Creates Buffer For Each Call To writeDouble
[
https://issues.apache.org/jira/browse/AVRO-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated AVRO-2056:
---
Resolution: Fixed
Fix Version/s: 1.9.0
Status: Resolved (was: Patch Available)
> DirectBinaryEncoder Creates Buffer For Each Call To writeDouble
> ---
>
> Key: AVRO-2056
> URL: https://issues.apache.org/jira/browse/AVRO-2056
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.7.7, 1.8.2
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Fix For: 1.9.0
>
> Attachments: AVRO-2056.1.patch
>
>
> Each call to {{writeDouble}} creates a new buffer and promptly throws it away
> even though the class has a re-usable buffer and is used in other methods
> such as {{writeFloat}}. Remove this extra buffer.
> {code:title=org.apache.avro.io.DirectBinaryEncoder}
> // the buffer is used for writing floats, doubles, and large longs.
> private final byte[] buf = new byte[12];
> @Override
> public void writeFloat(float f) throws IOException {
> int len = BinaryData.encodeFloat(f, buf, 0);
> out.write(buf, 0, len);
> }
> @Override
> public void writeDouble(double d) throws IOException {
> byte[] buf = new byte[8];
> int len = BinaryData.encodeDouble(d, buf, 0);
> out.write(buf, 0, len);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AVRO-2048) Avro Binary Decoding - Gracefully Handle Long Strings
[
https://issues.apache.org/jira/browse/AVRO-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated AVRO-2048:
---
Resolution: Fixed
Fix Version/s: 1.9.0
Status: Resolved (was: Patch Available)
> Avro Binary Decoding - Gracefully Handle Long Strings
> -
>
> Key: AVRO-2048
> URL: https://issues.apache.org/jira/browse/AVRO-2048
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.7.7, 1.8.2
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Fix For: 1.9.0
>
> Attachments: AVRO-2048.1.patch, AVRO-2048.2.patch, AVRO-2048.3.patch
>
>
> According to the
> [specs|https://avro.apache.org/docs/1.8.2/spec.html#binary_encode_primitive]:
> bq. a string is encoded as a *long* followed by that many bytes of UTF-8
> encoded character data.
> However, that is currently not being adhered to:
> {code:title=org.apache.avro.io.BinaryDecoder}
> @Override
> public Utf8 readString(Utf8 old) throws IOException {
> int length = readInt();
> Utf8 result = (old != null ? old : new Utf8());
> result.setByteLength(length);
> if (0 != length) {
> doReadBytes(result.getBytes(), 0, length);
> }
> return result;
> }
> {code}
> The first thing the code does here is to load an *int* value, not a *long*.
> Because of the variable length nature of the size, this will mostly work.
> However, there may be edge-cases where the serializer is putting in large
> length values erroneously or nefariously. Let us gracefully detect such
> scenarios and more closely adhere to the spec.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AVRO-2050) Clear Array To Allow GC
[
https://issues.apache.org/jira/browse/AVRO-2050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated AVRO-2050:
---
Resolution: Fixed
Fix Version/s: 1.9.0
Status: Resolved (was: Patch Available)
> Clear Array To Allow GC
> ---
>
> Key: AVRO-2050
> URL: https://issues.apache.org/jira/browse/AVRO-2050
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.7.7, 1.8.2
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Fix For: 1.9.0
>
> Attachments: AVRO-2050.1.patch, AVRO-2050.2.patch, AVRO-2050.3.patch
>
>
> Java's {{ArrayList}} implementation clears all Objects from the internal
> buffer when the {{clear()}} method is called. This allows the Objects to be
> free for GC. We should do the same in Avro
> {{org.apache.avro.generic.GenericData}}
> [ArrayList
> Source|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8u40-b25/java/util/ArrayList.java#ArrayList.clear%28%29]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AVRO-2049) Remove Superfluous Configuration From AvroSerializer
[
https://issues.apache.org/jira/browse/AVRO-2049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated AVRO-2049:
---
Resolution: Fixed
Fix Version/s: 1.9.0
Status: Resolved (was: Patch Available)
> Remove Superfluous Configuration From AvroSerializer
>
>
> Key: AVRO-2049
> URL: https://issues.apache.org/jira/browse/AVRO-2049
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.7.7, 1.8.2
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Fix For: 1.9.0
>
> Attachments: AVRO-2049.1.patch, AVRO-2049.2.patch, AVRO-2049.3.patch
>
>
> In the class {{org.apache.avro.hadoop.io.AvroSerializer}}, we see that the
> Avro block size is configured with a hard-coded value and there is a request
> to benchmark different buffer sizes.
> {code:title=org.apache.avro.hadoop.io.AvroSerializer}
> /**
>* The block size for the Avro encoder.
>*
>* This number was copied from the AvroSerialization of
> org.apache.avro.mapred in Avro 1.5.1.
>*
>* TODO(gwu): Do some benchmarking with different numbers here to see if it
> is important.
>*/
> private static final int AVRO_ENCODER_BLOCK_SIZE_BYTES = 512;
> /** An factory for creating Avro datum encoders. */
> private static EncoderFactory mEncoderFactory
> = new
> EncoderFactory().configureBlockSize(AVRO_ENCODER_BLOCK_SIZE_BYTES);
> {code}
> However, there is no need to benchmark, this setting is superfluous and is
> ignored with the current implementation.
> {code:title=org.apache.avro.hadoop.io.AvroSerializer}
> @Override
> public void open(OutputStream outputStream) throws IOException {
> mOutputStream = outputStream;
> mAvroEncoder = mEncoderFactory.binaryEncoder(outputStream, mAvroEncoder);
> }
> {code}
> {{org.apache.avro.io.EncoderFactory.binaryEncoder}} ignores this setting.
> This setting is only relevant for calls to
> {{org.apache.avro.io.EncoderFactory.blockingBinaryEncoder}}
> which considers the configured "Block Size" for doing binary encoding of
> blocked Array types as laid out in the
> [specs|https://avro.apache.org/docs/1.8.2/spec.html#binary_encode_complex].
> It can simply be removed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-2055) Remove Magic Value From org.apache.avro.hadoop.io.AvroSequenceFile
[ https://issues.apache.org/jira/browse/AVRO-2055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102915#comment-16102915 ] Gabor Szadovszky commented on AVRO-2055: [~belugabehr], sorry I've just realised during doing a build with your patches that we are in the same situation than with AVRO-2053. It does not even compile as we are using hadoop-1 by default. So, let's wait if we can get rid of Hadoop 1 in 1.9. > Remove Magic Value From org.apache.avro.hadoop.io.AvroSequenceFile > -- > > Key: AVRO-2055 > URL: https://issues.apache.org/jira/browse/AVRO-2055 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.7.7, 1.8.2 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: AVRO-2055.1.patch > > > Remove magic string _io.file.buffer.size_ and _DEFAULT_BUFFER_SIZE_BYTES_ and > instead rely on the Hadoop libraries to provide this information. Will help > to keep Avro in sync with changes in Hadoop. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2056) DirectBinaryEncoder Creates Buffer For Each Call To writeDouble
[
https://issues.apache.org/jira/browse/AVRO-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102845#comment-16102845
]
Gabor Szadovszky commented on AVRO-2056:
[~belugabehr], I've took a look on Perf and I see how it works now. So, there
is no relevant change there, it is more for testing your stuff offline like in
an IDE...
+1
> DirectBinaryEncoder Creates Buffer For Each Call To writeDouble
> ---
>
> Key: AVRO-2056
> URL: https://issues.apache.org/jira/browse/AVRO-2056
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.7.7, 1.8.2
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: AVRO-2056.1.patch
>
>
> Each call to {{writeDouble}} creates a new buffer and promptly throws it away
> even though the class has a re-usable buffer and is used in other methods
> such as {{writeFloat}}. Remove this extra buffer.
> {code:title=org.apache.avro.io.DirectBinaryEncoder}
> // the buffer is used for writing floats, doubles, and large longs.
> private final byte[] buf = new byte[12];
> @Override
> public void writeFloat(float f) throws IOException {
> int len = BinaryData.encodeFloat(f, buf, 0);
> out.write(buf, 0, len);
> }
> @Override
> public void writeDouble(double d) throws IOException {
> byte[] buf = new byte[8];
> int len = BinaryData.encodeDouble(d, buf, 0);
> out.write(buf, 0, len);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-2049) Remove Superfluous Configuration From AvroSerializer
[
https://issues.apache.org/jira/browse/AVRO-2049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102841#comment-16102841
]
Gabor Szadovszky commented on AVRO-2049:
+1 will commit tomorrow if no more comments
[~nkollar], I did some research related to the usage of
{{Serializer.open(OutputStream)}} and it seems they are not really reused. We
might set the previous {{encoder}} to
{{EncoderFactory.binaryEncoder(OutputStream, BinaryEncoder)}} to reuse, though.
However, it would recommend having another JIRA for that.
> Remove Superfluous Configuration From AvroSerializer
>
>
> Key: AVRO-2049
> URL: https://issues.apache.org/jira/browse/AVRO-2049
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.7.7, 1.8.2
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: AVRO-2049.1.patch, AVRO-2049.2.patch, AVRO-2049.3.patch
>
>
> In the class {{org.apache.avro.hadoop.io.AvroSerializer}}, we see that the
> Avro block size is configured with a hard-coded value and there is a request
> to benchmark different buffer sizes.
> {code:title=org.apache.avro.hadoop.io.AvroSerializer}
> /**
>* The block size for the Avro encoder.
>*
>* This number was copied from the AvroSerialization of
> org.apache.avro.mapred in Avro 1.5.1.
>*
>* TODO(gwu): Do some benchmarking with different numbers here to see if it
> is important.
>*/
> private static final int AVRO_ENCODER_BLOCK_SIZE_BYTES = 512;
> /** An factory for creating Avro datum encoders. */
> private static EncoderFactory mEncoderFactory
> = new
> EncoderFactory().configureBlockSize(AVRO_ENCODER_BLOCK_SIZE_BYTES);
> {code}
> However, there is no need to benchmark, this setting is superfluous and is
> ignored with the current implementation.
> {code:title=org.apache.avro.hadoop.io.AvroSerializer}
> @Override
> public void open(OutputStream outputStream) throws IOException {
> mOutputStream = outputStream;
> mAvroEncoder = mEncoderFactory.binaryEncoder(outputStream, mAvroEncoder);
> }
> {code}
> {{org.apache.avro.io.EncoderFactory.binaryEncoder}} ignores this setting.
> This setting is only relevant for calls to
> {{org.apache.avro.io.EncoderFactory.blockingBinaryEncoder}}
> which considers the configured "Block Size" for doing binary encoding of
> blocked Array types as laid out in the
> [specs|https://avro.apache.org/docs/1.8.2/spec.html#binary_encode_complex].
> It can simply be removed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-2048) Avro Binary Decoding - Gracefully Handle Long Strings
[
https://issues.apache.org/jira/browse/AVRO-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102826#comment-16102826
]
Gabor Szadovszky commented on AVRO-2048:
Thanks, [~belugabehr], I'll wait another day for additional comments before
committing.
+1
> Avro Binary Decoding - Gracefully Handle Long Strings
> -
>
> Key: AVRO-2048
> URL: https://issues.apache.org/jira/browse/AVRO-2048
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.7.7, 1.8.2
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: AVRO-2048.1.patch, AVRO-2048.2.patch, AVRO-2048.3.patch
>
>
> According to the
> [specs|https://avro.apache.org/docs/1.8.2/spec.html#binary_encode_primitive]:
> bq. a string is encoded as a *long* followed by that many bytes of UTF-8
> encoded character data.
> However, that is currently not being adhered to:
> {code:title=org.apache.avro.io.BinaryDecoder}
> @Override
> public Utf8 readString(Utf8 old) throws IOException {
> int length = readInt();
> Utf8 result = (old != null ? old : new Utf8());
> result.setByteLength(length);
> if (0 != length) {
> doReadBytes(result.getBytes(), 0, length);
> }
> return result;
> }
> {code}
> The first thing the code does here is to load an *int* value, not a *long*.
> Because of the variable length nature of the size, this will mostly work.
> However, there may be edge-cases where the serializer is putting in large
> length values erroneously or nefariously. Let us gracefully detect such
> scenarios and more closely adhere to the spec.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-2048) Avro Binary Decoding - Gracefully Handle Long Strings
[
https://issues.apache.org/jira/browse/AVRO-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16101657#comment-16101657
]
Gabor Szadovszky commented on AVRO-2048:
[~belugabehr], the actual string size (or any array size) usually cannot be
{{Integer.MAX_VALUE}} as "_Some VMs reserve some header words in an array._".
See e.g. {{java.util.ArrayList.MAX_ARRAY_SIZE}}.
> Avro Binary Decoding - Gracefully Handle Long Strings
> -
>
> Key: AVRO-2048
> URL: https://issues.apache.org/jira/browse/AVRO-2048
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.7.7, 1.8.2
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: AVRO-2048.1.patch, AVRO-2048.2.patch
>
>
> According to the
> [specs|https://avro.apache.org/docs/1.8.2/spec.html#binary_encode_primitive]:
> bq. a string is encoded as a *long* followed by that many bytes of UTF-8
> encoded character data.
> However, that is currently not being adhered to:
> {code:title=org.apache.avro.io.BinaryDecoder}
> @Override
> public Utf8 readString(Utf8 old) throws IOException {
> int length = readInt();
> Utf8 result = (old != null ? old : new Utf8());
> result.setByteLength(length);
> if (0 != length) {
> doReadBytes(result.getBytes(), 0, length);
> }
> return result;
> }
> {code}
> The first thing the code does here is to load an *int* value, not a *long*.
> Because of the variable length nature of the size, this will mostly work.
> However, there may be edge-cases where the serializer is putting in large
> length values erroneously or nefariously. Let us gracefully detect such
> scenarios and more closely adhere to the spec.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-2049) Remove Superfluous Configuration From AvroSerializer
[
https://issues.apache.org/jira/browse/AVRO-2049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16101604#comment-16101604
]
Gabor Szadovszky commented on AVRO-2049:
[~belugabehr], as far as I can see {{ResolvingGramarGenerator}} sets the
_bufferSize_ (not the _blockSize_) which is used when a new _binaryEncoder_ is
created. With your change the _bufferSize_ of the _binaryEncoder_ will be
{{2048}} instead of {{32}} in {{ResolvingGrammarGenerator.getBinary}}.
> Remove Superfluous Configuration From AvroSerializer
>
>
> Key: AVRO-2049
> URL: https://issues.apache.org/jira/browse/AVRO-2049
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.7.7, 1.8.2
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: AVRO-2049.1.patch, AVRO-2049.2.patch
>
>
> In the class {{org.apache.avro.hadoop.io.AvroSerializer}}, we see that the
> Avro block size is configured with a hard-coded value and there is a request
> to benchmark different buffer sizes.
> {code:title=org.apache.avro.hadoop.io.AvroSerializer}
> /**
>* The block size for the Avro encoder.
>*
>* This number was copied from the AvroSerialization of
> org.apache.avro.mapred in Avro 1.5.1.
>*
>* TODO(gwu): Do some benchmarking with different numbers here to see if it
> is important.
>*/
> private static final int AVRO_ENCODER_BLOCK_SIZE_BYTES = 512;
> /** An factory for creating Avro datum encoders. */
> private static EncoderFactory mEncoderFactory
> = new
> EncoderFactory().configureBlockSize(AVRO_ENCODER_BLOCK_SIZE_BYTES);
> {code}
> However, there is no need to benchmark, this setting is superfluous and is
> ignored with the current implementation.
> {code:title=org.apache.avro.hadoop.io.AvroSerializer}
> @Override
> public void open(OutputStream outputStream) throws IOException {
> mOutputStream = outputStream;
> mAvroEncoder = mEncoderFactory.binaryEncoder(outputStream, mAvroEncoder);
> }
> {code}
> {{org.apache.avro.io.EncoderFactory.binaryEncoder}} ignores this setting.
> This setting is only relevant for calls to
> {{org.apache.avro.io.EncoderFactory.blockingBinaryEncoder}}
> which considers the configured "Block Size" for doing binary encoding of
> blocked Array types as laid out in the
> [specs|https://avro.apache.org/docs/1.8.2/spec.html#binary_encode_complex].
> It can simply be removed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-2050) Clear Array To Allow GC
[
https://issues.apache.org/jira/browse/AVRO-2050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16101567#comment-16101567
]
Gabor Szadovszky commented on AVRO-2050:
+1
> Clear Array To Allow GC
> ---
>
> Key: AVRO-2050
> URL: https://issues.apache.org/jira/browse/AVRO-2050
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.7.7, 1.8.2
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: AVRO-2050.1.patch, AVRO-2050.2.patch, AVRO-2050.3.patch
>
>
> Java's {{ArrayList}} implementation clears all Objects from the internal
> buffer when the {{clear()}} method is called. This allows the Objects to be
> free for GC. We should do the same in Avro
> {{org.apache.avro.generic.GenericData}}
> [ArrayList
> Source|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8u40-b25/java/util/ArrayList.java#ArrayList.clear%28%29]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-2053) Remove Reference To Deprecated Property mapred.output.compression.type
[
https://issues.apache.org/jira/browse/AVRO-2053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16101457#comment-16101457
]
Gabor Szadovszky commented on AVRO-2053:
The property {{mapred.output.compression.type}} had been deprecated in hadoop
{{2.7.3}}. Based on the actual {{pom.xml}} we still supports haddop1 which uses
this property. So the question is when do we want to officially drop hadoop1
support.
Maybe, we should only add this change to the next major release ({{1.9.0}})
after the hadoop1 dependency is removed.
> Remove Reference To Deprecated Property mapred.output.compression.type
> --
>
> Key: AVRO-2053
> URL: https://issues.apache.org/jira/browse/AVRO-2053
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.7.7, 1.8.2
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: AVRO-2053.1.patch
>
>
> Avro utilizes
> [deprecated|https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/DeprecatedProperties.html]
> property _mapred.output.compression.type_. Update code to use the MRv2
> property and don't override default behaviors/settings. Use the appropriate
> facilities from {{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat}}
> and {{org.apache.hadoop.io.SequenceFile}}.
> {code:title=org.apache.avro.mapreduce.AvroSequenceFileOutputFormat}
> /** Configuration key for storing the type of compression for the target
> sequence file. */
> private static final String CONF_COMPRESSION_TYPE =
> "mapred.output.compression.type";
> /** The default compression type for the target sequence file. */
> private static final CompressionType DEFAULT_COMPRESSION_TYPE =
> CompressionType.RECORD;
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-2054) Use StringBuilder instead of StringBuffer
[
https://issues.apache.org/jira/browse/AVRO-2054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16101405#comment-16101405
]
Gabor Szadovszky commented on AVRO-2054:
+1
> Use StringBuilder instead of StringBuffer
> -
>
> Key: AVRO-2054
> URL: https://issues.apache.org/jira/browse/AVRO-2054
> Project: Avro
> Issue Type: Improvement
>Affects Versions: 1.7.7, 1.8.2
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: AVRO-2054.1.patch, AVRO-2054.2.patch
>
>
> Use the un-synchronized StringBuilder instead of StringBuffer. Use _char_
> values instead of Strings.
> {code:title=org.apache.trevni.MetaData}
> @Override public String toString() {
> StringBuffer buffer = new StringBuffer();
> buffer.append("{ ");
> for (Map.Entry e : entrySet()) {
> buffer.append(e.getKey());
> buffer.append("=");
> try {
> buffer.append(new String(e.getValue(), "ISO-8859-1"));
> } catch (java.io.UnsupportedEncodingException error) {
> throw new TrevniRuntimeException(error);
> }
> buffer.append(" ");
> }
> buffer.append("}");
> return buffer.toString();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-2055) Remove Magic Value From org.apache.avro.hadoop.io.AvroSequenceFile
[ https://issues.apache.org/jira/browse/AVRO-2055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16101345#comment-16101345 ] Gabor Szadovszky commented on AVRO-2055: +1 > Remove Magic Value From org.apache.avro.hadoop.io.AvroSequenceFile > -- > > Key: AVRO-2055 > URL: https://issues.apache.org/jira/browse/AVRO-2055 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.7.7, 1.8.2 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: AVRO-2055.1.patch > > > Remove magic string _io.file.buffer.size_ and _DEFAULT_BUFFER_SIZE_BYTES_ and > instead rely on the Hadoop libraries to provide this information. Will help > to keep Avro in sync with changes in Hadoop. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2056) DirectBinaryEncoder Creates Buffer For Each Call To writeDouble
[
https://issues.apache.org/jira/browse/AVRO-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16101342#comment-16101342
]
Gabor Szadovszky commented on AVRO-2056:
+1 from my side.
[~belugabehr], Could you please add the Perf change as well to the patch?
(hint: next time you should use pull requests instead of patch files. It is
easier to review, comment and track the changes.)
> DirectBinaryEncoder Creates Buffer For Each Call To writeDouble
> ---
>
> Key: AVRO-2056
> URL: https://issues.apache.org/jira/browse/AVRO-2056
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.7.7, 1.8.2
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: AVRO-2056.1.patch
>
>
> Each call to {{writeDouble}} creates a new buffer and promptly throws it away
> even though the class has a re-usable buffer and is used in other methods
> such as {{writeFloat}}. Remove this extra buffer.
> {code:title=org.apache.avro.io.DirectBinaryEncoder}
> // the buffer is used for writing floats, doubles, and large longs.
> private final byte[] buf = new byte[12];
> @Override
> public void writeFloat(float f) throws IOException {
> int len = BinaryData.encodeFloat(f, buf, 0);
> out.write(buf, 0, len);
> }
> @Override
> public void writeDouble(double d) throws IOException {
> byte[] buf = new byte[8];
> int len = BinaryData.encodeDouble(d, buf, 0);
> out.write(buf, 0, len);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AVRO-2043) Move to java8
[ https://issues.apache.org/jira/browse/AVRO-2043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky updated AVRO-2043: --- Resolution: Fixed Hadoop Flags: Incompatible change Release Note: Avro java source and target levels are now 1.8 Status: Resolved (was: Patch Available) > Move to java8 > - > > Key: AVRO-2043 > URL: https://issues.apache.org/jira/browse/AVRO-2043 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.9.0 >Reporter: Gabor Szadovszky >Assignee: Gabor Szadovszky >Priority: Blocker > Fix For: 1.9.0 > > > java6 is really old and java7 is already EOD for 2yrs. It is time to move to > java8 in the next major release (1.9.0): > * update source/target to 1.8.0 in the pom.xml > * update the Dockerfile to used jdk8 > * ensure that lang/java builds fine with jdk8 > * ensure that all unit tests pass in lang/java with jdk8 > * ensure that everything builds fine and tests pass by using the new docker > image -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AVRO-1485) Specification says Record field type can be record name but implementation allows any named type.
[ https://issues.apache.org/jira/browse/AVRO-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky updated AVRO-1485: --- Resolution: Fixed Fix Version/s: 1.8.3 1.9.0 1.7.8 Status: Resolved (was: Patch Available) Checked with the doc generation on each branches. Seems that the -1 from the bot is really not relevant. > Specification says Record field type can be record name but implementation > allows any named type. > - > > Key: AVRO-1485 > URL: https://issues.apache.org/jira/browse/AVRO-1485 > Project: Avro > Issue Type: Bug > Components: java, spec >Affects Versions: 1.7.6 >Reporter: Sean Busbey >Assignee: Nandor Kollar > Fix For: 1.7.8, 1.9.0, 1.8.3 > > Attachments: AVRO-1485_1.patch > > > The [specification for Record > fields|http://avro.apache.org/docs/1.7.6/spec.html#schema_record] says that > the type is > bq. A JSON object defining a schema, or a JSON string naming a record > definition (required). > AFAICT, the Java implementation allows for any [named > type|http://avro.apache.org/docs/1.7.6/spec.html#Names]. > The specification should be updated to state any named type is allowed or the > Java implementation should restrict what can be used. The former seems less > likely to disturb current users. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-1485) Specification says Record field type can be record name but implementation allows any named type.
[ https://issues.apache.org/jira/browse/AVRO-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16092947#comment-16092947 ] Gabor Szadovszky commented on AVRO-1485: +1 > Specification says Record field type can be record name but implementation > allows any named type. > - > > Key: AVRO-1485 > URL: https://issues.apache.org/jira/browse/AVRO-1485 > Project: Avro > Issue Type: Bug > Components: java, spec >Affects Versions: 1.7.6 >Reporter: Sean Busbey >Assignee: Nandor Kollar > Attachments: AVRO-1485_1.patch > > > The [specification for Record > fields|http://avro.apache.org/docs/1.7.6/spec.html#schema_record] says that > the type is > bq. A JSON object defining a schema, or a JSON string naming a record > definition (required). > AFAICT, the Java implementation allows for any [named > type|http://avro.apache.org/docs/1.7.6/spec.html#Names]. > The specification should be updated to state any named type is allowed or the > Java implementation should restrict what can be used. The former seems less > likely to disturb current users. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AVRO-2021) uuid logical type is not documented
[ https://issues.apache.org/jira/browse/AVRO-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky updated AVRO-2021: --- Resolution: Fixed Fix Version/s: 1.8.3 1.9.0 Status: Resolved (was: Patch Available) > uuid logical type is not documented > --- > > Key: AVRO-2021 > URL: https://issues.apache.org/jira/browse/AVRO-2021 > Project: Avro > Issue Type: Improvement >Reporter: Andrew Rosca >Assignee: Nandor Kollar >Priority: Minor > Fix For: 1.9.0, 1.8.3 > > Attachments: AVRO-2021_1.patch > > > The documentation does not mention anything about the _uuid_ logical type, > which is in fact implemented in LogicalTypes.java. > Add documentation for _uuid_ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2021) uuid logical type is not documented
[ https://issues.apache.org/jira/browse/AVRO-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066591#comment-16066591 ] Gabor Szadovszky commented on AVRO-2021: Thanks a lot, [~nkollar] for fixing this. +1 > uuid logical type is not documented > --- > > Key: AVRO-2021 > URL: https://issues.apache.org/jira/browse/AVRO-2021 > Project: Avro > Issue Type: Improvement >Reporter: Andrew Rosca >Assignee: Nandor Kollar >Priority: Minor > Attachments: AVRO-2021_1.patch > > > The documentation does not mention anything about the _uuid_ logical type, > which is in fact implemented in LogicalTypes.java. > Add documentation for _uuid_ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AVRO-2044) Enforce maven 3
Gabor Szadovszky created AVRO-2044: -- Summary: Enforce maven 3 Key: AVRO-2044 URL: https://issues.apache.org/jira/browse/AVRO-2044 Project: Avro Issue Type: Improvement Components: build Affects Versions: 1.9.0 Reporter: Gabor Szadovszky Fix For: 1.9.0 maven 2 is EOL for 3+ years. We shall use maven 3 instead: * Modify the root pom.xml to enforce maven 3.0.5 (or newer) * Ensure that the Dockerfile installs the appropriate maven version * Upgrade the maven version for maven-plugin module (it may require code change because of the differences between maven 2 and maven 3 API) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AVRO-2043) Move to java8
[ https://issues.apache.org/jira/browse/AVRO-2043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky updated AVRO-2043: --- Status: Patch Available (was: Open) > Move to java8 > - > > Key: AVRO-2043 > URL: https://issues.apache.org/jira/browse/AVRO-2043 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.9.0 >Reporter: Gabor Szadovszky >Assignee: Gabor Szadovszky >Priority: Blocker > Fix For: 1.9.0 > > > java6 is really old and java7 is already EOD for 2yrs. It is time to move to > java8 in the next major release (1.9.0): > * update source/target to 1.8.0 in the pom.xml > * update the Dockerfile to used jdk8 > * ensure that lang/java builds fine with jdk8 > * ensure that all unit tests pass in lang/java with jdk8 > * ensure that everything builds fine and tests pass by using the new docker > image -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AVRO-2043) Move to java8
[ https://issues.apache.org/jira/browse/AVRO-2043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky updated AVRO-2043: --- Description: java6 is really old and java7 is already EOD for 2yrs. It is time to move to java8 in the next major release (1.9.0): * update source/target to 1.8.0 in the pom.xml * update the Dockerfile to used jdk8 * ensure that lang/java builds fine with jdk8 * ensure that all unit tests pass in lang/java with jdk8 * ensure that everything builds fine and tests pass by using the new docker image was: java6 is really old and java7 is already EOD since 2yrs. It is time to move to java8 in the next major release (1.9.0): * update source/target to 1.8.0 in the pom.xml * update the Dockerfile to used jdk8 * ensure that everything builds fine with jdk8 * ensure that all unit tests pass > Move to java8 > - > > Key: AVRO-2043 > URL: https://issues.apache.org/jira/browse/AVRO-2043 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.9.0 >Reporter: Gabor Szadovszky >Assignee: Gabor Szadovszky > > java6 is really old and java7 is already EOD for 2yrs. It is time to move to > java8 in the next major release (1.9.0): > * update source/target to 1.8.0 in the pom.xml > * update the Dockerfile to used jdk8 > * ensure that lang/java builds fine with jdk8 > * ensure that all unit tests pass in lang/java with jdk8 > * ensure that everything builds fine and tests pass by using the new docker > image -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AVRO-1985) Unreleased resources
[ https://issues.apache.org/jira/browse/AVRO-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky updated AVRO-1985: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Unreleased resources > > > Key: AVRO-1985 > URL: https://issues.apache.org/jira/browse/AVRO-1985 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.1 >Reporter: Gabor Szadovszky >Assignee: Gabor Szadovszky > Fix For: 1.9.0 > > > Static code analyzer tool found the following unreleased resources: > - > [Json.java:59|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/data/Json.java#L59] > - > [DataFileWriter.java:216|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/file/DataFileWriter.java#L205] > - > [SpecificCompiler.java:275|https://github.com/apache/avro/blob/master/lang/java/compiler/src/main/java/org/apache/avro/compiler/specific/SpecificCompiler.java#L275] > - > [AvroKeyOutputFormat.java:107|https://github.com/apache/avro/blob/master/lang/java/mapred/src/main/java/org/apache/avro/mapreduce/AvroKeyOutputFormat.java#L107] > - > [AvroKeyValueOutputFormat.java:63|https://github.com/apache/avro/blob/master/lang/java/mapred/src/main/java/org/apache/avro/mapreduce/AvroKeyValueOutputFormat.java#L63] > - > [IdlTool.java:62|https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/IdlTool.java#L62] > - > [Main.java:92|https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/Main.java#L92] > - > [Main.java:94|https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/Main.java#L94] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AVRO-2043) Move to java8
Gabor Szadovszky created AVRO-2043: -- Summary: Move to java8 Key: AVRO-2043 URL: https://issues.apache.org/jira/browse/AVRO-2043 Project: Avro Issue Type: Improvement Components: java Affects Versions: 1.9.0 Reporter: Gabor Szadovszky Assignee: Gabor Szadovszky java6 is really old and java7 is already EOD since 2yrs. It is time to move to java8 in the next major release (1.9.0): * update source/target to 1.8.0 in the pom.xml * update the Dockerfile to used jdk8 * ensure that everything builds fine with jdk8 * ensure that all unit tests pass -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AVRO-1971) AvroAlias Annotation should work on field elements
[ https://issues.apache.org/jira/browse/AVRO-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky updated AVRO-1971: --- Resolution: Fixed Fix Version/s: 1.9.0 Status: Resolved (was: Patch Available) > AvroAlias Annotation should work on field elements > -- > > Key: AVRO-1971 > URL: https://issues.apache.org/jira/browse/AVRO-1971 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.8.1 >Reporter: Johannes Schulte >Assignee: Johannes Schulte >Priority: Minor > Fix For: 1.9.0 > > > @AvroAlias annotation works only for records / classes but not for fields > when using ReflectData-generated schemas. The specification allows aliases > for comple types AND for fields. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-1971) AvroAlias Annotation should work on field elements
[ https://issues.apache.org/jira/browse/AVRO-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16050112#comment-16050112 ] Gabor Szadovszky commented on AVRO-1971: Thanks a lot, [~baunz]. It looks great. I have some minor comments in the PR. I'll try to commit it tomorrow if there are no more objections. > AvroAlias Annotation should work on field elements > -- > > Key: AVRO-1971 > URL: https://issues.apache.org/jira/browse/AVRO-1971 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.8.1 >Reporter: Johannes Schulte >Assignee: Johannes Schulte >Priority: Minor > > @AvroAlias annotation works only for records / classes but not for fields > when using ReflectData-generated schemas. The specification allows aliases > for comple types AND for fields. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AVRO-1401) @Nullable does not work with byte[]
[
https://issues.apache.org/jira/browse/AVRO-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated AVRO-1401:
---
Resolution: Fixed
Fix Version/s: 1.8.3
1.9.0
Status: Resolved (was: Patch Available)
Was unable to backport to 1.7 as the fix depends on the improvement AVRO-680
added to 1.8.
> @Nullable does not work with byte[]
> ---
>
> Key: AVRO-1401
> URL: https://issues.apache.org/jira/browse/AVRO-1401
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.7.5
>Reporter: dennis lucero
>Assignee: Nandor Kollar
> Labels: java, reflection, union
> Fix For: 1.9.0, 1.8.3
>
>
> @Nullable does not seem to be compatible with byte[] (Avro type bytes)
> {code:java}
> public static void main(String[] args) throws IOException
> {
> Schema schema = ReflectData.get().getSchema(MyRecord.class);
> DatumWriter protocol = ReflectData.get().createDatumWriter(schema);
> DataFileWriter writer = new
> DataFileWriter(protocol).create(schema, System.out);
> writer.append(new MyRecord());
> writer.close();
> }
> public static class MyRecord {
> @Nullable
> byte[] bytes = "foo".getBytes();
> }
> {code}
> {code}
> org.apache.avro.UnresolvedUnionException: Not in union
> ["null",{"type":"bytes","java-class":"[B"}]: [B@6d3f1f92
> at
> org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:600)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:151)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:71)
> at
> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:114)
> at
> org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:175)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
> at
> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:257)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-1971) AvroAlias Annotation should work on field elements
[ https://issues.apache.org/jira/browse/AVRO-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16048807#comment-16048807 ] Gabor Szadovszky commented on AVRO-1971: Thanks [~baunz] for taking care of this issue. I agree that based on the spec namespaces for field aliases shall not be allowed. Therefore, I would expect some error handling for the case of the space is specified. > AvroAlias Annotation should work on field elements > -- > > Key: AVRO-1971 > URL: https://issues.apache.org/jira/browse/AVRO-1971 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.8.1 >Reporter: Johannes Schulte >Assignee: Johannes Schulte >Priority: Minor > > @AvroAlias annotation works only for records / classes but not for fields > when using ReflectData-generated schemas. The specification allows aliases > for comple types AND for fields. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-1401) @Nullable does not work with byte[]
[
https://issues.apache.org/jira/browse/AVRO-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16047857#comment-16047857
]
Gabor Szadovszky commented on AVRO-1401:
+1
Wait another day and will commit if there are no more comments.
> @Nullable does not work with byte[]
> ---
>
> Key: AVRO-1401
> URL: https://issues.apache.org/jira/browse/AVRO-1401
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.7.5
>Reporter: dennis lucero
>Assignee: Nandor Kollar
> Labels: java, reflection, union
>
> @Nullable does not seem to be compatible with byte[] (Avro type bytes)
> {code:java}
> public static void main(String[] args) throws IOException
> {
> Schema schema = ReflectData.get().getSchema(MyRecord.class);
> DatumWriter protocol = ReflectData.get().createDatumWriter(schema);
> DataFileWriter writer = new
> DataFileWriter(protocol).create(schema, System.out);
> writer.append(new MyRecord());
> writer.close();
> }
> public static class MyRecord {
> @Nullable
> byte[] bytes = "foo".getBytes();
> }
> {code}
> {code}
> org.apache.avro.UnresolvedUnionException: Not in union
> ["null",{"type":"bytes","java-class":"[B"}]: [B@6d3f1f92
> at
> org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:600)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:151)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:71)
> at
> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:114)
> at
> org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:175)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
> at
> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:257)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AVRO-1985) Unreleased resources
[ https://issues.apache.org/jira/browse/AVRO-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky updated AVRO-1985: --- Hadoop Flags: Incompatible change Fix Version/s: 1.9.0 Release Note: Incompatible change: DataFileWriter.appendTo(SeekableInput, OutputStream) does not close the input anymore. > Unreleased resources > > > Key: AVRO-1985 > URL: https://issues.apache.org/jira/browse/AVRO-1985 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.1 >Reporter: Gabor Szadovszky >Assignee: Gabor Szadovszky > Fix For: 1.9.0 > > > Static code analyzer tool found the following unreleased resources: > - > [Json.java:59|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/data/Json.java#L59] > - > [DataFileWriter.java:216|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/file/DataFileWriter.java#L205] > - > [SpecificCompiler.java:275|https://github.com/apache/avro/blob/master/lang/java/compiler/src/main/java/org/apache/avro/compiler/specific/SpecificCompiler.java#L275] > - > [AvroKeyOutputFormat.java:107|https://github.com/apache/avro/blob/master/lang/java/mapred/src/main/java/org/apache/avro/mapreduce/AvroKeyOutputFormat.java#L107] > - > [AvroKeyValueOutputFormat.java:63|https://github.com/apache/avro/blob/master/lang/java/mapred/src/main/java/org/apache/avro/mapreduce/AvroKeyValueOutputFormat.java#L63] > - > [IdlTool.java:62|https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/IdlTool.java#L62] > - > [Main.java:92|https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/Main.java#L92] > - > [Main.java:94|https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/Main.java#L94] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AVRO-1857) GenericDatumWriter.write using BufferedBinaryEncoder leaves ByteBuffer in indeterminate state
[
https://issues.apache.org/jira/browse/AVRO-1857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated AVRO-1857:
---
Resolution: Fixed
Fix Version/s: 1.9.0
Status: Resolved (was: Patch Available)
> GenericDatumWriter.write using BufferedBinaryEncoder leaves ByteBuffer in
> indeterminate state
> -
>
> Key: AVRO-1857
> URL: https://issues.apache.org/jira/browse/AVRO-1857
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.7.7
>Reporter: Matt Grimwade
>Assignee: Nandor Kollar
> Fix For: 1.9.0
>
>
> Calling {{GenericDatumWriter.write(Object, Encoder)}} using a
> BufferedBinaryEncoder leaves any ByteBuffers within the object (representing
> BYTES or FIXED types) in an indeterminate state. Specifically, each buffer
> may be left either in its initial state, with (position, remaining) = (0, N)
> or in its "consumed" state of (N, 0).
> Although I cannot find it documented, I believe the correct behaviour is that
> the state of the object being written should be unmodified.
> This is an actual problem in our project where we save a copy of an object to
> disk before performing some action on it. This later action fails
> indeterminately because some of the ByteBuffers in the object are "consumed"
> and some are not.
> I think the fault lies in
> {{org.apache.avro.io.BufferedBinaryEncoder#writeFixed(java.nio.ByteBuffer)}},
> wherein the first branch advances the buffer's position but the second does
> not:
> {code}
> @Override
> public void writeFixed(ByteBuffer bytes) throws IOException {
> if (!bytes.hasArray() && bytes.remaining() > bulkLimit) {
> flushBuffer();
> sink.innerWrite(bytes); // bypass the buffer
> } else {
> super.writeFixed(bytes);
> }
> }
> {code}
> Here is a failing test case:
> {code}
> import static java.util.Arrays.asList;
> import static org.hamcrest.Matchers.equalTo;
> import static org.hamcrest.Matchers.is;
> import static org.junit.Assert.assertThat;
> import java.io.ByteArrayOutputStream;
> import java.io.IOException;
> import java.nio.ByteBuffer;
> import java.nio.MappedByteBuffer;
> import java.nio.channels.FileChannel;
> import java.nio.file.Files;
> import java.nio.file.Path;
> import java.nio.file.StandardOpenOption;
> import org.apache.avro.Schema;
> import org.apache.avro.generic.GenericDatumWriter;
> import org.apache.avro.io.Encoder;
> import org.apache.avro.io.EncoderFactory;
> import org.junit.Test;
> public class BugTest {
> private static final int ENCODER_BUFFER_SIZE = 32;
> private static final int EXAMPLE_DATA_SIZE = 17;
> @Test
> public void testArrayBackedByteBuffer() throws IOException {
> ByteBuffer buffer = ByteBuffer.wrap(someBytes(EXAMPLE_DATA_SIZE));
> doTest(buffer);
> }
> @Test
> public void testMappedByteBuffer() throws IOException {
> Path file = Files.createTempFile("test", "data");
> Files.write(file, someBytes(EXAMPLE_DATA_SIZE));
> MappedByteBuffer buffer = FileChannel.open(file,
> StandardOpenOption.READ).map(FileChannel.MapMode.READ_ONLY, 0,
> EXAMPLE_DATA_SIZE);
> doTest(buffer);
> }
> private static void doTest(ByteBuffer buffer) throws IOException {
> assertThat(asList(buffer.position(), buffer.remaining()),
> is(asList(0, EXAMPLE_DATA_SIZE)));
> ByteArrayOutputStream output = new
> ByteArrayOutputStream(EXAMPLE_DATA_SIZE * 2);
> EncoderFactory encoderFactory = new EncoderFactory();
> encoderFactory.configureBufferSize(ENCODER_BUFFER_SIZE);
> Encoder encoder = encoderFactory.binaryEncoder(output, null);
> new
> GenericDatumWriter(Schema.create(Schema.Type.BYTES)).write(buffer,
> encoder);
> encoder.flush();
> assertThat(output.toByteArray(),
> equalTo(avroEncoded(someBytes(EXAMPLE_DATA_SIZE;
> assertThat(asList(buffer.position(), buffer.remaining()),
> is(asList(0, EXAMPLE_DATA_SIZE))); // fails if buffer is not array-backed and
> buffer overflow occurs
> }
> private static byte[] someBytes(int size) {
> byte[] result = new byte[size];
> for (int i = 0; i < size; i++) {
> result[i] = (byte) i;
> }
> return result;
> }
> private static byte[] avroEncoded(byte[] bytes) {
> assert bytes.length < 64;
> byte[] result = new byte[1 + bytes.length];
> result[0] = (byte) (bytes.length * 2); // zig-zag encoding
> System.arraycopy(bytes, 0, result, 1, bytes.length);
> return result;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (AVRO-1857) GenericDatumWriter.write using BufferedBinaryEncoder leaves ByteBuffer in indeterminate state
[
https://issues.apache.org/jira/browse/AVRO-1857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16021352#comment-16021352
]
Gabor Szadovszky commented on AVRO-1857:
+1
I'll commit it tomorrow if no nobody else has any comment.
> GenericDatumWriter.write using BufferedBinaryEncoder leaves ByteBuffer in
> indeterminate state
> -
>
> Key: AVRO-1857
> URL: https://issues.apache.org/jira/browse/AVRO-1857
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.7.7
>Reporter: Matt Grimwade
>Assignee: Nandor Kollar
>
> Calling {{GenericDatumWriter.write(Object, Encoder)}} using a
> BufferedBinaryEncoder leaves any ByteBuffers within the object (representing
> BYTES or FIXED types) in an indeterminate state. Specifically, each buffer
> may be left either in its initial state, with (position, remaining) = (0, N)
> or in its "consumed" state of (N, 0).
> Although I cannot find it documented, I believe the correct behaviour is that
> the state of the object being written should be unmodified.
> This is an actual problem in our project where we save a copy of an object to
> disk before performing some action on it. This later action fails
> indeterminately because some of the ByteBuffers in the object are "consumed"
> and some are not.
> I think the fault lies in
> {{org.apache.avro.io.BufferedBinaryEncoder#writeFixed(java.nio.ByteBuffer)}},
> wherein the first branch advances the buffer's position but the second does
> not:
> {code}
> @Override
> public void writeFixed(ByteBuffer bytes) throws IOException {
> if (!bytes.hasArray() && bytes.remaining() > bulkLimit) {
> flushBuffer();
> sink.innerWrite(bytes); // bypass the buffer
> } else {
> super.writeFixed(bytes);
> }
> }
> {code}
> Here is a failing test case:
> {code}
> import static java.util.Arrays.asList;
> import static org.hamcrest.Matchers.equalTo;
> import static org.hamcrest.Matchers.is;
> import static org.junit.Assert.assertThat;
> import java.io.ByteArrayOutputStream;
> import java.io.IOException;
> import java.nio.ByteBuffer;
> import java.nio.MappedByteBuffer;
> import java.nio.channels.FileChannel;
> import java.nio.file.Files;
> import java.nio.file.Path;
> import java.nio.file.StandardOpenOption;
> import org.apache.avro.Schema;
> import org.apache.avro.generic.GenericDatumWriter;
> import org.apache.avro.io.Encoder;
> import org.apache.avro.io.EncoderFactory;
> import org.junit.Test;
> public class BugTest {
> private static final int ENCODER_BUFFER_SIZE = 32;
> private static final int EXAMPLE_DATA_SIZE = 17;
> @Test
> public void testArrayBackedByteBuffer() throws IOException {
> ByteBuffer buffer = ByteBuffer.wrap(someBytes(EXAMPLE_DATA_SIZE));
> doTest(buffer);
> }
> @Test
> public void testMappedByteBuffer() throws IOException {
> Path file = Files.createTempFile("test", "data");
> Files.write(file, someBytes(EXAMPLE_DATA_SIZE));
> MappedByteBuffer buffer = FileChannel.open(file,
> StandardOpenOption.READ).map(FileChannel.MapMode.READ_ONLY, 0,
> EXAMPLE_DATA_SIZE);
> doTest(buffer);
> }
> private static void doTest(ByteBuffer buffer) throws IOException {
> assertThat(asList(buffer.position(), buffer.remaining()),
> is(asList(0, EXAMPLE_DATA_SIZE)));
> ByteArrayOutputStream output = new
> ByteArrayOutputStream(EXAMPLE_DATA_SIZE * 2);
> EncoderFactory encoderFactory = new EncoderFactory();
> encoderFactory.configureBufferSize(ENCODER_BUFFER_SIZE);
> Encoder encoder = encoderFactory.binaryEncoder(output, null);
> new
> GenericDatumWriter(Schema.create(Schema.Type.BYTES)).write(buffer,
> encoder);
> encoder.flush();
> assertThat(output.toByteArray(),
> equalTo(avroEncoded(someBytes(EXAMPLE_DATA_SIZE;
> assertThat(asList(buffer.position(), buffer.remaining()),
> is(asList(0, EXAMPLE_DATA_SIZE))); // fails if buffer is not array-backed and
> buffer overflow occurs
> }
> private static byte[] someBytes(int size) {
> byte[] result = new byte[size];
> for (int i = 0; i < size; i++) {
> result[i] = (byte) i;
> }
> return result;
> }
> private static byte[] avroEncoded(byte[] bytes) {
> assert bytes.length < 64;
> byte[] result = new byte[1 + bytes.length];
> result[0] = (byte) (bytes.length * 2); // zig-zag encoding
> System.arraycopy(bytes, 0, result, 1, bytes.length);
> return result;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (AVRO-1973) Upgrade netty to the latest 3.x release
[
https://issues.apache.org/jira/browse/AVRO-1973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated AVRO-1973:
---
Resolution: Fixed
Hadoop Flags: Incompatible change
Fix Version/s: 1.9.0
Status: Resolved (was: Patch Available)
> Upgrade netty to the latest 3.x release
> ---
>
> Key: AVRO-1973
> URL: https://issues.apache.org/jira/browse/AVRO-1973
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.8.1
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
> Fix For: 1.9.0
>
>
> Netty version {{3.5.13}} is pretty old. Shall be updated to the latest 3.x
> release: {{3.10.6}}.
> (Not suggesting to upgrade to 4.x as it requires much more effort due to API
> incompatibilities.)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (AVRO-1973) Upgrade netty to the latest 3.x release
[
https://issues.apache.org/jira/browse/AVRO-1973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15874161#comment-15874161
]
Gabor Szadovszky commented on AVRO-1973:
Thanks a lot, [~busbey]. Completely agree on putting this one to 1.9.0 as all
the other dependency upgrades were put there as well.
> Upgrade netty to the latest 3.x release
> ---
>
> Key: AVRO-1973
> URL: https://issues.apache.org/jira/browse/AVRO-1973
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.8.1
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
>
> Netty version {{3.5.13}} is pretty old. Shall be updated to the latest 3.x
> release: {{3.10.6}}.
> (Not suggesting to upgrade to 4.x as it requires much more effort due to API
> incompatibilities.)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (AVRO-1973) Upgrade netty to the latest 3.x release
[
https://issues.apache.org/jira/browse/AVRO-1973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated AVRO-1973:
---
Release Note: Netty was upgraded from 3.5.13.Final to 3.10.6.Final. After
3.6.0 Netty does not interrupt its worker threads therefore,
NettyServer.close() would hang until all the ongoing responder calls finish
their job. Normally it means that all the ongoing client requests will be
responded before the server stops. It also means that if any of the responder
calls are in dead lock NettyServer.close() will never return.
Added release note about the identified problem might occur due to the Netty
upgrade.
I've failed to check the Netty issue tracker whether any other breaking change
introduced.
> Upgrade netty to the latest 3.x release
> ---
>
> Key: AVRO-1973
> URL: https://issues.apache.org/jira/browse/AVRO-1973
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.8.1
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
>
> Netty version {{3.5.13}} is pretty old. Shall be updated to the latest 3.x
> release: {{3.10.6}}.
> (Not suggesting to upgrade to 4.x as it requires much more effort due to API
> incompatibilities.)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (AVRO-1965) Reparsing an existing schema mutates the schema
[
https://issues.apache.org/jira/browse/AVRO-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15863610#comment-15863610
]
Gabor Szadovszky commented on AVRO-1965:
It looks good to me.
> Reparsing an existing schema mutates the schema
> ---
>
> Key: AVRO-1965
> URL: https://issues.apache.org/jira/browse/AVRO-1965
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.1
>Reporter: David Maughan
>Assignee: Doug Cutting
> Fix For: 1.8.2
>
> Attachments: AVRO-1965.patch
>
>
> h2. Overview
> In certain scenarios re-parsing a schema constructed via {{SchemaBuilder}} or
> {{Schema.createRecord}} produces a schema that is no longer equal to the
> original. In the below example, after parsing the existing schema, the
> namespace of the top level record is cascaded down to the bottom level
> record. Note that the way the top level record is being created with the
> namespace as part of the name {{default.a}} is the same way Hive's
> {{AvroSerDe}} constructs a schema from a Hive table's metadata.
> h2. Failing test case
> {code}
> Schema d = SchemaBuilder.builder().intType();
> Schema c =
> SchemaBuilder.record("c").fields().name("d").type(d).noDefault().endRecord();
> Schema b =
> SchemaBuilder.record("b").fields().name("c").type(c).noDefault().endRecord();
> Schema a1 =
> SchemaBuilder.record("default.a").fields().name("b").type(b).noDefault().endRecord();
> Schema a2 = new Schema.Parser().parse(a1.toString());
> // a1 =
> {"type":"record","name":"a","namespace":"default","fields":[{"name":"b","type":{"type":"record","name":"b","namespace":"","fields":[{"name":"c","type":{"type":"record","name":"c","fields":[{"name":"d","type":"int"}]}}]}}]}
> // a2 =
> {"type":"record","name":"a","namespace":"default","fields":[{"name":"b","type":{"type":"record","name":"b","namespace":"","fields":[{"name":"c","type":{"type":"record","name":"c","namespace":"default","fields":[{"name":"d","type":"int"}]}}]}}]}
> assertThat(a2, is(a1));
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (AVRO-1971) AvroAlias Annotation should work on field elements
[ https://issues.apache.org/jira/browse/AVRO-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky reassigned AVRO-1971: -- Assignee: Johannes Schulte > AvroAlias Annotation should work on field elements > -- > > Key: AVRO-1971 > URL: https://issues.apache.org/jira/browse/AVRO-1971 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.8.1 >Reporter: Johannes Schulte >Assignee: Johannes Schulte >Priority: Minor > > @AvroAlias annotation works only for records / classes but not for fields > when using ReflectData-generated schemas. The specification allows aliases > for comple types AND for fields. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AVRO-1813) Incorrect link to build instructions in Java Getting Started doc
[ https://issues.apache.org/jira/browse/AVRO-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky updated AVRO-1813: --- Resolution: Fixed Fix Version/s: 1.8.2 1.9.0 1.7.8 Status: Resolved (was: Patch Available) > Incorrect link to build instructions in Java Getting Started doc > > > Key: AVRO-1813 > URL: https://issues.apache.org/jira/browse/AVRO-1813 > Project: Avro > Issue Type: Bug > Components: doc >Reporter: Erik Forsberg >Assignee: Pietro Menna >Priority: Trivial > Fix For: 1.7.8, 1.9.0, 1.8.2 > > > https://avro.apache.org/docs/current/gettingstartedjava.html has an incorrect > link to https://cwiki.apache.org/AVRO/build-documentation.html > Should link to > https://cwiki.apache.org/confluence/display/AVRO/Build+Documentation -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AVRO-1975) Upgrade java dependencies
[ https://issues.apache.org/jira/browse/AVRO-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky updated AVRO-1975: --- Resolution: Fixed Hadoop Flags: Incompatible change Status: Resolved (was: Patch Available) > Upgrade java dependencies > - > > Key: AVRO-1975 > URL: https://issues.apache.org/jira/browse/AVRO-1975 > Project: Avro > Issue Type: Improvement > Components: build, dependencies >Reporter: Gabor Szadovszky >Assignee: Gabor Szadovszky > Fix For: 1.9.0 > > > It was quite a long time ago we upgraded the dependencies. > This one is to track the upgrades that does not require any code/config > modification. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AVRO-1973) Upgrade netty to the latest 3.x release
[
https://issues.apache.org/jira/browse/AVRO-1973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856025#comment-15856025
]
Gabor Szadovszky commented on AVRO-1973:
Any thoughts about this one?
> Upgrade netty to the latest 3.x release
> ---
>
> Key: AVRO-1973
> URL: https://issues.apache.org/jira/browse/AVRO-1973
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.8.1
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
>
> Netty version {{3.5.13}} is pretty old. Shall be updated to the latest 3.x
> release: {{3.10.6}}.
> (Not suggesting to upgrade to 4.x as it requires much more effort due to API
> incompatibilities.)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (AVRO-1975) Upgrade java dependencies
[ https://issues.apache.org/jira/browse/AVRO-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856019#comment-15856019 ] Gabor Szadovszky commented on AVRO-1975: I'll push it tomorrow if there is no more comment. > Upgrade java dependencies > - > > Key: AVRO-1975 > URL: https://issues.apache.org/jira/browse/AVRO-1975 > Project: Avro > Issue Type: Improvement > Components: build, dependencies >Reporter: Gabor Szadovszky >Assignee: Gabor Szadovszky > Fix For: 1.9.0 > > > It was quite a long time ago we upgraded the dependencies. > This one is to track the upgrades that does not require any code/config > modification. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AVRO-1605) Remove Jackson classes from public API
[ https://issues.apache.org/jira/browse/AVRO-1605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856011#comment-15856011 ] Gabor Szadovszky commented on AVRO-1605: [~rdblue], do you have any additional comment here? Still against committing it? > Remove Jackson classes from public API > -- > > Key: AVRO-1605 > URL: https://issues.apache.org/jira/browse/AVRO-1605 > Project: Avro > Issue Type: Sub-task > Components: java >Affects Versions: 1.7.8 >Reporter: Tom White >Assignee: Gabor Szadovszky > Fix For: 1.9.0 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AVRO-1987) Add @Deprecated to getConverion(String fieldName) in 1.8.X
[ https://issues.apache.org/jira/browse/AVRO-1987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky updated AVRO-1987: --- Resolution: Fixed Fix Version/s: 1.8.2 Status: Resolved (was: Patch Available) I've pushed directly to branch-1.8 as AVRO-1972 will do the related change on master. > Add @Deprecated to getConverion(String fieldName) in 1.8.X > -- > > Key: AVRO-1987 > URL: https://issues.apache.org/jira/browse/AVRO-1987 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.1 >Reporter: Suraj Acharya >Assignee: Suraj Acharya >Priority: Trivial > Fix For: 1.8.2 > > > in Avro-1972 a fix was added to correct the spelling. > It will be a good idea to add a @Deprecated to the method in 1.8.X so people > can move over in 1.9.O -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AVRO-1975) Upgrade java dependencies
[ https://issues.apache.org/jira/browse/AVRO-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky updated AVRO-1975: --- Release Note: The following dependencies are upgraded: * scope:compile: ** commons-cli:commons-cli:1.2 -> 1.3.1 ** commons-codec:commons-codec:1.9 -> 1.10 ** commons-logging:commons:logging:1.1.1 -> 1.2 ** com.google.protobuf:protobuf-java:2.5.0 -> 2.6.1 ** joda-time:joda-time:2.7 -> 2.9.7 ** net.sf.jopt-simple:jopt-simple:4.7 -> 5.0.3 ** org.apache.commons:commons-compress:1.8.1 -> 1.13 ** org.apache.hadoop:hadoop-client:2.5.1 -> 2.7.3 ** org.apache.maven:maven-*:2.0.10 -> 2.0.22 ** org.apache.thrift:libthrift:0.9.1 -> 0.9.3 ** org.easymock:easymock:3.2 -> 3.4 ** org.slf4j:slf4j-*:1.7.7 -> 1.7.22 ** org.tukaani:xz:1.5 -> 1.6 ** org.xerial.snappy:snappy-java:1.1.1.3 -> 1.1.2.6 * scope:test: ** junit:junit:4.11 -> 4.12 * scope:provided: ** org.apache.ant:ant:1.9.0 -> 1.10.0 * plugins: ** com.thoughtworks.paranamer:paranamer-maven-plugin:2.7 -> 2.8 ** org.apache.felix:maven-bundle-plugin:2.5.3 -> 3.2.0 ** org.apache.maven.plugin-tools:maven-plugin-tools-javadoc:3.2 -> 3.5 ** org.apache.maven.plugins:maven-archetype-plugin:2.2 -> 2.4 ** org.apache.maven.plugins:maven-compiler-plugin:3.1 -> 3.6.0 ** org.apache.maven.plugins:maven-jar-plugin:2.5 -> 2.6 ** org.apache.maven.plugins:maven-javadoc-plugin:2.9.1 -> 2.10.4 ** org.apache.maven.plugins:maven-plugin-plugin:3.3 -> 3.5 ** org.apache.maven.plugins:maven-site-plugin:3.3 -> 3.6 ** org.apache.maven.plugins:maven-source-plugin:2.3 -> 2.4 ** org.apache.maven.plugins:maven-surefire-plugin:2.17 -> 2.19.1 ** org.codehaus.mojo:exec-maven-plugin:1.3.2 -> 1.5.0 The dependency commons-httpclient:commons-httpclient has been removed. Listed all the upgrades in the release notes. Maybe listing the plugin upgrades and the dependencies not in the compile scope does not make sense. Feel free to edit. > Upgrade java dependencies > - > > Key: AVRO-1975 > URL: https://issues.apache.org/jira/browse/AVRO-1975 > Project: Avro > Issue Type: Improvement > Components: build, dependencies >Reporter: Gabor Szadovszky >Assignee: Gabor Szadovszky > Fix For: 1.9.0 > > > It was quite a long time ago we upgraded the dependencies. > This one is to track the upgrades that does not require any code/config > modification. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1985) Unreleased resources
[
https://issues.apache.org/jira/browse/AVRO-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833407#comment-15833407
]
Gabor Szadovszky commented on AVRO-1985:
Thanks for reviewing, [~sacharya].
You're right, it won't be closed but it is intentional. {{RecordWriter}} is
closeable and it is up to the invoker to close it when it is required. I've
added the closing of the newly created output streams in case of any exception
occurs during the creation of the record writer.
> Unreleased resources
>
>
> Key: AVRO-1985
> URL: https://issues.apache.org/jira/browse/AVRO-1985
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.1
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
>
> Static code analyzer tool found the following unreleased resources:
> -
> [Json.java:59|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/data/Json.java#L59]
> -
> [DataFileWriter.java:216|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/file/DataFileWriter.java#L205]
> -
> [SpecificCompiler.java:275|https://github.com/apache/avro/blob/master/lang/java/compiler/src/main/java/org/apache/avro/compiler/specific/SpecificCompiler.java#L275]
> -
> [AvroKeyOutputFormat.java:107|https://github.com/apache/avro/blob/master/lang/java/mapred/src/main/java/org/apache/avro/mapreduce/AvroKeyOutputFormat.java#L107]
> -
> [AvroKeyValueOutputFormat.java:63|https://github.com/apache/avro/blob/master/lang/java/mapred/src/main/java/org/apache/avro/mapreduce/AvroKeyValueOutputFormat.java#L63]
> -
> [IdlTool.java:62|https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/IdlTool.java#L62]
> -
> [Main.java:92|https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/Main.java#L92]
> -
> [Main.java:94|https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/Main.java#L94]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (AVRO-1985) Unreleased resources
[ https://issues.apache.org/jira/browse/AVRO-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky updated AVRO-1985: --- Status: Patch Available (was: Open) > Unreleased resources > > > Key: AVRO-1985 > URL: https://issues.apache.org/jira/browse/AVRO-1985 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.1 >Reporter: Gabor Szadovszky >Assignee: Gabor Szadovszky > > Static code analyzer tool found the following unreleased resources: > - > [Json.java:59|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/data/Json.java#L59] > - > [DataFileWriter.java:216|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/file/DataFileWriter.java#L205] > - > [SpecificCompiler.java:275|https://github.com/apache/avro/blob/master/lang/java/compiler/src/main/java/org/apache/avro/compiler/specific/SpecificCompiler.java#L275] > - > [AvroKeyOutputFormat.java:107|https://github.com/apache/avro/blob/master/lang/java/mapred/src/main/java/org/apache/avro/mapreduce/AvroKeyOutputFormat.java#L107] > - > [AvroKeyValueOutputFormat.java:63|https://github.com/apache/avro/blob/master/lang/java/mapred/src/main/java/org/apache/avro/mapreduce/AvroKeyValueOutputFormat.java#L63] > - > [IdlTool.java:62|https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/IdlTool.java#L62] > - > [Main.java:92|https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/Main.java#L92] > - > [Main.java:94|https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/Main.java#L94] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AVRO-1985) Unreleased resources
Gabor Szadovszky created AVRO-1985: -- Summary: Unreleased resources Key: AVRO-1985 URL: https://issues.apache.org/jira/browse/AVRO-1985 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.8.1 Reporter: Gabor Szadovszky Assignee: Gabor Szadovszky Static code analyzer tool found the following unreleased resources: - [Json.java:59|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/data/Json.java#L59] - [DataFileWriter.java:216|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/file/DataFileWriter.java#L205] - [SpecificCompiler.java:275|https://github.com/apache/avro/blob/master/lang/java/compiler/src/main/java/org/apache/avro/compiler/specific/SpecificCompiler.java#L275] - [AvroKeyOutputFormat.java:107|https://github.com/apache/avro/blob/master/lang/java/mapred/src/main/java/org/apache/avro/mapreduce/AvroKeyOutputFormat.java#L107] - [AvroKeyValueOutputFormat.java:63|https://github.com/apache/avro/blob/master/lang/java/mapred/src/main/java/org/apache/avro/mapreduce/AvroKeyValueOutputFormat.java#L63] - [IdlTool.java:62|https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/IdlTool.java#L62] - [Main.java:92|https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/Main.java#L92] - [Main.java:94|https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/Main.java#L94] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1983) Unreleased resources
[ https://issues.apache.org/jira/browse/AVRO-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15821939#comment-15821939 ] Gabor Szadovszky commented on AVRO-1983: This one is created during we've had problems with JIRA. It is a bit strange as neither the button 'Resolve' nor 'Submit Patch' appear for me just as I wouldn't be a contributor for Avro. But I am and all the other issues have these buttons. Do anyone know how to fix it? > Unreleased resources > > > Key: AVRO-1983 > URL: https://issues.apache.org/jira/browse/AVRO-1983 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.1 >Reporter: Gabor Szadovszky >Assignee: Gabor Szadovszky > > Static code analyzer tool found the following unreleased resources: > - > [Json.java:59|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/data/Json.java#L59] > - > [DataFileWriter.java:216|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/file/DataFileWriter.java#L205] > - > [SpecificCompiler.java:275|https://github.com/apache/avro/blob/master/lang/java/compiler/src/main/java/org/apache/avro/compiler/specific/SpecificCompiler.java#L275] > - > [AvroKeyOutputFormat.java:107|https://github.com/apache/avro/blob/master/lang/java/mapred/src/main/java/org/apache/avro/mapreduce/AvroKeyOutputFormat.java#L107] > - > [AvroKeyValueOutputFormat.java:63|https://github.com/apache/avro/blob/master/lang/java/mapred/src/main/java/org/apache/avro/mapreduce/AvroKeyValueOutputFormat.java#L63] > - > [IdlTool.java:62|https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/IdlTool.java#L62] > - > [Main.java:92|https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/Main.java#L92] > - > [Main.java:94|https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/Main.java#L94] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1984) Unreleased resources
[ https://issues.apache.org/jira/browse/AVRO-1984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15821938#comment-15821938 ] Gabor Szadovszky commented on AVRO-1984: This one is created during we've had problems with JIRA. It shall be deleted or at least closed by duplicate (AVRO-1983). Seems, I do not have the privilege to do so. Actually, I cannot even resolve this one. > Unreleased resources > > > Key: AVRO-1984 > URL: https://issues.apache.org/jira/browse/AVRO-1984 > Project: Avro > Issue Type: Bug >Reporter: Gabor Szadovszky >Assignee: Gabor Szadovszky > > Static code analyzer tool found the following unreleased resources: > - > [Json.java:59|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/data/Json.java#L59] > - > [DataFileWriter.java:216|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/file/DataFileWriter.java#L205] > - > [SpecificCompiler.java:275|https://github.com/apache/avro/blob/master/lang/java/compiler/src/main/java/org/apache/avro/compiler/specific/SpecificCompiler.java#L275] > - > [AvroKeyOutputFormat.java:107|https://github.com/apache/avro/blob/master/lang/java/mapred/src/main/java/org/apache/avro/mapreduce/AvroKeyOutputFormat.java#L107] > - > [AvroKeyValueOutputFormat.java:63|https://github.com/apache/avro/blob/master/lang/java/mapred/src/main/java/org/apache/avro/mapreduce/AvroKeyValueOutputFormat.java#L63] > - > [IdlTool.java:62|https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/IdlTool.java#L62] > - > [Main.java:92|https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/Main.java#L92] > - > [Main.java:94|https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/Main.java#L94] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AVRO-1983) Unreleased resources
[ https://issues.apache.org/jira/browse/AVRO-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky updated AVRO-1983: --- Affects Version/s: 1.8.1 Component/s: java > Unreleased resources > > > Key: AVRO-1983 > URL: https://issues.apache.org/jira/browse/AVRO-1983 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.1 >Reporter: Gabor Szadovszky >Assignee: Gabor Szadovszky > > Static code analyzer tool found the following unreleased resources: > - > [Json.java:59|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/data/Json.java#L59] > - > [DataFileWriter.java:216|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/file/DataFileWriter.java#L205] > - > [SpecificCompiler.java:275|https://github.com/apache/avro/blob/master/lang/java/compiler/src/main/java/org/apache/avro/compiler/specific/SpecificCompiler.java#L275] > - > [AvroKeyOutputFormat.java:107|https://github.com/apache/avro/blob/master/lang/java/mapred/src/main/java/org/apache/avro/mapreduce/AvroKeyOutputFormat.java#L107] > - > [AvroKeyValueOutputFormat.java:63|https://github.com/apache/avro/blob/master/lang/java/mapred/src/main/java/org/apache/avro/mapreduce/AvroKeyValueOutputFormat.java#L63] > - > [IdlTool.java:62|https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/IdlTool.java#L62] > - > [Main.java:92|https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/Main.java#L92] > - > [Main.java:94|https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/Main.java#L94] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (AVRO-1983) Unreleased resources
[ https://issues.apache.org/jira/browse/AVRO-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky reassigned AVRO-1983: -- Assignee: (was: Gabor Szadovszky) > Unreleased resources > > > Key: AVRO-1983 > URL: https://issues.apache.org/jira/browse/AVRO-1983 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.1 >Reporter: Gabor Szadovszky > > Static code analyzer tool found the following unreleased resources: > - > [Json.java:59|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/data/Json.java#L59] > - > [DataFileWriter.java:216|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/file/DataFileWriter.java#L205] > - > [SpecificCompiler.java:275|https://github.com/apache/avro/blob/master/lang/java/compiler/src/main/java/org/apache/avro/compiler/specific/SpecificCompiler.java#L275] > - > [AvroKeyOutputFormat.java:107|https://github.com/apache/avro/blob/master/lang/java/mapred/src/main/java/org/apache/avro/mapreduce/AvroKeyOutputFormat.java#L107] > - > [AvroKeyValueOutputFormat.java:63|https://github.com/apache/avro/blob/master/lang/java/mapred/src/main/java/org/apache/avro/mapreduce/AvroKeyValueOutputFormat.java#L63] > - > [IdlTool.java:62|https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/IdlTool.java#L62] > - > [Main.java:92|https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/Main.java#L92] > - > [Main.java:94|https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/Main.java#L94] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (AVRO-1983) Unreleased resources
[ https://issues.apache.org/jira/browse/AVRO-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky reassigned AVRO-1983: -- Assignee: Gabor Szadovszky > Unreleased resources > > > Key: AVRO-1983 > URL: https://issues.apache.org/jira/browse/AVRO-1983 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.1 >Reporter: Gabor Szadovszky >Assignee: Gabor Szadovszky > > Static code analyzer tool found the following unreleased resources: > - > [Json.java:59|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/data/Json.java#L59] > - > [DataFileWriter.java:216|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/file/DataFileWriter.java#L205] > - > [SpecificCompiler.java:275|https://github.com/apache/avro/blob/master/lang/java/compiler/src/main/java/org/apache/avro/compiler/specific/SpecificCompiler.java#L275] > - > [AvroKeyOutputFormat.java:107|https://github.com/apache/avro/blob/master/lang/java/mapred/src/main/java/org/apache/avro/mapreduce/AvroKeyOutputFormat.java#L107] > - > [AvroKeyValueOutputFormat.java:63|https://github.com/apache/avro/blob/master/lang/java/mapred/src/main/java/org/apache/avro/mapreduce/AvroKeyValueOutputFormat.java#L63] > - > [IdlTool.java:62|https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/IdlTool.java#L62] > - > [Main.java:92|https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/Main.java#L92] > - > [Main.java:94|https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/Main.java#L94] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1980) Write to Avro File in Bulk
[
https://issues.apache.org/jira/browse/AVRO-1980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15817860#comment-15817860
]
Gabor Szadovszky commented on AVRO-1980:
DataFileWriter does not actually write record-by-record. It writes the records
in blocks instead. You can read more about it at
http://avro.apache.org/docs/1.8.1/spec.html#Object+Container+Files.
Or did I misunderstand your issue?
> Write to Avro File in Bulk
> ---
>
> Key: AVRO-1980
> URL: https://issues.apache.org/jira/browse/AVRO-1980
> Project: Avro
> Issue Type: Improvement
> Components: build, java
>Affects Versions: 1.8.1
>Reporter: Santosh Balasubramanya
>
> when writing to Avro files usually append happens record by record.
> Can't it be done by buffering and then committing it to file?
> Below example
> DatumWriter userDatumWriter = new SpecificDatumWriter(User.class);
> DataFileWriter dataFileWriter = new
> DataFileWriter(userDatumWriter);
> dataFileWriter.create(user1.getSchema(), new File("users.avro"));
> dataFileWriter.append(user1);
> dataFileWriter.append(user2);
> dataFileWriter.append(user3);
> dataFileWriter.close();
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (AVRO-1975) Upgrade java dependencies
[ https://issues.apache.org/jira/browse/AVRO-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky updated AVRO-1975: --- Status: Patch Available (was: Open) > Upgrade java dependencies > - > > Key: AVRO-1975 > URL: https://issues.apache.org/jira/browse/AVRO-1975 > Project: Avro > Issue Type: Improvement >Reporter: Gabor Szadovszky >Assignee: Gabor Szadovszky > > It was quite a long time ago we upgraded the dependencies. > This one is to track the upgrades that does not require any code/config > modification. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AVRO-1975) Upgrade java dependencies
Gabor Szadovszky created AVRO-1975: -- Summary: Upgrade java dependencies Key: AVRO-1975 URL: https://issues.apache.org/jira/browse/AVRO-1975 Project: Avro Issue Type: Improvement Reporter: Gabor Szadovszky Assignee: Gabor Szadovszky It was quite a long time ago we upgraded the dependencies. This one is to track the upgrades that does not require any code/config modification. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AVRO-1973) Upgrade netty to the latest 3.x release
[
https://issues.apache.org/jira/browse/AVRO-1973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated AVRO-1973:
---
Status: Patch Available (was: Open)
> Upgrade netty to the latest 3.x release
> ---
>
> Key: AVRO-1973
> URL: https://issues.apache.org/jira/browse/AVRO-1973
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.8.1
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
>
> Netty version {{3.5.13}} is pretty old. Shall be updated to the latest 3.x
> release: {{3.10.6}}.
> (Not suggesting to upgrade to 4.x as it requires much more effort due to API
> incompatibilities.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (AVRO-1973) Upgrade netty to the latest 3.x release
Gabor Szadovszky created AVRO-1973:
--
Summary: Upgrade netty to the latest 3.x release
Key: AVRO-1973
URL: https://issues.apache.org/jira/browse/AVRO-1973
Project: Avro
Issue Type: Improvement
Components: java
Affects Versions: 1.8.1
Reporter: Gabor Szadovszky
Assignee: Gabor Szadovszky
Netty version {{3.5.13}} is pretty old. Shall be updated to the latest 3.x
release: {{3.10.6}}.
(Not suggesting to upgrade to 4.x as it requires much more effort due to API
incompatibilities.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (AVRO-1972) Typo in method name
[ https://issues.apache.org/jira/browse/AVRO-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Szadovszky updated AVRO-1972: --- Fix Version/s: 1.9.0 Status: Patch Available (was: Open) [~cesco] provided a fix for this one. - Thanks a lot. As it is trivial typo fix I don't think the deprecation is necessary however, would not target it to 1.8.x as it would break backward compatibility. I suggest 1.9.0. > Typo in method name > --- > > Key: AVRO-1972 > URL: https://issues.apache.org/jira/browse/AVRO-1972 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.1 >Reporter: Francesco Versaci >Priority: Minor > Labels: typo > Fix For: 1.9.0 > > > There's a method of SpecificRecordBase.java which has a typo in the method > name, i.e., `getConverion(String fieldName)` instead of `getConversion(String > fieldName)`. > gszadovszky suggests two options, detailed below: > --- > This method is part of the public API and has already been released (1.8.1). > We would not like to break the code of any consumer in case of doing a minor > upgrade therefore, I would suggest deprecating the method with the typo and > create a new one with the correct naming. (Later on we will be able to remove > the deprecated method.) Another idea is to keep the fix as is but only > include in the next major release. > --- -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AVRO-1970) Flaky test: TestInputBytes
[
https://issues.apache.org/jira/browse/AVRO-1970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated AVRO-1970:
---
Status: Patch Available (was: Open)
> Flaky test: TestInputBytes
> --
>
> Key: AVRO-1970
> URL: https://issues.apache.org/jira/browse/AVRO-1970
> Project: Avro
> Issue Type: Bug
> Components: java, trevni
>Affects Versions: 1.8.1
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
>Priority: Minor
>
> The test {{org.apache.trevni.TestInputBytes}} is flaky. Sometimes it throws
> the following exception:
> {code}
> java.lang.IllegalArgumentException: bound must be positive
> at java.util.Random.nextInt(Random.java:388)
> at
> org.apache.trevni.TestInputBytes.testRandomReads(TestInputBytes.java:43)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (AVRO-1970) Flaky test: TestInputBytes
Gabor Szadovszky created AVRO-1970:
--
Summary: Flaky test: TestInputBytes
Key: AVRO-1970
URL: https://issues.apache.org/jira/browse/AVRO-1970
Project: Avro
Issue Type: Bug
Components: java, trevni
Affects Versions: 1.8.1
Reporter: Gabor Szadovszky
Assignee: Gabor Szadovszky
Priority: Minor
The test {{org.apache.trevni.TestInputBytes}} is flaky. Sometimes it throws the
following exception:
{code}
java.lang.IllegalArgumentException: bound must be positive
at java.util.Random.nextInt(Random.java:388)
at
org.apache.trevni.TestInputBytes.testRandomReads(TestInputBytes.java:43)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AVRO-1605) Remove Jackson classes from public API
[
https://issues.apache.org/jira/browse/AVRO-1605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15726078#comment-15726078
]
Gabor Szadovszky commented on AVRO-1605:
Currently the classes {{ResolvingGrammarGenerator}}, {{Idl}} and
{{ProtobufData}} are traversing/building/parsing jackson json object structures
and as they are outside of the package {{org.apache.avro}} they need the
Accessor to use the related methods (add property as a {{JsonNode}}, retrieve
default value as a {{JsonNode}} or creating a Field with a {{JsonNode}} as the
default value).
I am not sure we would like to rewrite these classes to use the public API.
Even not sure if it is feasible by choosing the second approach.
> Remove Jackson classes from public API
> --
>
> Key: AVRO-1605
> URL: https://issues.apache.org/jira/browse/AVRO-1605
> Project: Avro
> Issue Type: Sub-task
> Components: java
>Affects Versions: 1.7.8
>Reporter: Tom White
>Assignee: Gabor Szadovszky
> Fix For: 1.9.0
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AVRO-1605) Remove Jackson classes from public API
[
https://issues.apache.org/jira/browse/AVRO-1605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15725006#comment-15725006
]
Gabor Szadovszky commented on AVRO-1605:
Thanks for the reply, [~rdblue].
I completely agree that we should not commit half-ready changes or changes that
we already know will be removed later.
What I was trying to explain is that I can see the following approaches to get
rid of jackson classes from the public API:
* Rewrite the code base to not to use the jackson json object structure neither
public nor internally. In this case we shall use jackson for only mapping the
json structure to our own object structure. Currently, we have some classes
that use the jackson json object structure quite intensively (e.g. the
generated class {{Idl}}). I don't know how much effort would it require to do
this change neither if it would worth it.
* Keep using the jackson json object structure but only inside the packages.
Between the packages the public API is used with converting the internal
jackson json object representation to the public object types and vice-versa.
Another idea would be to restructure the related classes to have them in the
same package so package private visibility would be used easily. I don't think
it is feasible for the current Avro code base and I am not sure it would be the
best idea as packages used for not only controlling visibility but for adding
some semantic structuring.
* Keep the jackson json object structure used internally and use whatever
techniques to hide them from the public API. One way is the friend packages
suggested by [~tomwhite]. I think it is a completely acceptable approach for
kind of extending the original java visibility possibilities.
I would vote on the latter one.
> Remove Jackson classes from public API
> --
>
> Key: AVRO-1605
> URL: https://issues.apache.org/jira/browse/AVRO-1605
> Project: Avro
> Issue Type: Sub-task
> Components: java
>Affects Versions: 1.7.8
>Reporter: Tom White
>Assignee: Gabor Szadovszky
> Fix For: 1.9.0
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (AVRO-1942) Flaky test: TestNettyServerWithCompression.testConnectionsCount
[
https://issues.apache.org/jira/browse/AVRO-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky resolved AVRO-1942.
Resolution: Duplicate
Just realized that I've created this issue twice by mistake.
> Flaky test: TestNettyServerWithCompression.testConnectionsCount
> ---
>
> Key: AVRO-1942
> URL: https://issues.apache.org/jira/browse/AVRO-1942
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.1
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
>
> One of the unit tests is often failing in our jenkins environment. However, I
> was not able to reproduce it on my own environment; it might be related to
> either the CPU or the network load.
> Stack Trace:
> {code}
> junit.framework.AssertionFailedError: expected:<1> but was:<2>
> at junit.framework.Assert.fail(Assert.java:50)
> at junit.framework.Assert.failNotEquals(Assert.java:287)
> at junit.framework.Assert.assertEquals(Assert.java:67)
> at junit.framework.Assert.assertEquals(Assert.java:199)
> at junit.framework.Assert.assertEquals(Assert.java:205)
> at
> org.apache.avro.ipc.TestNettyServer.testConnectionsCount(TestNettyServer.java:152)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AVRO-1605) Remove Jackson classes from public API
[ https://issues.apache.org/jira/browse/AVRO-1605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15708893#comment-15708893 ] Gabor Szadovszky commented on AVRO-1605: What do you think, [~rdblue]? > Remove Jackson classes from public API > -- > > Key: AVRO-1605 > URL: https://issues.apache.org/jira/browse/AVRO-1605 > Project: Avro > Issue Type: Sub-task > Components: java >Affects Versions: 1.7.8 >Reporter: Tom White >Assignee: Gabor Szadovszky > Fix For: 1.9.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1961) [JAVA] Generate getters that return an Optional
[ https://issues.apache.org/jira/browse/AVRO-1961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15708244#comment-15708244 ] Gabor Szadovszky commented on AVRO-1961: Hi All, Thanks a lot for this issue. I really like using the new features of java8. I don't think it is that urgent to create setters for Optional. Optionals are more used for getting values and not for setting. However, having setters for both seems reasonable. I don't really like the two different flags for having separate getters for Optionals and returning Optionals by the original getters. I think, one flag for having Optionals returned by the original getters or keeping the actual behaviour is enough and less confusing. I don't think we really need both getters in the same time. This way, the default behaviour can also be switched easier in later major releases. What do you think? > [JAVA] Generate getters that return an Optional > --- > > Key: AVRO-1961 > URL: https://issues.apache.org/jira/browse/AVRO-1961 > Project: Avro > Issue Type: New Feature >Reporter: Niels Basjes >Assignee: Niels Basjes >Priority: Minor > > A colleague of mine indicated that having getters that return an Optional > (java 8 thingy) would be very useful for him. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AVRO-1947) Inconsistent maven profile activation
[
https://issues.apache.org/jira/browse/AVRO-1947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated AVRO-1947:
---
Status: Patch Available (was: Open)
> Inconsistent maven profile activation
> -
>
> Key: AVRO-1947
> URL: https://issues.apache.org/jira/browse/AVRO-1947
> Project: Avro
> Issue Type: Bug
> Components: build, java
>Affects Versions: 1.8.1
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
>
> The profiles {{hadoop1}} and {{hadoop2}} can be activated by setting the
> property {{hadoop.version}}. Currently it is implemented in a way that
> {{hadoop2}} is activated by default (if the property is not set) and
> {{hadoop1}} is activated if the property is set to {{1}}. BUT if the property
> is set to any other value (including {{2}}!) none of these profiles will be
> activated.
> Easily reproducible by using the following commands:
> - {{mvn help:active-profiles}}
> - {{mvn help:active-profiles -Dhadoop.version=2}}
> - {{mvn help:active-profiles -Dhadoop.version=1}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (AVRO-1947) Inconsistent maven profile activation
Gabor Szadovszky created AVRO-1947:
--
Summary: Inconsistent maven profile activation
Key: AVRO-1947
URL: https://issues.apache.org/jira/browse/AVRO-1947
Project: Avro
Issue Type: Bug
Components: build, java
Affects Versions: 1.8.1
Reporter: Gabor Szadovszky
Assignee: Gabor Szadovszky
The profiles {{hadoop1}} and {{hadoop2}} can be activated by setting the
property {{hadoop.version}}. Currently it is implemented in a way that
{{hadoop2}} is activated by default (if the property is not set) and
{{hadoop1}} is activated if the property is set to {{1}}. BUT if the property
is set to any other value (including {{2}}!) none of these profiles will be
activated.
Easily reproducible by using the following commands:
- {{mvn help:active-profiles}}
- {{mvn help:active-profiles -Dhadoop.version=2}}
- {{mvn help:active-profiles -Dhadoop.version=1}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AVRO-1605) Remove Jackson classes from public API
[
https://issues.apache.org/jira/browse/AVRO-1605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15628478#comment-15628478
]
Gabor Szadovszky commented on AVRO-1605:
- {{Accessor.defaultValue(Field)}} is used to invoke
{{ResolvingGrammarGenerator.encode(Encoder, Schema, JsonNode)}}. The
implementation of that method uses the jackson API. It might be refactored to
work on the common objects; I just didn't want to introduce regressions here
and let the patch grow too large.
- {{Accessor.createField}}, {{Accessor.addProp}}: These are mainly used by the
generated class {{Idl}}. This class extensively uses the Jackson API and build
structures from these objects. It seems to be a hard work to refactor this
class to not to use these objects.
Meanwhile, I was thinking about the concept of this friend package pattern. I
am not sure I share you opinion that it should be avoided. For me it is more
solving a problem that the java language should. Package private visibility is
simply not enough to handle such issues.
It might be a decision though, to not to use a lib like jackson such an
extensive way but in this case it is not enough to remove those classes from
the public methods. We should not use JsonNode objects even privately but use
jackson only for parsing and writing jsons. In the current situation I am not
sure it would worth the effort.
I'm sure I see things differently than you as I do not have too much xp in the
avro code base. Therefore, I am really curious about your opinion. :)
> Remove Jackson classes from public API
> --
>
> Key: AVRO-1605
> URL: https://issues.apache.org/jira/browse/AVRO-1605
> Project: Avro
> Issue Type: Sub-task
> Components: java
>Affects Versions: 1.7.8
>Reporter: Tom White
>Assignee: Gabor Szadovszky
> Fix For: 1.9.0
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (AVRO-1943) Flaky test: TestNettyServerWithCompression.testConnectionsCount
[
https://issues.apache.org/jira/browse/AVRO-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated AVRO-1943:
---
Status: Patch Available (was: Open)
> Flaky test: TestNettyServerWithCompression.testConnectionsCount
> ---
>
> Key: AVRO-1943
> URL: https://issues.apache.org/jira/browse/AVRO-1943
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.1
>Reporter: Gabor Szadovszky
>Assignee: Gabor Szadovszky
>
> One of the unit tests is often failing in our jenkins environment. However, I
> was not able to reproduce it on my own environment; it might be related to
> either the CPU or the network load.
> Stack Trace:
> {code}
> junit.framework.AssertionFailedError: expected:<1> but was:<2>
> at junit.framework.Assert.fail(Assert.java:50)
> at junit.framework.Assert.failNotEquals(Assert.java:287)
> at junit.framework.Assert.assertEquals(Assert.java:67)
> at junit.framework.Assert.assertEquals(Assert.java:199)
> at junit.framework.Assert.assertEquals(Assert.java:205)
> at
> org.apache.avro.ipc.TestNettyServer.testConnectionsCount(TestNettyServer.java:152)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (AVRO-1943) Flaky test: TestNettyServerWithCompression.testConnectionsCount
Gabor Szadovszky created AVRO-1943:
--
Summary: Flaky test:
TestNettyServerWithCompression.testConnectionsCount
Key: AVRO-1943
URL: https://issues.apache.org/jira/browse/AVRO-1943
Project: Avro
Issue Type: Bug
Components: java
Affects Versions: 1.8.1
Reporter: Gabor Szadovszky
Assignee: Gabor Szadovszky
One of the unit tests is often failing in our jenkins environment. However, I
was not able to reproduce it on my own environment; it might be related to
either the CPU or the network load.
Stack Trace:
{code}
junit.framework.AssertionFailedError: expected:<1> but was:<2>
at junit.framework.Assert.fail(Assert.java:50)
at junit.framework.Assert.failNotEquals(Assert.java:287)
at junit.framework.Assert.assertEquals(Assert.java:67)
at junit.framework.Assert.assertEquals(Assert.java:199)
at junit.framework.Assert.assertEquals(Assert.java:205)
at
org.apache.avro.ipc.TestNettyServer.testConnectionsCount(TestNettyServer.java:152)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (AVRO-1942) Flaky test: TestNettyServerWithCompression.testConnectionsCount
Gabor Szadovszky created AVRO-1942:
--
Summary: Flaky test:
TestNettyServerWithCompression.testConnectionsCount
Key: AVRO-1942
URL: https://issues.apache.org/jira/browse/AVRO-1942
Project: Avro
Issue Type: Bug
Components: java
Affects Versions: 1.8.1
Reporter: Gabor Szadovszky
Assignee: Gabor Szadovszky
One of the unit tests is often failing in our jenkins environment. However, I
was not able to reproduce it on my own environment; it might be related to
either the CPU or the network load.
Stack Trace:
{code}
junit.framework.AssertionFailedError: expected:<1> but was:<2>
at junit.framework.Assert.fail(Assert.java:50)
at junit.framework.Assert.failNotEquals(Assert.java:287)
at junit.framework.Assert.assertEquals(Assert.java:67)
at junit.framework.Assert.assertEquals(Assert.java:199)
at junit.framework.Assert.assertEquals(Assert.java:205)
at
org.apache.avro.ipc.TestNettyServer.testConnectionsCount(TestNettyServer.java:152)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AVRO-1605) Remove Jackson classes from public API
[ https://issues.apache.org/jira/browse/AVRO-1605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15580234#comment-15580234 ] Gabor Szadovszky commented on AVRO-1605: Removed some unnecessary calls of the Accessor. The other parts would require much more effort to eliminate the usage of the Accessor. [~rdblue], could you please check it out? > Remove Jackson classes from public API > -- > > Key: AVRO-1605 > URL: https://issues.apache.org/jira/browse/AVRO-1605 > Project: Avro > Issue Type: Sub-task > Components: java >Affects Versions: 1.7.8 >Reporter: Tom White >Assignee: Gabor Szadovszky > Fix For: 1.9.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
