Hello!
A bit tongue in cheek: the one advantage of the current Avro JSON
encoding is that it drives users rapidly to prefer the binary
encoding! In its current state, Avro isn't really a satisfactory
toolkit for JSON interoperability, while it shines for binary
interoperability. Using JSON with
Severity: low
Affected versions:
- Apache Avro Java SDK before 1.11.3
Description:
When deserializing untrusted or corrupted data, it is possible for a reader to
consume memory beyond the allowed constraints and thus lead to out of memory on
the system.
This issue affects Java applications
The Apache Avro community is pleased to announce the release of Avro 1.11.3!
All signed release artifacts, signatures and verification instructions can
be found here: https://avro.apache.org/releases.html
This is a minor release, specifically addressing known issues with the
1.11.2 release, but
Hello! While Avro doesn't have an official "end-of-life" statement or
policy, there is no active development on the 1.9 or 1.10 branch.
Our current policy is to add major features to the next major release
(1.12.0) while bug fixes, CVEs and minor features will be backported
to the next minor
The Apache Avro community is pleased to announce the release of Avro 1.11.2!
All signed release artifacts, signatures and verification instructions can
be found here: https://avro.apache.org/releases.html
This release addresses ~89 Avro JIRA, including some interesting highlights:
C#
-
Thanks Oscar!
Julien (or anyone else) -- do you think it would be useful to have a
category of "Schema" objects that are mutable for the Java SDK?
Something like:
MutableSchema ms = originalSchema.unlock();
ms.getField("quantity").setProperty("precision", 5);
The Apache Avro community is pleased to announce the release of Avro 1.11.0!
All signed release artifacts, signatures and verification instructions can
be found here: https://avro.apache.org/releases.html
This release includes ~250 Jira issues, including some interesting features:
Some
Hello! I was hoping someone has better news, but I'm afraid there's a
couple of constraints in using interfaces with ReflectData.
My recommendation would be to create a Schema from your actual
concrete implementation, and drop it onto your interface with an
@AvroSchema annotation. It's not
Hello!
Thanks, Dan, for your calm and measured response -- you've given some
excellent advice on how someone can make a positive contribution to
the project and the spec.
Askar, your approach in presenting your specification review should
have been more constructive: it isn't useful to
Hello,
As you note, It currently isn't possible to "pre-shade" Avro, but I
completely understand why you might want to do it! Shading avro is a
common thing to do (see
https://beam.apache.org/documentation/io/built-in/parquet/ for
example).
I guess we _might_ be able to fiddle with the maven
Hello!
This is a known issue in 1.11.0 and before, and has recently been
fixed. There's some discussion at AVRO-2498. The bad news is that
if you're using avro-tools to generate your code, there isn't any
released version yet that contains the fix. Martin sent the link to
the SNAPSHOT
This is really cool news -- it's always really interesting to see
benchmark studies and the trade-offs we make while choosing different
formats. Thanks for sharing!
I'd love to see links to some curated articles and papers on the
website! I created AVRO-3308 if you don't object :D
Ryan
On
Hello!
I realized that I haven't commented on this mailing list thread -- I
made some comments on https://issues.apache.org/jira/browse/AVRO-2175
This looks amazing and we should merge it very soon :D It's not
perfect, but it's really a great improvement and definitely not worst
than the
The Apache Avro community is pleased to announce the release of Avro 1.11.0!
All signed release artifacts, signatures and verification instructions can
be found here: https://avro.apache.org/releases.html
This release includes 120 Jira issues, including some interesting features:
Specification:
mockito/mockito/issues/2341 The regression test could be
>> run on emulators with different versions (or maybe robolectrics) and so
>> errors could be catched.
>>
>> Hope this answers roughly the questions.
>>
>> Thanks,
>>
>>
>>
>> On Wed, Aug 11,
Hello! I'm pretty sure that I've used enums with implementations and
ReflectData successfully, even with old versions of Avro.
It seems to work with 1.9.x+ with the following ReflectDatumWriter
(where datum is an instance of the TestEnum):
Encoder encoder =
Hello! I can reproduce it in Avro 1.10.2, and I think this is a bug.
I raised https://issues.apache.org/jira/browse/AVRO-3091 to track it.
Thanks so much for the full example!
It looks like the workaround is to validate the defId in your own code
before building. Until this is fixed, the
The Apache Avro community is pleased to announce the release of Avro 1.10.2!
All signed release artifacts, signatures and verification instructions can
be found here: https://avro.apache.org/releases.html
This release includes 31 Jira issues, including some interesting features:
C#: AVRO-3005
Hello!
As you noticed, the validate method deliberately ignores the actual schema
of a record datum, and validates the field values by position. It's
answering a slightly different question --> whether the datum (and it's
contents) could fit in the given schema.
For your use case, you might
Please note: I mistakenly sent this same message earlier today with the
wrong subject! It is, in fact, 1.10.1 that was released. My apologies!
The Apache Avro community is pleased to announce the release of Avro 1.10.1!
All signed release artifacts, signatures and verification instructions can
the full
> context but this certainly was not trivial to add DecimalConversion
> manually.
>
> Thanks again for your help.
>
> -B
>
> On Tue, Oct 6, 2020 at 10:56 AM Ryan Skraba wrote:
>
>> Hello! This is a frequent stumbling block for logical types.
>>
>> Y
Hello! This is a frequent stumbling block for logical types.
You should explicitly add the Decimal logical type conversion to the
data model that interprets the Java datum being serialized to your
file, like this:
GenericData model = new GenericData();
model.addLogicalTypeConversion(new
Hello Colin, you've hit one bit of fussiness with the Java SDK... you
can't reuse a Schema.Field object in two Records, because a field
knows its own position in the record[1]. If a field were to belong to
two records at different positions, this method would have an
ambiguous response.
As a
Hello! Thanks for the MCVE -- I could reproduce your symptoms easily!
Even when you're using JSON encoding, you should use a
GenericDatumReader<> to read generic datum.
The Json.ObjectReader sounds correct but is actually for a different
JSON use case (storing any arbitrary JSON snippet in a
Hi,
You've got it right: the DataFileReader and DataFileStream read a
block at a time, and "fileReader.tell()" sits at the sync marker
between blocks while records are being read from the current block.
You're probably aware that DataFileReader is only seekable to block
boundaries.
The entire
Hello! There's a policy for trademarks, service marks, and graphic
logos at https://www.apache.org/foundation/marks/
It sounds like you're using the Apache Avro logo to refer to the
Apache Avro project (see "nominative use" at the first link above),
which is usually OK. There are some additional
It looks like the "scale must be less than precision" rule comes from
Hive requirements[1] (although while searching, this is called into
question elsewhere in Hive[2]). From the design document, the
requirement was specifically to avoid variable (per-row scale):
> For instance, applications
/gems/avro/versions/1.9.2
Thanks to everyone for contributing!
Ryan Skraba
> <https://www.facebook.com/Feedzai/>[image:
> Follow Feedzai on Twitter!] <https://twitter.com/feedzai>[image: Connect
> with Feedzai on LinkedIn!] <https://www.linkedin.com/company/feedzai/>
> <https://feedzai.com/>[image: Feedzai in Forbes Fintech 50!]
> &
Hello!
It's a bit difficult to discover what's going wrong -- I'm not sure that
the code in the image corresponds to the exception you are encountering!
Notably, there's no reference to DataFileStream... Typically, it would be
easier with code as TXT than as PNG!
It is definitely possible to
On Fri, Jan 17, 2020 at 2:22 PM roger peppe wrote:
>
> On Thu, 16 Jan 2020 at 17:21, Ryan Skraba wrote:
>>
>> didn't find anything currently in the avro-tools that uses both
>> reader and writer schemas while deserializing data... It should be a
>> pretty ea
gt;>
>>
>> On Thu, 16 Jan 2020 at 13:57, Ryan Skraba wrote:
>>>
>>> Hello! Is it because you are using brew to install avro-tools? I'm
>>> not entirely familiar with how it packages the command, but using a
>>> direct bash-like solution ins
Hello! Is it because you are using brew to install avro-tools? I'm
not entirely familiar with how it packages the command, but using a
direct bash-like solution instead might solve this problem of mixing
stdout and stderr. This could be the simplest (and right) solution
for piping.
alias
Hello! You might be interested in this short discussion on the dev@
mailing list:
https://lists.apache.org/x/thread.html/dd7a23c303ef045c124050d7eac13356b20551a6a663a79cb8807f41@%3Cdev.avro.apache.org%3E
In short, it appears that the record name is already ignored in
record-to-record matching
gt;
> Austin
>
> On Tue, Dec 17, 2019 at 7:13 AM Michael Burr wrote:
>>
>> unsubscribe
>>
>> On Tue, Dec 17, 2019 at 4:43 AM Driesprong, Fokko
>> wrote:
>>>
>>> Folks,
>>>
>>> The Project Management Committee (PMC) for Apache
Related to the earlier question in the thread: there's one good
starting point for a language-agnostic set of test schemas here:
https://github.com/apache/avro/blob/master/share/test/data/schema-tests.txt#L24
There's a LOT of other schemas scattered throughout the project and
languages, of
I think the spec is OK with it. We've even used it in the Java API
(to refer to a table had been created but had no columns yet). It's
not *extremely* useful even as a starting point to add schema
evolutions, but maybe as a technique for forcing different Parsing
Canonical Forms for otherwise
finitely not a strict as it could be, but I've found it
> useful, and it has lots of room for improvement.
>
> cheers,
> rog.
>
>
>
> On Fri, 6 Dec 2019 at 17:43, Jonah H. Harris wrote:
>>
>> On Fri, Dec 6, 2019 at 12:16 PM Ryan Skraba wrote:
>>>
>
Hello! I had a Java unit test ready to go (looking at default values
for complex types for AVRO-2636), so just reporting back (the easy
work!):
1. In Java, the schema above is parsed without error, but when
attempting to use the default value, it fails with a
NullPointerException (trying to
arquet that Avro seems like a shift from time to time :)
>
> El mar., 6 ago. 2019 a las 12:01, Ryan Skraba () escribió:
>>
>> Hello -- Avro supports a map type:
>> https://avro.apache.org/docs/1.9.0/spec.html#Maps
>>
>> Generating an Avro schema from a JSON example can
Hello -- Avro supports a map type:
https://avro.apache.org/docs/1.9.0/spec.html#Maps
Generating an Avro schema from a JSON example can be ambiguous since a
JSON object can either be converted to a record or a map. You're
probably looking for something like this:
{
"type" : "record",
"name"
t uses int32 as schema Id. This is
>>>>> prepended (+a magic byte) to the binary avro. Thus using the schema
>>>>> registry again you can get the writer schema.
>>>>>
>>>>> /Svante
>>>>>
>>>>> On T
oder(byteArrayOutputStream, null)
> : EncoderFactory.get().jsonEncoder(schema,
> byteArrayOutputStream, pretty);
>
> DatumWriter datumWriter = new
> GenericDatumWriter<>(schema);
> datumWriter.write(data, binaryEncoder);
>
> binaryEnc
Hello! Schema evolution relies on both the writer and reader schemas
being available.
It looks like the allegro tool you are using is using the
GenericDatumReader that assumes the reader and writer schema are the
same:
Hello! I'm not sure I understand your question. Some names are
*required* with a specific format in the Avro specification
(http://avro.apache.org/docs/1.8.2/spec.html#names)
What are you looking to accomplish? I can think of two scenarios that
we've seen in the past: (1) anonymous records
> there.
>
>
> On Tue, Jul 16, 2019 at 11:16 AM Ryan Skraba wrote:
> >
> > Hello! Thanks to the reference to AVRO-1852. It's exactly what I was
> looking for.
> >
> > I agree that Java serialization shouldn't be used for anything
> cross-platform, or (in
ma 15 jul. 2019 om 23:03 schreef Doug Cutting :
> >
> > > I can't think of a reason Schema should not implement Serializable.
> > >
> > > There's actually already an issue & patch for this:
> > >
> > > https://issues.apache.org/jira/browse/AVRO-1852
&g
Hello!
I'm looking for any discussion or reference why the Schema object isn't
serializable -- I'm pretty sure this must have already been discussed (but
the keywords +avro +serializable +schema have MANY results in all the
searches I did: JIRA, stack overflow, mailing list, web)
In particular,
48 matches
Mail list logo
