Hi folks!
On behalf of the Apache Avro PMC I am pleased to announce that Ryan
Skraba has accepted our invitation to become a PMC member. We
appreciate Ryan stepping up to take more responsibility in the
project.
Please join me in welcoming Ryan to the Avro PMC!
As a reminder, if anyone would lik
Please join the dev@avro mailing list if you would like to try out things
prior to the 1.10 release. As an ASF project we must insist that downstream
users not use unreleased code i.e. development SNAPSHOTs.
If folks would rather have a chance to test things out as downstream prior
to a release th
I agree with Zoltan here. We have a really long history of maintaining
compatibility for encoders.
On Tue, Jan 7, 2020 at 10:06 AM Zoltan Farkas wrote:
> Fokko,
>
> I am not sure we should be changing the existing json encoder,
> I think we should just add another encoder, and devs can use eithe
Hi folks!
On behalf of the Apache Avro PMC I am pleased to announce that Nándor
Kollár has accepted our invitation to become a PMC member. We
appreciate Nándor stepping up to take more responsibility in the
project.
Please join me in welcoming Nándor to the Avro PMC!
As a reminder, if anyone wou
Hi folks!
On behalf of the Apache Avro PMC I am pleased to announce that Ismaël
Mejía has accepted our invitation to become a PMC member. We
appreciate Ismaël stepping up to take more responsibility in the
project.
Please join me in welcoming Ismaël to the Avro PMC!
As a reminder, if anyone woul
Hi folks!
On behalf of the Apache Avro PMC I am pleased to announce that Fokko
Driesprong has accepted our invitation to become a PMC member on the
Avro project. We appreciate Fokko stepping up to take more
responsibility in the project.
Please join me in welcoming Fokko to the Avro PMC!
As a
Hi David!
The fastest way to get a release with AVRO-1723 in it is to subscribe
to the dev@avro list and help push forward on getting a release out.
we can discussion options for it on dev@avro. For example, if there
are too many blockers waiting around for 1.8.3 it might make sense to
get a 1.8.2
;
>>> Can do. Thanks for the pointer.
>>>
>>> Edward
>>>
>>> On Wed, Mar 21, 2018 at 4:14 PM, Sean Busbey
>>> wrote:
>>>
>>>> It'd be great to get a new set of releases out. I don't recall what
>>>>
It'd be great to get a new set of releases out. I don't recall what bogged
us down last time we made a go of it.
Would you mind bringing the issue up over on dev@avro? we can figure out
what's missing, volunteers to get those things done, and if needed the PMC
can vote on giving people more powers
maybe use to rewrite the JSON records
with the right field name?
On Wed, Oct 25, 2017 at 11:05 AM, Sean Busbey wrote:
> +us...@nifi.apache.org[1]
>
> Could you can keep the data in Avro and then use Nifi's PutMongoRecord
> processor[2] with an AvroReader to insert?
>
>
>
+us...@nifi.apache.org[1]
Could you can keep the data in Avro and then use Nifi's PutMongoRecord
processor[2] with an AvroReader to insert?
[1]: https://lists.apache.org/list.html?us...@nifi.apache.org
[2]: https://s.apache.org/MmPG
On Wed, Oct 25, 2017 at 7:51 AM, Mike Thomsen
wrote:
> No, i
Hi Folks!
The appropriate place to discuss what gets merged and timing of
not-yet-done releases is the dev@avro list. There's even been a brief
discussion already about the tradeoff between trying to squeeze more
things in to the next 1.8.z release vs working on getting a regular
release cadence.
Your schema says that metadata is a map that has values of a few
different types, but it does not list a map type as one of them.
On Mon, Mar 27, 2017 at 6:11 AM, Dag Stockstad wrote:
> Hi Avro aficionados,
>
> I'm having trouble serializing a record with a nested map structure i.e. a
> map withi
The schemas are fine, but the JSON snippet isn't a valid instance of
the second schema.
In the default JSON encoding for Avro, you have to include the name of
the record as an object field[1].
For example, given test_schema_0.avsc with your first schema and
test_schema_1.avsc as your second, here
thanks!
On Tue, Aug 2, 2016 at 12:28 AM, Ross Black wrote:
> I raised https://issues.apache.org/jira/browse/AVRO-1891.
>
> Thanks.
> Ross
>
> On 2 August 2016 at 01:46, Sean Busbey wrote:
>>
>> That's definitely looks like a bug. Would you mind filing a JIRA f
That's definitely looks like a bug. Would you mind filing a JIRA for this issue?
On Sun, Jul 31, 2016 at 8:34 PM, Ross Black wrote:
> Hi,
>
> I just encountered an issue that I suspect is a bug when using a logical
> type within a union.
> I am using Avro 1.8.1 with JDK8.
>
>
> My schema is:
>
On Mon, May 9, 2016 at 12:21 PM, Koert Kuipers wrote:
> you cannot use avro to ensure the data comes in the format you expect (the
> negative numbers issue). you will have to parse these variations before
> converting to avro.
Unless, of course, you can get the folks sending you data to agree to
Hi folks!
Just a quick heads-up that the ASF JIRA is currently locked down to
counter a spam attack. Unfortunately, this lock down prevents our
normal open-policy that allows anyone with a JIRA account to create,
assign, and comment on issues.
If you are caught up in this, please drop me a note e
could you make a small maven project that reproduces the issue?
On Fri, Mar 25, 2016 at 5:24 PM, Matt Narrell
wrote:
> Avro and avro-maven-plugin 1.7.7
>
> No matter what I do, I'm unable to get this feature to work. I've
> exhausted my Google skills and continue to be unsuccessful. I'm lookin
any way when using the python library for it not to have consume
> all memory on the entire box?
>
> Regards,
>
> WU
>
>
>
> On Tue, Oct 27, 2015 at 2:08 PM, Sean Busbey wrote:
>>
>> It sounds like the file you are reading is malformed. Could you share
>&g
It sounds like the file you are reading is malformed. Could you share
the file or how it was written?
On Tue, Oct 27, 2015 at 1:01 PM, web user wrote:
> I ran this in a vm with much less memory and it immediately failed with a
> memory error:
>
> Traceback (most recent call last):
> File "testa
can the avro-tools jar read the schema from the datafile? Can it read
the entries from the datafile using tojson?
On Mon, Oct 5, 2015 at 6:09 AM, Balaji Vijayan
wrote:
> Windows 8.1, Python 2.7, Avro 1.7.7
>
> Using this avro schema and data in this format I am able to validate the
> data agains
Hi Aditya!
Glad to hear you're interested in getting started in open source generally
and the ASF specifically.
To get more info about how the foundation participates in GSoC, you should
check out the Community Development project:
http://community.apache.org/gsoc.html
Community Development als
Hiya Lan!
You need to use a container file instead of just writing via the datum
writer yourself.
Take a look at the "Getting Started (Java)" section on serialization[1].
The example there uses the GenericDatumWriter, but you ought to be able to
switch it out for your ProtobufDatumWriter.
[1]
Hi folks!
The dev list is working to get our next minor release line, 1.8.z, ready to
ship.
We're looking to modernize our Ruby support, and there is currently a
ticket[1] to drop support for Ruby 1.8. At the moment, the implementation
moves us to Ruby 2.0. We'd like to know how this will impact
t, than it's likely best to manage the schema externally
> to the container.
>
> On Fri, Jun 26, 2015 at 10:11 AM, Sean Busbey wrote:
>
>> Avro Container Files are always splittable[1]. They're the way you will
>> commonly interact with Avro serialized data.
>
Avro Container Files are always splittable[1]. They're the way you will
commonly interact with Avro serialized data.
Data serialized as Avro's binary encoding is not splittable by itself,
because the encoding includes no markers[2]. This may be the source of the
disconnect you're finding in online
The JSON listed is not the form that Avro's json encoder/decoder can
handle. Because the optional fields are unions, Avro's decoder expects you
to first list the type before the values.
Presuming CustomerEmails is an Avro record.
i.e.
{"emails" : { "CustomerEmails" : { "emails": ["a...@a.com", "b
that should be synchronize on the DataFileWriter instance, or whatever
writing object you're using.
On Fri, Mar 13, 2015 at 1:22 PM, Sean Busbey wrote:
> The various Avro writer / readers are not thread safe. You will need to do
> some sort of external synchronization. If the threads
The various Avro writer / readers are not thread safe. You will need to do
some sort of external synchronization. If the threads are in the same JVM,
the easiest way to write from multiple threads safely will be to
synchronize on the DataFileStream instance.
e.g.
synchronized(myDataFileWriter) {
.flush();
>> out.close();
>>
>> Here, record is of type org.apache.avro.generic.GenericData.Record.
>>
>> Thanks and Regards,
>> Arun G
>>
>>
>> On Wed, Feb 11, 2015 at 2:08 PM, Sean Busbey wrote:
>>
>>> On Wed, Feb 11, 2015
On Wed, Feb 11, 2015 at 1:24 AM, Arunasalam G wrote:
>
> Our scenario is we have stored the data with schema added to it.
>
> I would like to make it more simple without bringing the Hbase into
> consideration.
>
> We have an Avro data object which has both data and schema and is
> serialized to
On Tue, Feb 3, 2015 at 11:34 AM, Lukas Steiblys
wrote:
> On a related note, is there a tool that can check the backwards
> compatibility of schemas? I found some old messages talking about it, but
> no actual tool. I guess I could hack it together using some functions in
> the Avro library.
>
>
On Tue, Feb 3, 2015 at 11:01 AM, Burak Emre wrote:
> @Sean thanks for the explanation.
>
> I have multiple writers but only one reader and the only schema migration
> operation is adding a new field so I thought that I may use the same schema
> for all dataset since the ordering will be same in a
Schema evolution in Avro requires access to both the schema used when
writing the data and the desired Schema for reading the data.
Normally, Avro data is stored in some container format (i.e. the one in the
spec[1]) and the parsing library takes care of pulling the schema used when
writing out of
Sounds interesting. Any chance you could put up a patch for a new Encoder
that does json-id? (rather than changing the extant json encoder)
--
Sean
On Nov 25, 2014 1:41 PM, "peter" wrote:
> Hello everyone,
>
> Has anyone given any thought to a json-ld (http://json-ld.org/) encoding
> for Avro?
Hi Lukas!
Yes, this sounds like a bug please file an issue.
On Wed, Oct 8, 2014 at 2:26 PM, Lukas Steiblys wrote:
> I realized now that “default” is a keyword in Java and can’t be used as
> an enum value. The files were generated in python using the python Avro
> library, where “default” is
> [1]
> https://issues.apache.org/jira/secure/attachment/12640815/AVRO-1402.patch
> [2] https://issues.apache.org/jira/browse/AVRO-1402
> [3] https://issues.apache.org/jira/browse/AVRO-1497
>
>
> On Mon, Aug 11, 2014 at 10:36 PM, Sean Busbey wrote:
>
>> AVRO-1402
AVRO-1402 only updated the specification to include Decimal[1].
AVRO-1497 is the ticket for adding an implementation to the java library
and it is still open[2].
HTH.
[1]: http://avro.apache.org/docs/1.7.7/spec.html#Decimal
[2]: https://issues.apache.org/jira/browse/AVRO-1497
On Mon, Aug 11,
Hi Mike!
Yep, the specification calls out that only those types that are named
(records, enums, and fixed) can occur multiple times in a union[1].
[1]: http://avro.apache.org/docs/1.7.7/spec.html#Unions
On Tue, Aug 5, 2014 at 2:28 PM, Michael Pigott <
mpigott.subscripti...@gmail.com> wrote:
>
You could consider using the KiteSDK[1]. It uses Avro schemas to describe
datasets, can infer an avro schema from CSV headers, and includes tools for
importing from CSV to Avro for storage. There's a tool demo that can walk
you through getting things into avro and displaying a plain text version[2]
Can you share the schema? How big is it?
The schema itself is not compressed, so given your small data size it might
be dominating.
On Wed, Jul 9, 2014 at 1:20 AM, Sachin Goyal wrote:
> Hi,
>
> I have been trying to use Avro compression codecs to reduce the size of
> avro-output.
> The Java ob
namespace structure which can be globally
> unified across multiple instruments and receiver platforms into a common
> namespace on the filesystem.
> I have a whole list of other data representation constraints and Avro
> satisfies them all. This one is the only 'issue'.
> Thanks for
Hi Lewis!
Avro namespaces don't allow the characters '/', ':', or '-'. So your
specific example would not work. The allowed characters for a namespace
are defined in the Avro spec[1].
It would help if you could clarify what purpose namespacing serves in the
system.
[1]: http://avro.apache.org/d
On Sat, Apr 5, 2014 at 11:49 AM, Serkan Taş wrote:
> Hi all,
>
> I am faced to a common problem of tr locale settings for java application
> while trying to build dev environment according to
> http://wiki.apache.org/hadoop/EclipseEnvironment.
>
> Here is the error :
>
> [ERROR] Failed to execute
Hi Sachneet!
Can you describe your use case a little?
Far and away the recommended way to use Avro is via one of the container
files. The getting started guide for Java will walk you through writing and
reading via the default container format:
http://avro.apache.org/docs/current/gettingstartedj
Hi Shaq!
Could you describe your use case in more detail?
Generally, HDFS will behave poorly in the face of many small files. Could
you perhaps colocate several data in one file? This will help both with the
relative overhead of the schema and the pressure on the HDFS NameNode.
-Sean
On Mon, M
rote:
>> >> > Does anyone else might have an idea how I can resolve this
>> >> > "namespace-less"
>> >> > Avro schema code generation?
>> >> >
>> >> > Thanks,
>> >> > Vitaly
>> >> >
>> >&
Hi Vitay!
Can you give us a minimal schema and test program that illustrates the
problem you're describing?
--
Sean
On Oct 7, 2013 12:27 PM, "Vitaly Gordon" wrote:
> Hi All,
> I am trying to read Avro data that its schema does not have a namespace.
> The problem is that I cannot compile the cl
+user@pig
On Wed, Sep 25, 2013 at 9:33 AM, Anup wrote:
> > On Sep 24, 2013, at 11:15 PM, Phani wrote:
> >
> > wanted to know if avro files can be read using pig from hdfs.
> >
> > Thanks
> >
> > --
> > phani
>
> Yes. Use AvroStorage().
>
> Sent from my Turing Machine.
>
With slightly more d
Yep. Avro would be great at that (provided your central consumer is Avro
friendly, like a Hadoop system). Make sure that all of your schemas have
default values defined for fields so that schema evolution will be easier
in the future.
On Thu, May 23, 2013 at 4:29 PM, Mark wrote:
> We're thinki
ache.org/jira/browse/AVRO-488
On Mon, Oct 22, 2012 at 5:04 PM, Wolfgang Hoschek
wrote:
> What's the preferred mime type for binary avro and json formatted avro
> files that I should use in an app?
> For example "application/avro+binary", "application/avro+json&q
52 matches
Mail list logo
