Re: Formal spec for Avro Schema

problems I see in online questions arise from using raw / internal encodings, which I think is an easy problem to prevent. As to the specification, I think it’s a good start. Some things I really like: the introduction of an Avro Schema Document, limiting top types to a (union of) named type(s

RE: Formal spec for Avro Schema

Hi Martin, I am saying that the specification of the schema is currently entangled with the specification of the serialization framework. Avro Schema is useful and usable even if you never touch the Avro binaries (the framework, an implementation using the spec). I am indeed proposing

Re: Formal spec for Avro Schema

ces of the specification >> >> On Wed, May 15, 2024 at 9:28 AM Clemens Vasters >> wrote: >> >>> I wrote a formal spec for the Avro Schema format. >>> >>> >>> >>> https://gist.github.com/clemensv/498c481965c425b218e

Re: Formal spec for Avro Schema

Hi Clemens, On Wed, May 15, 2024 at 11:18 AM Clemens Vasters wrote: > Hi Martin, > > we find Avro Schema to be a great fit for describing application data > structures in general and even independent of wire-serialization scenarios. > Therefore, I would like to have a sp

RE: Formal spec for Avro Schema

Hi Martin, we find Avro Schema to be a great fit for describing application data structures in general and even independent of wire-serialization scenarios. Therefore, I would like to have a spec that focuses specifically on the schema format, is grounded in the IETF RFC specs, and which

Re: Formal spec for Avro Schema

emens Vasters > wrote: > >> I wrote a formal spec for the Avro Schema format. >> >> >> >> https://gist.github.com/clemensv/498c481965c425b218ee156b38b49333 >> >> >> >> Where would that go in the repo? >> >> >> >> &g

Re: Formal spec for Avro Schema

%2Bversion%2B%2B/Specification - sources of the specification On Wed, May 15, 2024 at 9:28 AM Clemens Vasters wrote: > I wrote a formal spec for the Avro Schema format. > > > > https://gist.github.com/clemensv/498c481965c425b218ee156b38b49333 > > > > Where would that go in th

Formal spec for Avro Schema

I wrote a formal spec for the Avro Schema format. https://gist.github.com/clemensv/498c481965c425b218ee156b38b49333 Where would that go in the repo? [cid:image001.jpg@01DAA6A1.96E35FC0]<http://www.microsoft.com/en-us/news/ImageDetail.aspx?id=4DABA54CBB4D25A9E9905BC59E4A6D44E33694EA> C

Re: Avro schema evolution support in AVRO CPP

ading.com/> >> +852 2108 7352 >> >> Follow us today on our online platforms >> [Facebook]<https://www.facebook.com/eclipsetrading/>[Linked-In]< >> https://www.linkedin.com/company/eclipse-trading>[Instagram]< >> https://www.instagram.com/eclips

Re: Avro schema evolution support in AVRO CPP

n Java as > well). > But, in C++ and in Java, if you generate code for the given Avro schema, > the generated code takes care of the field ordering issue. Similarly, in > both bindings, Generic data structures work properly with the field-order. > If you are using the resolving deco

Re: Avro schema evolution support in AVRO CPP

Out-of-order fields are not handled transparently in C++ if you are manually using the resolving decoder. (It's the same situation in Java as well). But, in C++ and in Java, if you generate code for the given Avro schema, the generated code takes care of the field ordering issue. Similarly

Re: Avro schema evolution support in AVRO CPP

gt; https://www.instagram.com/eclipsetrading> > > From: Vivek Kumar > Sent: Thursday, January 11, 2024 11:07 AM > To: user@avro.apache.org > Subject: Avro schema evolution support in AVRO CPP > > Hi Avro team, > > I am writing this e

Re: Manageable avro schema evolution in Java

Hi Niels, Thanks a lot for sharing. Very interesting talk! Adding thumbs up :) One comment around JSON Schema: in the talk you mention that JSON Schema is still on beta given it is a draft. While JSON Schema is a "draft" from the point of view of IETF, it is considered production-ready and the

Manageable avro schema evolution in Java

sible" demonstrator project that shows how I think this can be done. Note that this is only intended to show a _possible_ way of doing this. https://github.com/nielsbasjes/avro-schema-example Note that the commit history is also part of the demonstration ! I would love to hear your feedback

Re: Unable to get Avro schema alias working for an array item

er to describe in the context of a particular > language. > > > On Sat, Jan 15, 2022 at 8:37 AM Spencer Lu wrote: >> >> Hi everyone, >> >> We have an application that receives Avro data, and it needs to rename >> certain fields in the data before sendin

Re: Unable to get Avro schema alias working for an array item

of a particular language. On Sat, Jan 15, 2022 at 8:37 AM Spencer Lu wrote: > Hi everyone, > > We have an application that receives Avro data, and it needs to rename > certain fields in the data before sending it downstream. The > application is using the following Avro schema t

Unable to get Avro schema alias working for an array item

Hi everyone, We have an application that receives Avro data, and it needs to rename certain fields in the data before sending it downstream. The application is using the following Avro schema to send the data downstream (note that 2 of the fields have aliases defined): { "

Re: Getting ArrayIndexOutOfBoundException when decoding byte-array with new avro schema

avro pojo > (only one new Double field is added with default value 0), I am > getting ArrayIndexOutOfBoundException. > Now I know this might be expected as we are not transferring avro schema > in any header, so the exception might be legit. > But I do not want any down time in my syst

Getting ArrayIndexOutOfBoundException when decoding byte-array with new avro schema

as we are not transferring avro schema in any header, so the exception might be legit. But I do not want any down time in my system and want to deploy my changes with new pojo. Is there any way I can do that. For reference... Our Utility class for encoding/decoding import

Re: Announcement: Avro Schema Viewer

rmat). Below is a screenshot of an example >> deployment. >> >> >> Try it out yourself at: https://bolcom.github.io/avro-schema-viewer >> Learn more and view the source at: >> https://github.com/bolcom/avro-schema-viewer >> >> Regards, >> Mike Junge

Re: Announcement: Avro Schema Viewer

like to introduce you to a tool that we built at bol.com > <http://bol.com/> (largest online retailer of the Netherlands), that can be > used to visualize Avro schemas (in .avsc format). Below is a screenshot of an > example deployment. > > > Try it out yourself at: https://

Re: Avro schema having Map of Records

; within the parent POJO? > > > > Sorry if the questions sound too simple, but I'm too used to work with > Parquet that Avro seems like a shift from time to time :) > > > > El mar., 6 ago. 2019 a las 12:01, Ryan Skraba () > escribió: > >> > >> Hello

Re: Avro schema having Map of Records

arquet that Avro seems like a shift from time to time :) > > El mar., 6 ago. 2019 a las 12:01, Ryan Skraba () escribió: >> >> Hello -- Avro supports a map type: >> https://avro.apache.org/docs/1.9.0/spec.html#Maps >> >> Generating an Avro schema from a JSON example can

Re: Avro schema having Map of Records

that Avro seems like a shift from time to time :) El mar., 6 ago. 2019 a las 12:01, Ryan Skraba () escribió: > Hello -- Avro supports a map type: > https://avro.apache.org/docs/1.9.0/spec.html#Maps > > Generating an Avro schema from a JSON example can be ambiguous since a > JSON ob

Re: Avro schema having Map of Records

Hello -- Avro supports a map type: https://avro.apache.org/docs/1.9.0/spec.html#Maps Generating an Avro schema from a JSON example can be ambiguous since a JSON object can either be converted to a record or a map. You're probably looking for something like this: { "type" : "r

Avro schema having Map of Records

StructField("field1", StringType), StructField("field2", ArrayType(StringType)) ) ) ) ) )), nullable = false) However, in Avro I haven't seen any examples of Maps containing Record type objects... Tried a sample input with an online Avro schema gen

Re: AVRO schema evolution: adding optional column with default fails deserialization

ce in that >>>>>>>> > case I could have just use plain old JSON and deserialize it using >>>>>>>> > jackson having no schema evolution problems at all. If that's >>>>>>>> > correct, I'd rather stick to SpecificD

Re: AVRO schema evolution: adding optional column with default fails deserialization

gt;> > >>>>>>> > thanks! >>>>>>> > Martin. >>>>>>> > >>>>>>> > deserializer: >>>>>>> > >>>>>>> > public static T >>>>>&

Re: AVRO schema evolution: adding optional column with default fails deserialization

gt;>>> > >>>>>> > log.trace("data='{}'", >>>>>> DatatypeConverter.printHexBinary(data)); >>>>>> > >>>>>> > Schema schema = targetType.newInstance().getSchema(); >>>>>> >

Re: AVRO schema evolution: adding optional column with default fails deserialization

: DecoderFactory.get().jsonDecoder(schema, new >>>>> String(data)); >>>>> > >>>>> > T result = targetType.cast(datumReader.read(null, >>>>> decoder)); >>>>> > log.tr

Re: AVRO schema evolution: adding optional column with default fails deserialization

gt; } >>>> > } >>>> > >>>> > serializer: >>>> > public static byte[] serialize(T data, >>>> boolean useBinaryDecoder, boolean pretty) { >>>> > try { >>>> > if (data == null) { >

Re: AVRO schema evolution: adding optional column with default fails deserialization

utStream(); >>> > Encoder binaryEncoder = useBinaryDecoder >>> > ? >>> EncoderFactory.get().binaryEncoder(byteArrayOutputStream, null) >>> > : EncoderFactory.get().jsonEncoder(schema, >>> byteArrayOut

Re: AVRO schema evolution: adding optional column with default fails deserialization

byteArrayOutputStream, pretty); >> > >> > DatumWriter datumWriter = new >> GenericDatumWriter<>(schema); >> > datumWriter.write(data, binaryEncoder); >> > >> > binaryEncoder.flush(); >> > byteArrayOutput

Re: AVRO schema evolution: adding optional column with default fails deserialization

ion( > > "Can't serialize data='" + data, ex); > > } > > } > > > > út 30. 7. 2019 v 13:48 odesílatel Ryan Skraba napsal: > >> > >> Hello! Schema evolution relies on both the writer and reader schemas > >> being a

Re: AVRO schema evolution: adding optional column with default fails deserialization

nd reader schemas >> being available. >> >> It looks like the allegro tool you are using is using the >> GenericDatumReader that assumes the reader and writer schema are the >> same: >> >> https://github.com/allegro/json-avro-converter/blob/json-avro-

Re: AVRO schema evolution: adding optional column with default fails deserialization

nput, just when a field > is known to be in the reader schema but missing from the original > writer. > > You may have more luck reading the GenericRecord with a > GenericDatumReader with both schemas, and using the > `convertToJson(record)`. > > I hope this is useful

Re: AVRO schema evolution: adding optional column with default fails deserialization

e original writer. You may have more luck reading the GenericRecord with a GenericDatumReader with both schemas, and using the `convertToJson(record)`. I hope this is useful -- Ryan On Tue, Jul 30, 2019 at 10:20 AM Martin Mucha wrote: > > Hi, > > I've got some issues/misunders

Validate JSON data against an Avro schema

I want to use Avro to validate data in JSON objects against a schema. My expectation was that the schema validation process covers the following scenarios with appropriate error messages: 1. Required field X is missing in the data. Error message something like "field X not found" 2. Field X has

Re: Avro Schema with Large Number of fields AVRO-1642

Hi All, We have an avro schema that has around 2k fields .When we do not use a nested structure(nested record types) we run into jvm specification issues related to method size and number of parameters .My preference is to keep the avro schema completely flattened without any nested types .How

Avro Schema with Large Number of fields AVRO-1642

Hi All, We have an avro schema that has around 2k fields .When we do not use a nested structure(nested record types) we run into jvm specification issues related to method size and number of parameters .My preference is to keep the avro schema completely flattened without any nested types .How

Re: Avro Schema Question

, 2018 1:58 AM To: user@avro.apache.org Subject: Re: Avro Schema Question A word of caution on the union type. You may find support for unions very patchy if you are hoping to process records using well known data processing engines. We’ve been unable to usefully read union types in both Apache

Re: Avro Schema Question

san...@outlook.com> wrote: > Hi Michael, > > Thanks!! Yes, it does. > > Scott > -- > *From:* Michael Smith <micha...@syapse.com> > *Sent:* Friday, May 25, 2018 2:21 PM > *To:* user@avro.apache.org > *Subject:* Re: Avro Schema Ques

Re: Avro Schema Question

Hi Michael, Thanks!! Yes, it does. Scott From: Michael Smith <micha...@syapse.com> Sent: Friday, May 25, 2018 2:21 PM To: user@avro.apache.org Subject: Re: Avro Schema Question {"type": "int"}, {"type": "string"}

Re: Avro Schema Question

{"type": "int"}, {"type": "string"} is not valid json, so you definitely can't do that. But [{"type": "int"}, {"type": "string"}] is a valid schema -- it can encode a single value that is either an int or a string. At the highest level, your schema can only be one type, but that type may be (and

Avro Schema Question

Hi, I read the specification multiple times. In the specification, it says "A Schema is represented in JSON by one of:" in the Schema Declaration section. The "one" confuses me as I am interpreting it as exactly one of the 3 that it listed. In short, can I do this as a

Modify an existing Avro schema

I'm implementing an application which reads in structured data with an Avro schema applies some dynamically configurable transformations and outputs the data with Avro again. The problem I have is that for some transformations I need to modify the Avro schema. One transform could be for example

Re: Avro schema properties contention on multithread read

/master/spf4j-avro/src/main/java/org/spf4j/avro/schema/Schemas.java#L26 <https://github.com/zolyfarkas/spf4j/blob/master/spf4j-avro/src/main/java/org/spf4j/avro/schema/Schemas.java#L26> But you would have to use it it conjunction with a unsynchronized avro implementation. (which I do in m

Re: Avro schema properties contention on multithread read

On 05.07.2017 21:53, Zoltan Farkas wrote: > The synchronization in JsonProperties is curently inconsistent (see > getObjectProps()) which makes current implementation @NotThreadSafe > > I think it would be probably best to remove synchronization from those > methods... and add @NotThreadSafe

Re: Avro schema properties contention on multithread read

(…) and Schemas.unmodifiableSchema(…) could be added to help with various use cases… —Z > On Jun 29, 2017, at 2:21 AM, f...@legsem.com wrote: > > Hello, > > We are using Avro Schema properties and while running concurrent tests, we > noticed a

Re: Is this a valid Avro schema?

The schemas are fine, but the JSON snippet isn't a valid instance of the second schema. In the default JSON encoding for Avro, you have to include the name of the record as an object field[1]. For example, given test_schema_0.avsc with your first schema and test_schema_1.avsc as your second,

Is this a valid Avro schema?

Hi there, First, please look at the following schema {"name": "user_record", "namespace": "com.user", "type": "record", "fields" : [ {"name": "name", "type": "string"}, {"name": "id", "type": "long"} ]} and the following JSON: {"name": “Foo", “id": 42} When I

Trouble Serializing Record from a multi-Record Avro Schema using C++

I am getting a runtime exception when serializing an Avro record generated from JSON schema containing multiple records in a C++ program. I was able to reproduce this problem with both avrocpp-1.7.7 aas well as avrocpp-1.8.1 (latest stable release). Here is my schema: [ > { > "type" :

How to define byte[] and LocalDateTime in avro schema?

Hi all; Im new to Avro schema. I try to publish/consumer my java objects using kafka. I have java bean classes, which contains fields with LocalDateTime and byte[] . How can I define both in avro schema primitive types?What is the best primitive type i can use for LocalDateTime? private

RE: Want to Add New Column in Avro Schema

Thanks guys. Just updated the .avsc and it’s done. No need to recreated the table again. Regards, Dhaval From: Maulik Gandhi [mailto:mmg...@gmail.com] Sent: Wednesday, March 23, 2016 7:05 PM To: user Cc: er.dcpa...@gmail.com Subject: Re: Want to Add New Column in Avro Schema Create table DDL

Re: Want to Add New Column in Avro Schema

t; *Sent:* Wednesday, March 23, 2016 6:50 PM > *To:* user@avro.apache.org > *Cc:* 'er.dcpa...@gmail.com' > *Subject:* Re: Want to Add New Column in Avro Schema > > > > You shouldn’t have to drop the table, just update the .avsc. Can you > share the DDL you use to create the ta

Re: Want to Add New Column in Avro Schema

t; Cc: "'er.dcpa...@gmail.com'" <er.dcpa...@gmail.com> > Subject: RE: Want to Add New Column in Avro Schema > > Here is the DDL. > > > > DROP TABLE IF EXISTS TEST; > > > > CREATE EXTERNAL TABLE TEST > > PARTITIONED BY (

Re: Want to Add New Column in Avro Schema

..@gmail.com>'" <er.dcpa...@gmail.com<mailto:er.dcpa...@gmail.com>> Subject: RE: Want to Add New Column in Avro Schema Here is the DDL. DROP TABLE IF EXISTS TEST; CREATE EXTERNAL TABLE TEST PARTITIONED BY ( COL1 STRING, COL2 STRING ) ROW FORMAT SERDE 'org.apache.hadoop.hiv

RE: Want to Add New Column in Avro Schema

' Subject: Re: Want to Add New Column in Avro Schema You shouldn't have to drop the table, just update the .avsc. Can you share the DDL you use to create the table? From: "Lunagariya, Dhaval" <dhaval.lunagar...@citi.com<mailto:dhaval.lunagar...@citi.com>> Reply-To: "user@

RE: Want to Add New Column in Avro Schema

6:36 PM To: user@avro.apache.org Cc: 'er.dcpa...@gmail.com' Subject: Re: Want to Add New Column in Avro Schema If you create the external table by reference to the .avsc file (TBLPROPERTIES ( 'avro.schema.url'='hdfs://foo.avsc')) the all you have to do is update that avsc file in a compatible way

Want to Add New Column in Avro Schema

Hey folks, I want to add new column in existing Hive Table. We created external hive table with the help of .avsc. Now I want to add new column in that table. How can I do that without disturbing any data present in table? Please Help. Regards, Dhaval

Re: Avro schema doesn't honor backward compatibilty

o the datum reader knows that the field should be read. > > Does that make sense? > > rb > > On 02/01/2016 12:31 PM, Raghvendra Singh wrote: > >> down votefavorite >> < >> http://stackoverflow.com/questions/34733604/avro-schema-doesnt-honor-backward-compa

Re: Avro schema doesn't honor backward compatibilty

m reader will >>> know that it isn't present and will use your default instead. When you read >>> data encoded with the new schema, you need to use it as the written schema >>> instead so the datum reader knows that the field should be read. >>> >>> Does that ma

Re: Avro schema doesn't honor backward compatibilty

it as the written schema instead so the datum reader knows that the field should be read. Does that make sense? rb On 02/01/2016 12:31 PM, Raghvendra Singh wrote: down votefavorite <http://stackoverflow.com/questions/34733604/avro-schema-doesnt-honor-backward-compatibilty#> I have this avro

Avro schema doesn't honor backward compatibilty

down votefavorite <http://stackoverflow.com/questions/34733604/avro-schema-doesnt-honor-backward-compatibilty#> I have this avro schema { "namespace": "xx..x.x", "type": "record", "name": "MyPayLoad", "field

Re: Avro schema doesn't honor backward compatibilty

"fields" : >> [ >>{"name": "min", "type": "long"}, >> {"name": "max", "type": "long"}, >>

Re: Avro schema doesn't honor backward compatibilty

"fields" : > [ >{"name": "min", "type": "long"}, >{"name": "max", "type": "long"}, > {&quo

RE: add new attributes into avro schema

gous to the JSON case.” makes it look like you can. David Newberger From: John Smith [mailto:lenov...@gmail.com] Sent: Thursday, December 17, 2015 5:13 AM To: user@avro.apache.org Subject: add new attributes into avro schema Hi, is it possible to extend avro schema with custom attributes, for examp

add new attributes into avro schema

Hi, is it possible to extend avro schema with custom attributes, for example { "type":"record", "name":"X", "fields":[ {"name":"b3","type":"int","doc":"blabla",*"newField1"*:"test", *"newField2"*:"test2"} ]}'); Thank you!

Re: Json data to avro schema validation

any idea here? On Tue, Sep 1, 2015 at 1:36 AM, Jeetendra G <jeetendr...@housing.com> wrote: > Hi All > > I have Json data in object and I want to validate that object against > Avro schema .is there an API to validate this? > > > Regards > Jeetendra >

Json data to avro schema validation

Hi All I have Json data in object and I want to validate that object against Avro schema .is there an API to validate this? Regards Jeetendra

RE: Not able to load avro schema fully with all its contents

if it is useful?) Hope this helps. Pierre Date: Tue, 19 May 2015 18:37:56 + From: sgr...@yahoo-inc.com To: user@avro.apache.org Subject: Re: Not able to load avro schema fully with all its contents Just a guess, but I would assume that the schema object only stores fields that it cares about. This would

RE: Not able to load avro schema fully with all its contents

Hello, I posted the patch for the trunk branch of the git there (for it to be reviewed):https://issues.apache.org/jira/browse/AVRO-1256 Pierre From: theped...@hotmail.com To: user@avro.apache.org Subject: RE: Not able to load avro schema fully with all its contents Date: Wed, 20 May 2015 10:08

Re: Not able to load avro schema fully with all its contents

Avro in C++ and I am trying to load avro schema by using Avro C++ library. Everything works fine without any issues, only problem is - I have few doc in my Avro schema which is not getting shown at all in my AvroSchema when I try to load it and also print it out. DataSchema_ptr schema_data

Not able to load avro schema fully with all its contents

I am working with Apache Avro in C++ and I am trying to load avro schema by using Avro C++ library. Everything works fine without any issues, only problem is - I have few doc in my Avro schema which is not getting shown at all in my AvroSchema when I try to load it and also print it out

Re: Not able to load avro schema fully with all its contents

wrote: I am working with Apache Avro in C++ and I am trying to load avro schema by using Avro C++ library. Everything works fine without any issues, only problem is - I have few doc in my Avro schema which is not getting shown at all in my AvroSchema when I try to load it and also print it out

Re: Not able to load avro schema fully with all its contents

()) { os JSON data: ; m_schema.toJson(os); } return os.str(); } On Mon, May 18, 2015 at 12:54 PM, Check Peck comptechge...@gmail.com wrote: I am working with Apache Avro in C++ and I am trying to load avro schema by using Avro C++ library. Everything works fine without any

Issue with reading old data with a new Avro Schema

Hello, I'm trying to read old avro binary data with a new schema (I add a new field). This is the Avro Schema (OLD) I was using to write Avro binary data before: { namespace: com.hello.world, type: record, name: Toto, fields: { name: a, type

Re: Issue with reading old data with a new Avro Schema

The schema is not valid JSON. Maybe you forgot the “[“ after “fields:”? Lukas From: Nicolas Phung Sent: Wednesday, April 8, 2015 9:45 AM To: user@avro.apache.org Subject: Issue with reading old data with a new Avro Schema Hello, I'm trying to read old avro binary data with a new schema (I

Re: Issue with reading old data with a new Avro Schema

, type: string, default: na } ] } Sorry bad copy paste. The Avro Schema should be fine because I'm using sbt-avro to generate the class from it. On Wed, Apr 8, 2015 at 6:57 PM, Lukas Steiblys lu...@doubledutch.me wrote: The schema is not valid JSON. Maybe you forgot

Re: Adding new field with default value to an Avro schema

amount of space. [1]: http://avro.apache.org/docs/current/spec.html#Object+Container+Files On Tue, Feb 3, 2015 at 5:57 AM, Burak Emre emrekaba...@gmail.com wrote: I added a field with a default value to an Avro schema which is previously used for writing data. Is it possible to read the previous

Re: Adding new field with default value to an Avro schema

To: user@avro.apache.org Subject: Re: Adding new field with default value to an Avro schema @Sean thanks for the explanation. I have multiple writers but only one reader and the only schema migration operation is adding a new field so I thought that I may use the same schema for all dataset

Re: Adding new field with default value to an Avro schema

On Tue, Feb 3, 2015 at 11:34 AM, Lukas Steiblys lu...@doubledutch.me wrote: On a related note, is there a tool that can check the backwards compatibility of schemas? I found some old messages talking about it, but no actual tool. I guess I could hack it together using some functions in the

Re: Adding new field with default value to an Avro schema

On Tue, Feb 3, 2015 at 9:34 AM, Lukas Steiblys lu...@doubledutch.me wrote: On a related note, is there a tool that can check the backwards compatibility of schemas? https://avro.apache.org/docs/current/api/java/org/apache/avro/SchemaCompatibility.html Doug

Avro schema and data read with it.

I have a data that is persisted in Avro format. Each record has a certain schema and it contains 10 fields while it is persisted. When I read the same record(s) from other process, i also specify a schema with a subset of fields (5). Will only 5 columns be read from disk? or Will all the columns

Re: Avro schema and data read with it.

Avro skips over fields that were present in the writer's schema but are no longer present in the reader's schema. Skipping is substantially faster than reading for most types. For known-size types like string, bytes, fixed, double and float the file pointer can be incremented past skipped

Is it legal avro schema to have a name tie to different type in different record

Hi All, Is the following legal schema: { { metadata : { schema : { family : search, version : v1, attrs : [ srch ] } } }{ metadata : { schema : { family : UDB, version : v1, attrs : [ login, reg ] } } } } Note the

Map avro schema example in php

Good evening. This is snippet of avro schema field that I need to build data for and populate in php *{name: meta__kvpairs, type: [ null, { type: map, values: [null, string] } ] }*. Would appreciate if someone gives me an example of how data would look in php to satisfy meta__kvpairs avro rule

Re: Map avro schema example in php

Please disregard. On Tue, Sep 9, 2014 at 11:26 PM, Vadim Keylis vkeylis2...@gmail.com wrote: Good evening. This is snippet of avro schema field that I need to build data for and populate in php *{name: meta__kvpairs, type: [ null, { type: map, values: [null, string] } ] }*. Would

Encoding Avro schema as binary

Hi, I wanted to see how much overhead would be involved if we ship an avro schema along with message in a messaging context. Seems like it might simplify things to not always need a schema registry available with all schema versions. I found an old thread ( http://search-hadoop.com/m

Re: Avro schema in Ruby API

Hey Harsh, thanks. I can confirm that the first one works. Let me try the second one. Tomas On Sun, Feb 16, 2014 at 8:07 AM, Harsh J ha...@cloudera.com wrote: Hi, For (1) I believe you could do a Schema.parse meta['avro.schema'] to obtain the schema as an object from the meta entry of the

Avro schema in Ruby API

Hey, I wanted to ask couple of questions. 1) Let's assume I have 2 avro files. I would like to grab schemas of both. Compare them and decide what to do. The only way I found to get to the schema in a reader is through dr = Avro::DataFile::Reader.new(file, Avro::IO::DatumReader.new) dr.meta and

Avro schema

Hi all, When writing Avro schema to the a data file, what will be the expected behavior if the file is used as M/R input. How does the second/third/... splits get the schema (the schema is always written to the first split) ? Thanks, Lior

Re: Avro schema

We read it from the top of the file at start (just the schema bytes) and then initialize the reader. On Thu, Aug 1, 2013 at 8:32 PM, Lior Schachter lior...@gmail.com wrote: Hi all, When writing Avro schema to the a data file, what will be the expected behavior if the file is used as M/R input

Re: Avro schema

...@cloudera.com wrote: We read it from the top of the file at start (just the schema bytes) and then initialize the reader. On Thu, Aug 1, 2013 at 8:32 PM, Lior Schachter lior...@gmail.com wrote: Hi all, When writing Avro schema to the a data file, what will be the expected behavior

Re: Avro Schema to SQL

work depending on what database is involved. Traversing an Avro Schema in Java is done via the Schema API, the Javadoc explains it and there are many examples in the avro source code. The type of schema must be checked, and for each nested type a different decent into its contained types can occur

Re: Avro Schema to SQL

Might be worth looking at Sqoop's source. On 6/19/13 02:31 AM, Avinash Dongre wrote: Is there know tool/framework available to convert Avro Schema into SQL. If now , How Do i iterate over the schema to find out what records, enums are there. I can think of how to achieve this with simple

How to process different types of avro schema

Hi All, In my application I am getting a stream of avro events. This stream contains different types of avro events belonging to different schemas. I was wondering what is the right way to process this data and do analytics on top of this. Can I use hive? I did study the avro serde that could

Re: AvroStorage/Avro Schema Question

ago I started https://issues.apache.org/jira/browse/AVRO-592 but before finishing it AvroStorage was written elsewhere. I don't recall exactly what I did with the schema translation there, but I recall the mapping from an Avro schema to pig tried to hide the nullable wrappers more. In Avro

AVRO Schema Validator

Is there a command line way to validate an AVRO schema?

Re: Exposing a constant in an Avro Schema

Kenworthy adwkenwor...@yahoo.com Sent: Monday, November 14, 2011 9:09 PM Subject: Re: Exposing a constant in an Avro Schema Named types (records, fields, fixed, enum) can store arbitrary user properties attached to the schema ( similar to doc but no special meaning). Do you want this constant

  1   2   >