Re: How to extract String from GenericRecorcd as null instead of null string?

[Check Peck]

Hi John,

Thanks for the link. I understood now. So that means we should check for
the object as null and if it is not null then convert back to string? Is
this the right way then?
Or there is any better way by which we can extract all the typed value from
GenericRecord? Below is the question I had long time back but everyone
suggested directly to use "String.valueOf".

http://stackoverflow.com/questions/40792614/how-to-get-typed-value-from-genericrecord

On Fri, Dec 16, 2016 at 6:02 PM, John McClean <jmccl...@gmail.com> wrote:

> This isn't an avro issue, it's to do with your use of 'valueOf'. You need
> to check for null before you call it. More info's here.
>
> http://stackoverflow.com/questions/13577892/what-is-
> difference-between-null-and-null-of-string-valueofstring-object
>
> J
>
> On Fri, Dec 16, 2016 at 5:54 PM, Check Peck <comptechge...@gmail.com>
> wrote:
>
>> I am working with Avro and in my avro schema I have one field like this:
>>
>>  {
>> "name" : "userId",
>> "type" : [ "null", "string" ],
>> "doc" : "some doc"
>>   },
>>
>> This is how I am extracting userId field from GenericRecord:
>>
>> GenericRecord payload = decoder.decode(record.value());
>> String userid = String.valueOf(payload.get("userId"));
>> // sometimes userid comes as null string meaning like this "null"
>> System.out.println(Strings.isNullOrEmpty(userid));
>>
>> And because of that "null" string, my sysout prints out as false.  Is
>> there any way to extract userid as null instead of "null" String?
>>
>> Bcoz when I check for null string it fails and if I have to accommodate
>> this fix, I have to add extra check with ".equals" which I want to avoid if
>> possible? Is there any way?
>>
>>
>

How to extract String from GenericRecorcd as null instead of null string?

[Check Peck]

I am working with Avro and in my avro schema I have one field like this:

 {
"name" : "userId",
"type" : [ "null", "string" ],
"doc" : "some doc"
  },

This is how I am extracting userId field from GenericRecord:

GenericRecord payload = decoder.decode(record.value());
String userid = String.valueOf(payload.get("userId"));
// sometimes userid comes as null string meaning like this "null"
System.out.println(Strings.isNullOrEmpty(userid));

And because of that "null" string, my sysout prints out as false.  Is there
any way to extract userid as null instead of "null" String?

Bcoz when I check for null string it fails and if I have to accommodate
this fix, I have to add extra check with ".equals" which I want to avoid if
possible? Is there any way?

Re: How to get typed value from GenericRecord?

[Check Peck]

Can anyone help me with this?

On Thu, Nov 24, 2016 at 10:29 AM, Check Peck <comptechge...@gmail.com>
wrote:

> I am working with Avro and I have a GenericRecord. I want to extract
> clientId and deviceName from it. In the Avro Schema, clientId is integer
> and deviceName is String. My question is - what is the recommended way to
> retrieve a typed value, as opposed to an Object or extracting everything as
> a String and then converting it back to original data type?
>
> In the below code, payload is GenericRecord. This is what I am doing
> right now, extracting everything as a String. But how can I just get typed
> value instead. Is there any way? I mean whatever the data type is there in
> the avro schema, I want to extract that only.
>
> String id = String.valueOf(payload.get("clientId"));String name = 
> String.valueOf(payload.get("deviceName"));
>
>

How to get typed value from GenericRecord?

[Check Peck]

I am working with Avro and I have a GenericRecord. I want to extract
clientId and deviceName from it. In the Avro Schema, clientId is integer
and deviceName is String. My question is - what is the recommended way to
retrieve a typed value, as opposed to an Object or extracting everything as
a String and then converting it back to original data type?

In the below code, payload is GenericRecord. This is what I am doing right
now, extracting everything as a String. But how can I just get typed value
instead. Is there any way? I mean whatever the data type is there in the
avro schema, I want to extract that only.

String id = String.valueOf(payload.get("clientId"));String name =
String.valueOf(payload.get("deviceName"));

Re: Not able to load avro schema fully with all its contents

[Check Peck]

Can anyone help me with this?

On Mon, May 18, 2015 at 2:04 PM, Check Peck comptechge...@gmail.com wrote:

 Does anyone have any idea on this why it is behaving like this?

 On Mon, May 18, 2015 at 1:03 PM, Check Peck comptechge...@gmail.com
 wrote:

 And this is my to_string method I forgot to provide.

 std::string DataSchema::to_string() const
 {
 ostringstream os;
 if (valid())
 {
 os  JSON data: ;
 m_schema.toJson(os);
 }
 return os.str();

 }


 On Mon, May 18, 2015 at 12:54 PM, Check Peck comptechge...@gmail.com
 wrote:

 I am working with Apache Avro in C++ and I am trying to load avro schema
 by using Avro C++ library. Everything works fine without any issues, only
 problem is - I have few doc in my Avro schema which is not getting shown
 at all in my AvroSchema when I try to load it and also print it out.

 DataSchema_ptr schema_data(new DataSchema());
 schema_data-m_schema = load(avro_schema_file_name.c_str());
 const avro::NodePtr node_data_ptr = schema_data-m_schema.root();
 if (node_data_ptr  node_data_ptr-hasName())
 {
 // is there any problem with this node_data_ptr usage here?
 schema_data-m_name = node_data_ptr-name().fullname().c_str();

 // this line prints out whole AVRO but it doesn't have doc which
 is there in my AVRO
 coutFile String :   schema_data-to_string()  endl;
 }

 Here m_schema is avro::ValidSchema m_schema;

 Can anyone help me with this. In general I don't see my doc which I have
 in Avro Schema getting shown when I print it out.

Not able to load avro schema fully with all its contents

[Check Peck]

I am working with Apache Avro in C++ and I am trying to load avro schema by
using Avro C++ library. Everything works fine without any issues, only
problem is - I have few doc in my Avro schema which is not getting shown
at all in my AvroSchema when I try to load it and also print it out.

DataSchema_ptr schema_data(new DataSchema());
schema_data-m_schema = load(avro_schema_file_name.c_str());
const avro::NodePtr node_data_ptr = schema_data-m_schema.root();
if (node_data_ptr  node_data_ptr-hasName())
{
// is there any problem with this node_data_ptr usage here?
schema_data-m_name = node_data_ptr-name().fullname().c_str();

// this line prints out whole AVRO but it doesn't have doc which is
there in my AVRO
coutFile String :   schema_data-to_string()  endl;
}

Here m_schema is avro::ValidSchema m_schema;

Can anyone help me with this. In general I don't see my doc which I have in
Avro Schema getting shown when I print it out.

Re: Not able to load avro schema fully with all its contents

[Check Peck]

And this is my to_string method I forgot to provide.

std::string DataSchema::to_string() const
{
ostringstream os;
if (valid())
{
os  JSON data: ;
m_schema.toJson(os);
}
return os.str();
}


On Mon, May 18, 2015 at 12:54 PM, Check Peck comptechge...@gmail.com
wrote:

 I am working with Apache Avro in C++ and I am trying to load avro schema
 by using Avro C++ library. Everything works fine without any issues, only
 problem is - I have few doc in my Avro schema which is not getting shown
 at all in my AvroSchema when I try to load it and also print it out.

 DataSchema_ptr schema_data(new DataSchema());
 schema_data-m_schema = load(avro_schema_file_name.c_str());
 const avro::NodePtr node_data_ptr = schema_data-m_schema.root();
 if (node_data_ptr  node_data_ptr-hasName())
 {
 // is there any problem with this node_data_ptr usage here?
 schema_data-m_name = node_data_ptr-name().fullname().c_str();

 // this line prints out whole AVRO but it doesn't have doc which
 is there in my AVRO
 coutFile String :   schema_data-to_string()  endl;
 }

 Here m_schema is avro::ValidSchema m_schema;

 Can anyone help me with this. In general I don't see my doc which I have
 in Avro Schema getting shown when I print it out.

Re: Not able to load avro schema fully with all its contents

[Check Peck]

Does anyone have any idea on this why it is behaving like this?

On Mon, May 18, 2015 at 1:03 PM, Check Peck comptechge...@gmail.com wrote:

 And this is my to_string method I forgot to provide.

 std::string DataSchema::to_string() const
 {
 ostringstream os;
 if (valid())
 {
 os  JSON data: ;
 m_schema.toJson(os);
 }
 return os.str();

 }


 On Mon, May 18, 2015 at 12:54 PM, Check Peck comptechge...@gmail.com
 wrote:

 I am working with Apache Avro in C++ and I am trying to load avro schema
 by using Avro C++ library. Everything works fine without any issues, only
 problem is - I have few doc in my Avro schema which is not getting shown
 at all in my AvroSchema when I try to load it and also print it out.

 DataSchema_ptr schema_data(new DataSchema());
 schema_data-m_schema = load(avro_schema_file_name.c_str());
 const avro::NodePtr node_data_ptr = schema_data-m_schema.root();
 if (node_data_ptr  node_data_ptr-hasName())
 {
 // is there any problem with this node_data_ptr usage here?
 schema_data-m_name = node_data_ptr-name().fullname().c_str();

 // this line prints out whole AVRO but it doesn't have doc which
 is there in my AVRO
 coutFile String :   schema_data-to_string()  endl;
 }

 Here m_schema is avro::ValidSchema m_schema;

 Can anyone help me with this. In general I don't see my doc which I have
 in Avro Schema getting shown when I print it out.

Avro Encoding Options?

[Check Peck]

If we want to encode the data then we have two options as of now.

   - A binary encoder, which encodes into binary Avro data
   avro::binaryEncoder()
   
http://avro.apache.org/docs/current/api/cpp/html/namespaceavro.html#ad0158bd2fc76615b68db68d7e4f7c4f6
   - A JSON encoder, which encodes into JSON Avro data avro::jsonEncoder()
   
http://avro.apache.org/docs/current/api/cpp/html/namespaceavro.html#a0847ef62f42f6f0d0af28da9f7c7cf15

Do we have XML Encoding option as well? I may be asking silly question here
but just wanted to make sure do we have XML Encoding option or plan to have
it in future version?

Re: How to change a particular field value after decoding the original byte array?

[Check Peck]

Yes that's what I am looking for. Question is - How do I update the
GenericRecord, I want to update user_id field with some new random long
number.?

Can you provide an example how can I update GenericRecord and then
serialize it to a new Byte Array?

On Mon, Jan 12, 2015 at 4:34 PM, Joey Echeverria j...@cloudera.com wrote:

 I'm not sure I understand the question. You can certainly deserialize
 the byte array back into a GenericRecord. You can then update that
 GenericRecord and serialize that to a new byte array.

 Is that what you're looking for?

 -Joey

 On Mon, Jan 12, 2015 at 3:42 PM, Check Peck comptechge...@gmail.com
 wrote:
  I have an Avro Schema which is like this -
 
  {
 type:record,
 name:new_user,
 namespace:com.hello,
 fields:[
{
   name:user_id,
   type:[
  long,
  null
   ]
},
{
   name:segment,
   type:[
  string,
  null
   ]
}
 ]
  }
 
  I am using my above Avro Schema like this to serialize the data and which
  gives me a Byte Array and works fine -
 
  public static void main(String[] args) throws IOException {
  Schema schema = new Parser()
  .parse({ \type\:\record\, \name\:\new_user\,
  \namespace\:\com.hello\, \fields\:[ { \name\:\user_id\,
 \type\:[
  \long\, \null\ ] }, { \name\:\segment\, \type\:[ \string\,
  \null\ ] } ] });
 
  byte[] originalAvrodata = getAvroBinaryData(schema);
 
  // how to get newAvroData byte array in which user_id
  // is change to some other random long number?
  }
 
  private static byte[] getAvroBinaryData(Schema schema) throws
  IOException {
  GenericRecord record = new GenericData.Record(schema);
  record.put(user_id, 123456L);
  record.put(segment, hello);
 
  GenericDatumWriterGenericRecord writer = new
  GenericDatumWriterGenericRecord(schema);
  ByteArrayOutputStream os = new ByteArrayOutputStream();
 
  Encoder e = EncoderFactory.get().binaryEncoder(os, null);
 
  writer.write(record, e);
  e.flush();
  byte[] byteData = os.toByteArray();
  return byteData;
  }
 
 
 
  I need to decode the `originalAvrodata` byte array and then change the
  `user_id` field value to some other `long` number and then construct a
  `newAvroData` byte array using the same schema which should have
 `user_id`
  field value to some random `long` number. Is this possible to do by any
  chance using Avro?



 --
 Joey Echeverria