Re: Optional string in avro scheme

2015-05-19 Thread Emre Sevinc 
Hello Dmitry,

Does the following SO answer help you:

 http://stackoverflow.com/a/27499930/236007


Kind regards,
Emre Sevinç


On Fri, May 15, 2015 at 9:44 AM, Dzmitry Hancharou goncharo...@gmail.com
wrote:

 Hello all,

 I want to convert json with optional string field into binary avro file.

 The following scheme is used:
 Optional.avsc:
 {
 namespace: Namespace,
 type: record,
 name: Optional,
 fields: [
 {
 name: opt,
 type: [null, string],
 default: null
 },
 {
 name: req,
 type: string
 }
 ]
 }

 optional.json: {req: xxx}

 When I run java -jar avro-tools-1.7.7.jar fromjson --schema-file
 Optional.avsc optional.json  optional.avro, get exception:
 Exception in thread main org.apache.avro.AvroTypeException: Expected
 field name not found: opt
 at org.apache.avro.io.JsonDecoder.doAction(JsonDecoder.java:477)
 at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
 at org.apache.avro.io.JsonDecoder.advance(JsonDecoder.java:139)
 at org.apache.avro.io.JsonDecoder.readIndex(JsonDecoder.java:429)
 at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:290)
 at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
 at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:267)
 at
 org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:155)
 at
 org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
 at
 org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
 at
 org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
 at
 org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
 at org.apache.avro.tool.DataFileWriteTool.run(DataFileWriteTool.java:99)
 at org.apache.avro.tool.Main.run(Main.java:84)
 at org.apache.avro.tool.Main.main(Main.java:73)

 When I change optional.json to {opt: yyy, req: xxx}, I get
 Exception in thread main org.apache.avro.AvroTypeException: Expected
 start-union. Got VALUE_STRING
 at org.apache.avro.io.JsonDecoder.error(JsonDecoder.java:697)
 at org.apache.avro.io.JsonDecoder.readIndex(JsonDecoder.java:441)
 at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:290)
 at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
 at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:267)
 at
 org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:155)
 at
 org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
 at
 org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
 at
 org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
 at
 org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
 at org.apache.avro.tool.DataFileWriteTool.run(DataFileWriteTool.java:99)
 at org.apache.avro.tool.Main.run(Main.java:84)
 at org.apache.avro.tool.Main.main(Main.java:73)

 What I'm doing wrong?

 Thanks,
 Dmitry





-- 
Emre Sevinc

Re: Optional string in avro scheme

2015-05-19 Thread mani dunga 
Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but
class was expected


how can we solve this.




On Tue, May 19, 2015 at 6:34 AM, Emre Sevinc emre.sev...@gmail.com wrote:

 Hello Dmitry,

 Does the following SO answer help you:

  http://stackoverflow.com/a/27499930/236007


 Kind regards,
 Emre Sevinç


 On Fri, May 15, 2015 at 9:44 AM, Dzmitry Hancharou goncharo...@gmail.com
 wrote:

 Hello all,

 I want to convert json with optional string field into binary avro file.

 The following scheme is used:
 Optional.avsc:
 {
 namespace: Namespace,
 type: record,
 name: Optional,
 fields: [
 {
 name: opt,
 type: [null, string],
 default: null
 },
 {
 name: req,
 type: string
 }
 ]
 }

 optional.json: {req: xxx}

 When I run java -jar avro-tools-1.7.7.jar fromjson --schema-file
 Optional.avsc optional.json  optional.avro, get exception:
 Exception in thread main org.apache.avro.AvroTypeException: Expected
 field name not found: opt
 at org.apache.avro.io.JsonDecoder.doAction(JsonDecoder.java:477)
 at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
 at org.apache.avro.io.JsonDecoder.advance(JsonDecoder.java:139)
 at org.apache.avro.io.JsonDecoder.readIndex(JsonDecoder.java:429)
 at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:290)
 at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
 at
 org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:267)
 at
 org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:155)
 at
 org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
 at
 org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
 at
 org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
 at
 org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
 at org.apache.avro.tool.DataFileWriteTool.run(DataFileWriteTool.java:99)
 at org.apache.avro.tool.Main.run(Main.java:84)
 at org.apache.avro.tool.Main.main(Main.java:73)

 When I change optional.json to {opt: yyy, req: xxx}, I get
 Exception in thread main org.apache.avro.AvroTypeException: Expected
 start-union. Got VALUE_STRING
 at org.apache.avro.io.JsonDecoder.error(JsonDecoder.java:697)
 at org.apache.avro.io.JsonDecoder.readIndex(JsonDecoder.java:441)
 at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:290)
 at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
 at
 org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:267)
 at
 org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:155)
 at
 org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
 at
 org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
 at
 org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
 at
 org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
 at org.apache.avro.tool.DataFileWriteTool.run(DataFileWriteTool.java:99)
 at org.apache.avro.tool.Main.run(Main.java:84)
 at org.apache.avro.tool.Main.main(Main.java:73)

 What I'm doing wrong?

 Thanks,
 Dmitry





 --
 Emre Sevinc




-- 
Regards,

*MANI*

Re: Not able to load avro schema fully with all its contents

2015-05-19 Thread Check Peck 
Can anyone help me with this?

On Mon, May 18, 2015 at 2:04 PM, Check Peck comptechge...@gmail.com wrote:

 Does anyone have any idea on this why it is behaving like this?

 On Mon, May 18, 2015 at 1:03 PM, Check Peck comptechge...@gmail.com
 wrote:

 And this is my to_string method I forgot to provide.

 std::string DataSchema::to_string() const
 {
 ostringstream os;
 if (valid())
 {
 os  JSON data: ;
 m_schema.toJson(os);
 }
 return os.str();

 }


 On Mon, May 18, 2015 at 12:54 PM, Check Peck comptechge...@gmail.com
 wrote:

 I am working with Apache Avro in C++ and I am trying to load avro schema
 by using Avro C++ library. Everything works fine without any issues, only
 problem is - I have few doc in my Avro schema which is not getting shown
 at all in my AvroSchema when I try to load it and also print it out.

 DataSchema_ptr schema_data(new DataSchema());
 schema_data-m_schema = load(avro_schema_file_name.c_str());
 const avro::NodePtr node_data_ptr = schema_data-m_schema.root();
 if (node_data_ptr  node_data_ptr-hasName())
 {
 // is there any problem with this node_data_ptr usage here?
 schema_data-m_name = node_data_ptr-name().fullname().c_str();

 // this line prints out whole AVRO but it doesn't have doc which
 is there in my AVRO
 coutFile String :   schema_data-to_string()  endl;
 }

 Here m_schema is avro::ValidSchema m_schema;

 Can anyone help me with this. In general I don't see my doc which I have
 in Avro Schema getting shown when I print it out.