RE: Not able to load avro schema fully with all its contents
Hello, Sam is right in his previous answer.More precisely, the field doc is read by the Compiler, but not stored at the moment in the Node object. The reason might be that the field doc is optional is the avro specification (see: https://avro.apache.org/docs/1.7.7/spec.html, Complex types). If you want to store the field doc, you'll have to modify the source code yourself to:- create a new member doc in the Node API (Node.hh),- store the doc field in Node as it is read by the Compiler (Compiler.cc),- serialize the field doc in NodeImpl.cc I did a patch for my own use were I store and read fields doc for a NodeRecord, and I serialize fields doc for the root Node of a NodeRecord. You can find it at:the corresponding branch (created for the patch):https://github.com/pidefrem/avro/tree/branch-1.7-specificrecord the corresponding commit for the field doc:https://github.com/pidefrem/avro/commit/795a0805b8ea8d3228bd92a483c9cbb405e11a62 Rem: if you want to serialize all fields doc of a NodeRecord, just change line 195 of NodeImpl.cc fromif (depth == 1 getDoc().size()) { to if (getDoc().size()) { (Maybe my patch could be added in the trunk of the source code if it is useful?) Hope this helps. Pierre Date: Tue, 19 May 2015 18:37:56 + From: sgr...@yahoo-inc.com To: user@avro.apache.org Subject: Re: Not able to load avro schema fully with all its contents Just a guess, but I would assume that the schema object only stores fields that it cares about. This would exclude your docs. If you want to know for sure, the source code is here: https://github.com/apache/avro/tree/trunk/lang/c%2B%2B Sam On Tuesday, May 19, 2015 1:13 PM, Check Peck comptechge...@gmail.com wrote: Can anyone help me with this?On Mon, May 18, 2015 at 2:04 PM, Check Peck comptechge...@gmail.com wrote:Does anyone have any idea on this why it is behaving like this?On Mon, May 18, 2015 at 1:03 PM, Check Peck comptechge...@gmail.com wrote:And this is my to_string method I forgot to provide.std::string DataSchema::to_string() const{ostringstream os;if (valid()){os JSON data: ;m_schema.toJson(os); } return os.str();}On Mon, May 18, 2015 at 12:54 PM, Check Peck comptechge...@gmail.com wrote:I am working with Apache Avro in C++ and I am trying to load avro schema by using Avro C++ library. Everything works fine without any issues, only problem is - I have few doc in my Avro schema which is not getting shown at all in my AvroSchema when I try to load it and also print it out.DataSchema_ptr schema_data(new DataSchema()); schema_data-m_schema = load(avro_schema_file_name.c_str());const avro::NodePtr node_data_ptr = schema_data-m_schema.root();if (node_data_ptr node_data_ptr-hasName()){// is there any problem with this node_data_ptr usage here?schema_data-m_name = node_data_ptr-name().fullname().c_str(); // this line prints out whole AVRO but it doesn't have doc which is there in my AVROcoutFile String : schema_data-to_string() endl;} Here m_schema is avro::ValidSchema m_schema; Can anyone help me with this. In general I don't see my doc which I have in Avro Schema getting shown when I print it out.
RE: Not able to load avro schema fully with all its contents
Hello, I posted the patch for the trunk branch of the git there (for it to be reviewed):https://issues.apache.org/jira/browse/AVRO-1256 Pierre From: theped...@hotmail.com To: user@avro.apache.org Subject: RE: Not able to load avro schema fully with all its contents Date: Wed, 20 May 2015 10:08:22 + Hello, Sam is right in his previous answer.More precisely, the field doc is read by the Compiler, but not stored at the moment in the Node object. The reason might be that the field doc is optional is the avro specification (see: https://avro.apache.org/docs/1.7.7/spec.html, Complex types). If you want to store the field doc, you'll have to modify the source code yourself to:- create a new member doc in the Node API (Node.hh),- store the doc field in Node as it is read by the Compiler (Compiler.cc),- serialize the field doc in NodeImpl.cc I did a patch for my own use were I store and read fields doc for a NodeRecord, and I serialize fields doc for the root Node of a NodeRecord. You can find it at:the corresponding branch (created for the patch):https://github.com/pidefrem/avro/tree/branch-1.7-specificrecord the corresponding commit for the field doc:https://github.com/pidefrem/avro/commit/795a0805b8ea8d3228bd92a483c9cbb405e11a62 Rem: if you want to serialize all fields doc of a NodeRecord, just change line 195 of NodeImpl.cc fromif (depth == 1 getDoc().size()) { to if (getDoc().size()) { (Maybe my patch could be added in the trunk of the source code if it is useful?) Hope this helps. Pierre Date: Tue, 19 May 2015 18:37:56 + From: sgr...@yahoo-inc.com To: user@avro.apache.org Subject: Re: Not able to load avro schema fully with all its contents Just a guess, but I would assume that the schema object only stores fields that it cares about. This would exclude your docs. If you want to know for sure, the source code is here: https://github.com/apache/avro/tree/trunk/lang/c%2B%2B Sam On Tuesday, May 19, 2015 1:13 PM, Check Peck comptechge...@gmail.com wrote: Can anyone help me with this?On Mon, May 18, 2015 at 2:04 PM, Check Peck comptechge...@gmail.com wrote:Does anyone have any idea on this why it is behaving like this?On Mon, May 18, 2015 at 1:03 PM, Check Peck comptechge...@gmail.com wrote:And this is my to_string method I forgot to provide.std::string DataSchema::to_string() const{ostringstream os;if (valid()){os JSON data: ;m_schema.toJson(os); } return os.str();}On Mon, May 18, 2015 at 12:54 PM, Check Peck comptechge...@gmail.com wrote:I am working with Apache Avro in C++ and I am trying to load avro schema by using Avro C++ library. Everything works fine without any issues, only problem is - I have few doc in my Avro schema which is not getting shown at all in my AvroSchema when I try to load it and also print it out.DataSchema_ptr schema_data(new DataSchema()); schema_data-m_schema = load(avro_schema_file_name.c_str());const avro::NodePtr node_data_ptr = schema_data-m_schema.root();if (node_data_ptr node_data_ptr-hasName()){// is there any problem with this node_data_ptr usage here?schema_data-m_name = node_data_ptr-name().fullname().c_str(); // this line prints out whole AVRO but it doesn't have doc which is there in my AVROcoutFile String : schema_data-to_string() endl;} Here m_schema is avro::ValidSchema m_schema; Can anyone help me with this. In general I don't see my doc which I have in Avro Schema getting shown when I print it out.
Schema default values in C++ implementation
Hello, I'm using the avro c++ api (1.7.8) to serialize/deserialize avro files. The schema I want to use is the following: { name: cpx, type: record, fields: [ {name: numbername, type: string}, {name: re, type: double}, {name: im, type: double, default: 0} ] }I try to write programmatically the schema above in c++ in the following way: #include avro/Schema.hh static avro::Schema getMySchema() {avro::RecordSchema node0(cpx); {avro::Schema node1 = avro::StringSchema(); node0.addField(numbername, node1);}{avro::Schema node1 = avro::DoubleSchema();node0.addField(re, node1);}{ avro::Schema node1 = avro::DoubleSchema();node0.addField(im, node1); } return node0;} Then I write the file using something like: ... ... avro::DataFileWriterObjAvro pDfw_m = new avro::DataFileWriterObjAvro(filename_p, *( new avro::ValidSchema(getMySchema()); ... ... const ObjAvro avro_l = ... pDfw_m.write(avro_l); ... The problem is that the schema at the top of the output file created is different tothe input schema. It looks like: { name: cpx, type: record, fields: [ {name: numbername, type: string}, {name: re, type: double}, {name: im, type: double} ] } Is there a way to specify the default values directly in the c++ code above (in the method getMySchema)? (So that the schema of my output file will match the one of my input file.) After a look in the source code, I see where the defaultAttributes are stored for a record node, and what the method .addField does. The method addField adds the name of the leaf to the Node member leafNameAttributes_ and the leaf to the Node member leafAttributes_, but I can't see a simple way to add default values to the NodeRecord member defaultValues (NodeImpl.hh). Thanks in advance. Regards. Pierre