Toivo, At a framework level NiFi itself is format/schema agnostic. It holds maps of strings (attributes) and a chunk of zero or more bytes (payload) on a per flowfile basis. Those bytes could be anything. In looking at the majority of processors they exist to deal with protocols of exchange between systems and generally are also type agnostic. In looking at others we see things that work with JSON, XML, Avro, CSV, text files, etc.. These processors inherently are meant to deal with those formats.
Now, with the inclusion of processors that pull from and send to databases we've hit the need for a way to serialize that data while it lives in NiFi. We also need these for things like writing to Accumulo or HBase and presumably other systems as well. These are systems for which the 'structure' of the data is in many ways controlled by their model (mutations, rows, etc...) So, for these I definitely see what you mean about us centering on some recommended and fully tooled formats. The devil of course is always in the schema details. When pulling from Solr, or a database, or pushing to accumulo or hbase or other such systems I do think we can/should find a standard. Joe On Sat, Jan 2, 2016 at 5:03 AM, Toivo Adams <toivo.ad...@gmail.com> wrote: > I was occupied with other things lately and didn't have time to deal with > this. > > In my opinion clear statement which formats are recommended by NiFi helps a > lot newcomers and processor writers. Also it helps to create Lego like > pieces which fits together without incompatibility worries. > > In ideal world I'd like have one or two “correct data exchange format”. > But reality is different. > Different users have different needs and habits, so there is a need for many > formats. > > I prefer to have few 'recommended core formats' which processors should > support in the way or the other. > And Convert processors for all other formats. > > But I am not sure how to reach consensus. > > > Thanks > Toivo > > > > > -- > View this message in context: > http://apache-nifi-developer-list.39713.n7.nabble.com/Common-data-exchange-formats-and-tabular-data-tp3508p6009.html > Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
