Hi Alparslan, On Wed, Feb 19, 2014 at 12:04 PM, <[email protected]> wrote:
> > I think. After Nutch provides this info, a data analysis process (with > help of Pig, for example) can be run over the collected datum. (Google also > saves this kind of info. I know that Alfonso has been a user of Pig over data persisted through Gora so maybe he can help with you on this one. > We can create a new field for this, however this will change the data > model. Generally speaking, as you've indicated, changing the data model should be used as a last resort. Avro schema evolution is something which I do not quite fully understand yet. This article [0] helped a _bit_ as the basics are covered e.g. "Avro requires schemas when data is written or read. Most interesting is that you can use different schemas for serialization and deserialization, and Avro will handle the missing/extra/modified fields.", however in my experience, typically, if the tools you've created to read data from your data set are reading from what is essentially unstructured data e.g. data that has been changed over time with new fields being added to the data model, then you run in to NPE. In reflection however I feel that this was most likely because I was accessing the WebPage object fields directly instead of using Avro to access and deserialize the data. > Instead, we can add the tag info into the metadata map. (We can also add a > prefix to map key to differ the tag content data from other info.) > In short, yes, I find that this is the most efficient method of persisting additional information to WebPage's. It may have some overhead if for example in Cassandra your metadata is persisted into a super column and that you need to unpack, read the contents of the super column, then pack it up again before pushing the data over the wire, however on the other hand this may not be an issue if your metadata is not too large for example. > > What do you think about this? Any comments or suggestions? > > Generally speaking, I think that you're gunning for the right target by saving the overhead of changing your data model, however I am by far no expert using Pig for data analysis on Gora data so I cannot comment too much on that side of the issue. hth Lewis [0] http://blog.cloudera.com/blog/2011/05/three-reasons-why-apache-avro-data-serialization-is-a-good-choice-for-openrtb/ -- *Lewis*
