Hi Will, You're right that PigStorage doesn't handle \n. PigStorage is really a dummy reference implementation of Load/StoreFunc, so I'd not recommend to use it in production. In particular, when you have complex data structures and special characters in data, advanced file formats work far better. Will the built-in ParquetLoader<http://pig.apache.org/docs/r0.12.0/api/org/apache/pig/builtin/ParquetLoader.html>/Storer or AvroStorage<http://pig.apache.org/docs/r0.12.0/api/org/apache/pig/builtin/AvroStorage.html>work for you?
Thanks, Cheolsoo On Wed, Mar 26, 2014 at 10:51 AM, William Oberman <ober...@civicscience.com>wrote: > I was debugging some warnings in a script I had: > FIELD_DISCARDED_TYPE_CONVERSION_FAILED > ACCESSING_NON_EXISTENT_FIELD > > I got it down to basically these two lines: > --foo was stored using PigStorage > foo = LOAD '....' AS (key:chararray, value:map[chararray]); > STORE foo INTO '...'; > > The problem is some of the map values have line feeds (\n) in them, which I > think breaks the PigStorage mind in the load path. > > Bug? Or is it "user error" to allow map values with \n's in them. I mean, > I agree it's weird. But, I didn't expect Pig to have such trouble with > it... > > will >
