Document BinStorage behaviour
------------------------------
Key: PIG-1711
URL: https://issues.apache.org/jira/browse/PIG-1711
Project: Pig
Issue Type: Bug
Components: documentation
Affects Versions: 0.7.0, 0.6.0
Reporter: Viraj Bhat
Fix For: 0.9.0
We need to document some features of BinStorage that can cause indeterminate
results.
I have a Pig script of this type:
{code}
raw = load 'sampledata' using BinStorage() as (col1,col2, col3);
--filter out null columns
A = filter raw by col1#'bcookie' is not null;
B = foreach A generate col1#'bcookie' as reqcolumn;
describe B;
--B: {regcolumn: bytearray}
X = limit B 5;
dump X;
B = foreach A generate (chararray)col1#'bcookie' as convertedcol;
describe B;
--B: {convertedcol: chararray}
X = limit B 5;
dump X;
{code}
The first dump produces:
(36co9b55onr8s)
(36co9b55onr8s)
(36hilul5oo1q1)
(36hilul5oo1q1)
(36l4cj15ooa8a)
The second dump produces:
()
()
()
()
()
So we need to write correct documentation on why this happens. One good
explanation seems to be:
According to Alan:
BinStorage should not track data lineage. In the case where Pig is using
BinStorage (or whatever) for moving data between MR jobs then Pig can figure
out the correct cast function to use and apply it. For cases such as the one
here where users are storing data using BinStorage and then in a separate Pig
Latin script reading it (and thus loosing the type information) it is the users
responsibility to correctly cast the data before storing it in BinStorage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
