RE: Custom Slicer

2011-03-23 Thread Lai Will
ache.org Cc: Lai Will Subject: Re: Custom Slicer Slicers are deprecated -- Pig now uses Hadoop InputFormats directly; you can read up what those entail in Hadoop documentation and books. As far as dealing with partial records at the beginning and end of the slice, the normal pattern is to al

Re: Custom Slicer

2011-03-01 Thread Lai Will
Ok, thanks.. i'll definitely keep that in mind. As i'm still using pig 0.6.0 because the old code base i'm working on was using it. Ah so does the behavior you mentioned also apply to 0.6.0 then i would have no issue here:) I was just thinking that as different chunks get processed in different

Re: Custom Slicer

2011-03-01 Thread Dmitriy Ryaboy
Slicers are deprecated -- Pig now uses Hadoop InputFormats directly; you can read up what those entail in Hadoop documentation and books. As far as dealing with partial records at the beginning and end of the slice, the normal pattern is to always read a full record even if it takes you past the c