The input is simply too large to relay to others. A simplified schema is below. I only have INT columns with some null values in them. This is my Pig code snippet:
D= LOAD 'src_locatn' as IS_REPORTED:INT, PROCESSING_STATUS_ID:INT, PROGRAM_ID:INT, AFFINITY_GROUP_ID:INT; X = FILTER D BY (IS_REPORTED is not null) AND (PROCESSING_STATUS_ID is not null) AND (IS_REPORTED==1) AND (PROGRAM_ID==1) AND (PROCESSING_STATUS_ID==2) AND (AFFINITY_GROUP_ID==76); Thanks On Thu, Feb 18, 2016 at 12:59 PM, Chandeep Singh <c...@chandeep.com> wrote: > Any chance you could share a sample record which has NULL’s in it? as well > as your pig script? > > > On Feb 18, 2016, at 8:36 PM, Parth Sawant <parth.sawan...@gmail.com> > wrote: > > > > I had anticipated it would throw a similar error with this suggestion as > > the last one... and it did. My fields are declared as INT, just to > > re-iterate. I don't think they can be compared to regexes. Here is the > > error: > > > > ERROR 1037: > > <file LeadSales.pig, line 19, column 29> Operands of Regex can be > > CharArray only :(Name: Regex Type: null Uid: null) > > > > org.apache.pig.impl.logicalLayer.validators.TypeCheckerException: ERROR > 1037: > > <file LeadSales.pig, line 19, column 29> Operands of Regex can be > > CharArray only :(Name: Regex Type: null Uid: null) > > > > > > > > Thanks. > > > > > > On Thu, Feb 18, 2016 at 5:24 AM, Chandeep Singh <c...@chandeep.com> wrote: > > > >> Since you integers in this field can you try matching to a regular > >> expression? > >> > >> Something like: X matches '\\d+' > >> > >>> On Feb 18, 2016, at 12:55 AM, Parth Sawant <parth.sawan...@gmail.com> > >> wrote: > >>> > >>> Hi Chandeep. I tried that already but it gave me the following error: > >>> > >>> ERROR 1039: > >>> <file LeadSales.pig, line 19, column 27> In alias X, incompatible > >>> types in NotEqual Operator left hand side:int right hand > >>> side:chararray. > >>> > >>> The error makes sense cause the fields I have are INT type and hence > >>> cannot be compared to a chararray. > >>> > >>> > >>> Thanks for the prompt response though. > >>> > >>> > >>> > >>> On Feb 17, 2016 16:32, "Chandeep Singh" <c...@chandeep.com> wrote: > >>> > >>> Try adding != '' along with IS NOT NULL. > >>>> > >>>>> On Feb 18, 2016, at 12:26 AM, Parth Sawant <parth.sawan...@gmail.com > > > >>>> wrote: > >>>>> > >>>>> I'm trying to Filter some null fields in Pig using 'IS NOT NULL' . > For > >>>> some > >>>>> reason the null data values persist. > >>>>> For eg: the following filter on storing it's contents, contains null > >>>> values > >>>>> for ABC and PQR. > >>>>> > >>>>> X = FILTER D BY (ABC IS NOT NULL) AND (ABC==1) AND (PQR==1) AND (PQR > IS > >>>> NOT > >>>>> NULL) ; > >>>>> > >>>>> > >>>>> Can someone help with this? > >>>>> > >>>>> Thanks > >>>>> > >>>>> Parth S > >>>> > >>>> > >> > >> > >
