I’m just looking for one sample record (which has NULL's) and not the entire input so that its easier for me to debug.
> On Feb 18, 2016, at 9:40 PM, Parth Sawant <parth.sawan...@gmail.com> wrote: > > The input is simply too large to relay to others. A simplified schema is > below. I only have INT columns with some null values in them. This is my > Pig code snippet: > > D= LOAD 'src_locatn' as > IS_REPORTED:INT, PROCESSING_STATUS_ID:INT, PROGRAM_ID:INT, > AFFINITY_GROUP_ID:INT; > > X = FILTER D BY (IS_REPORTED is not null) AND (PROCESSING_STATUS_ID is not > null) AND (IS_REPORTED==1) AND (PROGRAM_ID==1) AND > (PROCESSING_STATUS_ID==2) AND (AFFINITY_GROUP_ID==76); > > Thanks > > On Thu, Feb 18, 2016 at 12:59 PM, Chandeep Singh <c...@chandeep.com> wrote: > >> Any chance you could share a sample record which has NULL’s in it? as well >> as your pig script? >> >>> On Feb 18, 2016, at 8:36 PM, Parth Sawant <parth.sawan...@gmail.com> >> wrote: >>> >>> I had anticipated it would throw a similar error with this suggestion as >>> the last one... and it did. My fields are declared as INT, just to >>> re-iterate. I don't think they can be compared to regexes. Here is the >>> error: >>> >>> ERROR 1037: >>> <file LeadSales.pig, line 19, column 29> Operands of Regex can be >>> CharArray only :(Name: Regex Type: null Uid: null) >>> >>> org.apache.pig.impl.logicalLayer.validators.TypeCheckerException: ERROR >> 1037: >>> <file LeadSales.pig, line 19, column 29> Operands of Regex can be >>> CharArray only :(Name: Regex Type: null Uid: null) >>> >>> >>> >>> Thanks. >>> >>> >>> On Thu, Feb 18, 2016 at 5:24 AM, Chandeep Singh <c...@chandeep.com> wrote: >>> >>>> Since you integers in this field can you try matching to a regular >>>> expression? >>>> >>>> Something like: X matches '\\d+' >>>> >>>>> On Feb 18, 2016, at 12:55 AM, Parth Sawant <parth.sawan...@gmail.com> >>>> wrote: >>>>> >>>>> Hi Chandeep. I tried that already but it gave me the following error: >>>>> >>>>> ERROR 1039: >>>>> <file LeadSales.pig, line 19, column 27> In alias X, incompatible >>>>> types in NotEqual Operator left hand side:int right hand >>>>> side:chararray. >>>>> >>>>> The error makes sense cause the fields I have are INT type and hence >>>>> cannot be compared to a chararray. >>>>> >>>>> >>>>> Thanks for the prompt response though. >>>>> >>>>> >>>>> >>>>> On Feb 17, 2016 16:32, "Chandeep Singh" <c...@chandeep.com> wrote: >>>>> >>>>> Try adding != '' along with IS NOT NULL. >>>>>> >>>>>>> On Feb 18, 2016, at 12:26 AM, Parth Sawant <parth.sawan...@gmail.com >>> >>>>>> wrote: >>>>>>> >>>>>>> I'm trying to Filter some null fields in Pig using 'IS NOT NULL' . >> For >>>>>> some >>>>>>> reason the null data values persist. >>>>>>> For eg: the following filter on storing it's contents, contains null >>>>>> values >>>>>>> for ABC and PQR. >>>>>>> >>>>>>> X = FILTER D BY (ABC IS NOT NULL) AND (ABC==1) AND (PQR==1) AND (PQR >> IS >>>>>> NOT >>>>>>> NULL) ; >>>>>>> >>>>>>> >>>>>>> Can someone help with this? >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Parth S >>>>>> >>>>>> >>>> >>>> >> >>
