[ https://issues.apache.org/jira/browse/PIG-4910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rohini Palaniswamy updated PIG-4910: ------------------------------------ Fix Version/s: (was: 0.17.0) 0.18.0 > Assert wrongly pushed up in optimizer > ------------------------------------- > > Key: PIG-4910 > URL: https://issues.apache.org/jira/browse/PIG-4910 > Project: Pig > Issue Type: Bug > Reporter: Daniel Dai > Assignee: Daniel Dai > Fix For: 0.18.0 > > > The following script fail: > {code} > TEST_DATA = LOAD 'input' USING PigStorage() AS (c1:int); > GR = FOREACH (GROUP TEST_DATA BY c1) GENERATE group as c1, > COUNT_STAR(TEST_DATA) as count, TEST_DATA; > ROWS_WITH_C1_EQUALS_ZERO = FILTER GR BY count > 1L; > ROWS_WITH_C1_EQUALS_ZERO_FLATTENED = FOREACH ROWS_WITH_C1_EQUALS_ZERO > GENERATE FLATTEN($0); > -- Assert shouldn't fail as it should be applied after group by but because > assert is getting pushed to mapper, it is failing. > ASSERT ROWS_WITH_C1_EQUALS_ZERO_FLATTENED BY c1 == 0, 'Should have never seen > this message, assert has a bug.'; > DUMP ROWS_WITH_C1_EQUALS_ZERO_FLATTENED; > {code} > input: > 0 > 0 > 1 > The reason is assert is pushed before FILTER: > {code} > ROWS_WITH_C1_EQUALS_ZERO_FLATTENED: (Name: LOStore Schema: c1#14:int) > | > |---ROWS_WITH_C1_EQUALS_ZERO_FLATTENED: (Name: LOForEach Schema: c1#14:int) > | | > | (Name: LOGenerate[true] Schema: > c1#14:int)ColumnPrune:InputUids=[14]ColumnPrune:OutputUids=[14] > | | | > | | c1:(Name: Project Type: int Uid: 14 Input: 0 Column: (*)) > | | > | |---(Name: LOInnerLoad[0] Schema: c1#14:int) > | > |---ROWS_WITH_C1_EQUALS_ZERO: (Name: LOFilter Schema: > c1#14:int,count#31:long,ROWS_WITH_C1_EQUALS_ZERO_FLATTENED#29:bag{#30:tuple(c1#14:int)}) > | | > | (Name: GreaterThan Type: boolean Uid: 33) > | | > | |---count:(Name: Project Type: long Uid: 31 Input: 0 Column: 1) > | | > | |---(Name: Constant Type: long Uid: 32) > | > |---GR: (Name: LOForEach Schema: > c1#14:int,count#31:long,ROWS_WITH_C1_EQUALS_ZERO_FLATTENED#29:bag{#30:tuple(c1#14:int)}) > | | > | (Name: LOGenerate[false,false,false] Schema: > c1#14:int,count#31:long,ROWS_WITH_C1_EQUALS_ZERO_FLATTENED#29:bag{#30:tuple(c1#14:int)})ColumnPrune:InputUids=[29, > 14]ColumnPrune:OutputUids=[14, 31] > | | | > | | group:(Name: Project Type: int Uid: 14 Input: 0 Column: > (*)) > | | | > | | (Name: UserFunc(org.apache.pig.builtin.COUNT_STAR) Type: > long Uid: 31) > | | | > | | |---ROWS_WITH_C1_EQUALS_ZERO_FLATTENED:(Name: Project > Type: bag Uid: 29 Input: 1 Column: (*)) > | | | > | | ROWS_WITH_C1_EQUALS_ZERO_FLATTENED:(Name: Project Type: > bag Uid: 29 Input: 2 Column: (*)) > | | > | |---(Name: LOInnerLoad[0] Schema: group#14:int) > | | > | |---ROWS_WITH_C1_EQUALS_ZERO_FLATTENED: (Name: LOInnerLoad[1] > Schema: c1#14:int) > | | > | |---ROWS_WITH_C1_EQUALS_ZERO_FLATTENED: (Name: LOInnerLoad[1] > Schema: c1#14:int) > | > |---1-3: (Name: LOCogroup Schema: > group#14:int,ROWS_WITH_C1_EQUALS_ZERO_FLATTENED#29:bag{#44:tuple(c1#14:int)}) > | | > | c1:(Name: Project Type: int Uid: 14 Input: 0 Column: 0) > | > |---ROWS_WITH_C1_EQUALS_ZERO_FLATTENED: (Name: LOFilter > Schema: c1#14:int) > | | > | (Name: UserFunc(org.apache.pig.builtin.Assert) Type: > boolean Uid: 40) > | | > | |---(Name: BinCond Type: boolean Uid: 38) > | | | > | | |---(Name: Equal Type: boolean Uid: 35) > | | | | > | | | |---c1:(Name: Project Type: int Uid: 14 > Input: 0 Column: 0) > | | | | > | | | |---(Name: Constant Type: int Uid: 34) > | | | > | | |---(Name: Constant Type: boolean Uid: 36) > | | | > | | |---(Name: Constant Type: boolean Uid: 37) > | | > | |---(Name: Constant Type: chararray Uid: 39) > | > |---TEST_DATA: (Name: LOForEach Schema: c1#14:int) > | | > | (Name: LOGenerate[false] Schema: > c1#14:int)ColumnPrune:InputUids=[14]ColumnPrune:OutputUids=[14] > | | | > | | (Name: Cast Type: int Uid: 14) > | | | > | | |---c1:(Name: Project Type: bytearray Uid: 14 > Input: 0 Column: (*)) > | | > | |---(Name: LOInnerLoad[0] Schema: c1#14:bytearray) > | > |---TEST_DATA: (Name: LOLoad Schema: > c1#14:bytearray)RequiredFields:null > {code} > Runs fine by turning off PushUpFilter ("-t PushUpFilter"). -- This message was sent by Atlassian JIRA (v6.3.15#6346)