Large number of FILTER matching clauses causes internal error 2998
------------------------------------------------------------------
Key: PIG-1475
URL: https://issues.apache.org/jira/browse/PIG-1475
Project: Pig
Issue Type: Bug
Components: grunt
Affects Versions: 0.6.0
Environment: Amazon Elastic MapReduce with Pig 0.6 and Hadoop 0.20.2
Reporter: Mike Subelsky
Priority: Minor
I'm generating reports using Pig where I only want to report on rows matching a
set of regular expressions, but those regular expressions are pretty numerous.
Pig fails with this internal error when I run FILTER with 500 terms:
2010-06-29 15:46:04,579 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
2998: Unhandled internal error. null
This only occurred when I ran my Pig script against Hadoop with the full
dataset. When I ran Pig in local mode, with a smaller sample file, Pig handled
the FILTER command just fine.
The workaround has been to split my list into two separate lists of 250 then
UNION the results, but I assume this is something that could be addressed in
the code.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
