[jira] Commented: (PIG-909) Allow Pig executable to use hadoop jars not bundled with pig
[
https://issues.apache.org/jira/browse/PIG-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12739351#action_12739351
]
Daniel Dai commented on PIG-909:
Hi, Dmitriy
Sorry if my previous comments confuse you. What I mean is pig script is
generating the following command line:
java -Xmx1000m -Dpig.log.dir=$PIG_HOME/bin/../logs -Dpig.log.file=pig.log
-Dpig.home.dir=$PIG_HOME/bin/.. -Dpig.root.logger=INFO,console,DRFA -classpath
$PIG_HOME/bin/../conf:/usr/java/default//lib/tools.jar:$PIG_HOME/bin/../build/classes:$PIG_HOME/bin/../build/test/classes:$PIG_HOME/bin/../pig-*-core.jar:$PIG_HOME/bin/../build/pig-*-core.jar:$PIG_HOME/bin/../lib/hadoop18.jar:$PIG_HOME/bin/../lib/hbase-0.18.1.jar:$PIG_HOME/bin/../lib/hbase-0.18.1-test.jar
org.apache.pig.Main
-classpath does not include all libraries required by Pig, so just run the
scripts will generate error. It does not related to your patch.
Allow Pig executable to use hadoop jars not bundled with pig
Key: PIG-909
URL: https://issues.apache.org/jira/browse/PIG-909
Project: Pig
Issue Type: Improvement
Reporter: Dmitriy V. Ryaboy
Priority: Minor
Attachments: pig_909.patch
The current pig executable (bin/pig) looks for a file named
hadoop${PIG_HADOOP_VERSION}.jar that comes bundled with Pig.
The proposed change will allow Pig to look in $HADOOP_HOME for the hadoop
jars, if that variable is set.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-905) TOKENIZE throws exception on null data
[ https://issues.apache.org/jira/browse/PIG-905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated PIG-905: --- Attachment: PIG-905-1.patch Include null handling suggested by Olga. For Jeff's comment on DataByteArray, it is because TOKENIZE do not declare getArgToFuncMapping. If UDF declare getArgToFuncMapping, Pig can handle type cast correctly. I also include getArgToFuncMapping in the patch. TOKENIZE throws exception on null data -- Key: PIG-905 URL: https://issues.apache.org/jira/browse/PIG-905 Project: Pig Issue Type: Bug Reporter: Olga Natkovich Attachments: PIG-905-1.patch it should just return null -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-901) InputSplit (SliceWrapper) created by Pig is big in size due to serialized PigContext
[ https://issues.apache.org/jira/browse/PIG-901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan updated PIG-901: --- Status: Open (was: Patch Available) resubmitting to hudson patch queue InputSplit (SliceWrapper) created by Pig is big in size due to serialized PigContext Key: PIG-901 URL: https://issues.apache.org/jira/browse/PIG-901 Project: Pig Issue Type: Bug Affects Versions: 0.3.1 Reporter: Pradeep Kamath Assignee: Pradeep Kamath Fix For: 0.4.0 Attachments: PIG-901-1.patch, PIG-901-branch-0.3.patch, PIG-901-trunk.patch InputSplit (SliceWrapper) created by Pig is big in size due to serialized PigContext. SliceWrapper only needs ExecType - so the entire PigContext should not be serialized and only the ExecType should be serialized. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-901) InputSplit (SliceWrapper) created by Pig is big in size due to serialized PigContext
[ https://issues.apache.org/jira/browse/PIG-901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan updated PIG-901: --- Status: Patch Available (was: Open) InputSplit (SliceWrapper) created by Pig is big in size due to serialized PigContext Key: PIG-901 URL: https://issues.apache.org/jira/browse/PIG-901 Project: Pig Issue Type: Bug Affects Versions: 0.3.1 Reporter: Pradeep Kamath Assignee: Pradeep Kamath Fix For: 0.4.0 Attachments: PIG-901-1.patch, PIG-901-branch-0.3.patch, PIG-901-trunk.patch InputSplit (SliceWrapper) created by Pig is big in size due to serialized PigContext. SliceWrapper only needs ExecType - so the entire PigContext should not be serialized and only the ExecType should be serialized. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
pig patch test builds
Sendmail issue on hudson.zones is fixed and the pig patch build is restarted. Thanks, -Giri
RE: pig patch test builds
If you have changed the jira status to patch available in the last couple of days please resubmit your patch for hudson to pick your patch for testing. -Giri From: Giridharan Kesavan Sent: Wednesday, August 05, 2009 7:31 PM To: 'pig-dev@hadoop.apache.org' Cc: Nigel Daley; Olga Natkovich Subject: pig patch test builds Sendmail issue on hudson.zones is fixed and the pig patch build is restarted. Thanks, -Giri
[jira] Updated: (PIG-893) support cast of chararray to other simple types
[ https://issues.apache.org/jira/browse/PIG-893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated PIG-893: --- Status: Open (was: Patch Available) support cast of chararray to other simple types --- Key: PIG-893 URL: https://issues.apache.org/jira/browse/PIG-893 Project: Pig Issue Type: New Feature Affects Versions: 0.4.0 Reporter: Thejas M Nair Assignee: Jeff Zhang Fix For: 0.4.0 Attachments: Pig_893_Patch.txt Pig should support casting of chararray to integer,long,float,double,bytearray. If the conversion fails for reasons such as overflow, cast should return null and log a warning. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-893) support cast of chararray to other simple types
[ https://issues.apache.org/jira/browse/PIG-893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated PIG-893: --- Status: Patch Available (was: Open) support cast of chararray to other simple types --- Key: PIG-893 URL: https://issues.apache.org/jira/browse/PIG-893 Project: Pig Issue Type: New Feature Affects Versions: 0.4.0 Reporter: Thejas M Nair Assignee: Jeff Zhang Fix For: 0.4.0 Attachments: Pig_893_Patch.txt Pig should support casting of chararray to integer,long,float,double,bytearray. If the conversion fails for reasons such as overflow, cast should return null and log a warning. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-905) TOKENIZE throws exception on null data
[ https://issues.apache.org/jira/browse/PIG-905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12739527#action_12739527 ] Olga Natkovich commented on PIG-905: I think we should also check for null or empty input tuple and return null in those cases as well. Other than that the patch looks good. TOKENIZE throws exception on null data -- Key: PIG-905 URL: https://issues.apache.org/jira/browse/PIG-905 Project: Pig Issue Type: Bug Reporter: Olga Natkovich Attachments: PIG-905-1.patch it should just return null -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-893) support cast of chararray to other simple types
[ https://issues.apache.org/jira/browse/PIG-893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12739568#action_12739568 ] Dmitriy V. Ryaboy commented on PIG-893: --- Jeff, Thanks for the contribution! Just a few comments: 0) could you name your patch files *.patch? That makes them easier to review, as the proper highlighting mode is chosen. 1) Other class names in the utils package imply that the class name for this should be CastUtils 2) Spacing in POCast.java is a bit messed up. Please make sure all spacing is to project conventions 3) In TestSchema -- Numberic isn't a word, you mean Numeric (no b) 4) I am not sure about naming the methods chararrayTo . Since they take String as an argument, being in Java-land, I think it would be more straightforward to say stringToxxx . 5) Implementation of the casts -- you call str.toBytes(), and hand off to bytesToXXX method. That method, in turn, converts bytes back into a string, and proceeds to do the conversion. That seems like redundant work. Wouldn't it be better to have stringToXXX peform the conversion, and have bytesToXXX covert to string, then call the stringToXXX method? 6) TestCharArray2Numeric.java -- the convention is to spell out To instead of using the number 2 7) The tests in TestCharArray2Numeric look very similar to each other. Could you pull out the common functionality so the code is only repeated once? About the tests themselves: Since you are just testing conversions, this can be a straightforward unit test -- make a few strings, assert that they convert to the expected value. Hit the edge cases (overflows, special cases for parsing, etc). We don't need to spin up a whole Pig query. 8) I don't like testing random values, as this creates tests that might sometimes pass, and sometimes not. Recommend using known data for reproducible test results. 9) You extracted functionality from Utf8StorageConverter by duplicating the code; I would prefer to see Utf8StorageConverter modified to hand off conversions to CastUtils support cast of chararray to other simple types --- Key: PIG-893 URL: https://issues.apache.org/jira/browse/PIG-893 Project: Pig Issue Type: New Feature Affects Versions: 0.4.0 Reporter: Thejas M Nair Assignee: Jeff Zhang Fix For: 0.4.0 Attachments: Pig_893_Patch.txt Pig should support casting of chararray to integer,long,float,double,bytearray. If the conversion fails for reasons such as overflow, cast should return null and log a warning. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-660) Integration with Hadoop 0.20
[ https://issues.apache.org/jira/browse/PIG-660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy V. Ryaboy updated PIG-660: -- Attachment: pig_660_shims_2.patch Sure is.. uploading a patch with the fixed package name. Integration with Hadoop 0.20 Key: PIG-660 URL: https://issues.apache.org/jira/browse/PIG-660 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.2.0 Environment: Hadoop 0.20 Reporter: Santhosh Srinivasan Assignee: Santhosh Srinivasan Fix For: 0.4.0 Attachments: PIG-660-for-branch-0.3.patch, PIG-660.patch, PIG-660_1.patch, PIG-660_2.patch, PIG-660_3.patch, PIG-660_4.patch, PIG-660_5.patch, pig_660_shims.patch, pig_660_shims_2.patch With Hadoop 0.20, it will be possible to query the status of each map and reduce in a map reduce job. This will allow better error reporting. Some of the other items that could be on Hadoop's feature requests/bugs are documented here for tracking. 1. Hadoop should return objects instead of strings when exceptions are thrown 2. The JobControl should handle all exceptions and report them appropriately. For example, when the JobControl fails to launch jobs, it should handle exceptions appropriately and should support APIs that query this state, i.e., failure to launch jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-909) Allow Pig executable to use hadoop jars not bundled with pig
[
https://issues.apache.org/jira/browse/PIG-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12739570#action_12739570
]
Dmitriy V. Ryaboy commented on PIG-909:
---
Sorry I am being slow -- which libraries are missing from the classpath you
posted?
Allow Pig executable to use hadoop jars not bundled with pig
Key: PIG-909
URL: https://issues.apache.org/jira/browse/PIG-909
Project: Pig
Issue Type: Improvement
Reporter: Dmitriy V. Ryaboy
Priority: Minor
Attachments: pig_909.patch
The current pig executable (bin/pig) looks for a file named
hadoop${PIG_HADOOP_VERSION}.jar that comes bundled with Pig.
The proposed change will allow Pig to look in $HADOOP_HOME for the hadoop
jars, if that variable is set.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Hudson build is back to normal: Pig-Patch-minerva.apache.org #147
See http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/147/changes
[jira] Commented: (PIG-901) InputSplit (SliceWrapper) created by Pig is big in size due to serialized PigContext
[ https://issues.apache.org/jira/browse/PIG-901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12739583#action_12739583 ] Hadoop QA commented on PIG-901: --- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12415495/PIG-901-trunk.patch against trunk revision 800494. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/147/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/147/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/147/console This message is automatically generated. InputSplit (SliceWrapper) created by Pig is big in size due to serialized PigContext Key: PIG-901 URL: https://issues.apache.org/jira/browse/PIG-901 Project: Pig Issue Type: Bug Affects Versions: 0.3.1 Reporter: Pradeep Kamath Assignee: Pradeep Kamath Fix For: 0.4.0 Attachments: PIG-901-1.patch, PIG-901-branch-0.3.patch, PIG-901-trunk.patch InputSplit (SliceWrapper) created by Pig is big in size due to serialized PigContext. SliceWrapper only needs ExecType - so the entire PigContext should not be serialized and only the ExecType should be serialized. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-909) Allow Pig executable to use hadoop jars not bundled with pig
[
https://issues.apache.org/jira/browse/PIG-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12739643#action_12739643
]
Dmitriy V. Ryaboy commented on PIG-909:
---
Oh I see.
I have this in my bashrc:
export PIG_CLASSPATH=$PIGDIR/pig.jar
I thought this was included in a README somewhere. I guess we can modify
bin/pig to use this as a default value (so a user can still override by setting
PIG_CLASSPATH to something else).
Allow Pig executable to use hadoop jars not bundled with pig
Key: PIG-909
URL: https://issues.apache.org/jira/browse/PIG-909
Project: Pig
Issue Type: Improvement
Reporter: Dmitriy V. Ryaboy
Priority: Minor
Attachments: pig_909.patch
The current pig executable (bin/pig) looks for a file named
hadoop${PIG_HADOOP_VERSION}.jar that comes bundled with Pig.
The proposed change will allow Pig to look in $HADOOP_HOME for the hadoop
jars, if that variable is set.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-905) TOKENIZE throws exception on null data
[ https://issues.apache.org/jira/browse/PIG-905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated PIG-905: --- Attachment: PIG-905-2.patch Address Olga's comments and add unit test TOKENIZE throws exception on null data -- Key: PIG-905 URL: https://issues.apache.org/jira/browse/PIG-905 Project: Pig Issue Type: Bug Reporter: Olga Natkovich Attachments: PIG-905-1.patch, PIG-905-2.patch it should just return null -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-905) TOKENIZE throws exception on null data
[ https://issues.apache.org/jira/browse/PIG-905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated PIG-905: --- Fix Version/s: 0.4.0 Affects Version/s: 0.3.0 Status: Patch Available (was: Open) TOKENIZE throws exception on null data -- Key: PIG-905 URL: https://issues.apache.org/jira/browse/PIG-905 Project: Pig Issue Type: Bug Affects Versions: 0.3.0 Reporter: Olga Natkovich Fix For: 0.4.0 Attachments: PIG-905-1.patch, PIG-905-2.patch it should just return null -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-901) InputSplit (SliceWrapper) created by Pig is big in size due to serialized PigContext
[ https://issues.apache.org/jira/browse/PIG-901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12739669#action_12739669 ] Olga Natkovich commented on PIG-901: +1 on the patch to the trunk InputSplit (SliceWrapper) created by Pig is big in size due to serialized PigContext Key: PIG-901 URL: https://issues.apache.org/jira/browse/PIG-901 Project: Pig Issue Type: Bug Affects Versions: 0.3.1 Reporter: Pradeep Kamath Assignee: Pradeep Kamath Fix For: 0.4.0 Attachments: PIG-901-1.patch, PIG-901-branch-0.3.patch, PIG-901-trunk.patch InputSplit (SliceWrapper) created by Pig is big in size due to serialized PigContext. SliceWrapper only needs ExecType - so the entire PigContext should not be serialized and only the ExecType should be serialized. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (PIG-910) PushUpFilter: Filter after skew LOJoin generates an error
PushUpFilter: Filter after skew LOJoin generates an error - Key: PIG-910 URL: https://issues.apache.org/jira/browse/PIG-910 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.3.0 Reporter: Daniel Dai Fix For: 0.4.0 The script is: A = load 'a' as (user:chararray, action, timespent, query_term, ip_addr, timestamp, estimated_revenue, page_info, page_links); alpha = load 'b' as (user:chararray, action, timespent, query_term, ip_addr, timestamp, estimated_revenue, page_info, page_links); C = join A by user, alpha by user using skewed parallel 40; D = filter C by A::user == 'aaa'; store D into 'L5out'; Stack trace is: ERROR 1093: Swap supports swap of operators with at most one input. Found first operator with 2 inputs. org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2151: Internal error while pushing filters up. at org.apache.pig.impl.logicalLayer.optimizer.PushUpFilter.transform(PushUpFilter.java: 325) at org.apache.pig.impl.logicalLayer.optimizer.LogicalOptimizer.optimize(LogicalOptimizer.java:185) at org.apache.pig.PigServer.compileLp(PigServer.java:854) at org.apache.pig.PigServer.compileLp(PigServer.java:791) at org.apache.pig.PigServer.execute(PigServer.java:760) at org.apache.pig.PigServer.access$100(PigServer.java:95) at org.apache.pig.PigServer$Graph.execute(PigServer.java:943) at org.apache.pig.PigServer.executeBatch(PigServer.java:255) at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java: 112) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java: 169) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java: 141) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:89) at org.apache.pig.Main.main(Main.java:320) Caused by: org.apache.pig.impl.plan.PlanException: ERROR 1093: Swap supports swap of operators with at most one input. Found first operator with 2 inputs. at org.apache.pig.impl.plan.OperatorPlan.swap(OperatorPlan.java:964) at org.apache.pig.impl.logicalLayer.optimizer.PushUpFilter.transform(PushUpFilter.java: 310) ... 12 more -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-910) PushUpFilter: Filter after skew LOJoin generates an error
[ https://issues.apache.org/jira/browse/PIG-910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated PIG-910: --- Attachment: PIG-910-1.patch Can you take a look of this patch, Santhosh? Thanks PushUpFilter: Filter after skew LOJoin generates an error - Key: PIG-910 URL: https://issues.apache.org/jira/browse/PIG-910 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.3.0 Reporter: Daniel Dai Fix For: 0.4.0 Attachments: PIG-910-1.patch The script is: A = load 'a' as (user:chararray, action, timespent, query_term, ip_addr, timestamp, estimated_revenue, page_info, page_links); alpha = load 'b' as (user:chararray, action, timespent, query_term, ip_addr, timestamp, estimated_revenue, page_info, page_links); C = join A by user, alpha by user using skewed parallel 40; D = filter C by A::user == 'aaa'; store D into 'L5out'; Stack trace is: ERROR 1093: Swap supports swap of operators with at most one input. Found first operator with 2 inputs. org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2151: Internal error while pushing filters up. at org.apache.pig.impl.logicalLayer.optimizer.PushUpFilter.transform(PushUpFilter.java: 325) at org.apache.pig.impl.logicalLayer.optimizer.LogicalOptimizer.optimize(LogicalOptimizer.java:185) at org.apache.pig.PigServer.compileLp(PigServer.java:854) at org.apache.pig.PigServer.compileLp(PigServer.java:791) at org.apache.pig.PigServer.execute(PigServer.java:760) at org.apache.pig.PigServer.access$100(PigServer.java:95) at org.apache.pig.PigServer$Graph.execute(PigServer.java:943) at org.apache.pig.PigServer.executeBatch(PigServer.java:255) at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java: 112) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java: 169) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java: 141) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:89) at org.apache.pig.Main.main(Main.java:320) Caused by: org.apache.pig.impl.plan.PlanException: ERROR 1093: Swap supports swap of operators with at most one input. Found first operator with 2 inputs. at org.apache.pig.impl.plan.OperatorPlan.swap(OperatorPlan.java:964) at org.apache.pig.impl.logicalLayer.optimizer.PushUpFilter.transform(PushUpFilter.java: 310) ... 12 more -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-911) [Piggybank] SequenceFileLoader
[ https://issues.apache.org/jira/browse/PIG-911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy V. Ryaboy updated PIG-911: -- Attachment: pig_sequencefile.patch The attached patch is an initial implementation of a loader for SequenceFiles. It works with keys and values of the following types: Text, IntWritable, LongWritable, FloatWritable, DoubleWritable, BooleanWritable, ByteWritable I would appreciate some comments on how to properly handle errors (casting errors, IO errors, etc). [Piggybank] SequenceFileLoader --- Key: PIG-911 URL: https://issues.apache.org/jira/browse/PIG-911 Project: Pig Issue Type: New Feature Reporter: Dmitriy V. Ryaboy Attachments: pig_sequencefile.patch The proposed piggybank contribution adds a SequenceFileLoader to the piggybank. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-910) PushUpFilter: Filter after skew LOJoin generates an error
[ https://issues.apache.org/jira/browse/PIG-910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12739783#action_12739783 ] Daniel Dai commented on PIG-910: I discussed with Santhosh, this issue is addressed by PIG-697 phase 5 patch (LOJoin patch). PIG-910-1.patch is a temporary solution before LOJoin patch, and is not intended to commit. PushUpFilter: Filter after skew LOJoin generates an error - Key: PIG-910 URL: https://issues.apache.org/jira/browse/PIG-910 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.3.0 Reporter: Daniel Dai Fix For: 0.4.0 Attachments: PIG-910-1.patch The script is: A = load 'a' as (user:chararray, action, timespent, query_term, ip_addr, timestamp, estimated_revenue, page_info, page_links); alpha = load 'b' as (user:chararray, action, timespent, query_term, ip_addr, timestamp, estimated_revenue, page_info, page_links); C = join A by user, alpha by user using skewed parallel 40; D = filter C by A::user == 'aaa'; store D into 'L5out'; Stack trace is: ERROR 1093: Swap supports swap of operators with at most one input. Found first operator with 2 inputs. org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2151: Internal error while pushing filters up. at org.apache.pig.impl.logicalLayer.optimizer.PushUpFilter.transform(PushUpFilter.java: 325) at org.apache.pig.impl.logicalLayer.optimizer.LogicalOptimizer.optimize(LogicalOptimizer.java:185) at org.apache.pig.PigServer.compileLp(PigServer.java:854) at org.apache.pig.PigServer.compileLp(PigServer.java:791) at org.apache.pig.PigServer.execute(PigServer.java:760) at org.apache.pig.PigServer.access$100(PigServer.java:95) at org.apache.pig.PigServer$Graph.execute(PigServer.java:943) at org.apache.pig.PigServer.executeBatch(PigServer.java:255) at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java: 112) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java: 169) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java: 141) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:89) at org.apache.pig.Main.main(Main.java:320) Caused by: org.apache.pig.impl.plan.PlanException: ERROR 1093: Swap supports swap of operators with at most one input. Found first operator with 2 inputs. at org.apache.pig.impl.plan.OperatorPlan.swap(OperatorPlan.java:964) at org.apache.pig.impl.logicalLayer.optimizer.PushUpFilter.transform(PushUpFilter.java: 310) ... 12 more -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
build failures on hudson zones
Build on hudson.zones are failing as the zonestorage for hudson is full. I 've sent an email to the ASF infra team about the space issues on hudson zones. Once the issues is resolved I would restart hudson for builds. Thanks, Giri
