[jira] [Commented] (PIG-2878) Pig current releases lack a UDF equalIgnoreCase.This function returns a Boolean value indicating whether string left is equal to string right. This check is case insensit
[ https://issues.apache.org/jira/browse/PIG-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13565308#comment-13565308 ] Shami B commented on PIG-2878: -- Thanks for the review comments. I am working on the unit test cases and will submit it soon. Pig current releases lack a UDF equalIgnoreCase.This function returns a Boolean value indicating whether string left is equal to string right. This check is case insensitive. -- Key: PIG-2878 URL: https://issues.apache.org/jira/browse/PIG-2878 Project: Pig Issue Type: Bug Components: internal-udfs Affects Versions: 0.10.0 Reporter: Arjun K R Assignee: Arjun K R Labels: features Attachments: PIG-2878.patch Pig current releases lack a UDF equalIgnoreCase.This function returns a Boolean value indicating whether string left is equal to string right. This check is case insensitive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2878) Pig current releases lack a UDF equalIgnoreCase.This function returns a Boolean value indicating whether string left is equal to string right. This check is case insensitiv
[ https://issues.apache.org/jira/browse/PIG-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shami B updated PIG-2878: - Attachment: PIG-2878-UnitTest.patch Please find the attached unit test case for the EqualsIgnore UDF. Pig current releases lack a UDF equalIgnoreCase.This function returns a Boolean value indicating whether string left is equal to string right. This check is case insensitive. -- Key: PIG-2878 URL: https://issues.apache.org/jira/browse/PIG-2878 Project: Pig Issue Type: Bug Components: internal-udfs Affects Versions: 0.10.0 Reporter: Arjun K R Assignee: Arjun K R Labels: features Attachments: PIG-2878.patch, PIG-2878-UnitTest.patch Pig current releases lack a UDF equalIgnoreCase.This function returns a Boolean value indicating whether string left is equal to string right. This check is case insensitive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2878) Pig current releases lack a UDF equalIgnoreCase.This function returns a Boolean value indicating whether string left is equal to string right. This check is case insensitiv
[ https://issues.apache.org/jira/browse/PIG-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shami B updated PIG-2878: - Status: Patch Available (was: Open) Please find the attached unit test case for EqualsIgnoreCase UDF Pig current releases lack a UDF equalIgnoreCase.This function returns a Boolean value indicating whether string left is equal to string right. This check is case insensitive. -- Key: PIG-2878 URL: https://issues.apache.org/jira/browse/PIG-2878 Project: Pig Issue Type: Bug Components: internal-udfs Affects Versions: 0.10.0 Reporter: Arjun K R Assignee: Arjun K R Labels: features Attachments: PIG-2878.patch, PIG-2878-UnitTest.patch Pig current releases lack a UDF equalIgnoreCase.This function returns a Boolean value indicating whether string left is equal to string right. This check is case insensitive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-1824) Support import modules in Jython UDF
[
https://issues.apache.org/jira/browse/PIG-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13565541#comment-13565541
]
Russell Jurney commented on PIG-1824:
-
This does not actually work for me, in either Pig 0.10 or Pig 0.10.1. I can't
include the 're' module via 'import re', or I get this error:
Caused by: Traceback (most recent call last):
File udfs.py, line 20, in module
import re
ImportError: No module named re
Support import modules in Jython UDF
Key: PIG-1824
URL: https://issues.apache.org/jira/browse/PIG-1824
Project: Pig
Issue Type: Improvement
Affects Versions: 0.8.0, 0.9.0
Reporter: Richard Ding
Assignee: Woody Anderson
Fix For: 0.10.0
Attachments: 1824a.patch, 1824b.patch, 1824c.patch, 1824d.patch,
1824_final.patch, 1824.patch, 1824x.patch,
TEST-org.apache.pig.test.TestGrunt.txt,
TEST-org.apache.pig.test.TestScriptLanguage.txt,
TEST-org.apache.pig.test.TestScriptUDF.txt
Currently, Jython UDF script doesn't support Jython import statement as in
the following example:
{code}
#!/usr/bin/python
import re
@outputSchema(word:chararray)
def resplit(content, regex, index):
return re.compile(regex).split(content)[index]
{code}
Can Pig automatically locate the Jython module file and ship it to the
backend? Or should we add a ship clause to let user explicitly specify the
module to ship?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (PIG-3146) Can't 'import re' in Pig 0.10/0.10.1: ImportError: No module named re
Russell Jurney created PIG-3146: --- Summary: Can't 'import re' in Pig 0.10/0.10.1: ImportError: No module named re Key: PIG-3146 URL: https://issues.apache.org/jira/browse/PIG-3146 Project: Pig Issue Type: Bug Affects Versions: 0.10.0, 0.10.1 Reporter: Russell Jurney Fix For: 0.12 Caused by: Traceback (most recent call last): File udfs.py, line 20, in module import re ImportError: No module named re -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2878) Pig current releases lack a UDF equalIgnoreCase.This function returns a Boolean value indicating whether string left is equal to string right. This check is case insensit
[ https://issues.apache.org/jira/browse/PIG-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13565550#comment-13565550 ] Alan Gates commented on PIG-2878: - I'll review this. Pig current releases lack a UDF equalIgnoreCase.This function returns a Boolean value indicating whether string left is equal to string right. This check is case insensitive. -- Key: PIG-2878 URL: https://issues.apache.org/jira/browse/PIG-2878 Project: Pig Issue Type: Bug Components: internal-udfs Affects Versions: 0.10.0 Reporter: Arjun K R Assignee: Arjun K R Labels: features Attachments: PIG-2878.patch, PIG-2878-UnitTest.patch Pig current releases lack a UDF equalIgnoreCase.This function returns a Boolean value indicating whether string left is equal to string right. This check is case insensitive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (PIG-3147) Spill failing with java.lang.RuntimeException: InternalCachedBag.spill() should not be called
Koji Noguchi created PIG-3147:
-
Summary: Spill failing with java.lang.RuntimeException:
InternalCachedBag.spill() should not be called
Key: PIG-3147
URL: https://issues.apache.org/jira/browse/PIG-3147
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: 0.11
Reporter: Koji Noguchi
Priority: Blocker
Tried 0.11 jar with spilling, my job failed to spill with the following stack
trace. Anyone else seeing this?
{noformat}
java.lang.RuntimeException: InternalCachedBag.spill() should not be called
at
org.apache.pig.data.InternalCachedBag.spill(InternalCachedBag.java:167)
at
org.apache.pig.impl.util.SpillableMemoryManager.handleNotification(SpillableMemoryManager.java:243)
at
sun.management.NotificationEmitterSupport.sendNotification(NotificationEmitterSupport.java:138)
at sun.management.MemoryImpl.createNotification(MemoryImpl.java:171)
at
sun.management.MemoryPoolImpl$PoolSensor.triggerAction(MemoryPoolImpl.java:272)
at sun.management.Sensor.trigger(Sensor.java:120)
Exception in thread Low Memory Detector java.lang.InternalError: Error in
invoking listener
at
sun.management.NotificationEmitterSupport.sendNotification(NotificationEmitterSupport.java:141)
at sun.management.MemoryImpl.createNotification(MemoryImpl.java:171)
at
sun.management.MemoryPoolImpl$PoolSensor.triggerAction(MemoryPoolImpl.java:272)
at sun.management.Sensor.trigger(Sensor.java:120)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3147) Spill failing with java.lang.RuntimeException: InternalCachedBag.spill() should not be called
[
https://issues.apache.org/jira/browse/PIG-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13565698#comment-13565698
]
Koji Noguchi commented on PIG-3147:
---
Dumping a stacktrace when InternalCacheBag is added to the
SpillableMemoryManager.spillables,
{noformat}
java.lang.Exception: Stack trace
at java.lang.Thread.dumpStack(Thread.java:1206)
at
org.apache.pig.impl.util.SpillableMemoryManager.registerSpillable(SpillableMemoryManager.java:296)
at
org.apache.pig.data.DefaultAbstractBag.markSpillableIfNecessary(DefaultAbstractBag.java:101)
at
org.apache.pig.data.InternalCachedBag.addDone(InternalCachedBag.java:131)
at
org.apache.pig.data.InternalCachedBag.iterator(InternalCachedBag.java:159)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject.getNext(POProject.java:456)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:308)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFilter.getNext(POFilter.java:95)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:308)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:241)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:308)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POSortedDistinct.getNext(POSortedDistinct.java:62)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:432)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject.processInputBag(POProject.java:581)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.PORelationToExprProject.getNext(PORelationToExprProject.java:107)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:334)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.processInput(POUserFunc.java:228)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:282)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:416)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:348)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:372)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:297)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.runPipeline(PigGenericMapReduce.java:465)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:433)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:413)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:257)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
at
org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
{noformat}
Is this from a change in PIG-2923?
Spill failing with java.lang.RuntimeException: InternalCachedBag.spill()
should not be called
---
Key: PIG-3147
URL: https://issues.apache.org/jira/browse/PIG-3147
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: 0.11
Reporter: Koji Noguchi
Priority: Blocker
Tried 0.11 jar with spilling, my job failed to spill with the following stack
trace. Anyone else seeing this?
{noformat}
java.lang.RuntimeException: InternalCachedBag.spill() should not be called
at
Re: Pig 11.0
Sounds like we can roll a release. Who wants to do the honors? I think Olga and Daniel have done it in the past, not sure if they have the time? On Mon, Jan 28, 2013 at 11:37 PM, Bill Graham billgra...@gmail.com wrote: I've just committed the 2 documentation jiras and all Pig 0.11 issues are resolved once again. On Mon, Jan 28, 2013 at 7:22 PM, Rohini Palaniswamy rohini.adi...@gmail.com wrote: e2e tests are fine on my end too except for hcat tests (Did not have hcat setup) and few transient Errors test failures which is a known issue. Ran with Hadoop 0.23. Regards, Rohini On Mon, Jan 28, 2013 at 4:44 PM, Julien Le Dem jul...@twitter.com wrote: It sounds like we are ready to go. There are two remaining documentation JIRAs that will be committed soon. Daniel, do you want to build the release? Julien On Mon, Jan 28, 2013 at 10:12 AM, Daniel Dai da...@hortonworks.com wrote: All tests pass for me, no single failures or aborts. Daniel On Mon, Jan 28, 2013 at 8:33 AM, Cheolsoo Park cheol...@cloudera.com wrote: Here are my results: Hadoop-1.0.x: [exec] Final results ,PASSED: 612 FAILED: 2 SKIPPED: 24 ABORTED: 1FAILED DEPENDENCY: 0 - The failures are simply because I didn't install Hcatalog. Hadoop-2.0.x: [exec] Final results ,PASSED: 567 FAILED: 3 SKIPPED: 27 ABORTED: 42 FAILED DEPENDENCY: 0 - The failures seem due to issues in my cluster rather than Pig issues. I will re-run them to verify. On Fri, Jan 25, 2013 at 5:31 PM, Cheolsoo Park cheol...@cloudera.com wrote: I will also run e2e on Hadoop-1.x and Hadoop-2.x. On Fri, Jan 25, 2013 at 5:02 PM, Daniel Dai da...@hortonworks.com wrote: I will run e2e tests on Hadoop 1.x over the weekend. Thanks, Daniel On Fri, Jan 25, 2013 at 4:27 PM, Rohini Palaniswamy rohini.adi...@gmail.com wrote: Thats good :). Unit tests have all been passing. I haven't run e2e tests on pig 0.11 for sometime. Will kick off one this weekend. It would be nice if Cheolsoo and Daniel can also kick off one run. Regards, Rohini On Fri, Jan 25, 2013 at 4:08 PM, Julien Le Dem jul...@twitter.com wrote: It looks like all the tickets for Pig 0.11 have been resolved as of today. See: https://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%20%220.11%22%20AND%20project%20%3D%20PIG%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20updated%20DESC%2C%20created%20ASC%2C%20priority%20DESC I propose we make the release 0.11.0 next week. Julien -- *Note that I'm no longer using my Yahoo! email address. Please email me at billgra...@gmail.com going forward.*
[jira] [Commented] (PIG-2923) Lazily register bags with SpillableMemoryManager
[ https://issues.apache.org/jira/browse/PIG-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13565705#comment-13565705 ] Koji Noguchi commented on PIG-2923: --- Hi Dmitriy, I'm seeing a weird error when pig 0.11 tries to spill. Can this change be related? Opened PIG-3147. Lazily register bags with SpillableMemoryManager Key: PIG-2923 URL: https://issues.apache.org/jira/browse/PIG-2923 Project: Pig Issue Type: Improvement Reporter: Dmitriy V. Ryaboy Assignee: Dmitriy V. Ryaboy Fix For: 0.11 Attachments: bagspill_delayed_register.patch, bagspill_delay.patch Currently, all Spillable DataBags get registered by the BagFactory at the moment of creation. In practice, a lot of these bags will not get large enough to be worth spilling; we can avoid a lot of memory overhead and cheapen the process of finding a bag to spill when we do need it, by allowing Bags themselves to register when they grow to some respectable threshold. Related JIRAs: PIG-2917, PIG-2918 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Getting Pig 0.11 out
Somehow 0.11 is failing when trying to spill. Has anyone else seen this? Opened PIG-3147. Koji On Jan 21, 2013, at 10:43 PM, Bill Graham wrote: Great, thanks all. I've just removed 0.11 from PIG-2973 and subtasks and we're now down to 5 issues. So close! On Mon, Jan 21, 2013 at 1:51 PM, John Gordon john.gor...@microsoft.comwrote: I agree with Daniel. PIG-2973 does best when pig is run against a version of Hadoop core with Windows fixes. I don't think it would offer as much value to the community until we have a release branch of Hadoop so that there is a combination of release branches that can be documented to accomplish this goal for devs/users. Thanks! John -Original Message- From: Daniel Dai [mailto:da...@hortonworks.com] Sent: Monday, January 21, 2013 1:34 PM To: dev@pig.apache.org Subject: Re: Getting Pig 0.11 out I am fine moving PIG-2973 to 0.12. I probably will check several in before 0.11 release, but this is not a blocker. Thanks, Daniel On Sun, Jan 20, 2013 at 7:23 AM, Jarek Jarcec Cecho jar...@apache.org wrote: I've uploaded patch for PIG-3109. I'll try to take a look on the failing tests, but can't promise anything for next two weeks. Jarcec On Fri, Jan 18, 2013 at 06:23:59PM -0500, Bill Graham wrote: Hi, Re-starting a discussion around what it will take to release Pig 11. There are 15 unresolved issues out there currently: https://issues.apache.org/jira/issues/?jql=project%20%3D%20PIG%20AND% 20resolution%20%3D%20Unresolved%20AND%20fixVersion%20%3D%20%220.11%22 %20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC - PIG-2973: Make Pig Work on Windows without Cygwin This issue and it's subtasks account for 11 of the 15 issues. Can we move this to Pig 0.12? - PIG-2972: Umbrella ticket for test failures in 0.11/trunk This one has 4 unassigned open subtasks which are not tagged with 0.11. Any takers? - PIG-3109: License headers. This is a easy blocker to tackle. - PIG-2756: Pig 0.11 documentation umbrella. Sub-tasks are complete, no work required here. - PIG-3105: Fix TestJobSubmission unit test failure. Patch available, should be easy to verify. thanks, Bill -- *Note that I'm no longer using my Yahoo! email address. Please email me at billgra...@gmail.com going forward.*
[jira] [Updated] (PIG-2266) bug with input file joining optimization in Pig
[ https://issues.apache.org/jira/browse/PIG-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheolsoo Park updated PIG-2266: --- Resolution: Fixed Fix Version/s: 0.12 Status: Resolved (was: Patch Available) Committed to trunk. Thanks Joe! bug with input file joining optimization in Pig --- Key: PIG-2266 URL: https://issues.apache.org/jira/browse/PIG-2266 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.9.0, 0.10.0 Reporter: Joseph Adler Assignee: Joseph Adler Fix For: 0.12 Attachments: PIG-2266.patch In src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MRCompiler.java, the function hasTooManyInputFiles instantiated a LoadFunc instance, then calls setLocation before calling setUDFContextSignature. This is inconsistent with the documentation for the LoadFunc interface (see http://pig.apache.org/docs/r0.9.0/api/org/apache/pig/LoadFunc.html#setUDFContextSignature(java.lang.String)). (We've written UDFs that assume that setUDFContextSignature is called first.) I think you can fix this by adding loader.setUDFContextSignature(ld.getSignature()); Before loader.setLocation(location, job); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-3147) Spill failing with java.lang.RuntimeException: InternalCachedBag.spill() should not be called
[
https://issues.apache.org/jira/browse/PIG-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Noguchi updated PIG-3147:
--
Attachment: pig-3147-v01.txt
Reading PIG-975, InternalCachedBag should not register with
SpillableMemoryManager.
This is just a pure guess, but uploading a patch that takes out
markSpillableIfNecessary from InternalCachedBag.java.
Spill failing with java.lang.RuntimeException: InternalCachedBag.spill()
should not be called
---
Key: PIG-3147
URL: https://issues.apache.org/jira/browse/PIG-3147
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: 0.11
Reporter: Koji Noguchi
Priority: Blocker
Attachments: pig-3147-v01.txt
Tried 0.11 jar with spilling, my job failed to spill with the following stack
trace. Anyone else seeing this?
{noformat}
java.lang.RuntimeException: InternalCachedBag.spill() should not be called
at
org.apache.pig.data.InternalCachedBag.spill(InternalCachedBag.java:167)
at
org.apache.pig.impl.util.SpillableMemoryManager.handleNotification(SpillableMemoryManager.java:243)
at
sun.management.NotificationEmitterSupport.sendNotification(NotificationEmitterSupport.java:138)
at sun.management.MemoryImpl.createNotification(MemoryImpl.java:171)
at
sun.management.MemoryPoolImpl$PoolSensor.triggerAction(MemoryPoolImpl.java:272)
at sun.management.Sensor.trigger(Sensor.java:120)
Exception in thread Low Memory Detector java.lang.InternalError: Error in
invoking listener
at
sun.management.NotificationEmitterSupport.sendNotification(NotificationEmitterSupport.java:141)
at sun.management.MemoryImpl.createNotification(MemoryImpl.java:171)
at
sun.management.MemoryPoolImpl$PoolSensor.triggerAction(MemoryPoolImpl.java:272)
at sun.management.Sensor.trigger(Sensor.java:120)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (PIG-3148) OutOfMemory exception while spilling stale DefaultDataBag. Extra option to gc() before spilling large bag.
Koji Noguchi created PIG-3148:
-
Summary: OutOfMemory exception while spilling stale
DefaultDataBag. Extra option to gc() before spilling large bag.
Key: PIG-3148
URL: https://issues.apache.org/jira/browse/PIG-3148
Project: Pig
Issue Type: Improvement
Components: impl
Reporter: Koji Noguchi
Assignee: Koji Noguchi
Our user reported that one of their jobs in pig 0.10 occasionally failed with
'Error: GC overhead limit exceeded' or 'Error: Java heap space', but rerunning
it sometimes finishes successfully.
For 1G heap reducer, heap dump showed it contained two huge DefaultDataBag with
300-400MBytes each when failing with OOM.
Jstack at the time of OOM always showed that spill was running.
{noformat}
Low Memory Detector daemon prio=10 tid=0xb9c11800 nid=0xa52 runnable
[0xb9afc000]
java.lang.Thread.State: RUNNABLE
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:260)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
- locked 0xe57c6390 (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
- locked 0xe57c60b8 (a java.io.DataOutputStream)
at java.io.FilterOutputStream.write(FilterOutputStream.java:80)
at org.apache.pig.data.utils.SedesHelper.writeBytes(SedesHelper.java:46)
at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:537)
at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:435)
at
org.apache.pig.data.utils.SedesHelper.writeGenericTuple(SedesHelper.java:135)
at org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:613)
at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:443)
at org.apache.pig.data.DefaultDataBag.spill(DefaultDataBag.java:106)
- locked 0xceb16190 (a java.util.ArrayList)
at
org.apache.pig.impl.util.SpillableMemoryManager.handleNotification(SpillableMemoryManager.java:243)
- locked 0xbeb86318 (a java.util.LinkedList)
at
sun.management.NotificationEmitterSupport.sendNotification(NotificationEmitterSupport.java:138)
at sun.management.MemoryImpl.createNotification(MemoryImpl.java:171)
at
sun.management.MemoryPoolImpl$PoolSensor.triggerAction(MemoryPoolImpl.java:272)
at sun.management.Sensor.trigger(Sensor.java:120)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3148) OutOfMemory exception while spilling stale DefaultDataBag. Extra option to gc() before spilling large bag.
[
https://issues.apache.org/jira/browse/PIG-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13565975#comment-13565975
]
Koji Noguchi commented on PIG-3148:
---
I cannot attach the original query, but to give you an idea
{noformat}
A = LOAD '$INPUT' USING MyLoader('\u0001') AS
( val1, val2, val3, val4, val5, val6, val7, val8, val9, val10, val11, val12,
val13, val14, val15, val16);
B = GROUP A BY (val1, val2, val3, val8) PARALLEL $NUM_REDUCERS;
C = FOREACH B {
D = FILTER A BY (val3 == 'status1' AND val5 == 'status2');
E = D.val4;
F = DISTINCT E;
G = FILTER D BY val7 == 'status3';
GENERATE group.val1, group.val2, group.val8, COUNT(F), COUNT(G), SUM(G.val9),
SUM(G.val10), SUM(G.val11), SUM(A.val12), SUM(A.val13), SUM(A.val15),
SUM(A.val14), SUM(A.val16);
}
STORE C INTO '$OUTPUT' USING PigStorage('\u0001');
{noformat}
Assuming that this script does not require a two huge DefaultDataBag, looked
into SpillableMemoryManager.handleNotification. 'handleNotification' is
called whenever certain memory condition is met but not necessary after the
gc().
What was happening on this user's case was,
(i) 400MB of DefaultDataBag#1 goes stale.
(ii) SpillableMemoryManager.handleNotification is triggered.
(iii) Since gc() is not called yet, WeakReference is still valid and pig
decides to spill holding the lock of this DefaultDataBag#1.mContents(ArrayList).
(iv) While reduce task is working on another 400MB DefaultDataBag#2, jvm heap
gets full and gc is called. Even though no one is using the stale
DefaultDataBag#1, it cannot be GC-ed since spill is holding the lock.
As a result, we end up with two DefaultDataBag leading to OOM.
OutOfMemory exception while spilling stale DefaultDataBag. Extra option to
gc() before spilling large bag.
--
Key: PIG-3148
URL: https://issues.apache.org/jira/browse/PIG-3148
Project: Pig
Issue Type: Improvement
Components: impl
Reporter: Koji Noguchi
Assignee: Koji Noguchi
Our user reported that one of their jobs in pig 0.10 occasionally failed with
'Error: GC overhead limit exceeded' or 'Error: Java heap space', but
rerunning it sometimes finishes successfully.
For 1G heap reducer, heap dump showed it contained two huge DefaultDataBag
with 300-400MBytes each when failing with OOM.
Jstack at the time of OOM always showed that spill was running.
{noformat}
Low Memory Detector daemon prio=10 tid=0xb9c11800 nid=0xa52 runnable
[0xb9afc000]
java.lang.Thread.State: RUNNABLE
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:260)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
- locked 0xe57c6390 (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
- locked 0xe57c60b8 (a java.io.DataOutputStream)
at java.io.FilterOutputStream.write(FilterOutputStream.java:80)
at org.apache.pig.data.utils.SedesHelper.writeBytes(SedesHelper.java:46)
at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:537)
at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:435)
at
org.apache.pig.data.utils.SedesHelper.writeGenericTuple(SedesHelper.java:135)
at org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:613)
at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:443)
at org.apache.pig.data.DefaultDataBag.spill(DefaultDataBag.java:106)
- locked 0xceb16190 (a java.util.ArrayList)
at
org.apache.pig.impl.util.SpillableMemoryManager.handleNotification(SpillableMemoryManager.java:243)
- locked 0xbeb86318 (a java.util.LinkedList)
at
sun.management.NotificationEmitterSupport.sendNotification(NotificationEmitterSupport.java:138)
at sun.management.MemoryImpl.createNotification(MemoryImpl.java:171)
at
sun.management.MemoryPoolImpl$PoolSensor.triggerAction(MemoryPoolImpl.java:272)
at sun.management.Sensor.trigger(Sensor.java:120)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-3148) OutOfMemory exception while spilling stale DefaultDataBag. Extra option to gc() before spilling large bag.
[
https://issues.apache.org/jira/browse/PIG-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Noguchi updated PIG-3148:
--
Attachment: pig-3148-v01.patch
Uploading a patch that adds a feature that would call System.gc() when
Spillable is bigger than 'pig.spill.extragc.size.threshold' before spilling.
This extra gc() is called at most once per handleNotification and also disabled
as default since adding GC has a risk of changing the performance drastically.
For the job I was looking, adding
'-Dpig.spill.extragc.size.threshold=1' let the job run successfully
with no OOM errors.
(Note: Separate spill issues before this patch on 0.11 is tracked at PIG-3147.)
OutOfMemory exception while spilling stale DefaultDataBag. Extra option to
gc() before spilling large bag.
--
Key: PIG-3148
URL: https://issues.apache.org/jira/browse/PIG-3148
Project: Pig
Issue Type: Improvement
Components: impl
Reporter: Koji Noguchi
Assignee: Koji Noguchi
Attachments: pig-3148-v01.patch
Our user reported that one of their jobs in pig 0.10 occasionally failed with
'Error: GC overhead limit exceeded' or 'Error: Java heap space', but
rerunning it sometimes finishes successfully.
For 1G heap reducer, heap dump showed it contained two huge DefaultDataBag
with 300-400MBytes each when failing with OOM.
Jstack at the time of OOM always showed that spill was running.
{noformat}
Low Memory Detector daemon prio=10 tid=0xb9c11800 nid=0xa52 runnable
[0xb9afc000]
java.lang.Thread.State: RUNNABLE
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:260)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
- locked 0xe57c6390 (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
- locked 0xe57c60b8 (a java.io.DataOutputStream)
at java.io.FilterOutputStream.write(FilterOutputStream.java:80)
at org.apache.pig.data.utils.SedesHelper.writeBytes(SedesHelper.java:46)
at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:537)
at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:435)
at
org.apache.pig.data.utils.SedesHelper.writeGenericTuple(SedesHelper.java:135)
at org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:613)
at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:443)
at org.apache.pig.data.DefaultDataBag.spill(DefaultDataBag.java:106)
- locked 0xceb16190 (a java.util.ArrayList)
at
org.apache.pig.impl.util.SpillableMemoryManager.handleNotification(SpillableMemoryManager.java:243)
- locked 0xbeb86318 (a java.util.LinkedList)
at
sun.management.NotificationEmitterSupport.sendNotification(NotificationEmitterSupport.java:138)
at sun.management.MemoryImpl.createNotification(MemoryImpl.java:171)
at
sun.management.MemoryPoolImpl$PoolSensor.triggerAction(MemoryPoolImpl.java:272)
at sun.management.Sensor.trigger(Sensor.java:120)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Pig 11.0
My hands are full these several days, I can try after several days but no promise. Thanks, Daniel On Tue, Jan 29, 2013 at 11:40 AM, Dmitriy Ryaboy dvrya...@gmail.com wrote: Sounds like we can roll a release. Who wants to do the honors? I think Olga and Daniel have done it in the past, not sure if they have the time? On Mon, Jan 28, 2013 at 11:37 PM, Bill Graham billgra...@gmail.com wrote: I've just committed the 2 documentation jiras and all Pig 0.11 issues are resolved once again. On Mon, Jan 28, 2013 at 7:22 PM, Rohini Palaniswamy rohini.adi...@gmail.com wrote: e2e tests are fine on my end too except for hcat tests (Did not have hcat setup) and few transient Errors test failures which is a known issue. Ran with Hadoop 0.23. Regards, Rohini On Mon, Jan 28, 2013 at 4:44 PM, Julien Le Dem jul...@twitter.com wrote: It sounds like we are ready to go. There are two remaining documentation JIRAs that will be committed soon. Daniel, do you want to build the release? Julien On Mon, Jan 28, 2013 at 10:12 AM, Daniel Dai da...@hortonworks.com wrote: All tests pass for me, no single failures or aborts. Daniel On Mon, Jan 28, 2013 at 8:33 AM, Cheolsoo Park cheol...@cloudera.com wrote: Here are my results: Hadoop-1.0.x: [exec] Final results ,PASSED: 612 FAILED: 2 SKIPPED: 24 ABORTED: 1FAILED DEPENDENCY: 0 - The failures are simply because I didn't install Hcatalog. Hadoop-2.0.x: [exec] Final results ,PASSED: 567 FAILED: 3 SKIPPED: 27 ABORTED: 42 FAILED DEPENDENCY: 0 - The failures seem due to issues in my cluster rather than Pig issues. I will re-run them to verify. On Fri, Jan 25, 2013 at 5:31 PM, Cheolsoo Park cheol...@cloudera.com wrote: I will also run e2e on Hadoop-1.x and Hadoop-2.x. On Fri, Jan 25, 2013 at 5:02 PM, Daniel Dai da...@hortonworks.com wrote: I will run e2e tests on Hadoop 1.x over the weekend. Thanks, Daniel On Fri, Jan 25, 2013 at 4:27 PM, Rohini Palaniswamy rohini.adi...@gmail.com wrote: Thats good :). Unit tests have all been passing. I haven't run e2e tests on pig 0.11 for sometime. Will kick off one this weekend. It would be nice if Cheolsoo and Daniel can also kick off one run. Regards, Rohini On Fri, Jan 25, 2013 at 4:08 PM, Julien Le Dem jul...@twitter.com wrote: It looks like all the tickets for Pig 0.11 have been resolved as of today. See: https://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%20%220.11%22%20AND%20project%20%3D%20PIG%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20updated%20DESC%2C%20created%20ASC%2C%20priority%20DESC I propose we make the release 0.11.0 next week. Julien -- *Note that I'm no longer using my Yahoo! email address. Please email me at billgra...@gmail.com going forward.*
[jira] Subscription: PIG patch available
Issue Subscription Filter: PIG patch available (25 issues) Subscriber: pigdaily Key Summary PIG-3136Introduce a syntax making declared aliases optional https://issues.apache.org/jira/browse/PIG-3136 PIG-3123Simplify Logical Plans By Removing Unneccessary Identity Projections https://issues.apache.org/jira/browse/PIG-3123 PIG-3122Operators should not implicitly become reserved keywords https://issues.apache.org/jira/browse/PIG-3122 PIG-3114Duplicated macro name error when using pigunit https://issues.apache.org/jira/browse/PIG-3114 PIG-3108HBaseStorage returns empty maps when mixing wildcard- with other columns https://issues.apache.org/jira/browse/PIG-3108 PIG-3105Fix TestJobSubmission unit test failure. https://issues.apache.org/jira/browse/PIG-3105 PIG-3098Add another test for the self join case https://issues.apache.org/jira/browse/PIG-3098 PIG-3088Add a builtin udf which removes prefixes https://issues.apache.org/jira/browse/PIG-3088 PIG-3069Native Windows Compatibility for Pig E2E Tests and Harness https://issues.apache.org/jira/browse/PIG-3069 PIG-3028testGrunt dev test needs some command filters to run correctly without cygwin https://issues.apache.org/jira/browse/PIG-3028 PIG-3027pigTest unit test needs a newline filter for comparisons of golden multi-line https://issues.apache.org/jira/browse/PIG-3027 PIG-3026Pig checked-in baseline comparisons need a pre-filter to address OS-specific newline differences https://issues.apache.org/jira/browse/PIG-3026 PIG-3025TestPruneColumn unit test - SimpleEchoStreamingCommand perl inline script needs simplification https://issues.apache.org/jira/browse/PIG-3025 PIG-3024TestEmptyInputDir unit test - hadoop version detection logic is brittle https://issues.apache.org/jira/browse/PIG-3024 PIG-3015Rewrite of AvroStorage https://issues.apache.org/jira/browse/PIG-3015 PIG-3010Allow UDF's to flatten themselves https://issues.apache.org/jira/browse/PIG-3010 PIG-2959Add a pig.cmd for Pig to run under Windows https://issues.apache.org/jira/browse/PIG-2959 PIG-2955 Fix bunch of Pig e2e tests on Windows https://issues.apache.org/jira/browse/PIG-2955 PIG-2878Pig current releases lack a UDF equalIgnoreCase.This function returns a Boolean value indicating whether string left is equal to string right. This check is case insensitive. https://issues.apache.org/jira/browse/PIG-2878 PIG-2873Converting bin/pig shell script to python https://issues.apache.org/jira/browse/PIG-2873 PIG-2834MultiStorage requires unused constructor argument https://issues.apache.org/jira/browse/PIG-2834 PIG-2661Pig uses an extra job for loading data in Pigmix L9 https://issues.apache.org/jira/browse/PIG-2661 PIG-1942script UDF (jython) should utilize the intended output schema to more directly convert Py objects to Pig objects https://issues.apache.org/jira/browse/PIG-1942 PIG-1914Support load/store JSON data in Pig https://issues.apache.org/jira/browse/PIG-1914 PIG-1237Piggybank MutliStorage - specify field to write in output https://issues.apache.org/jira/browse/PIG-1237 You may edit this subscription at: https://issues.apache.org/jira/secure/FilterSubscription!default.jspa?subId=13225filterId=12322384
[jira] [Updated] (PIG-3137) fix Piggybank test to not using /tmp dir
[ https://issues.apache.org/jira/browse/PIG-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johnny Zhang updated PIG-3137: -- Status: Patch Available (was: Open) fix Piggybank test to not using /tmp dir Key: PIG-3137 URL: https://issues.apache.org/jira/browse/PIG-3137 Project: Pig Issue Type: Bug Components: piggybank Affects Versions: 0.11 Reporter: Johnny Zhang Fix For: 0.12 right now several Piggybank tests create directory under /tmp to store test data, the test could fail because user doesn't have permission to create directory under /tmp. It is better to move test data dir under build dir to avoid this problem. I will submit a patch soon. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-3137) fix Piggybank test to not using /tmp dir
[ https://issues.apache.org/jira/browse/PIG-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johnny Zhang updated PIG-3137: -- Attachment: PIG-3137.patch.txt this patch changes TestDBStorage and TestAvroStorage not use /tmp to store the testing data. It uses the FileLocalizer to gain the temp dir from PigContext. The good side about it is user will have fully permission on the dir used by PigServer, so test won't fail in case user doesn't have enough permission on /tmp. For example, the TestAvroStorage right not using $PIG_DIR/contrib/piggybank/java/file:/tmp/temp1371024088/tmp1070213954/TestAvroStorage/ instead of /tmp/TestAvroStorage to store testing data. The fix for TestAvroStorage is straightforward, the fix for TestDBStorage is a little bit different, since hsqldb doesn't work in minicluster hdfs, we need to create another local PigServer to host the hsqldb database. [~cheolsoo], what do you think about it? Thanks. fix Piggybank test to not using /tmp dir Key: PIG-3137 URL: https://issues.apache.org/jira/browse/PIG-3137 Project: Pig Issue Type: Bug Components: piggybank Affects Versions: 0.11 Reporter: Johnny Zhang Fix For: 0.12 Attachments: PIG-3137.patch.txt right now several Piggybank tests create directory under /tmp to store test data, the test could fail because user doesn't have permission to create directory under /tmp. It is better to move test data dir under build dir to avoid this problem. I will submit a patch soon. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Pig 11.0
Olga had previously volunteered to be RM for this release. Olga, did you still want to do this or are you open to someone else taking it on? BTW, I don't have time to drive it right now either, though I'm happy to coach a first time RM through it if Olga doesn't want to do it. Alan. On Jan 29, 2013, at 7:00 PM, Daniel Dai wrote: My hands are full these several days, I can try after several days but no promise. Thanks, Daniel On Tue, Jan 29, 2013 at 11:40 AM, Dmitriy Ryaboy dvrya...@gmail.com wrote: Sounds like we can roll a release. Who wants to do the honors? I think Olga and Daniel have done it in the past, not sure if they have the time? On Mon, Jan 28, 2013 at 11:37 PM, Bill Graham billgra...@gmail.com wrote: I've just committed the 2 documentation jiras and all Pig 0.11 issues are resolved once again. On Mon, Jan 28, 2013 at 7:22 PM, Rohini Palaniswamy rohini.adi...@gmail.com wrote: e2e tests are fine on my end too except for hcat tests (Did not have hcat setup) and few transient Errors test failures which is a known issue. Ran with Hadoop 0.23. Regards, Rohini On Mon, Jan 28, 2013 at 4:44 PM, Julien Le Dem jul...@twitter.com wrote: It sounds like we are ready to go. There are two remaining documentation JIRAs that will be committed soon. Daniel, do you want to build the release? Julien On Mon, Jan 28, 2013 at 10:12 AM, Daniel Dai da...@hortonworks.com wrote: All tests pass for me, no single failures or aborts. Daniel On Mon, Jan 28, 2013 at 8:33 AM, Cheolsoo Park cheol...@cloudera.com wrote: Here are my results: Hadoop-1.0.x: [exec] Final results ,PASSED: 612 FAILED: 2 SKIPPED: 24 ABORTED: 1FAILED DEPENDENCY: 0 - The failures are simply because I didn't install Hcatalog. Hadoop-2.0.x: [exec] Final results ,PASSED: 567 FAILED: 3 SKIPPED: 27 ABORTED: 42 FAILED DEPENDENCY: 0 - The failures seem due to issues in my cluster rather than Pig issues. I will re-run them to verify. On Fri, Jan 25, 2013 at 5:31 PM, Cheolsoo Park cheol...@cloudera.com wrote: I will also run e2e on Hadoop-1.x and Hadoop-2.x. On Fri, Jan 25, 2013 at 5:02 PM, Daniel Dai da...@hortonworks.com wrote: I will run e2e tests on Hadoop 1.x over the weekend. Thanks, Daniel On Fri, Jan 25, 2013 at 4:27 PM, Rohini Palaniswamy rohini.adi...@gmail.com wrote: Thats good :). Unit tests have all been passing. I haven't run e2e tests on pig 0.11 for sometime. Will kick off one this weekend. It would be nice if Cheolsoo and Daniel can also kick off one run. Regards, Rohini On Fri, Jan 25, 2013 at 4:08 PM, Julien Le Dem jul...@twitter.com wrote: It looks like all the tickets for Pig 0.11 have been resolved as of today. See: https://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%20%220.11%22%20AND%20project%20%3D%20PIG%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20updated%20DESC%2C%20created%20ASC%2C%20priority%20DESC I propose we make the release 0.11.0 next week. Julien -- *Note that I'm no longer using my Yahoo! email address. Please email me at billgra...@gmail.com going forward.*
[jira] [Commented] (PIG-2994) Grunt shortcuts
[ https://issues.apache.org/jira/browse/PIG-2994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13566225#comment-13566225 ] Prasanth J commented on PIG-2994: - Ping! Can someone please review this patch? Not sure to whom I should send review request in RB. Grunt shortcuts --- Key: PIG-2994 URL: https://issues.apache.org/jira/browse/PIG-2994 Project: Pig Issue Type: New Feature Components: grunt Reporter: Prasanth J Assignee: Prasanth J Priority: Minor Attachments: PIG-2994.1.git.patch This feature is aimed at providing shortcuts for frequently used commands like illustrate, dump, explain, describe, quit, help etc. This feature is inspired from postgres(psql) shortcuts. I tried implementing a simple shortcut for quitting the grunt shell using \q with very minimal changes. I think this feature will help save many keystrokes for users. If this feature looks useful I can submit the current patch for review and go ahead with implementing the following shortcuts \i alias - illustrate \e alias - explain \de alias - describe \du alias - dump \h - help This will also be useful to view information about tables/statistics stored in HCatalog similar to the way psql does. \dt alias - display table \dm - display metadata etc.. except \t, \r and \n delimiters we should be able to use all other characters as shortcuts. Please let me know your thoughts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
