[jira] [Updated] (PIG-2221) Couldnt find documentation for ColumnMapKeyPrune optimization rule
[ https://issues.apache.org/jira/browse/PIG-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Corinne Chandel updated PIG-2221: - Attachment: pig-2221.patch Includes fixes for (1) optimzation rules (perf.xml) and (2) required updates to the Pig Index (pig-index.xml) > Couldnt find documentation for ColumnMapKeyPrune optimization rule > -- > > Key: PIG-2221 > URL: https://issues.apache.org/jira/browse/PIG-2221 > Project: Pig > Issue Type: Bug > Components: documentation >Affects Versions: 0.8.1 >Reporter: Vivek Padmanabhan >Assignee: Corinne Chandel > Fix For: 0.9.1 > > Attachments: PIG-2221-help.patch, pig-2221.patch > > > There are no documentations for some of the Optimization Rules, > For ex; ColumnMapKeyPrune > in http://pig.apache.org/docs/r0.8.1/piglatin_ref1.html#Optimization+Rules > And moreover I believe the documentaion should be saying how to disable these > rules using -t option. > It would be nice if the documentation could talk of some uses cases where it > makes sense to disable the optimization rule. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2221) Couldnt find documentation for ColumnMapKeyPrune optimization rule
[ https://issues.apache.org/jira/browse/PIG-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Corinne Chandel updated PIG-2221: - Status: Patch Available (was: Open) Apply patch to Trunk and Branch-9. > Couldnt find documentation for ColumnMapKeyPrune optimization rule > -- > > Key: PIG-2221 > URL: https://issues.apache.org/jira/browse/PIG-2221 > Project: Pig > Issue Type: Bug > Components: documentation >Affects Versions: 0.8.1 >Reporter: Vivek Padmanabhan >Assignee: Corinne Chandel > Fix For: 0.9.1 > > Attachments: PIG-2221-help.patch, pig-2221.patch > > > There are no documentations for some of the Optimization Rules, > For ex; ColumnMapKeyPrune > in http://pig.apache.org/docs/r0.8.1/piglatin_ref1.html#Optimization+Rules > And moreover I believe the documentaion should be saying how to disable these > rules using -t option. > It would be nice if the documentation could talk of some uses cases where it > makes sense to disable the optimization rule. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (PIG-2257) AvroStorage doesn't recognize schema_file field when JSON isn't used in the constructor
AvroStorage doesn't recognize schema_file field when JSON isn't used in the
constructor
---
Key: PIG-2257
URL: https://issues.apache.org/jira/browse/PIG-2257
Project: Pig
Issue Type: Bug
Reporter: Bill Graham
Assignee: Bill Graham
PIG-2195 introduced the {{schema_file}} constructor param to {{AvroStorage}}.
This field is currently only supported when passing constructor data via JSON
though (a different code path is used when constructor data is a String array).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2256) Upgrade Avro dependency to 1.5.3
[ https://issues.apache.org/jira/browse/PIG-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094158#comment-13094158 ] Tom White commented on PIG-2256: Thanks Dmitriy! 1.5.3 is a bugfix release, and it contains AVRO-880 which is useful for allowing more control over loading native snappy libraries. The format is forwards and backwards compatible between these releases. (In fact, my understanding is that incompatible format changes are only allowed on major versions.) > Upgrade Avro dependency to 1.5.3 > > > Key: PIG-2256 > URL: https://issues.apache.org/jira/browse/PIG-2256 > Project: Pig > Issue Type: Improvement > Components: build >Reporter: Alejandro Abdelnur > Attachments: PIG-2256v1.patch > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2256) Upgrade Avro dependency to 1.5.3
[ https://issues.apache.org/jira/browse/PIG-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094141#comment-13094141 ] Bill Graham commented on PIG-2256: -- FYI, PIG-2202 upgraded AvroStorage to work with both the 1.4.1 and 1.5.1 Avro APIs. I'll let someone else from Avro comment on the serialization and release schedule questions. > Upgrade Avro dependency to 1.5.3 > > > Key: PIG-2256 > URL: https://issues.apache.org/jira/browse/PIG-2256 > Project: Pig > Issue Type: Improvement > Components: build >Reporter: Alejandro Abdelnur > Attachments: PIG-2256v1.patch > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2115) Pig HBaseStorage configuration and setup issues
[ https://issues.apache.org/jira/browse/PIG-2115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094130#comment-13094130 ] Greg Bowyer commented on PIG-2115: -- did we get this looked at ? I just found this patch at the end of my todo list :S > Pig HBaseStorage configuration and setup issues > --- > > Key: PIG-2115 > URL: https://issues.apache.org/jira/browse/PIG-2115 > Project: Pig > Issue Type: Bug >Affects Versions: 0.9.0 >Reporter: Greg Bowyer > Labels: hbase, load, storage > Attachments: > HBaseStorage-correctly-configure-child-MapR-task-v1.patch, > HBaseStorage-correctly-configure-child-MapR-task-v2.patch > > > HBase storage currently configures child MapR jobs assuming that the HBase > jars and cluster configuration are present on the hadoop cluster that will > run the pig program. In some circumstances, and for some configurations this > causes problems with both the configuration suddenly becoming defaulted > during a job run as well as ClassNotFound exceptions occurring due to the > child MapReduce job being incorrectly configured. > As such, the hbase built in configuration is merged with the job > configuration using the inbuilt pig classes for this functionality, this > prevents the jobs configuration from being overwritten by defaults. > At the same time this change has been mirrored in the relevant setup methods > called when the HBaseStorage is used as a StoreFunc; this means that the > StoreFunc also correctly sets up the relevant classpath preventing > ClassNotFound exceptions in clusters that are not adding the HBase classes to > the HADOOP_CLASSPATH. > Attached is a patch that hopefully fixes these issues for others. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2256) Upgrade Avro dependency to 1.5.3
[ https://issues.apache.org/jira/browse/PIG-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094129#comment-13094129 ] Dmitriy V. Ryaboy commented on PIG-2256: I *just* bumped the version to 1.5.2 for Tom :-) Does Avro have a release schedule and notes I can look at? Is the serialization format backwards and forwards compatible? > Upgrade Avro dependency to 1.5.3 > > > Key: PIG-2256 > URL: https://issues.apache.org/jira/browse/PIG-2256 > Project: Pig > Issue Type: Improvement > Components: build >Reporter: Alejandro Abdelnur > Attachments: PIG-2256v1.patch > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2256) Upgrade Avro dependency to 1.5.3
[ https://issues.apache.org/jira/browse/PIG-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated PIG-2256: Status: Patch Available (was: Open) > Upgrade Avro dependency to 1.5.3 > > > Key: PIG-2256 > URL: https://issues.apache.org/jira/browse/PIG-2256 > Project: Pig > Issue Type: Improvement > Components: build >Reporter: Alejandro Abdelnur > Attachments: PIG-2256v1.patch > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2256) Upgrade Avro dependency to 1.5.3
[ https://issues.apache.org/jira/browse/PIG-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated PIG-2256: Attachment: PIG-2256v1.patch > Upgrade Avro dependency to 1.5.3 > > > Key: PIG-2256 > URL: https://issues.apache.org/jira/browse/PIG-2256 > Project: Pig > Issue Type: Improvement > Components: build >Reporter: Alejandro Abdelnur > Attachments: PIG-2256v1.patch > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (PIG-2256) Upgrade Avro dependency to 1.5.3
Upgrade Avro dependency to 1.5.3 Key: PIG-2256 URL: https://issues.apache.org/jira/browse/PIG-2256 Project: Pig Issue Type: Improvement Components: build Reporter: Alejandro Abdelnur -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2031) NPE in TOP
[
https://issues.apache.org/jira/browse/PIG-2031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan updated PIG-2031:
-
Assignee: Jacob Perkins
> NPE in TOP
> --
>
> Key: PIG-2031
> URL: https://issues.apache.org/jira/browse/PIG-2031
> Project: Pig
> Issue Type: Bug
>Reporter: Jacob Perkins
>Assignee: Jacob Perkins
> Labels: newbie
> Attachments: toppatch.txt
>
>
> If a NULL DataBag is passed to org.apache.pig.builtin.TOP then a NPE is
> thrown. Consider:
> {code}
> $: cat foo.tsv
> a {(foo,1),(bar,2)}
> b
> c {(fyha,4),(asdf,9)}
> {code}
> then:
> {code}
> data = LOAD 'foo.tsv' AS (key:chararray, a_bag:bag {t:tuple (name:chararray,
> value:int)});
> tpd = FOREACH data {
> top_n = TOP(1, 1, a_bag);
> GENERATE
> key AS key,
> top_n AS top_n
> ;
> };
> DUMP tpd;
> {code}
> will throw an NPE when it gets to the row with no bag.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Sometimes-Algebraic functions
Wow this is so much less trivial than I thought it was going to be! :) D On Mon, Aug 29, 2011 at 9:36 PM, Ashutosh Chauhan wrote: > Yup, that makes sense. Class returned by argToFuncMapping needs to > implement > Algebraic instead of EvalFunc itself. > > Ashutosh > On Mon, Aug 29, 2011 at 15:46, Dmitriy Ryaboy wrote: > > > Had a fun discovery today: a user tried to do the following (perfectly > > reasonable) thing: > > > > bag_sizes = foreach grouped_data genrate > > group, SIZE(grouped_data); > > > > .. and it was excessively slow, because SIZE is not algebraic. > > > > I had him switch to COUNT_STAR, and that worked, of course. > > > > But he reasonably pointed out that SIZE should work, too. It's documented > > to > > work on bags, after all. > > > > I tried switching SIZE's argToFunc mapping to return > > COUNT_STAR.class.getName(), but discovered much to my dismay that > algebraic > > optimization still does not get invoked. > > > > Presumably because we check if SIZE is instanceof Algebraic, instead of > > checking the class that's returned by argToFuncMapping. > > > > Is that about right? Do you guys agree that's a bug? > > > > D > > >
[jira] [Commented] (PIG-2163) Improve nested cross to stream one relation
[ https://issues.apache.org/jira/browse/PIG-2163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13093990#comment-13093990 ] Daniel Dai commented on PIG-2163: - Thanks Zhijie, patch works good. One comment, as I mentioned in my last comment, Pig usually stream the rightmost relation. We wanna that here as well, that will improves user experience, and simplify the documentation. > Improve nested cross to stream one relation > --- > > Key: PIG-2163 > URL: https://issues.apache.org/jira/browse/PIG-2163 > Project: Pig > Issue Type: Improvement > Components: impl >Affects Versions: 0.10 >Reporter: Daniel Dai >Assignee: Zhijie Shen > Fix For: 0.10 > > Attachments: PIG-2163.patch > > > PIG-1916 added nested cross support for PIG. One optimization is instead of > materialize all bags before producing result, we can stream one of the input > to save on memory. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (PIG-2255) SIZE should be algebraic when invoked on a bag
SIZE should be algebraic when invoked on a bag -- Key: PIG-2255 URL: https://issues.apache.org/jira/browse/PIG-2255 Project: Pig Issue Type: Bug Reporter: Dmitriy V. Ryaboy Assignee: Dmitriy V. Ryaboy SIZE of a bag is equivalent to COUNT_STAR, and should get optimized accordingly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (PIG-2254) Algebraic and Accumulative behavior should depend on udf from ArgToFuncMapping
[ https://issues.apache.org/jira/browse/PIG-2254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy V. Ryaboy reassigned PIG-2254: -- Assignee: Dmitriy V. Ryaboy > Algebraic and Accumulative behavior should depend on udf from ArgToFuncMapping > -- > > Key: PIG-2254 > URL: https://issues.apache.org/jira/browse/PIG-2254 > Project: Pig > Issue Type: Bug >Reporter: Dmitriy V. Ryaboy >Assignee: Dmitriy V. Ryaboy > > Currently, we check whether the UDF specified in the user script is Algebraic > or Accumulative in order to determine whether these optimizations can kick > in. When argToFuncMapping is implemented, the UDF we actually run might not > be the one we checked; we should be checking the UDF we are going to run, > instead. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (PIG-2254) Algebraic and Accumulative behavior should depend on udf from ArgToFuncMapping
Algebraic and Accumulative behavior should depend on udf from ArgToFuncMapping -- Key: PIG-2254 URL: https://issues.apache.org/jira/browse/PIG-2254 Project: Pig Issue Type: Bug Reporter: Dmitriy V. Ryaboy Currently, we check whether the UDF specified in the user script is Algebraic or Accumulative in order to determine whether these optimizations can kick in. When argToFuncMapping is implemented, the UDF we actually run might not be the one we checked; we should be checking the UDF we are going to run, instead. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (PIG-2206) Make Pig unit test work on Mac
[ https://issues.apache.org/jira/browse/PIG-2206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai resolved PIG-2206. - Resolution: Invalid I get a clear run on Mac. > Make Pig unit test work on Mac > -- > > Key: PIG-2206 > URL: https://issues.apache.org/jira/browse/PIG-2206 > Project: Pig > Issue Type: Bug > Components: impl >Affects Versions: 0.9.1, 0.10 >Reporter: Daniel Dai >Priority: Minor > Fix For: 0.10 > > > I saw two failures on Mac: > TestGrunt.testShellCommand > TestInvoker.testSpeed -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: going to request yourkit license for committers
+1 I have never used Yourkit before, but it looks more professional than what I have used up to now to profile Pig. Cheers, -- Gianmarco Il giorno 30/ago/11, alle ore 07:04, Ashutosh Chauhan ha scritto: +1 I have used Yourkit previously for profiling Pig and its more then useful. Ashutosh On Mon, Aug 29, 2011 at 15:02, Santhosh Srinivasan inc.com>wrote: +1 - this will be very beneficial! -Original Message- From: Thejas Nair [mailto:the...@hortonworks.com] Sent: Monday, August 29, 2011 2:57 PM To: dev@pig.apache.org Subject: going to request yourkit license for committers FYI- Yourkit is very useful java profiling tool and they give license for free for use by open source projects. I am planning to request license for use by pig committers. But they need a reference from the web pages of the project to their website. - http://www.yourkit.com/purchase/index.jsp . I believe a link from a credits page should be sufficient. As the project would need to thank them, I am sharing my plan before contacting them. Thanks, Thejas
[jira] [Updated] (PIG-2253) --config option does not work
[ https://issues.apache.org/jira/browse/PIG-2253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Butler updated PIG-2253: - Attachment: pig.patch The solution works but is rather inelegant, I have just reordered the code to the avoid the problem. The real problem is the secretDebugCmd and config option are parsed in different ways. > --config option does not work > - > > Key: PIG-2253 > URL: https://issues.apache.org/jira/browse/PIG-2253 > Project: Pig > Issue Type: Bug > Components: impl >Affects Versions: 0.8.1, 0.9.0 >Reporter: Mark Butler >Priority: Minor > Attachments: pig.patch > > > The pig shell script has a --config option like hadoop. However it does not, > because the "secretDebugCmd" option makes a copy of the arguments and places > it in a variable called "remaining". This means "--config > /my/config/directory" is passed to the Java class org.apache.pig.Main. The > class does not recognize the "--config" option so instead of grunt starting > it prints the usage message and exits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2253) --config option does not work
[ https://issues.apache.org/jira/browse/PIG-2253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13093696#comment-13093696 ] Mark Butler commented on PIG-2253: -- Sorry, I did not understand Jira markup. What I meant to say was: The pig shell script has a config option like Hadoop. However it does not work, because the code processing the "secretDebugCmd" option makes a copy of the command line arguments and places it in a variable called "remaining". This is passed to the Java class, so if the command line contains a config option this is passed to the Java class. The class does not recognize the "config" option so instead of grunt starting it prints the usage message and exits. > --config option does not work > - > > Key: PIG-2253 > URL: https://issues.apache.org/jira/browse/PIG-2253 > Project: Pig > Issue Type: Bug > Components: impl >Affects Versions: 0.8.1, 0.9.0 >Reporter: Mark Butler >Priority: Minor > > The pig shell script has a --config option like hadoop. However it does not, > because the "secretDebugCmd" option makes a copy of the arguments and places > it in a variable called "remaining". This means "--config > /my/config/directory" is passed to the Java class org.apache.pig.Main. The > class does not recognize the "--config" option so instead of grunt starting > it prints the usage message and exits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (PIG-2253) --config option does not work
--config option does not work - Key: PIG-2253 URL: https://issues.apache.org/jira/browse/PIG-2253 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.9.0, 0.8.1 Reporter: Mark Butler Priority: Minor The pig shell script has a --config option like hadoop. However it does not, because the "secretDebugCmd" option makes a copy of the arguments and places it in a variable called "remaining". This means "--config /my/config/directory" is passed to the Java class org.apache.pig.Main. The class does not recognize the "--config" option so instead of grunt starting it prints the usage message and exits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (PIG-2250) Pig 0.9 error message not useful as compared to 0.8
[
https://issues.apache.org/jira/browse/PIG-2250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vivek Padmanabhan resolved PIG-2250.
Resolution: Fixed
Fix Version/s: 0.10
Verified this with trunk. Sorry for the trouble.
> Pig 0.9 error message not useful as compared to 0.8
> ---
>
> Key: PIG-2250
> URL: https://issues.apache.org/jira/browse/PIG-2250
> Project: Pig
> Issue Type: Bug
>Affects Versions: 0.9.0
>Reporter: Vivek Padmanabhan
> Fix For: 0.10
>
>
> Another instance of change in error message from 0.8 to 0.9 due to parser
> modifications.
> This improper error message is due to \n in the UDF arguments.
> The below is a sample script;
> a = load 'input' using myLoader('a1,a2,
> a3,a4');
> dump a;
> Error Message from 0.9
> --
> ERROR 1200: Pig script failed to parse: MismatchedTokenException(93!=3)
> Error Message from 0.8
>
> ERROR 1000: Error during parsing. Lexical error at line 1, column 40.
> Encountered: "\n" (10), after : "\'a1,a2,"
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2250) Pig 0.9 error message not useful as compared to 0.8
[
https://issues.apache.org/jira/browse/PIG-2250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13093553#comment-13093553
]
Daniel Dai commented on PIG-2250:
-
Seems it is fixed by PIG-2215 on trunk.
> Pig 0.9 error message not useful as compared to 0.8
> ---
>
> Key: PIG-2250
> URL: https://issues.apache.org/jira/browse/PIG-2250
> Project: Pig
> Issue Type: Bug
>Affects Versions: 0.9.0
>Reporter: Vivek Padmanabhan
>
> Another instance of change in error message from 0.8 to 0.9 due to parser
> modifications.
> This improper error message is due to \n in the UDF arguments.
> The below is a sample script;
> a = load 'input' using myLoader('a1,a2,
> a3,a4');
> dump a;
> Error Message from 0.9
> --
> ERROR 1200: Pig script failed to parse: MismatchedTokenException(93!=3)
> Error Message from 0.8
>
> ERROR 1000: Error during parsing. Lexical error at line 1, column 40.
> Encountered: "\n" (10), after : "\'a1,a2,"
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2249) Enable pig e2e testing on EC2
[ https://issues.apache.org/jira/browse/PIG-2249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13093544#comment-13093544 ] Daniel Dai commented on PIG-2249: - Works for me. Need to mention: 1. Tested with whirr 0.6. It fails in my first try with whirr 0.4 2. I need to install svn and DBI by myself +1 for commit. > Enable pig e2e testing on EC2 > - > > Key: PIG-2249 > URL: https://issues.apache.org/jira/browse/PIG-2249 > Project: Pig > Issue Type: New Feature > Components: tools >Affects Versions: 0.10 >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 0.10 > > Attachments: PIG-2249.patch > > > We need to enable users to test Pig on actual Hadoop clusters. We can use > Whirr to allow users to easily spin up instances on EC2 and run the > end-to-end tests. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
