[jira] Commented: (PIG-794) Use Avro serialization in Pig
[ https://issues.apache.org/jira/browse/PIG-794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884577#action_12884577 ] Jeff Zhang commented on PIG-794: We can leverage AvroInputFormat and AvroOutputFormat in Avro trunk, (see AVRO-493) > Use Avro serialization in Pig > - > > Key: PIG-794 > URL: https://issues.apache.org/jira/browse/PIG-794 > Project: Pig > Issue Type: Improvement > Components: impl >Affects Versions: 0.2.0 >Reporter: Rakesh Setty >Assignee: Dmitriy V. Ryaboy > Attachments: avro-0.1-dev-java_r765402.jar, AvroStorage.patch, > jackson-asl-0.9.4.jar, PIG-794.patch > > > We would like to use Avro serialization in Pig to pass data between MR jobs > instead of the current BinStorage. Attached is an implementation of > AvroBinStorage which performs significantly better compared to BinStorage on > our benchmarks. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1478) Add progress notification listener to PigRunner API
[
https://issues.apache.org/jira/browse/PIG-1478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884677#action_12884677
]
Hadoop QA commented on PIG-1478:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12448532/PIG-1478.patch
against trunk revision 959865.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/358/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/358/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/358/console
This message is automatically generated.
> Add progress notification listener to PigRunner API
> ---
>
> Key: PIG-1478
> URL: https://issues.apache.org/jira/browse/PIG-1478
> Project: Pig
> Issue Type: Improvement
>Reporter: Richard Ding
>Assignee: Richard Ding
> Fix For: 0.8.0
>
> Attachments: PIG-1478.patch
>
>
> PIG-1333 added PigRunner API to allow Pig users and tools to get a
> status/stats object back after executing a Pig script. The new API, however,
> is synchronous (blocking). It's known that a Pig script can spawn tens (even
> hundreds) MR jobs and take hours to complete. Therefore it'll be nice to give
> progress feedback to the callers during the execution.
> The proposal is to add an optional parameter to the API:
> {code}
> public abstract class PigRunner {
> public static PigStats run(String[] args, PigProgressNotificationListener
> listener) {...}
> }
> {code}
> The new listener is defined as following:
> {code}
> package org.apache.pig.tools.pigstats;
> public interface PigProgressNotificationListener extends
> java.util.EventListener {
> // just before the launch of MR jobs for the script
> public void LaunchStartedNotification(int numJobsToLaunch);
> // number of jobs submitted in a batch
> public void jobsSubmittedNotification(int numJobsSubmitted);
> // a job is started
> public void jobStartedNotification(String assignedJobId);
> // a job is completed successfully
> public void jobFinishedNotification(JobStats jobStats);
> // a job is failed
> public void jobFailedNotification(JobStats jobStats);
> // a user output is completed successfully
> public void outputCompletedNotification(OutputStats outputStats);
> // updates the progress as percentage
> public void progressUpdatedNotification(int progress);
> // the script execution is done
> public void launchCompletedNotification(int numJobsSucceeded);
> }
> {code}
> Any thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1478) Add progress notification listener to PigRunner API
[
https://issues.apache.org/jira/browse/PIG-1478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884732#action_12884732
]
Hadoop QA commented on PIG-1478:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12448532/PIG-1478.patch
against trunk revision 959865.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/337/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/337/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/337/console
This message is automatically generated.
> Add progress notification listener to PigRunner API
> ---
>
> Key: PIG-1478
> URL: https://issues.apache.org/jira/browse/PIG-1478
> Project: Pig
> Issue Type: Improvement
>Reporter: Richard Ding
>Assignee: Richard Ding
> Fix For: 0.8.0
>
> Attachments: PIG-1478.patch
>
>
> PIG-1333 added PigRunner API to allow Pig users and tools to get a
> status/stats object back after executing a Pig script. The new API, however,
> is synchronous (blocking). It's known that a Pig script can spawn tens (even
> hundreds) MR jobs and take hours to complete. Therefore it'll be nice to give
> progress feedback to the callers during the execution.
> The proposal is to add an optional parameter to the API:
> {code}
> public abstract class PigRunner {
> public static PigStats run(String[] args, PigProgressNotificationListener
> listener) {...}
> }
> {code}
> The new listener is defined as following:
> {code}
> package org.apache.pig.tools.pigstats;
> public interface PigProgressNotificationListener extends
> java.util.EventListener {
> // just before the launch of MR jobs for the script
> public void LaunchStartedNotification(int numJobsToLaunch);
> // number of jobs submitted in a batch
> public void jobsSubmittedNotification(int numJobsSubmitted);
> // a job is started
> public void jobStartedNotification(String assignedJobId);
> // a job is completed successfully
> public void jobFinishedNotification(JobStats jobStats);
> // a job is failed
> public void jobFailedNotification(JobStats jobStats);
> // a user output is completed successfully
> public void outputCompletedNotification(OutputStats outputStats);
> // updates the progress as percentage
> public void progressUpdatedNotification(int progress);
> // the script execution is done
> public void launchCompletedNotification(int numJobsSucceeded);
> }
> {code}
> Any thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (PIG-1449) RegExLoader hangs on lines that don't match the regular expression
[
https://issues.apache.org/jira/browse/PIG-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmitriy V. Ryaboy closed PIG-1449.
--
> RegExLoader hangs on lines that don't match the regular expression
> --
>
> Key: PIG-1449
> URL: https://issues.apache.org/jira/browse/PIG-1449
> Project: Pig
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Justin Sanders
>Priority: Minor
> Fix For: 0.8.0
>
> Attachments: PIG-1449-RegExLoaderInfiniteLoopFix.patch,
> RegExLoader.patch
>
>
> In the 0.7.0 changes to RegExLoader there was a bug introduced where the code
> will stay in the while loop if the line isn't matched. Before 0.7.0 these
> lines would be skipped if they didn't match the regular expression. The
> result is the mapper will not respond and will time out with "Task attempt_X
> failed to report status for 600 seconds. Killing!".
> Here are the steps to recreate the bug:
> Create a text file in HDFS with the following lines:
> test1
> testA
> test2
> Run the following pig script:
> REGISTER /usr/local/pig/contrib/piggybank/java/piggybank.jar;
> test = LOAD '/path/to/test.txt' using
> org.apache.pig.piggybank.storage.MyRegExLoader('(test\\d)') AS (line);
> dump test;
> Expected result:
> (test1)
> (test3)
> Actual result:
> Job fails to complete after 600 second timeout waiting on the mapper to
> complete. The mapper hangs at 33% since it can process the first line but
> gets stuck into the while loop on the second line.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1469) DefaultDataBag assumes ArrayList as default List type
[ https://issues.apache.org/jira/browse/PIG-1469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy V. Ryaboy updated PIG-1469: --- Status: Resolved (was: Patch Available) Resolution: Fixed I committed this. > DefaultDataBag assumes ArrayList as default List type > - > > Key: PIG-1469 > URL: https://issues.apache.org/jira/browse/PIG-1469 > Project: Pig > Issue Type: Bug > Components: data >Affects Versions: 0.8.0 >Reporter: Gianmarco De Francisci Morales >Assignee: Gianmarco De Francisci Morales > Fix For: 0.8.0 > > Attachments: PIG-1469.patch > > > In org.apache.pig.data.DefaultDataBag, the field mContents is assumed to be > of type ArrayList but the user can actually pass a different List to the > constructor. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Closed: (PIG-1469) DefaultDataBag assumes ArrayList as default List type
[ https://issues.apache.org/jira/browse/PIG-1469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy V. Ryaboy closed PIG-1469. -- > DefaultDataBag assumes ArrayList as default List type > - > > Key: PIG-1469 > URL: https://issues.apache.org/jira/browse/PIG-1469 > Project: Pig > Issue Type: Bug > Components: data >Affects Versions: 0.8.0 >Reporter: Gianmarco De Francisci Morales >Assignee: Gianmarco De Francisci Morales > Fix For: 0.8.0 > > Attachments: PIG-1469.patch > > > In org.apache.pig.data.DefaultDataBag, the field mContents is assumed to be > of type ArrayList but the user can actually pass a different List to the > constructor. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1434) Allow casting relations to scalars
[
https://issues.apache.org/jira/browse/PIG-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884750#action_12884750
]
Richard Ding commented on PIG-1434:
---
How about a "replicated" cross?
{code}
A = load 'data' as (x, y, z);
B = group A all;
C = foreach B generate COUNT(A) as count, MAX(A.y) as max;
.
X =
Y = cross X, C using 'repl';
Z = foreach Y generate X::$1/(long) C.count, X::$2-(long) C.max;
{code}
> Allow casting relations to scalars
> --
>
> Key: PIG-1434
> URL: https://issues.apache.org/jira/browse/PIG-1434
> Project: Pig
> Issue Type: Improvement
>Reporter: Olga Natkovich
>Assignee: Aniket Mokashi
> Fix For: 0.8.0
>
> Attachments: scalarImpl.patch
>
>
> This jira is to implement a simplified version of the functionality described
> in https://issues.apache.org/jira/browse/PIG-801.
> The proposal is to allow casting relations to scalar types in foreach.
> Example:
> A = load 'data' as (x, y, z);
> B = group A all;
> C = foreach B generate COUNT(A);
> .
> X =
> Y = foreach X generate $1/(long) C;
> Couple of additional comments:
> (1) You can only cast relations including a single value or an error will be
> reported
> (2) Name resolution is needed since relation X might have field named C in
> which case that field takes precedence.
> (3) Y will look for C closest to it.
> Implementation thoughts:
> The idea is to store C into a file and then convert it into scalar via a UDF.
> I believe we already have a UDF that Ben Reed contributed for this purpose.
> Most of the work would be to update the logical plan to
> (1) Store C
> (2) convert the cast to the UDF
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1434) Allow casting relations to scalars
[ https://issues.apache.org/jira/browse/PIG-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884755#action_12884755 ] Dmitriy V. Ryaboy commented on PIG-1434: Richard, In my opinion -- Yes in principle, but not as a replacement for this. Cross is dangerous, I would rather have a constrained "implicit scalar tuple" that people use for the common case, and leave something like a replicated cross for power users (and then not limit it to number of rows in replicated relation). Agreed with not constraining this feature on the frontend, failing at runtime instead. > Allow casting relations to scalars > -- > > Key: PIG-1434 > URL: https://issues.apache.org/jira/browse/PIG-1434 > Project: Pig > Issue Type: Improvement >Reporter: Olga Natkovich >Assignee: Aniket Mokashi > Fix For: 0.8.0 > > Attachments: scalarImpl.patch > > > This jira is to implement a simplified version of the functionality described > in https://issues.apache.org/jira/browse/PIG-801. > The proposal is to allow casting relations to scalar types in foreach. > Example: > A = load 'data' as (x, y, z); > B = group A all; > C = foreach B generate COUNT(A); > . > X = > Y = foreach X generate $1/(long) C; > Couple of additional comments: > (1) You can only cast relations including a single value or an error will be > reported > (2) Name resolution is needed since relation X might have field named C in > which case that field takes precedence. > (3) Y will look for C closest to it. > Implementation thoughts: > The idea is to store C into a file and then convert it into scalar via a UDF. > I believe we already have a UDF that Ben Reed contributed for this purpose. > Most of the work would be to update the logical plan to > (1) Store C > (2) convert the cast to the UDF -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1434) Allow casting relations to scalars
[ https://issues.apache.org/jira/browse/PIG-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884759#action_12884759 ] Richard Ding commented on PIG-1434: --- I agree that we should use the right syntax. What I meant was that it can be implemented as a 'replicated' cross which seems to solve the problems of implicit dependency and using distributed cache. > Allow casting relations to scalars > -- > > Key: PIG-1434 > URL: https://issues.apache.org/jira/browse/PIG-1434 > Project: Pig > Issue Type: Improvement >Reporter: Olga Natkovich >Assignee: Aniket Mokashi > Fix For: 0.8.0 > > Attachments: scalarImpl.patch > > > This jira is to implement a simplified version of the functionality described > in https://issues.apache.org/jira/browse/PIG-801. > The proposal is to allow casting relations to scalar types in foreach. > Example: > A = load 'data' as (x, y, z); > B = group A all; > C = foreach B generate COUNT(A); > . > X = > Y = foreach X generate $1/(long) C; > Couple of additional comments: > (1) You can only cast relations including a single value or an error will be > reported > (2) Name resolution is needed since relation X might have field named C in > which case that field takes precedence. > (3) Y will look for C closest to it. > Implementation thoughts: > The idea is to store C into a file and then convert it into scalar via a UDF. > I believe we already have a UDF that Ben Reed contributed for this purpose. > Most of the work would be to update the logical plan to > (1) Store C > (2) convert the cast to the UDF -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-928) UDFs in scripting languages
[ https://issues.apache.org/jira/browse/PIG-928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884763#action_12884763 ] Dmitriy V. Ryaboy commented on PIG-928: --- Aniket, the patch does not apply cleanly to trunk, can you rebase it? > UDFs in scripting languages > --- > > Key: PIG-928 > URL: https://issues.apache.org/jira/browse/PIG-928 > Project: Pig > Issue Type: New Feature >Reporter: Alan Gates >Assignee: Aniket Mokashi > Fix For: 0.8.0 > > Attachments: calltrace.png, package.zip, pig-greek.tgz, > pig.scripting.patch.arnab, pyg.tgz, RegisterPythonUDF2.patch, > RegisterPythonUDF3.patch, RegisterPythonUDF4.patch, > RegisterScriptUDFDefineParse.patch, scripting.tgz, scripting.tgz, test.zip > > > It should be possible to write UDFs in scripting languages such as python, > ruby, etc. This frees users from needing to compile Java, generate a jar, > etc. It also opens Pig to programmers who prefer scripting languages over > Java. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-928) UDFs in scripting languages
[ https://issues.apache.org/jira/browse/PIG-928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy V. Ryaboy updated PIG-928: -- Attachment: PIG-928.patch I rebased the patch and made it pull jython down via maven. 2.5.1 doesn't appear to be available right now, so this pulls down 2.5.0. Hope that's ok. Looks like the tabulation is wrong in most of this patch.. someone please hit ctrl-a, ctrl-i next time :). Needless to say, this thing needs tests, desperately. Also imho in order for it to make it into trunk, it should be a compile-time option to support (and pull down) jython or jruby or whatnot, not a default option. Otherwise we are well on our way to making people pull down the internet in order to compile pig. > UDFs in scripting languages > --- > > Key: PIG-928 > URL: https://issues.apache.org/jira/browse/PIG-928 > Project: Pig > Issue Type: New Feature >Reporter: Alan Gates >Assignee: Aniket Mokashi > Fix For: 0.8.0 > > Attachments: calltrace.png, package.zip, PIG-928.patch, > pig-greek.tgz, pig.scripting.patch.arnab, pyg.tgz, RegisterPythonUDF2.patch, > RegisterPythonUDF3.patch, RegisterPythonUDF4.patch, > RegisterScriptUDFDefineParse.patch, scripting.tgz, scripting.tgz, test.zip > > > It should be possible to write UDFs in scripting languages such as python, > ruby, etc. This frees users from needing to compile Java, generate a jar, > etc. It also opens Pig to programmers who prefer scripting languages over > Java. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1309) Map-side Cogroup
[ https://issues.apache.org/jira/browse/PIG-1309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated PIG-1309: -- Attachment: PIG_1309_7.patch Backport of merge cogroup for 0.7 branch. Since, hudson can test only for trunk. Manually ran all the tests, all passed. > Map-side Cogroup > > > Key: PIG-1309 > URL: https://issues.apache.org/jira/browse/PIG-1309 > Project: Pig > Issue Type: Bug > Components: impl >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Fix For: 0.8.0 > > Attachments: mapsideCogrp.patch, pig-1309_1.patch, pig-1309_2.patch, > PIG_1309_7.patch > > > In never ending quest to make Pig go faster, we want to parallelize as many > relational operations as possible. Its already possible to do Group-by( > PIG-984 ) and Joins( PIG-845 , PIG-554 ) purely in map-side in Pig. This jira > is to add map-side implementation of Cogroup in Pig. Details to follow. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1434) Allow casting relations to scalars
[
https://issues.apache.org/jira/browse/PIG-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884791#action_12884791
]
Thejas M Nair commented on PIG-1434:
I think the replicated cross is a good alternative to this feature, though this
feature is probably more friendly for a beginner pig user. But if this feature
makes the pig code very complicated/hacky (the dependency order and stuff), I
think it might not be a bad idea to encourage the use of replicated-join
instead .
As a side note, we can actually get 'replicated cross' working using replicated
join -
eg -
{code}
j = join l1 by 1, l2 by 1 using 'replicated';
{code}
> Allow casting relations to scalars
> --
>
> Key: PIG-1434
> URL: https://issues.apache.org/jira/browse/PIG-1434
> Project: Pig
> Issue Type: Improvement
>Reporter: Olga Natkovich
>Assignee: Aniket Mokashi
> Fix For: 0.8.0
>
> Attachments: scalarImpl.patch
>
>
> This jira is to implement a simplified version of the functionality described
> in https://issues.apache.org/jira/browse/PIG-801.
> The proposal is to allow casting relations to scalar types in foreach.
> Example:
> A = load 'data' as (x, y, z);
> B = group A all;
> C = foreach B generate COUNT(A);
> .
> X =
> Y = foreach X generate $1/(long) C;
> Couple of additional comments:
> (1) You can only cast relations including a single value or an error will be
> reported
> (2) Name resolution is needed since relation X might have field named C in
> which case that field takes precedence.
> (3) Y will look for C closest to it.
> Implementation thoughts:
> The idea is to store C into a file and then convert it into scalar via a UDF.
> I believe we already have a UDF that Ben Reed contributed for this purpose.
> Most of the work would be to update the logical plan to
> (1) Store C
> (2) convert the cast to the UDF
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1434) Allow casting relations to scalars
[
https://issues.apache.org/jira/browse/PIG-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884794#action_12884794
]
Richard Ding commented on PIG-1434:
---
So all one needs to do is internally replace the line:
{code}
Y = foreach X generate $1/(long) C.count, $2-(long) C.max;
{code}
with
{code}
Z = join X by 1, C by 1 using 'replicated';
Y = foreach Z generate X::$1/(long) C.count, X::$2-(long) C.max;
{code}
> Allow casting relations to scalars
> --
>
> Key: PIG-1434
> URL: https://issues.apache.org/jira/browse/PIG-1434
> Project: Pig
> Issue Type: Improvement
>Reporter: Olga Natkovich
>Assignee: Aniket Mokashi
> Fix For: 0.8.0
>
> Attachments: scalarImpl.patch
>
>
> This jira is to implement a simplified version of the functionality described
> in https://issues.apache.org/jira/browse/PIG-801.
> The proposal is to allow casting relations to scalar types in foreach.
> Example:
> A = load 'data' as (x, y, z);
> B = group A all;
> C = foreach B generate COUNT(A);
> .
> X =
> Y = foreach X generate $1/(long) C;
> Couple of additional comments:
> (1) You can only cast relations including a single value or an error will be
> reported
> (2) Name resolution is needed since relation X might have field named C in
> which case that field takes precedence.
> (3) Y will look for C closest to it.
> Implementation thoughts:
> The idea is to store C into a file and then convert it into scalar via a UDF.
> I believe we already have a UDF that Ben Reed contributed for this purpose.
> Most of the work would be to update the logical plan to
> (1) Store C
> (2) convert the cast to the UDF
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1434) Allow casting relations to scalars
[
https://issues.apache.org/jira/browse/PIG-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884807#action_12884807
]
Dmitriy V. Ryaboy commented on PIG-1434:
I think it's important in this particular case to ensure that C only contains
one tuple, since multiple tuples will lead to very confusing output.
Maybe
{code}
C = LIMIT C 1;
Z = join X by 1, C by 1 using 'replicated';
Y = foreach Z generate X::$1/(long) C.count, X::$2-(long) C.max;
{code}
(this is where an optimization that notes that you don't need m/r boundary if
the relation you are limiting only has one partition would come in handy).
> Allow casting relations to scalars
> --
>
> Key: PIG-1434
> URL: https://issues.apache.org/jira/browse/PIG-1434
> Project: Pig
> Issue Type: Improvement
>Reporter: Olga Natkovich
>Assignee: Aniket Mokashi
> Fix For: 0.8.0
>
> Attachments: scalarImpl.patch
>
>
> This jira is to implement a simplified version of the functionality described
> in https://issues.apache.org/jira/browse/PIG-801.
> The proposal is to allow casting relations to scalar types in foreach.
> Example:
> A = load 'data' as (x, y, z);
> B = group A all;
> C = foreach B generate COUNT(A);
> .
> X =
> Y = foreach X generate $1/(long) C;
> Couple of additional comments:
> (1) You can only cast relations including a single value or an error will be
> reported
> (2) Name resolution is needed since relation X might have field named C in
> which case that field takes precedence.
> (3) Y will look for C closest to it.
> Implementation thoughts:
> The idea is to store C into a file and then convert it into scalar via a UDF.
> I believe we already have a UDF that Ben Reed contributed for this purpose.
> Most of the work would be to update the logical plan to
> (1) Store C
> (2) convert the cast to the UDF
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1321) Logical Optimizer: Merge cascading foreach
[ https://issues.apache.org/jira/browse/PIG-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884811#action_12884811 ] Xuefu Zhang commented on PIG-1321: -- Add one more pre-condition: 3. The first foreach statement cannot contain flatten due to its complexity. > Logical Optimizer: Merge cascading foreach > -- > > Key: PIG-1321 > URL: https://issues.apache.org/jira/browse/PIG-1321 > Project: Pig > Issue Type: Sub-task > Components: impl >Affects Versions: 0.7.0 >Reporter: Daniel Dai >Assignee: Xuefu Zhang > > We can merge consecutive foreach statement. > Eg: > b = foreach a generate a0#'key1' as b0, a0#'key2' as b1, a1; > c = foreach b generate b0#'kk1', b0#'kk2', b1, a1; > => c = foreach a generate a0#'key1'#'kk1', a0#'key1'#'kk2', a0#'key2', a1; -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1404) PigUnit - Pig script testing simplified.
[
https://issues.apache.org/jira/browse/PIG-1404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Romain Rigaux updated PIG-1404:
---
Status: Open (was: Patch Available)
> PigUnit - Pig script testing simplified.
> -
>
> Key: PIG-1404
> URL: https://issues.apache.org/jira/browse/PIG-1404
> Project: Pig
> Issue Type: New Feature
>Reporter: Romain Rigaux
>Assignee: Romain Rigaux
> Fix For: 0.8.0
>
> Attachments: commons-lang-2.4.jar, PIG-1404-2.patch,
> PIG-1404-3-doc.patch, PIG-1404-3.patch, PIG-1404.patch
>
>
> The goal is to provide a simple xUnit framework that enables our Pig scripts
> to be easily:
> - unit tested
> - regression tested
> - quickly prototyped
> No cluster set up is required.
> For example:
> TestCase
> {code}
> @Test
> public void testTop3Queries() {
> String[] args = {
> "n=3",
> };
> test = new PigTest("top_queries.pig", args);
> String[] input = {
> "yahoo\t10",
> "twitter\t7",
> "facebook\t10",
> "yahoo\t15",
> "facebook\t5",
>
> };
> String[] output = {
> "(yahoo,25L)",
> "(facebook,15L)",
> "(twitter,7L)",
> };
> test.assertOutput("data", input, "queries_limit", output);
> }
> {code}
> top_queries.pig
> {code}
> data =
> LOAD '$input'
> AS (query:CHARARRAY, count:INT);
>
> ...
>
> queries_sum =
> FOREACH queries_group
> GENERATE
> group AS query,
> SUM(queries.count) AS count;
>
> ...
>
> queries_limit = LIMIT queries_ordered $n;
> STORE queries_limit INTO '$output';
> {code}
> They are 3 modes:
> * LOCAL (if "pigunit.exectype.local" properties is present)
> * MAPREDUCE (use the cluster specified in the classpath, same as
> HADOOP_CONF_DIR)
> ** automatic mini cluster (is the default and the HADOOP_CONF_DIR to have in
> the class path will be: ~/pigtest/conf)
> ** pointing to an existing cluster (if "pigunit.exectype.cluster" properties
> is present)
> For now, it would be nice to see how this idea could be integrated in
> Piggybank and if PigParser/PigServer could improve their interfaces in order
> to make PigUnit simple.
> Other components based on PigUnit could be built later:
> - standalone MiniCluster
> - notion of workspaces for each test
> - standalone utility that reads test configuration and generates a test
> report...
> It is a first prototype, open to suggestions and can definitely take
> advantage of feedbacks.
> How to test, in pig_trunk:
> {code}
> Apply patch
> $pig_trunk ant compile-test
> $pig_trunk ant
> $pig_trunk/contrib/piggybank/java ant test -Dtest.timeout=99
> {code}
> (it takes 15 min in MAPREDUCE minicluster, tests will need to be split in the
> future between 'unit' and 'integration')
> Many examples are in:
> {code}
> contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/pigunit/TestPigTest.java
> {code}
> When used as a standalone, do not forget commons-lang-2.4.jar and the
> HADOOP_CONF_DIR to your cluster in your CLASSPATH.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1404) PigUnit - Pig script testing simplified.
[
https://issues.apache.org/jira/browse/PIG-1404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Romain Rigaux updated PIG-1404:
---
Status: Patch Available (was: Open)
> PigUnit - Pig script testing simplified.
> -
>
> Key: PIG-1404
> URL: https://issues.apache.org/jira/browse/PIG-1404
> Project: Pig
> Issue Type: New Feature
>Reporter: Romain Rigaux
>Assignee: Romain Rigaux
> Fix For: 0.8.0
>
> Attachments: commons-lang-2.4.jar, PIG-1404-2.patch,
> PIG-1404-3-doc.patch, PIG-1404-3.patch, PIG-1404.patch
>
>
> The goal is to provide a simple xUnit framework that enables our Pig scripts
> to be easily:
> - unit tested
> - regression tested
> - quickly prototyped
> No cluster set up is required.
> For example:
> TestCase
> {code}
> @Test
> public void testTop3Queries() {
> String[] args = {
> "n=3",
> };
> test = new PigTest("top_queries.pig", args);
> String[] input = {
> "yahoo\t10",
> "twitter\t7",
> "facebook\t10",
> "yahoo\t15",
> "facebook\t5",
>
> };
> String[] output = {
> "(yahoo,25L)",
> "(facebook,15L)",
> "(twitter,7L)",
> };
> test.assertOutput("data", input, "queries_limit", output);
> }
> {code}
> top_queries.pig
> {code}
> data =
> LOAD '$input'
> AS (query:CHARARRAY, count:INT);
>
> ...
>
> queries_sum =
> FOREACH queries_group
> GENERATE
> group AS query,
> SUM(queries.count) AS count;
>
> ...
>
> queries_limit = LIMIT queries_ordered $n;
> STORE queries_limit INTO '$output';
> {code}
> They are 3 modes:
> * LOCAL (if "pigunit.exectype.local" properties is present)
> * MAPREDUCE (use the cluster specified in the classpath, same as
> HADOOP_CONF_DIR)
> ** automatic mini cluster (is the default and the HADOOP_CONF_DIR to have in
> the class path will be: ~/pigtest/conf)
> ** pointing to an existing cluster (if "pigunit.exectype.cluster" properties
> is present)
> For now, it would be nice to see how this idea could be integrated in
> Piggybank and if PigParser/PigServer could improve their interfaces in order
> to make PigUnit simple.
> Other components based on PigUnit could be built later:
> - standalone MiniCluster
> - notion of workspaces for each test
> - standalone utility that reads test configuration and generates a test
> report...
> It is a first prototype, open to suggestions and can definitely take
> advantage of feedbacks.
> How to test, in pig_trunk:
> {code}
> Apply patch
> $pig_trunk ant compile-test
> $pig_trunk ant
> $pig_trunk/contrib/piggybank/java ant test -Dtest.timeout=99
> {code}
> (it takes 15 min in MAPREDUCE minicluster, tests will need to be split in the
> future between 'unit' and 'integration')
> Many examples are in:
> {code}
> contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/pigunit/TestPigTest.java
> {code}
> When used as a standalone, do not forget commons-lang-2.4.jar and the
> HADOOP_CONF_DIR to your cluster in your CLASSPATH.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (PIG-1481) PigServer throws exception if it cannot find hadoop-site.xml or core-site.xml
PigServer throws exception if it cannot find hadoop-site.xml or core-site.xml
-
Key: PIG-1481
URL: https://issues.apache.org/jira/browse/PIG-1481
Project: Pig
Issue Type: Bug
Affects Versions: 0.8.0
Reporter: Sameer M
Hi
We've been using the Hadoop MiniCluster to do unit testing of our pig scripts
in the following way.
MiniCluster minicluster = MiniCluster.buildCluster(2,2);
pigServer = new PigServer(ExecType.MAPREDUCE, minicluster.getProperties());
This has been working fine for 0.6 and 0.7.
However in the trunk (0.8) looks like there is change due to which an exception
is thrown if hadoop-site.xml or core-site.xml is not found in the classpath.
org.apache.pig.backend.executionengine.ExecException: ERROR 4010: Cannot find
hadoop configurations in classpath (neither hadoop-site.xml nor core-site.xml
was found in the classpath).If you plan to use local mode, please put -x local
option in command line
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:149)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:114)
at org.apache.pig.impl.PigContext.connect(PigContext.java:177)
at org.apache.pig.PigServer.(PigServer.java:215)
at org.apache.pig.PigServer.(PigServer.java:204)
at org.apache.pig.PigServer.(PigServer.java:200)
The problem seems to be
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine: 148
if( hadoop_site == null && core_site == null ) {
throw new ExecException("Cannot find hadoop configurations in
classpath (neither hadoop-site.xml nor core-site.xml was found in the
classpath)." +
"If you plan to use local mode, please put -x
local option in command line",
4010);
}
We would like to use the mapreduce mode but with the minicluster and have a lot
of unit test with that setup.
Can this check be removed from this level ?
Thanks
Sameer
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1389) Implement Pig counter to track number of rows for each input files
[
https://issues.apache.org/jira/browse/PIG-1389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884834#action_12884834
]
Richard Ding commented on PIG-1389:
---
We use Hadoop Path as a parser to parse the input/output locations for two use
cases:
* Determine the scheme of the location (e.g., hdfs, hbase, file, har, ...), and
* Get the short file name of the location
Ashutosh is right that this approach doesn't work with location string as in
PIG-1229:
{code}
jdbc:hsqldb:file:/tmp/batchtest;hsqldb.default_table_type=cached;hsqldb.cache_rows=100
{code}
A RuntimeException is thrown when trying to parse it.
The proposal is to use Java URI as parser instead for these use cases (Java URI
throws a checked exception for invalid syntax).
> Implement Pig counter to track number of rows for each input files
> ---
>
> Key: PIG-1389
> URL: https://issues.apache.org/jira/browse/PIG-1389
> Project: Pig
> Issue Type: Improvement
>Affects Versions: 0.7.0
>Reporter: Richard Ding
>Assignee: Richard Ding
> Fix For: 0.8.0
>
> Attachments: PIG-1389.patch, PIG-1389.patch, PIG-1389_1.patch
>
>
> A MR job generated by Pig not only can have multiple outputs (in the case of
> multiquery) but also can have multiple inputs (in the case of join or
> cogroup). In both cases, the existing Hadoop counters (e.g.
> MAP_INPUT_RECORDS, REDUCE_OUTPUT_RECORDS) can not be used to count the number
> of records in the given input or output. PIG-1299 addressed the case of
> multiple outputs. We need to add new counters for jobs with multiple inputs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-928) UDFs in scripting languages
[ https://issues.apache.org/jira/browse/PIG-928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884841#action_12884841 ] Aniket Mokashi commented on PIG-928: The fix needed some changes in queryparser to support namespace, I found this in test cases I added. Current EvalFuncSpec logic is convoluted, I replaced it with a cleaner one. I have attached the updated patch with changes mentioned above. I am not sure what needs to be done for jython.jar, my guess was to check-in that in /lib. Thoughts? > UDFs in scripting languages > --- > > Key: PIG-928 > URL: https://issues.apache.org/jira/browse/PIG-928 > Project: Pig > Issue Type: New Feature >Reporter: Alan Gates >Assignee: Aniket Mokashi > Fix For: 0.8.0 > > Attachments: calltrace.png, package.zip, PIG-928.patch, > pig-greek.tgz, pig.scripting.patch.arnab, pyg.tgz, RegisterPythonUDF2.patch, > RegisterPythonUDF3.patch, RegisterPythonUDF4.patch, > RegisterScriptUDFDefineParse.patch, scripting.tgz, scripting.tgz, test.zip > > > It should be possible to write UDFs in scripting languages such as python, > ruby, etc. This frees users from needing to compile Java, generate a jar, > etc. It also opens Pig to programmers who prefer scripting languages over > Java. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-928) UDFs in scripting languages
[ https://issues.apache.org/jira/browse/PIG-928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884845#action_12884845 ] Dmitriy V. Ryaboy commented on PIG-928: --- Aniket, I already made the changes you need to pull down jython -- take a look at the patch I attached. One more general note -- let's say jython instead of python (in the grammar, the keywords, everywhere), as there may be slight incompatibilities between the two and we want to be clear on what we are using. > UDFs in scripting languages > --- > > Key: PIG-928 > URL: https://issues.apache.org/jira/browse/PIG-928 > Project: Pig > Issue Type: New Feature >Reporter: Alan Gates >Assignee: Aniket Mokashi > Fix For: 0.8.0 > > Attachments: calltrace.png, package.zip, PIG-928.patch, > pig-greek.tgz, pig.scripting.patch.arnab, pyg.tgz, RegisterPythonUDF2.patch, > RegisterPythonUDF3.patch, RegisterPythonUDF4.patch, > RegisterPythonUDFFinale.patch, RegisterScriptUDFDefineParse.patch, > scripting.tgz, scripting.tgz, test.zip > > > It should be possible to write UDFs in scripting languages such as python, > ruby, etc. This frees users from needing to compile Java, generate a jar, > etc. It also opens Pig to programmers who prefer scripting languages over > Java. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-928) UDFs in scripting languages
[ https://issues.apache.org/jira/browse/PIG-928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aniket Mokashi updated PIG-928: --- Attachment: RegisterPythonUDFFinale.patch Changes needed for script UDF. TODO- jython.jar related changes > UDFs in scripting languages > --- > > Key: PIG-928 > URL: https://issues.apache.org/jira/browse/PIG-928 > Project: Pig > Issue Type: New Feature >Reporter: Alan Gates >Assignee: Aniket Mokashi > Fix For: 0.8.0 > > Attachments: calltrace.png, package.zip, PIG-928.patch, > pig-greek.tgz, pig.scripting.patch.arnab, pyg.tgz, RegisterPythonUDF2.patch, > RegisterPythonUDF3.patch, RegisterPythonUDF4.patch, > RegisterPythonUDFFinale.patch, RegisterScriptUDFDefineParse.patch, > scripting.tgz, scripting.tgz, test.zip > > > It should be possible to write UDFs in scripting languages such as python, > ruby, etc. This frees users from needing to compile Java, generate a jar, > etc. It also opens Pig to programmers who prefer scripting languages over > Java. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-928) UDFs in scripting languages
[ https://issues.apache.org/jira/browse/PIG-928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884856#action_12884856 ] Aniket Mokashi commented on PIG-928: I had added an interface: getStandardScriptJarPath to find the path of jython jar to be shipped as part of job.jar only when user uses this feature. How do I incorporate this into new changes? Do we want to go for compile time support option? > UDFs in scripting languages > --- > > Key: PIG-928 > URL: https://issues.apache.org/jira/browse/PIG-928 > Project: Pig > Issue Type: New Feature >Reporter: Alan Gates >Assignee: Aniket Mokashi > Fix For: 0.8.0 > > Attachments: calltrace.png, package.zip, PIG-928.patch, > pig-greek.tgz, pig.scripting.patch.arnab, pyg.tgz, RegisterPythonUDF2.patch, > RegisterPythonUDF3.patch, RegisterPythonUDF4.patch, > RegisterPythonUDFFinale.patch, RegisterScriptUDFDefineParse.patch, > scripting.tgz, scripting.tgz, test.zip > > > It should be possible to write UDFs in scripting languages such as python, > ruby, etc. This frees users from needing to compile Java, generate a jar, > etc. It also opens Pig to programmers who prefer scripting languages over > Java. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-928) UDFs in scripting languages
[
https://issues.apache.org/jira/browse/PIG-928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884863#action_12884863
]
Julien Le Dem commented on PIG-928:
---
Aniket, this is assuming the ScriptEngine requires only one jar.
I would suggest instead having a method ScriptEngine.init(PigContext) that
would be called after the ScriptEngine instance has been retrieved from the
factory.
That would let the script engine add whatever is needed to the job.
{code}
if(scriptingLang != null) {
ScriptEngine se = ScriptEngine.getInstance(scriptingLang);
//pigContext.scriptJars.add(se.getStandardScriptJarPath());
se.init(pigContext);
se.registerFunctions(path, namespace, pigContext);
}
{code}
Have a good week end, Julien
> UDFs in scripting languages
> ---
>
> Key: PIG-928
> URL: https://issues.apache.org/jira/browse/PIG-928
> Project: Pig
> Issue Type: New Feature
>Reporter: Alan Gates
>Assignee: Aniket Mokashi
> Fix For: 0.8.0
>
> Attachments: calltrace.png, package.zip, PIG-928.patch,
> pig-greek.tgz, pig.scripting.patch.arnab, pyg.tgz, RegisterPythonUDF2.patch,
> RegisterPythonUDF3.patch, RegisterPythonUDF4.patch,
> RegisterPythonUDFFinale.patch, RegisterScriptUDFDefineParse.patch,
> scripting.tgz, scripting.tgz, test.zip
>
>
> It should be possible to write UDFs in scripting languages such as python,
> ruby, etc. This frees users from needing to compile Java, generate a jar,
> etc. It also opens Pig to programmers who prefer scripting languages over
> Java.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
