[jira] Created: (PIG-1166) A bit change of the interface of Tuple & DataBag ( make the set and append method return this)
A bit change of the interface of Tuple & DataBag ( make the set and append method return this) -- Key: PIG-1166 URL: https://issues.apache.org/jira/browse/PIG-1166 Project: Pig Issue Type: Improvement Reporter: Jeff Zhang Priority: Minor When people write unit test for UDF, they always need to build a tuple or bag. If we change the interface of Tuple and DataBag, make the set and append method return this, it can decrease the code size. e.g. Now people have to write the following code to build a Tuple: {code} Tuple tuple=TupleFactory.getInstance().newTuple(3); tuple.set(0,"item_0"); tuple.set(1,"item_1"); tuple.set(2,"item_2"); {code} If we change the interface, make the set and append method return this, we can rewrite the above code like this: {code} Tuple tuple=TupleFactory.getInstance().newTuple(3); tuple.set(0,"item_0").set(1,"item_1").set(2,"item_2"); {code} This interface change won't have back compatibility problem and I think there's no performance problem too. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1166) A bit change of the interface of Tuple & DataBag ( make the set and append method return this)
[ https://issues.apache.org/jira/browse/PIG-1166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793118#action_12793118 ] Jeff Zhang commented on PIG-1166: - A better example of illustrating my idea is to build a DataBag. The current method: {code} BagFactory BAGFACTORY = BagFactory.getInstance(); TupleFactory TUPLEFACTORY = TupleFactory.getInstance(); DataBag bag = BAGFACTORY.newDefaultBag(); Tuple tuple_1 = TUPLEFACTORY.newTuple(1); tuple_1.set(0, "item_1"); bag.add(tuple_1); Tuple tuple_2 = TUPLEFACTORY.newTuple(1); tuple_2.set(0, "item_2"); bag.add(tuple_2); {code} and if we change the interface, we can write the code as following: {code} BagFactory BAGFACTORY = BagFactory.getInstance(); TupleFactory TUPLEFACTORY = TupleFactory.getInstance(); DataBag bag = BAGFACTORY.newDefaultBag(); bag.add(TUPLEFACTORY.newTuple(1).set(0,"item_1")).add(TUPLEFACTORY.newTuple(1).set(0,"item_2")); {code} The second piece of code snippet is more readable and concise in my opinion. > A bit change of the interface of Tuple & DataBag ( make the set and append > method return this) > -- > > Key: PIG-1166 > URL: https://issues.apache.org/jira/browse/PIG-1166 > Project: Pig > Issue Type: Improvement >Reporter: Jeff Zhang >Priority: Minor > > When people write unit test for UDF, they always need to build a tuple or > bag. If we change the interface of Tuple and DataBag, make the set and > append method return this, it can decrease the code size. e.g. Now people > have to write the following code to build a Tuple: > {code} > Tuple tuple=TupleFactory.getInstance().newTuple(3); > tuple.set(0,"item_0"); > tuple.set(1,"item_1"); > tuple.set(2,"item_2"); > {code} > If we change the interface, make the set and append method return this, we > can rewrite the above code like this: > {code} > Tuple tuple=TupleFactory.getInstance().newTuple(3); > tuple.set(0,"item_0").set(1,"item_1").set(2,"item_2"); > {code} > This interface change won't have back compatibility problem and I think > there's no performance problem too. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-919) Type mismatch in key from map: expected org.apache.pig.impl.io.NullableBytesWritable, recieved org.apache.pig.impl.io.NullableText when doing simple group
[ https://issues.apache.org/jira/browse/PIG-919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793121#action_12793121 ] George Mavromatis commented on PIG-919: --- > This was closed Fixed when it should have been closed Won't Fix or Later. Can we then resolve it with the correct resolution (Won't Fix or Later)? > Where are you seeing this error? I am seeing it in product code that I cannot refer to here. It has occurred twice, one instance of which was referred to this ticket and closed. I will send you more information offline. > Type mismatch in key from map: expected > org.apache.pig.impl.io.NullableBytesWritable, recieved > org.apache.pig.impl.io.NullableText when doing simple group > -- > > Key: PIG-919 > URL: https://issues.apache.org/jira/browse/PIG-919 > Project: Pig > Issue Type: Bug > Components: impl >Affects Versions: 0.3.0 >Reporter: Viraj Bhat >Assignee: Viraj Bhat > Fix For: 0.3.0 > > Attachments: GenHashList.java, mapscript.pig, mymapudf.jar > > > I have a Pig script, which takes in a student file and generates a bag of > maps. I later want to group on the value of the key "name0" which > corresponds to the first name of the student. > {code} > register mymapudf.jar; > data = LOAD '/user/viraj/studenttab10k' AS > (somename:chararray,age:long,marks:float); > genmap = foreach data generate flatten(mymapudf.GenHashList(somename,' ')) as > bp:map[], age, marks; > getfirstnames = foreach genmap generate bp#'name0' as firstname, age, marks; > filternonnullfirstnames = filter getfirstnames by firstname is not null; > groupgenmap = group filternonnullfirstnames by firstname; > dump groupgenmap; > {code} > When I execute this code, I get an error in the Map Phase: > === > java.io.IOException: Type mismatch in key from map: expected > org.apache.pig.impl.io.NullableBytesWritable, recieved > org.apache.pig.impl.io.NullableText > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:415) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:108) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:253) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:242) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.map(PigMapReduce.java:93) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:47) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227) > at > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2209) > === -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1149) Allow instantiation of SampleLoaders with parametrized LoadFuncs
[ https://issues.apache.org/jira/browse/PIG-1149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793269#action_12793269 ] Thejas M Nair commented on PIG-1149: +1 to the lsr branch version. But the FIXME comment in the test case is not correct. There does not have to be > 1 samples sampled for every map, if the number of rows are very small. Though this behavior is different from earlier version of the trunk version of poisson sampler, it satisfies the requirements as per http://wiki.apache.org/pig/PigSampler and PIG-1062. I can remove the FIXME comment as part of the patch I am going to submit to fix the other test case. > Allow instantiation of SampleLoaders with parametrized LoadFuncs > > > Key: PIG-1149 > URL: https://issues.apache.org/jira/browse/PIG-1149 > Project: Pig > Issue Type: Bug >Reporter: Dmitriy V. Ryaboy >Assignee: Dmitriy V. Ryaboy >Priority: Minor > Fix For: 0.7.0 > > Attachments: pig_1149.patch, pig_1149_lsr-branch.patch > > > Currently, it is not possible to instantiate a SampleLoader with something > like PigStorage(':'). We should allow passing parameters to the loaders > being sampled. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (PIG-1110) Handle compressed file formats -- Gz, BZip with the new proposal
[ https://issues.apache.org/jira/browse/PIG-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pradeep Kamath resolved PIG-1110. - Resolution: Fixed Hadoop Flags: [Incompatible change, Reviewed] (was: [Incompatible change]) +1, Patch committed to load-store-redesign branch - thanks Richard! > Handle compressed file formats -- Gz, BZip with the new proposal > > > Key: PIG-1110 > URL: https://issues.apache.org/jira/browse/PIG-1110 > Project: Pig > Issue Type: Sub-task >Reporter: Richard Ding >Assignee: Richard Ding > Attachments: PIG-1110.patch, PIG-1110.patch, PIG_1110_Jeff.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1094) Fix unit tests corresponding to source changes so far
[ https://issues.apache.org/jira/browse/PIG-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793286#action_12793286 ] Pradeep Kamath commented on PIG-1094: - +1, PIG-1094_4.patch checked in, thanks Richard! > Fix unit tests corresponding to source changes so far > - > > Key: PIG-1094 > URL: https://issues.apache.org/jira/browse/PIG-1094 > Project: Pig > Issue Type: Sub-task >Reporter: Pradeep Kamath >Assignee: Pradeep Kamath > Attachments: PIG-1094.patch, PIG-1094_2.patch, PIG-1094_3.patch, > PIG-1094_4.patch > > > The check-in's so far on load-store-redesign branch have nor addressed unit > test failures due to interface changes. This jira is to track the task of > making the common case unit tests work with the new interfaces. Some aspects > of the new proposal like using LoadCaster interface for casting, making local > mode work have not been completed yet. Tests which are failing due to those > reasons will not be fixed in this jira and addressed in the jiras > corresponding to those tasks -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1165) Signature of loader does not set correctly for order by
[ https://issues.apache.org/jira/browse/PIG-1165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated PIG-1165: Attachment: PIG-1165-1.patch > Signature of loader does not set correctly for order by > --- > > Key: PIG-1165 > URL: https://issues.apache.org/jira/browse/PIG-1165 > Project: Pig > Issue Type: Bug > Components: impl >Affects Versions: 0.6.0 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.6.0 > > Attachments: PIG-1165-1.patch > > > In pig, we need to set signature for each LoadFunc. Currently, we use alias > of the LOAD statement in Pig script of the signature of the LoadFunc. One use > case we have is in LoadFunc, we use signature to retrieve pruned columns of > each specific loader. However, in "order by" statement, we do not set > signature for the loader correctly. In this case, we do not prune the loader > correctly. > For example, the following script produce wrong result: > {code} > a = load '1.txt' as (a0, a1); > b = order a by a1; > c = order b by a1; > d = foreach c generate a1; > dump d; > {code} > 1.txt: > {code} > 1 a > 2 b > 3 c > 6 d > 5 e > {code} > expected result: > a > b > c > d > e > current result: > 1 > 2 > 3 > 5 > 6 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1165) Signature of loader does not set correctly for order by
[ https://issues.apache.org/jira/browse/PIG-1165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated PIG-1165: Status: Patch Available (was: Open) > Signature of loader does not set correctly for order by > --- > > Key: PIG-1165 > URL: https://issues.apache.org/jira/browse/PIG-1165 > Project: Pig > Issue Type: Bug > Components: impl >Affects Versions: 0.6.0 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.6.0 > > Attachments: PIG-1165-1.patch > > > In pig, we need to set signature for each LoadFunc. Currently, we use alias > of the LOAD statement in Pig script of the signature of the LoadFunc. One use > case we have is in LoadFunc, we use signature to retrieve pruned columns of > each specific loader. However, in "order by" statement, we do not set > signature for the loader correctly. In this case, we do not prune the loader > correctly. > For example, the following script produce wrong result: > {code} > a = load '1.txt' as (a0, a1); > b = order a by a1; > c = order b by a1; > d = foreach c generate a1; > dump d; > {code} > 1.txt: > {code} > 1 a > 2 b > 3 c > 6 d > 5 e > {code} > expected result: > a > b > c > d > e > current result: > 1 > 2 > 3 > 5 > 6 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1143) Poisson Sample Loader should compute the number of samples required only once
[ https://issues.apache.org/jira/browse/PIG-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793296#action_12793296 ] Olga Natkovich commented on PIG-1143: - +1 on the code changes. There is a extra debug trace in the code that I will remove as part of the commit > Poisson Sample Loader should compute the number of samples required only once > - > > Key: PIG-1143 > URL: https://issues.apache.org/jira/browse/PIG-1143 > Project: Pig > Issue Type: Bug >Reporter: Sriranjan Manjunath >Assignee: Sriranjan Manjunath > Attachments: PIG_1143.patch, PIG_1143.patch.1 > > > The current poisson sampler forces each of the maps to compute the sample > number. This is redundant and causes issues when a large directory is > specified in the join. The sampler should be changed to calculate the sample > count only once and this information should be shared with the remaining > mappers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Locking trunk for commits to merge on load-store-redesign branch
Hi, PIG-1143 and PIG-1149 need special handling on the load-store-redesign branch. PIG-1143 should not be applied to the branch since the code is not applicable and for PIG-1149 there is a separate patch. I am beginning a merge of load-store-redesign branch with current head of trunk. These two patches will be committed to trunk once I complete the merge and in the svn commit message for the merge on load-store-redesign branch, I will record the revision after these two patches are committed. This is so that the next time we merge from trunk to load-store-redesign branch we merge from the point after these patches. This mail is to let all committers know that they should hold off commits till this process is done at which point I will send an "all clear" email. Thanks, Pradeep
[jira] Updated: (PIG-1143) Poisson Sample Loader should compute the number of samples required only once
[ https://issues.apache.org/jira/browse/PIG-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Olga Natkovich updated PIG-1143: Fix Version/s: 0.6.0 > Poisson Sample Loader should compute the number of samples required only once > - > > Key: PIG-1143 > URL: https://issues.apache.org/jira/browse/PIG-1143 > Project: Pig > Issue Type: Bug >Reporter: Sriranjan Manjunath >Assignee: Sriranjan Manjunath > Fix For: 0.6.0 > > Attachments: PIG_1143.patch, PIG_1143.patch.1 > > > The current poisson sampler forces each of the maps to compute the sample > number. This is redundant and causes issues when a large directory is > specified in the join. The sampler should be changed to calculate the sample > count only once and this information should be shared with the remaining > mappers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1149) Allow instantiation of SampleLoaders with parametrized LoadFuncs
[ https://issues.apache.org/jira/browse/PIG-1149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793321#action_12793321 ] Olga Natkovich commented on PIG-1149: - patch pig_1149.patch is committed to the trunk. > Allow instantiation of SampleLoaders with parametrized LoadFuncs > > > Key: PIG-1149 > URL: https://issues.apache.org/jira/browse/PIG-1149 > Project: Pig > Issue Type: Bug >Reporter: Dmitriy V. Ryaboy >Assignee: Dmitriy V. Ryaboy >Priority: Minor > Fix For: 0.7.0 > > Attachments: pig_1149.patch, pig_1149_lsr-branch.patch > > > Currently, it is not possible to instantiate a SampleLoader with something > like PigStorage(':'). We should allow passing parameters to the loaders > being sampled. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1143) Poisson Sample Loader should compute the number of samples required only once
[ https://issues.apache.org/jira/browse/PIG-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793322#action_12793322 ] Olga Natkovich commented on PIG-1143: - patch committed to the trunk. Will commit to 0.6 branch tomorrow. > Poisson Sample Loader should compute the number of samples required only once > - > > Key: PIG-1143 > URL: https://issues.apache.org/jira/browse/PIG-1143 > Project: Pig > Issue Type: Bug >Reporter: Sriranjan Manjunath >Assignee: Sriranjan Manjunath > Fix For: 0.6.0 > > Attachments: PIG_1143.patch, PIG_1143.patch.1 > > > The current poisson sampler forces each of the maps to compute the sample > number. This is redundant and causes issues when a large directory is > specified in the join. The sampler should be changed to calculate the sample > count only once and this information should be shared with the remaining > mappers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
RE: Locking trunk for commits to merge on load-store-redesign branch
Hi, The process outlined below is now completed and the trunk is open for commits which do not conflict with load-store-redesign branch. Thanks, Pradeep -Original Message- From: Pradeep Kamath [mailto:prade...@yahoo-inc.com] Sent: Monday, December 21, 2009 10:50 AM To: pig-dev@hadoop.apache.org Subject: Locking trunk for commits to merge on load-store-redesign branch Hi, PIG-1143 and PIG-1149 need special handling on the load-store-redesign branch. PIG-1143 should not be applied to the branch since the code is not applicable and for PIG-1149 there is a separate patch. I am beginning a merge of load-store-redesign branch with current head of trunk. These two patches will be committed to trunk once I complete the merge and in the svn commit message for the merge on load-store-redesign branch, I will record the revision after these two patches are committed. This is so that the next time we merge from trunk to load-store-redesign branch we merge from the point after these patches. This mail is to let all committers know that they should hold off commits till this process is done at which point I will send an "all clear" email. Thanks, Pradeep
[jira] Updated: (PIG-1149) Allow instantiation of SampleLoaders with parametrized LoadFuncs
[ https://issues.apache.org/jira/browse/PIG-1149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pradeep Kamath updated PIG-1149: Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Patch for lsr branch also committed, thanks Dmitriy! > Allow instantiation of SampleLoaders with parametrized LoadFuncs > > > Key: PIG-1149 > URL: https://issues.apache.org/jira/browse/PIG-1149 > Project: Pig > Issue Type: Bug >Reporter: Dmitriy V. Ryaboy >Assignee: Dmitriy V. Ryaboy >Priority: Minor > Fix For: 0.7.0 > > Attachments: pig_1149.patch, pig_1149_lsr-branch.patch > > > Currently, it is not possible to instantiate a SampleLoader with something > like PigStorage(':'). We should allow passing parameters to the loaders > being sampled. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1158) pig command line -M option doesn't support table union correctly (comma seperated paths)
[ https://issues.apache.org/jira/browse/PIG-1158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Olga Natkovich updated PIG-1158: Resolution: Fixed Status: Resolved (was: Patch Available) patch committed to the trunk. Thanks, Richard > pig command line -M option doesn't support table union correctly (comma > seperated paths) > > > Key: PIG-1158 > URL: https://issues.apache.org/jira/browse/PIG-1158 > Project: Pig > Issue Type: Bug >Affects Versions: 0.6.0 >Reporter: Jing Huang >Assignee: Richard Ding > Fix For: 0.7.0 > > Attachments: PIG-1158.patch > > > for example, load (1.txt,2.txt) USING > org.apache.hadoop.zebra.pig.TableLoader() > i see this errror from stand out: > [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2100: > hdfs://gsbl90380.blue.ygrid.yahoo.com/user/hadoopqa/1.txt,2.txt does not > exist. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1141) Make streaming work with the new load-store interfaces
[ https://issues.apache.org/jira/browse/PIG-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Ding updated PIG-1141: -- Attachment: PIG-1141.patch This patch made changes following Alan's comments. > Make streaming work with the new load-store interfaces > --- > > Key: PIG-1141 > URL: https://issues.apache.org/jira/browse/PIG-1141 > Project: Pig > Issue Type: Sub-task >Reporter: Richard Ding >Assignee: Richard Ding > Attachments: PIG-1141.patch, PIG-1141.patch, PIG-1141.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1166) A bit change of the interface of Tuple & DataBag ( make the set and append method return this)
[ https://issues.apache.org/jira/browse/PIG-1166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793350#action_12793350 ] Dmitriy V. Ryaboy commented on PIG-1166: +1 to the idea, and we don't have to stop at Tuple and Bag factories. There are plenty of other places that this can be useful in (like all of the Logical and Physical operators). > A bit change of the interface of Tuple & DataBag ( make the set and append > method return this) > -- > > Key: PIG-1166 > URL: https://issues.apache.org/jira/browse/PIG-1166 > Project: Pig > Issue Type: Improvement >Reporter: Jeff Zhang >Priority: Minor > > When people write unit test for UDF, they always need to build a tuple or > bag. If we change the interface of Tuple and DataBag, make the set and > append method return this, it can decrease the code size. e.g. Now people > have to write the following code to build a Tuple: > {code} > Tuple tuple=TupleFactory.getInstance().newTuple(3); > tuple.set(0,"item_0"); > tuple.set(1,"item_1"); > tuple.set(2,"item_2"); > {code} > If we change the interface, make the set and append method return this, we > can rewrite the above code like this: > {code} > Tuple tuple=TupleFactory.getInstance().newTuple(3); > tuple.set(0,"item_0").set(1,"item_1").set(2,"item_2"); > {code} > This interface change won't have back compatibility problem and I think > there's no performance problem too. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1159) merge join right side table does not support comma seperated paths
[ https://issues.apache.org/jira/browse/PIG-1159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Olga Natkovich updated PIG-1159: Resolution: Fixed Status: Resolved (was: Patch Available) patch committed. Thanks, Richard > merge join right side table does not support comma seperated paths > -- > > Key: PIG-1159 > URL: https://issues.apache.org/jira/browse/PIG-1159 > Project: Pig > Issue Type: Bug >Affects Versions: 0.6.0 >Reporter: Jing Huang >Assignee: Richard Ding > Fix For: 0.7.0 > > Attachments: PIG-1159.patch > > > For example this is my script:(join_jira1.pig) > register /grid/0/dev/hadoopqa/jars/zebra.jar; > --a1 = load '1.txt' as (a:int, > b:float,c:long,d:double,e:chararray,f:bytearray,r1(f1:chararray,f2:chararray),m1:map[]); > --a2 = load '2.txt' as (a:int, > b:float,c:long,d:double,e:chararray,f:bytearray,r1(f1:chararray,f2:chararray),m1:map[]); > --sort1 = order a1 by a parallel 6; > --sort2 = order a2 by a parallel 5; > --store sort1 into 'asort1' using > org.apache.hadoop.zebra.pig.TableStorer('[a,b,c,d]'); > --store sort2 into 'asort2' using > org.apache.hadoop.zebra.pig.TableStorer('[a,b,c,d]'); > --store sort1 into 'asort3' using > org.apache.hadoop.zebra.pig.TableStorer('[a,b,c,d]'); > --store sort2 into 'asort4' using > org.apache.hadoop.zebra.pig.TableStorer('[a,b,c,d]'); > joinl = LOAD 'asort1,asort2' USING > org.apache.hadoop.zebra.pig.TableLoader('a,b,c,d', 'sorted'); > joinr = LOAD 'asort3,asort4' USING > org.apache.hadoop.zebra.pig.TableLoader('a,b,c,d', 'sorted'); > joina = join joinl by a, joinr by a using "merge" ; > dump joina; > == > here is the log: > Backend error message > - > java.lang.IllegalArgumentException: Pathname > /user/hadoopqa/asort3,hdfs:/gsbl90380.blue.ygrid.yahoo.com/user/hadoopqa/asort4 > from > hdfs://gsbl90380.blue.ygrid.yahoo.com/user/hadoopqa/asort3,hdfs:/gsbl90380.blue.ygrid.yahoo.com/user/hadoopqa/asort4 > is not a valid DFS filename. > at > org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:158) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:453) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:648) > at > org.apache.pig.backend.hadoop.datastorage.HDataStorage.isContainer(HDataStorage.java:203) > at > org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:131) > at > org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:147) > at > org.apache.pig.impl.io.FileLocalizer.fullPath(FileLocalizer.java:534) > at org.apache.pig.impl.io.FileLocalizer.open(FileLocalizer.java:338) > at > org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.seekInRightStream(POMergeJoin.java:398) > at > org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.getNext(POMergeJoin.java:184) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:253) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:244) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.map(PigMapOnly.java:65) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) > at org.apache.hadoop.mapred.Child.main(Child.java:159) > Pig Stack Trace > --- > ERROR 6015: During execution, encountered a Hadoop error. > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to > open iterator for alias joina > at org.apache.pig.PigServer.openIterator(PigServer.java:482) > at > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:539) > at > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:241) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:168) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:144) > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:89) > at org.apache.pig.Main.main(Main.java:386) > Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 6015: > During execution, encountered a Hadoop error. > at > .apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:158) > at > .apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:453) >
[jira] Commented: (PIG-1141) Make streaming work with the new load-store interfaces
[ https://issues.apache.org/jira/browse/PIG-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793354#action_12793354 ] Alan Gates commented on PIG-1141: - +1, changes look good. > Make streaming work with the new load-store interfaces > --- > > Key: PIG-1141 > URL: https://issues.apache.org/jira/browse/PIG-1141 > Project: Pig > Issue Type: Sub-task >Reporter: Richard Ding >Assignee: Richard Ding > Attachments: PIG-1141.patch, PIG-1141.patch, PIG-1141.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1102) Collect number of spills per job
[ https://issues.apache.org/jira/browse/PIG-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793369#action_12793369 ] Olga Natkovich commented on PIG-1102: - A few questions/comments on the patch: (1) I think the count should default to 0, not -1. (2) Does increment of count have to be combined with warn statement. Does this mean that users will see this many warnings? If so, should we combine this with spill message we already print? (3) I thought we discussed having increment per buffer not per record and to approximate that based on the buffer size. I did not see the code that did this. (4) I don't think you correctly separated bags that practively spill vs the bags that are spilled by memory manager. All the bags created by DefaultBagFactory get registerf with SpillableMemoryManager and belong to the second category. > Collect number of spills per job > > > Key: PIG-1102 > URL: https://issues.apache.org/jira/browse/PIG-1102 > Project: Pig > Issue Type: Improvement >Reporter: Olga Natkovich >Assignee: Sriranjan Manjunath > Fix For: 0.7.0 > > Attachments: PIG_1102.patch > > > Memory shortage is one of the main performance issues in Pig. Knowing when we > spill do the disk is useful for understanding query performance and also to > see how certain changes in Pig effect that. > Other interesting stats to collect would be average CPU usage and max mem > usage but I am not sure if this information is easily retrievable. > Using Hadoop counters for this would make sense. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (PIG-1167) [zebra] Zebra does not support Hadoop Globs
[zebra] Zebra does not support Hadoop Globs --- Key: PIG-1167 URL: https://issues.apache.org/jira/browse/PIG-1167 Project: Pig Issue Type: Improvement Reporter: Olga Natkovich Assignee: Jay Tang Pssing the following path to Zebra causing error but works with Hadoop directly: /projects/FETL/sample/ABF1/{2009120204} -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (PIG-1141) Make streaming work with the new load-store interfaces
[ https://issues.apache.org/jira/browse/PIG-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pradeep Kamath resolved PIG-1141. - Resolution: Fixed Hadoop Flags: [Incompatible change, Reviewed] Patch committed to load-store-redesign branch, thanks Richard! > Make streaming work with the new load-store interfaces > --- > > Key: PIG-1141 > URL: https://issues.apache.org/jira/browse/PIG-1141 > Project: Pig > Issue Type: Sub-task >Reporter: Richard Ding >Assignee: Richard Ding > Attachments: PIG-1141.patch, PIG-1141.patch, PIG-1141.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1167) [zebra] Zebra does not support Hadoop Globs
[ https://issues.apache.org/jira/browse/PIG-1167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793419#action_12793419 ] Yan Zhou commented on PIG-1167: --- Zebra's TableLoader, the implementation of PIG's LoadFunc, does support glob by design. But Map/Reduce interface does not. TableLoader just expands the glob and passes the paths to Map/Reduce interface so a union of underlying Zebra tables will be loaded. It looks like that not enough test coverage is present in this area. > [zebra] Zebra does not support Hadoop Globs > --- > > Key: PIG-1167 > URL: https://issues.apache.org/jira/browse/PIG-1167 > Project: Pig > Issue Type: Improvement >Reporter: Olga Natkovich >Assignee: Jay Tang > > Pssing the following path to Zebra causing error but works with Hadoop > directly: /projects/FETL/sample/ABF1/{2009120204} -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1167) [zebra] Zebra does not support Hadoop Globs
[ https://issues.apache.org/jira/browse/PIG-1167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793420#action_12793420 ] Olga Natkovich commented on PIG-1167: - Just to clarify. Pig does support it for all loaders that use default pig slicer. Zebra, however, uses its own and that's why it is not getting this functionality for free. > [zebra] Zebra does not support Hadoop Globs > --- > > Key: PIG-1167 > URL: https://issues.apache.org/jira/browse/PIG-1167 > Project: Pig > Issue Type: Improvement >Reporter: Olga Natkovich >Assignee: Jay Tang > > Pssing the following path to Zebra causing error but works with Hadoop > directly: /projects/FETL/sample/ABF1/{2009120204} -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1165) Signature of loader does not set correctly for order by
[ https://issues.apache.org/jira/browse/PIG-1165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793428#action_12793428 ] Hadoop QA commented on PIG-1165: +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12428644/PIG-1165-1.patch against trunk revision 892939. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/149/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/149/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/149/console This message is automatically generated. > Signature of loader does not set correctly for order by > --- > > Key: PIG-1165 > URL: https://issues.apache.org/jira/browse/PIG-1165 > Project: Pig > Issue Type: Bug > Components: impl >Affects Versions: 0.6.0 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.6.0 > > Attachments: PIG-1165-1.patch > > > In pig, we need to set signature for each LoadFunc. Currently, we use alias > of the LOAD statement in Pig script of the signature of the LoadFunc. One use > case we have is in LoadFunc, we use signature to retrieve pruned columns of > each specific loader. However, in "order by" statement, we do not set > signature for the loader correctly. In this case, we do not prune the loader > correctly. > For example, the following script produce wrong result: > {code} > a = load '1.txt' as (a0, a1); > b = order a by a1; > c = order b by a1; > d = foreach c generate a1; > dump d; > {code} > 1.txt: > {code} > 1 a > 2 b > 3 c > 6 d > 5 e > {code} > expected result: > a > b > c > d > e > current result: > 1 > 2 > 3 > 5 > 6 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1165) Signature of loader does not set correctly for order by
[ https://issues.apache.org/jira/browse/PIG-1165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793434#action_12793434 ] Olga Natkovich commented on PIG-1165: - +1; changes look good! > Signature of loader does not set correctly for order by > --- > > Key: PIG-1165 > URL: https://issues.apache.org/jira/browse/PIG-1165 > Project: Pig > Issue Type: Bug > Components: impl >Affects Versions: 0.6.0 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.6.0 > > Attachments: PIG-1165-1.patch > > > In pig, we need to set signature for each LoadFunc. Currently, we use alias > of the LOAD statement in Pig script of the signature of the LoadFunc. One use > case we have is in LoadFunc, we use signature to retrieve pruned columns of > each specific loader. However, in "order by" statement, we do not set > signature for the loader correctly. In this case, we do not prune the loader > correctly. > For example, the following script produce wrong result: > {code} > a = load '1.txt' as (a0, a1); > b = order a by a1; > c = order b by a1; > d = foreach c generate a1; > dump d; > {code} > 1.txt: > {code} > 1 a > 2 b > 3 c > 6 d > 5 e > {code} > expected result: > a > b > c > d > e > current result: > 1 > 2 > 3 > 5 > 6 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1165) Signature of loader does not set correctly for order by
[ https://issues.apache.org/jira/browse/PIG-1165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated PIG-1165: Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Patch committed to both trunk and 0.6 branch. > Signature of loader does not set correctly for order by > --- > > Key: PIG-1165 > URL: https://issues.apache.org/jira/browse/PIG-1165 > Project: Pig > Issue Type: Bug > Components: impl >Affects Versions: 0.6.0 >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 0.6.0 > > Attachments: PIG-1165-1.patch > > > In pig, we need to set signature for each LoadFunc. Currently, we use alias > of the LOAD statement in Pig script of the signature of the LoadFunc. One use > case we have is in LoadFunc, we use signature to retrieve pruned columns of > each specific loader. However, in "order by" statement, we do not set > signature for the loader correctly. In this case, we do not prune the loader > correctly. > For example, the following script produce wrong result: > {code} > a = load '1.txt' as (a0, a1); > b = order a by a1; > c = order b by a1; > d = foreach c generate a1; > dump d; > {code} > 1.txt: > {code} > 1 a > 2 b > 3 c > 6 d > 5 e > {code} > expected result: > a > b > c > d > e > current result: > 1 > 2 > 3 > 5 > 6 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1153) [zebra] spliting columns at different levels in a complex record column into different column groups throws exception
[ https://issues.apache.org/jira/browse/PIG-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yan Zhou updated PIG-1153: -- Status: Open (was: Patch Available) > [zebra] spliting columns at different levels in a complex record column into > different column groups throws exception > - > > Key: PIG-1153 > URL: https://issues.apache.org/jira/browse/PIG-1153 > Project: Pig > Issue Type: Bug > Components: impl >Affects Versions: 0.7.0 >Reporter: Xuefu Zhang >Assignee: Yan Zhou > Attachments: PIG-1153.patch, PIG-1153.patch > > > The following code sample: > String strSch = "r1:record(f1:int, f2:int), r2:record(f5:int, > r3:record(f3:float, f4))"; > String strStorage = "[r1.f1, r2.r3.f3, r2.f5]; [r1.f2, r2.r3.f4]"; > Partition p = new Partition(schema.toString(), strStorage, null); > gives the following exception: > org.apache.hadoop.zebra.parser.ParseException: Different Split Types Set > on the same field: r2.f5 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1153) [zebra] spliting columns at different levels in a complex record column into different column groups throws exception
[ https://issues.apache.org/jira/browse/PIG-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yan Zhou updated PIG-1153: -- Attachment: PIG-1153.patch > [zebra] spliting columns at different levels in a complex record column into > different column groups throws exception > - > > Key: PIG-1153 > URL: https://issues.apache.org/jira/browse/PIG-1153 > Project: Pig > Issue Type: Bug > Components: impl >Affects Versions: 0.7.0 >Reporter: Xuefu Zhang >Assignee: Yan Zhou > Attachments: PIG-1153.patch, PIG-1153.patch > > > The following code sample: > String strSch = "r1:record(f1:int, f2:int), r2:record(f5:int, > r3:record(f3:float, f4))"; > String strStorage = "[r1.f1, r2.r3.f3, r2.f5]; [r1.f2, r2.r3.f4]"; > Partition p = new Partition(schema.toString(), strStorage, null); > gives the following exception: > org.apache.hadoop.zebra.parser.ParseException: Different Split Types Set > on the same field: r2.f5 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1153) [zebra] spliting columns at different levels in a complex record column into different column groups throws exception
[ https://issues.apache.org/jira/browse/PIG-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yan Zhou updated PIG-1153: -- Status: Patch Available (was: Open) > [zebra] spliting columns at different levels in a complex record column into > different column groups throws exception > - > > Key: PIG-1153 > URL: https://issues.apache.org/jira/browse/PIG-1153 > Project: Pig > Issue Type: Bug > Components: impl >Affects Versions: 0.7.0 >Reporter: Xuefu Zhang >Assignee: Yan Zhou > Attachments: PIG-1153.patch, PIG-1153.patch > > > The following code sample: > String strSch = "r1:record(f1:int, f2:int), r2:record(f5:int, > r3:record(f3:float, f4))"; > String strStorage = "[r1.f1, r2.r3.f3, r2.f5]; [r1.f2, r2.r3.f4]"; > Partition p = new Partition(schema.toString(), strStorage, null); > gives the following exception: > org.apache.hadoop.zebra.parser.ParseException: Different Split Types Set > on the same field: r2.f5 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1164) [zebra]smoke test
[ https://issues.apache.org/jira/browse/PIG-1164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Huang updated PIG-1164: Attachment: (was: smoke.patch) > [zebra]smoke test > - > > Key: PIG-1164 > URL: https://issues.apache.org/jira/browse/PIG-1164 > Project: Pig > Issue Type: Test >Affects Versions: 0.6.0 >Reporter: Jing Huang > Fix For: 0.7.0 > > > Change zebra build.xml file to add smoke target. > And env.sh and run script under zebra/src/test/smoke dir -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (PIG-1168) Dump produces wrong results
Dump produces wrong results --- Key: PIG-1168 URL: https://issues.apache.org/jira/browse/PIG-1168 Project: Pig Issue Type: Bug Reporter: Ankur For a map-only job, dump just re-executes every pig-latin statement from the begininng assuming that they would produce same result. the assumption is not valid if there are UDFs that are invoked. Consider the following script:- raw = LOAD '$input' USING PigStorage() AS (text_string:chararray); DUMP raw; ccm = FOREACH raw GENERATE MyUDF(text_string); DUMP ccm; bug = FOREACH ccm GENERATE ccmObj; DUMP bug; The UDF MyUDF generates a tuple with one of the fields being a randomly generated UUID. So even though one would expect relations 'ccm' and 'bug' to contain identical data, they are different because of re-execution from the begininng. This breaks the application logic. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1153) [zebra] spliting columns at different levels in a complex record column into different column groups throws exception
[ https://issues.apache.org/jira/browse/PIG-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793522#action_12793522 ] Hadoop QA commented on PIG-1153: +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12428683/PIG-1153.patch against trunk revision 893053. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/150/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/150/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/150/console This message is automatically generated. > [zebra] spliting columns at different levels in a complex record column into > different column groups throws exception > - > > Key: PIG-1153 > URL: https://issues.apache.org/jira/browse/PIG-1153 > Project: Pig > Issue Type: Bug > Components: impl >Affects Versions: 0.7.0 >Reporter: Xuefu Zhang >Assignee: Yan Zhou > Attachments: PIG-1153.patch, PIG-1153.patch > > > The following code sample: > String strSch = "r1:record(f1:int, f2:int), r2:record(f5:int, > r3:record(f3:float, f4))"; > String strStorage = "[r1.f1, r2.r3.f3, r2.f5]; [r1.f2, r2.r3.f4]"; > Partition p = new Partition(schema.toString(), strStorage, null); > gives the following exception: > org.apache.hadoop.zebra.parser.ParseException: Different Split Types Set > on the same field: r2.f5 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.