[jira] [Assigned] (PIG-2924) PigStats should not be assuming all Storage classes to be file-based storage

[Cheolsoo Park (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cheolsoo Park reassigned PIG-2924:
--

Assignee: Cheolsoo Park

> PigStats should not be assuming all Storage classes to be file-based storage
> 
>
> Key: PIG-2924
> URL: https://issues.apache.org/jira/browse/PIG-2924
> Project: Pig
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 0.9.2
>Reporter: Harsh J
>Assignee: Cheolsoo Park
>
> Using PigStatsUtil (like Oozie does) to collect JobStats for jobs that use a 
> HBaseStorage blows up when the stats are asked to be accumulated.
> This is because JobStats (which adds stuff up) is assuming all storages are 
> file based and that it can do listStatus/etc. operations on their 
> filespec-provided filename. For HBaseStorage, this is set to the tablename 
> and there's no such file, leading to an exception (FileNotFound or Invalid 
> URI - depending on using 'tablename' or 'hbase://tablename').

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (PIG-2945) newline messes up deep casting

[Reza Zadeh (JIRA)]

Reza Zadeh created PIG-2945:
---

 Summary: newline messes up deep casting
 Key: PIG-2945
 URL: https://issues.apache.org/jira/browse/PIG-2945
 Project: Pig
  Issue Type: Bug
  Components: parser
Affects Versions: 0.10.0
Reporter: Reza Zadeh


Having a newline between GENERATE and a deep cast like (bag{tuple(long)}) 
causes a syntax error.

For example:

grunt> a = load '/etc/passwd' using PigStorage(':') as (foo, bar);
grunt> b = foreach a generate  (bag{tuple(long)}) foo;

Works. But this causes a syntax error:

grunt> b = foreach a generate   
>> (bag{tuple(long)}) foo;
2012-10-05 00:34:40,962 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 
1200:   Syntax error, unexpected symbol at or near 'bag'
Details at logfile: /var/log/pig/pig_1349397242524.log


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request: PIG-2898: allow to run pig e2e tests in parallel mode.

[Rohini Palaniswamy]

> On Oct. 4, 2012, 5:55 p.m., Rohini Palaniswamy wrote:
> > +1 non-binding. 
> > 
> > The tests passed fine for H20 and H23 when I ran them for branch-0.10. It 
> > hangs in local mode when run with Y! distribution of pig using the 
> > -Dpig.dir option. I think we can address that in a separate jira.  
> > 
> > The patch does not apply cleanly on 0.10 due to PERL5LIB recent change. You 
> > will have to create one for 0.10. Also need to upload this patch in the 
> > jira.

The problem with Y! distribution of pig was that it was not picking up 
PIG_OPTS. Worked fine after fixing that. 


- Rohini


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7053/#review12175
---


On Oct. 3, 2012, 12:53 p.m., Ivan Veselovsky wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7053/
> ---
> 
> (Updated Oct. 3, 2012, 12:53 p.m.)
> 
> 
> Review request for pig and Rohini Palaniswamy.
> 
> 
> Description
> ---
> 
> Please see https://issues.apache.org/jira/browse/PIG-2898 for details.
> 
> 
> This addresses bug https://issues.apache.org/jira/browse/PIG-2898.
> 
> https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/PIG-2898
> 
> 
> Diffs
> -
> 
>   http://svn.apache.org/repos/asf/pig/trunk/test/e2e/harness/TestDriver.pm 
> 1393450 
>   http://svn.apache.org/repos/asf/pig/trunk/test/e2e/harness/test_harness.pl 
> 1393450 
>   http://svn.apache.org/repos/asf/pig/trunk/test/e2e/pig/build.xml 1393450 
>   http://svn.apache.org/repos/asf/pig/trunk/test/e2e/pig/conf/local.conf 
> 1393450 
>   
> http://svn.apache.org/repos/asf/pig/trunk/test/e2e/pig/deployers/ExistingClusterDeployer.pm
>  1393450 
>   
> http://svn.apache.org/repos/asf/pig/trunk/test/e2e/pig/drivers/TestDriverPig.pm
>  1393450 
> 
> Diff: https://reviews.apache.org/r/7053/diff/
> 
> 
> Testing
> ---
> 
> Tested e2e tests execution in both sequential (default) and parallel modes.
> The test run duration measurement data (in dependency on the fork parameters) 
> will be available soon.
> 
> 
> Thanks,
> 
> Ivan Veselovsky
> 
>

Build failed in Jenkins: Pig-trunk #1328

[Apache Jenkins Server]

See 

Changes:

[jcoveney] PIG-2737: [piggybank] TestIndexedStorage is failing, should be 
refactored (jcoveney)

[daijy] PIG-2944: ivysettings.xml does not let you override .m2/repository

--
[...truncated 35949 lines...]
[junit] at 
org.apache.pig.test.MiniGenericCluster.shutdownMiniDfsAndMrClusters(MiniGenericCluster.java:77)
[junit] at 
org.apache.pig.test.MiniGenericCluster.shutDown(MiniGenericCluster.java:68)
[junit] at 
org.apache.pig.test.TestStore.oneTimeTearDown(TestStore.java:141)
[junit] Shutting down DataNode 2
[junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit] at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[junit] at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit] at java.lang.reflect.Method.invoke(Method.java:597)
[junit] at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
[junit] at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
[junit] at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
[junit] at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
[junit] at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
[junit] at 
junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
[junit] at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420)
[junit] at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911)
[junit] at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768)
[junit] 12/10/04 22:59:00 WARN datanode.FSDatasetAsyncDiskService: 
AsyncDiskService has already shut down.
[junit] 12/10/04 22:59:00 INFO mortbay.log: Stopped 
SelectChannelConnector@localhost:0
[junit] 12/10/04 22:59:00 INFO ipc.Server: Stopping server on 36826
[junit] 12/10/04 22:59:00 INFO ipc.Server: IPC Server handler 0 on 36826: 
exiting
[junit] 12/10/04 22:59:00 INFO ipc.Server: IPC Server handler 2 on 36826: 
exiting
[junit] 12/10/04 22:59:00 INFO ipc.Server: Stopping IPC Server listener on 
36826
[junit] 12/10/04 22:59:00 INFO metrics.RpcInstrumentation: shut down
[junit] 12/10/04 22:59:00 INFO ipc.Server: Stopping IPC Server Responder
[junit] 12/10/04 22:59:00 INFO ipc.Server: IPC Server handler 1 on 36826: 
exiting
[junit] 12/10/04 22:59:00 INFO datanode.DataNode: Waiting for threadgroup 
to exit, active threads is 1
[junit] 12/10/04 22:59:00 WARN datanode.DataNode: 
DatanodeRegistration(127.0.0.1:53526, 
storageID=DS-1699195994-67.195.138.20-53526-1349391046987, infoPort=57376, 
ipcPort=36826):DataXceiveServer:java.nio.channels.AsynchronousCloseException
[junit] at 
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
[junit] at 
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:159)
[junit] at 
sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
[junit] at 
org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
[junit] at java.lang.Thread.run(Thread.java:662)
[junit] 
[junit] 12/10/04 22:59:00 INFO datanode.DataNode: Exiting DataXceiveServer
[junit] 12/10/04 22:59:01 INFO datanode.DataBlockScanner: Exiting 
DataBlockScanner thread.
[junit] 12/10/04 22:59:01 INFO datanode.DataNode: Scheduling block 
blk_-2159142538880085882_1133 file 
build/test/data/dfs/data/data2/current/blk_-2159142538880085882 for deletion
[junit] 12/10/04 22:59:01 INFO datanode.DataNode: Scheduling block 
blk_-1890238983732947252_1134 file 
build/test/data/dfs/data/data1/current/blk_-1890238983732947252 for deletion
[junit] 12/10/04 22:59:01 INFO datanode.DataNode: Deleted block 
blk_-2159142538880085882_1133 at file 
build/test/data/dfs/data/data2/current/blk_-2159142538880085882
[junit] 12/10/04 22:59:01 INFO datanode.DataNode: Scheduling block 
blk_6523039248380329171_1134 file 
build/test/data/dfs/data/data2/current/blk_6523039248380329171 for deletion
[junit] 12/10/04 22:59:01 INFO datanode.DataNode: Deleted block 
blk_6523039248380329171_1134 at file 
build/test/data/dfs/data/data2/current/blk_6523039248380329171
[junit] 12/10/04 22:59:01 INFO datanode.DataNode: Deleted block 
blk_-1890238983732947252_1134 at file 
build/test/data/dfs/data/data1/current/blk_-1890238983732947252
[junit] 12/10/04 22:59:01 INFO datanode.DataNode: Waiting for threadgroup 
to exit, active threads is 0
[junit] 12/10/04 22:59:01 INFO datanode.DataNode: 
DatanodeRegistration(127.0.0.1:53526, 
storageID=DS-1699195994-67.195.138.20-535

[jira] [Updated] (PIG-2737) TestIndexedStorage is failing, should be refactored

[Jonathan Coveney (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Coveney updated PIG-2737:
--

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> TestIndexedStorage is failing, should be refactored
> ---
>
> Key: PIG-2737
> URL: https://issues.apache.org/jira/browse/PIG-2737
> Project: Pig
>  Issue Type: Bug
>Reporter: Jonathan Coveney
>Assignee: Jonathan Coveney
> Fix For: 0.11
>
> Attachments: PIG-2737-0.patch, PIG-2737-1.patch
>
>
> TestIndexedStorage is failing because it is not properly creating the output 
> directory. I've refactored it to use real temp directories instead of 
> something in the root.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request: Expand use of SchemaTuple to FOREACH

[Julien Le Dem]

> On Oct. 3, 2012, 6:29 p.m., Julien Le Dem wrote:
> > src/org/apache/pig/data/SchemaTupleFrontend.java, lines 124-125
> > 
> >
> > why did you add this?
> 
> Jonathan Coveney wrote:
> Whoops. This is an artifact of some bugtesting.
> 
> Jonathan Coveney wrote:
> It turns out that it is in fact necessary. There are some code paths 
> where it still needs to read this flag, but without short circuiting viz. 
> local mode.

Ok. then add a comment about it


> On Oct. 3, 2012, 6:29 p.m., Julien Le Dem wrote:
> > src/org/apache/pig/data/SchemaTupleFactory.java, line 55
> > 
> >
> > is empty schema the same as no schema?
> 
> Jonathan Coveney wrote:
> I guess my thought was that there isn't much value in generating empty 
> Tuples. But I guess there could be if there are a lot of them?

It is easier to read if there are no special cases


> On Oct. 3, 2012, 6:29 p.m., Julien Le Dem wrote:
> > test/org/apache/pig/test/TestLoad.java, line 333
> > 
> >
> > when using assertTrue you can pass a message to help debugging. Like 
> > op.getClass().getName()
> 
> Jonathan Coveney wrote:
> I just changed the Assert. => static import (for a consistent style), but 
> you're right in general.

I know :)


- Julien


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7336/#review12139
---


On Sept. 28, 2012, 12:21 a.m., Jonathan Coveney wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7336/
> ---
> 
> (Updated Sept. 28, 2012, 12:21 a.m.)
> 
> 
> Review request for pig.
> 
> 
> Description
> ---
> 
> This uses the SchemaTuple in Foreach, which means it will work after Loads
> 
> 
> This addresses bug PIG-2877.
> https://issues.apache.org/jira/browse/PIG-2877
> 
> 
> Diffs
> -
> 
>   src/org/apache/pig/PigConfiguration.java bf13d9d 
>   src/org/apache/pig/PigConstants.java PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java 
> d5c9315 
>   src/org/apache/pig/backend/hadoop/executionengine/HJob.java 2ff294a 
>   
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java
>  86d04f8 
>   
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigGenericMapBase.java
>  6ab33a4 
>   
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigGenericMapReduce.java
>  2e70462 
>   
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/partitioners/WeightedRangePartitioner.java
>  d7504ae 
>   
> src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/PhysicalOperator.java
>  26d5389 
>   
> src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POForEach.java
>  4e6876a 
>   
> src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POSort.java
>  702a01b 
>   src/org/apache/pig/data/BinInterSedes.java a7ba39b 
>   src/org/apache/pig/data/SchemaTuple.java 4f04834 
>   src/org/apache/pig/data/SchemaTupleBackend.java 2746316 
>   src/org/apache/pig/data/SchemaTupleClassGenerator.java ef15318 
>   src/org/apache/pig/data/SchemaTupleFactory.java b467d55 
>   src/org/apache/pig/data/SchemaTupleFrontend.java a8bfbab 
>   src/org/apache/pig/impl/builtin/FindQuantiles.java 1fcbdb1 
>   src/org/apache/pig/impl/io/ReadToEndLoader.java afb6ebd 
>   
> src/org/apache/pig/newplan/logical/relational/LogToPhyTranslationVisitor.java 
> 5c3dc43 
>   src/org/apache/pig/parser/LogicalPlanBuilder.java b0a8aa0 
>   test/org/apache/pig/data/TestSchemaTuple.java 24142c2 
>   test/org/apache/pig/test/TestBuiltin.java a835993 
>   test/org/apache/pig/test/TestCommit.java 5baaedc 
>   test/org/apache/pig/test/TestExampleGenerator.java 4babccb 
>   test/org/apache/pig/test/TestLoad.java a34e2bb 
>   test/org/apache/pig/test/TestPigServer.java 18a3dff 
>   test/org/apache/pig/test/TestPigServerWithMacros.java c6d8c4e 
> 
> Diff: https://reviews.apache.org/r/7336/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jonathan Coveney
> 
>

[jira] [Commented] (PIG-2737) TestIndexedStorage is failing, should be refactored

[Julien Le Dem (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469764#comment-13469764
 ] 

Julien Le Dem commented on PIG-2737:


+1

> TestIndexedStorage is failing, should be refactored
> ---
>
> Key: PIG-2737
> URL: https://issues.apache.org/jira/browse/PIG-2737
> Project: Pig
>  Issue Type: Bug
>Reporter: Jonathan Coveney
>Assignee: Jonathan Coveney
> Fix For: 0.11
>
> Attachments: PIG-2737-0.patch, PIG-2737-1.patch
>
>
> TestIndexedStorage is failing because it is not properly creating the output 
> directory. I've refactored it to use real temp directories instead of 
> something in the root.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request: Expand use of SchemaTuple to FOREACH

[Jonathan Coveney]

> On Oct. 3, 2012, 6:29 p.m., Julien Le Dem wrote:
> > src/org/apache/pig/data/SchemaTupleFrontend.java, lines 124-125
> > 
> >
> > why did you add this?
> 
> Jonathan Coveney wrote:
> Whoops. This is an artifact of some bugtesting.

It turns out that it is in fact necessary. There are some code paths where it 
still needs to read this flag, but without short circuiting viz. local mode.


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7336/#review12139
---


On Sept. 28, 2012, 12:21 a.m., Jonathan Coveney wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7336/
> ---
> 
> (Updated Sept. 28, 2012, 12:21 a.m.)
> 
> 
> Review request for pig.
> 
> 
> Description
> ---
> 
> This uses the SchemaTuple in Foreach, which means it will work after Loads
> 
> 
> This addresses bug PIG-2877.
> https://issues.apache.org/jira/browse/PIG-2877
> 
> 
> Diffs
> -
> 
>   src/org/apache/pig/PigConfiguration.java bf13d9d 
>   src/org/apache/pig/PigConstants.java PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java 
> d5c9315 
>   src/org/apache/pig/backend/hadoop/executionengine/HJob.java 2ff294a 
>   
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java
>  86d04f8 
>   
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigGenericMapBase.java
>  6ab33a4 
>   
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigGenericMapReduce.java
>  2e70462 
>   
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/partitioners/WeightedRangePartitioner.java
>  d7504ae 
>   
> src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/PhysicalOperator.java
>  26d5389 
>   
> src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POForEach.java
>  4e6876a 
>   
> src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POSort.java
>  702a01b 
>   src/org/apache/pig/data/BinInterSedes.java a7ba39b 
>   src/org/apache/pig/data/SchemaTuple.java 4f04834 
>   src/org/apache/pig/data/SchemaTupleBackend.java 2746316 
>   src/org/apache/pig/data/SchemaTupleClassGenerator.java ef15318 
>   src/org/apache/pig/data/SchemaTupleFactory.java b467d55 
>   src/org/apache/pig/data/SchemaTupleFrontend.java a8bfbab 
>   src/org/apache/pig/impl/builtin/FindQuantiles.java 1fcbdb1 
>   src/org/apache/pig/impl/io/ReadToEndLoader.java afb6ebd 
>   
> src/org/apache/pig/newplan/logical/relational/LogToPhyTranslationVisitor.java 
> 5c3dc43 
>   src/org/apache/pig/parser/LogicalPlanBuilder.java b0a8aa0 
>   test/org/apache/pig/data/TestSchemaTuple.java 24142c2 
>   test/org/apache/pig/test/TestBuiltin.java a835993 
>   test/org/apache/pig/test/TestCommit.java 5baaedc 
>   test/org/apache/pig/test/TestExampleGenerator.java 4babccb 
>   test/org/apache/pig/test/TestLoad.java a34e2bb 
>   test/org/apache/pig/test/TestPigServer.java 18a3dff 
>   test/org/apache/pig/test/TestPigServerWithMacros.java c6d8c4e 
> 
> Diff: https://reviews.apache.org/r/7336/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jonathan Coveney
> 
>

Re: Review Request: Expand use of SchemaTuple to FOREACH

[Jonathan Coveney]

> On Oct. 3, 2012, 6:29 p.m., Julien Le Dem wrote:
> > src/org/apache/pig/data/SchemaTupleBackend.java, line 311
> > 
> >
> > why would you not throw an exception here?

Good call.


> On Oct. 3, 2012, 6:29 p.m., Julien Le Dem wrote:
> > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POForEach.java,
> >  line 126
> > 
> >
> > feel free to remove constructors that are not used anymore

There's only one that isn't used anywhere (which I have now removed). The rest 
are used in niche places.


> On Oct. 3, 2012, 6:29 p.m., Julien Le Dem wrote:
> > src/org/apache/pig/data/SchemaTupleFactory.java, line 55
> > 
> >
> > is empty schema the same as no schema?

I guess my thought was that there isn't much value in generating empty Tuples. 
But I guess there could be if there are a lot of them?


> On Oct. 3, 2012, 6:29 p.m., Julien Le Dem wrote:
> > src/org/apache/pig/data/SchemaTupleFrontend.java, lines 124-125
> > 
> >
> > why did you add this?

Whoops. This is an artifact of some bugtesting.


> On Oct. 3, 2012, 6:29 p.m., Julien Le Dem wrote:
> > src/org/apache/pig/newplan/logical/relational/LogToPhyTranslationVisitor.java,
> >  line 864
> > 
> >
> > when would that happen? If we believe it can't happen let's throw an 
> > exception here.
> > That's the best way to get it fixed.

amen


> On Oct. 3, 2012, 6:29 p.m., Julien Le Dem wrote:
> > test/org/apache/pig/test/TestLoad.java, line 333
> > 
> >
> > when using assertTrue you can pass a message to help debugging. Like 
> > op.getClass().getName()

I just changed the Assert. => static import (for a consistent style), but 
you're right in general.


> On Oct. 3, 2012, 6:29 p.m., Julien Le Dem wrote:
> > test/org/apache/pig/test/TestLoad.java, lines 354-357
> > 
> >
> > probably not what you want

lol. death to IDEs


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7336/#review12139
---


On Sept. 28, 2012, 12:21 a.m., Jonathan Coveney wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7336/
> ---
> 
> (Updated Sept. 28, 2012, 12:21 a.m.)
> 
> 
> Review request for pig.
> 
> 
> Description
> ---
> 
> This uses the SchemaTuple in Foreach, which means it will work after Loads
> 
> 
> This addresses bug PIG-2877.
> https://issues.apache.org/jira/browse/PIG-2877
> 
> 
> Diffs
> -
> 
>   src/org/apache/pig/PigConfiguration.java bf13d9d 
>   src/org/apache/pig/PigConstants.java PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java 
> d5c9315 
>   src/org/apache/pig/backend/hadoop/executionengine/HJob.java 2ff294a 
>   
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java
>  86d04f8 
>   
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigGenericMapBase.java
>  6ab33a4 
>   
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigGenericMapReduce.java
>  2e70462 
>   
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/partitioners/WeightedRangePartitioner.java
>  d7504ae 
>   
> src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/PhysicalOperator.java
>  26d5389 
>   
> src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POForEach.java
>  4e6876a 
>   
> src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POSort.java
>  702a01b 
>   src/org/apache/pig/data/BinInterSedes.java a7ba39b 
>   src/org/apache/pig/data/SchemaTuple.java 4f04834 
>   src/org/apache/pig/data/SchemaTupleBackend.java 2746316 
>   src/org/apache/pig/data/SchemaTupleClassGenerator.java ef15318 
>   src/org/apache/pig/data/SchemaTupleFactory.java b467d55 
>   src/org/apache/pig/data/SchemaTupleFrontend.java a8bfbab 
>   src/org/apache/pig/impl/builtin/FindQuantiles.java 1fcbdb1 
>   src/org/apache/pig/impl/io/ReadToEndLoader.java afb6ebd 
>   
> src/org/apache/pig/newplan/logical/relational/LogToPhyTranslationVisitor.java 
> 5c3dc43 
>   src/org/apache/pig/parser/LogicalPlanBuilder.java b0a8aa0 
>   test/org/apache/pig/data/TestSchemaTuple.java 24142c2 
>   test/org/apache/pig/test/TestBuilti

[jira] [Commented] (PIG-2769) a simple logic causes very long compiling time on pig 0.10.0

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469592#comment-13469592
 ] 

Thejas M Nair commented on PIG-2769:


I have assigned it to you. I have also added you to contributors list, now you 
should be able to assign jiras to yourself.


> a simple logic causes very long compiling time on pig 0.10.0
> 
>
> Key: PIG-2769
> URL: https://issues.apache.org/jira/browse/PIG-2769
> Project: Pig
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.10.0
> Environment: Apache Pig version 0.10.0-SNAPSHOT (rexported)
>Reporter: Dan Li
>Assignee: Timothy Chen
> Fix For: 0.11
>
> Attachments: case1.tar
>
>
> We found the following simple logic will cause very long compiling time for 
> pig 0.10.0, while using pig 0.8.1, everything is fine.
> A = load 'A.txt' using PigStorage()  AS (m: int);
> B = FOREACH A {
> days_str = (chararray)
> (m == 1 ? 31: 
> (m == 2 ? 28: 
> (m == 3 ? 31: 
> (m == 4 ? 30: 
> (m == 5 ? 31: 
> (m == 6 ? 30: 
> (m == 7 ? 31: 
> (m == 8 ? 31: 
> (m == 9 ? 30: 
> (m == 10 ? 31: 
> (m == 11 ? 30:31)));
> GENERATE
>days_str as days_str;
> }   
> store B into 'B';
> and here's a simple input file example: A.txt
> 1
> 2
> 3
> The pig version we used in the test
> Apache Pig version 0.10.0-SNAPSHOT (rexported)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2769) a simple logic causes very long compiling time on pig 0.10.0

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated PIG-2769:
---

Assignee: Timothy Chen

> a simple logic causes very long compiling time on pig 0.10.0
> 
>
> Key: PIG-2769
> URL: https://issues.apache.org/jira/browse/PIG-2769
> Project: Pig
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.10.0
> Environment: Apache Pig version 0.10.0-SNAPSHOT (rexported)
>Reporter: Dan Li
>Assignee: Timothy Chen
> Fix For: 0.11
>
> Attachments: case1.tar
>
>
> We found the following simple logic will cause very long compiling time for 
> pig 0.10.0, while using pig 0.8.1, everything is fine.
> A = load 'A.txt' using PigStorage()  AS (m: int);
> B = FOREACH A {
> days_str = (chararray)
> (m == 1 ? 31: 
> (m == 2 ? 28: 
> (m == 3 ? 31: 
> (m == 4 ? 30: 
> (m == 5 ? 31: 
> (m == 6 ? 30: 
> (m == 7 ? 31: 
> (m == 8 ? 31: 
> (m == 9 ? 30: 
> (m == 10 ? 31: 
> (m == 11 ? 30:31)));
> GENERATE
>days_str as days_str;
> }   
> store B into 'B';
> and here's a simple input file example: A.txt
> 1
> 2
> 3
> The pig version we used in the test
> Apache Pig version 0.10.0-SNAPSHOT (rexported)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2769) a simple logic causes very long compiling time on pig 0.10.0

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469587#comment-13469587
 ] 

Timothy Chen commented on PIG-2769:
---

I'd like to look at this for my first bug fix for pig, as I've already glanced 
through the parsing and query plan code. Can't find a way to assign this to 
myself via Jira though, probably need someone else to do so?

> a simple logic causes very long compiling time on pig 0.10.0
> 
>
> Key: PIG-2769
> URL: https://issues.apache.org/jira/browse/PIG-2769
> Project: Pig
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.10.0
> Environment: Apache Pig version 0.10.0-SNAPSHOT (rexported)
>Reporter: Dan Li
> Fix For: 0.11
>
> Attachments: case1.tar
>
>
> We found the following simple logic will cause very long compiling time for 
> pig 0.10.0, while using pig 0.8.1, everything is fine.
> A = load 'A.txt' using PigStorage()  AS (m: int);
> B = FOREACH A {
> days_str = (chararray)
> (m == 1 ? 31: 
> (m == 2 ? 28: 
> (m == 3 ? 31: 
> (m == 4 ? 30: 
> (m == 5 ? 31: 
> (m == 6 ? 30: 
> (m == 7 ? 31: 
> (m == 8 ? 31: 
> (m == 9 ? 30: 
> (m == 10 ? 31: 
> (m == 11 ? 30:31)));
> GENERATE
>days_str as days_str;
> }   
> store B into 'B';
> and here's a simple input file example: A.txt
> 1
> 2
> 3
> The pig version we used in the test
> Apache Pig version 0.10.0-SNAPSHOT (rexported)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2944) ivysettings.xml does not let you override .m2/repository

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated PIG-2944:


  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Patch committed to 0.9/0.10/trunk. Thanks Raja!

> ivysettings.xml does not let you override .m2/repository
> 
>
> Key: PIG-2944
> URL: https://issues.apache.org/jira/browse/PIG-2944
> Project: Pig
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.10.0, 0.9.3, 0.11
>Reporter: Raja Aluri
>Assignee: Raja Aluri
> Fix For: 0.9.3, 0.11, 0.10.1
>
> Attachments: PIG-2944-1.patch
>
>
> ivysettings.xml does not let you override .m2/repository. This will be useful 
> for people who wants to use a localrepository, thats outside of 
> ~/.m2/repository

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2944) ivysettings.xml does not let you override .m2/repository

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated PIG-2944:


Affects Version/s: 0.11
   0.10.0
Fix Version/s: 0.10.1
   0.11

> ivysettings.xml does not let you override .m2/repository
> 
>
> Key: PIG-2944
> URL: https://issues.apache.org/jira/browse/PIG-2944
> Project: Pig
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.10.0, 0.9.3, 0.11
>Reporter: Raja Aluri
>Assignee: Raja Aluri
> Fix For: 0.9.3, 0.11, 0.10.1
>
> Attachments: PIG-2944-1.patch
>
>
> ivysettings.xml does not let you override .m2/repository. This will be useful 
> for people who wants to use a localrepository, thats outside of 
> ~/.m2/repository

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2944) ivysettings.xml does not let you override .m2/repository

[Raja Aluri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raja Aluri updated PIG-2944:


Status: Patch Available  (was: Open)

> ivysettings.xml does not let you override .m2/repository
> 
>
> Key: PIG-2944
> URL: https://issues.apache.org/jira/browse/PIG-2944
> Project: Pig
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.9.3
>Reporter: Raja Aluri
>Assignee: Raja Aluri
> Fix For: 0.9.3
>
> Attachments: PIG-2944-1.patch
>
>
> ivysettings.xml does not let you override .m2/repository. This will be useful 
> for people who wants to use a localrepository, thats outside of 
> ~/.m2/repository

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2944) ivysettings.xml does not let you override .m2/repository

[Raja Aluri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raja Aluri updated PIG-2944:


Attachment: PIG-2944-1.patch

> ivysettings.xml does not let you override .m2/repository
> 
>
> Key: PIG-2944
> URL: https://issues.apache.org/jira/browse/PIG-2944
> Project: Pig
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.9.3
>Reporter: Raja Aluri
>Assignee: Raja Aluri
> Fix For: 0.9.3
>
> Attachments: PIG-2944-1.patch
>
>
> ivysettings.xml does not let you override .m2/repository. This will be useful 
> for people who wants to use a localrepository, thats outside of 
> ~/.m2/repository

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (PIG-2944) ivysettings.xml does not let you override .m2/repository

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai reassigned PIG-2944:
---

Assignee: Raja Aluri

> ivysettings.xml does not let you override .m2/repository
> 
>
> Key: PIG-2944
> URL: https://issues.apache.org/jira/browse/PIG-2944
> Project: Pig
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.9.3
>Reporter: Raja Aluri
>Assignee: Raja Aluri
> Fix For: 0.9.3
>
>
> ivysettings.xml does not let you override .m2/repository. This will be useful 
> for people who wants to use a localrepository, thats outside of 
> ~/.m2/repository

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (PIG-2944) ivysettings.xml does not let you override .m2/repository

[Raja Aluri (JIRA)]

Raja Aluri created PIG-2944:
---

 Summary: ivysettings.xml does not let you override .m2/repository
 Key: PIG-2944
 URL: https://issues.apache.org/jira/browse/PIG-2944
 Project: Pig
  Issue Type: Bug
  Components: build
Affects Versions: 0.9.3
Reporter: Raja Aluri
 Fix For: 0.9.3


ivysettings.xml does not let you override .m2/repository. This will be useful 
for people who wants to use a localrepository, thats outside of ~/.m2/repository

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request: PIG-2898: allow to run pig e2e tests in parallel mode.

[Rohini Palaniswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7053/#review12175
---

Ship it!


+1 non-binding. 

The tests passed fine for H20 and H23 when I ran them for branch-0.10. It hangs 
in local mode when run with Y! distribution of pig using the -Dpig.dir option. 
I think we can address that in a separate jira.  

The patch does not apply cleanly on 0.10 due to PERL5LIB recent change. You 
will have to create one for 0.10. Also need to upload this patch in the jira. 

- Rohini Palaniswamy


On Oct. 3, 2012, 12:53 p.m., Ivan Veselovsky wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7053/
> ---
> 
> (Updated Oct. 3, 2012, 12:53 p.m.)
> 
> 
> Review request for pig and Rohini Palaniswamy.
> 
> 
> Description
> ---
> 
> Please see https://issues.apache.org/jira/browse/PIG-2898 for details.
> 
> 
> This addresses bug https://issues.apache.org/jira/browse/PIG-2898.
> 
> https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/PIG-2898
> 
> 
> Diffs
> -
> 
>   http://svn.apache.org/repos/asf/pig/trunk/test/e2e/harness/TestDriver.pm 
> 1393450 
>   http://svn.apache.org/repos/asf/pig/trunk/test/e2e/harness/test_harness.pl 
> 1393450 
>   http://svn.apache.org/repos/asf/pig/trunk/test/e2e/pig/build.xml 1393450 
>   http://svn.apache.org/repos/asf/pig/trunk/test/e2e/pig/conf/local.conf 
> 1393450 
>   
> http://svn.apache.org/repos/asf/pig/trunk/test/e2e/pig/deployers/ExistingClusterDeployer.pm
>  1393450 
>   
> http://svn.apache.org/repos/asf/pig/trunk/test/e2e/pig/drivers/TestDriverPig.pm
>  1393450 
> 
> Diff: https://reviews.apache.org/r/7053/diff/
> 
> 
> Testing
> ---
> 
> Tested e2e tests execution in both sequential (default) and parallel modes.
> The test run duration measurement data (in dependency on the fork parameters) 
> will be available soon.
> 
> 
> Thanks,
> 
> Ivan Veselovsky
> 
>

[jira] Subscription: PIG patch available

[jira]

Issue Subscription
Filter: PIG patch available (25 issues)

Subscriber: pigdaily

Key Summary
PIG-2940HBaseStorage store fails in secure cluster
https://issues.apache.org/jira/browse/PIG-2940
PIG-2932Setting high default_parallel causes IOException in local mode
https://issues.apache.org/jira/browse/PIG-2932
PIG-2931$ signs in the replacement string make parameter substitution fail
https://issues.apache.org/jira/browse/PIG-2931
PIG-2928Fix e2e test failures in trunk: FilterBoolean_23/24
https://issues.apache.org/jira/browse/PIG-2928
PIG-2925Extremely long JobConf values should not be added to Streaming 
environment
https://issues.apache.org/jira/browse/PIG-2925
PIG-2908Fix unit tests to work with jdk7
https://issues.apache.org/jira/browse/PIG-2908
PIG-2898Parallel execution of e2e tests
https://issues.apache.org/jira/browse/PIG-2898
PIG-2881Add SUBTRACT eval function
https://issues.apache.org/jira/browse/PIG-2881
PIG-2877Make SchemaTuple work in foreach (and thus, in loads) 
https://issues.apache.org/jira/browse/PIG-2877
PIG-2873Converting bin/pig shell script to python
https://issues.apache.org/jira/browse/PIG-2873
PIG-2834MultiStorage requires unused constructor argument
https://issues.apache.org/jira/browse/PIG-2834
PIG-2824Pushing checking number of fields into LoadFunc
https://issues.apache.org/jira/browse/PIG-2824
PIG-2778Add 'matches' operator to predicate pushdown
https://issues.apache.org/jira/browse/PIG-2778
PIG-2737TestIndexedStorage is failing, should be refactored
https://issues.apache.org/jira/browse/PIG-2737
PIG-2689JsonStorage fails to find schema when LimitAdjuster runs
https://issues.apache.org/jira/browse/PIG-2689
PIG-2661Pig uses an extra job for loading data in Pigmix L9
https://issues.apache.org/jira/browse/PIG-2661
PIG-2657Print warning if using wrong jython version
https://issues.apache.org/jira/browse/PIG-2657
PIG-2579Support for multiple input schemas in AvroStorage
https://issues.apache.org/jira/browse/PIG-2579
PIG-2495Using merge JOIN from a HBaseStorage produces an error
https://issues.apache.org/jira/browse/PIG-2495
PIG-2417Streaming UDFs -  allow users to easily write UDFs in scripting 
languages with no JVM implementation.
https://issues.apache.org/jira/browse/PIG-2417
PIG-2405svn tags/release-0.9.1: some unit test case failed with open JDK
https://issues.apache.org/jira/browse/PIG-2405
PIG-2362Rework Ant build.xml to use macrodef instead of antcall
https://issues.apache.org/jira/browse/PIG-2362
PIG-2312NPE when relation and column share the same name and used in Nested 
Foreach 
https://issues.apache.org/jira/browse/PIG-2312
PIG-1942script UDF (jython) should utilize the intended output schema to 
more directly convert Py objects to Pig objects
https://issues.apache.org/jira/browse/PIG-1942
PIG-1237Piggybank MutliStorage - specify field to write in output
https://issues.apache.org/jira/browse/PIG-1237

You may edit this subscription at:
https://issues.apache.org/jira/secure/FilterSubscription!default.jspa?subId=13225&filterId=12322384

[jira] [Updated] (PIG-2942) DevTests, TestLoad has a false failure on Windows

[John Gordon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Gordon updated PIG-2942:
-

Issue Type: Sub-task  (was: Bug)
Parent: PIG-2793

> DevTests, TestLoad has a false failure on Windows
> -
>
> Key: PIG-2942
> URL: https://issues.apache.org/jira/browse/PIG-2942
> Project: Pig
>  Issue Type: Sub-task
>  Components: build
>Affects Versions: 0.10.0
>Reporter: John Gordon
> Attachments: 0003-One-off-test-fix-for-TestLoad.patch
>
>
> TestLoad has a "ParseCygPath" call that doesn't work under Cygwin or without 
> it on Windows.  Removing it actually stabilizes the test under both 
> conditions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2943) DevTests, Refactor Windows checks to use new Util.WINDOWS method for code health

[John Gordon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Gordon updated PIG-2943:
-

Issue Type: Sub-task  (was: Task)
Parent: PIG-2793

> DevTests, Refactor Windows checks to use new Util.WINDOWS method for code 
> health
> 
>
> Key: PIG-2943
> URL: https://issues.apache.org/jira/browse/PIG-2943
> Project: Pig
>  Issue Type: Sub-task
>  Components: build
>Affects Versions: 0.10.0
>Reporter: John Gordon
> Attachments: 0004-One-off-for-testParamSubPreproc.patch, 
> 0005-One-off-for-TypeCheckingValidator.patch
>
>
> SOme tests still look for Windows via a search of the OS string.  Since we 
> have a utility function, now, it makes sense to refactor all of those to use 
> that function.  That way, if we ever need to change the logic there is only 
> one place to worry about it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2943) DevTests, Refactor Windows checks to use new Util.WINDOWS method for code health

[John Gordon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Gordon updated PIG-2943:
-

Attachment: 0004-One-off-for-testParamSubPreproc.patch

> DevTests, Refactor Windows checks to use new Util.WINDOWS method for code 
> health
> 
>
> Key: PIG-2943
> URL: https://issues.apache.org/jira/browse/PIG-2943
> Project: Pig
>  Issue Type: Task
>  Components: build
>Affects Versions: 0.10.0
>Reporter: John Gordon
> Attachments: 0004-One-off-for-testParamSubPreproc.patch, 
> 0005-One-off-for-TypeCheckingValidator.patch
>
>
> SOme tests still look for Windows via a search of the OS string.  Since we 
> have a utility function, now, it makes sense to refactor all of those to use 
> that function.  That way, if we ever need to change the logic there is only 
> one place to worry about it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2942) DevTests, TestLoad has a false failure on Windows

[John Gordon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Gordon updated PIG-2942:
-

Attachment: (was: 0004-One-off-for-testParamSubPreproc.patch)

> DevTests, TestLoad has a false failure on Windows
> -
>
> Key: PIG-2942
> URL: https://issues.apache.org/jira/browse/PIG-2942
> Project: Pig
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.10.0
>Reporter: John Gordon
> Attachments: 0003-One-off-test-fix-for-TestLoad.patch
>
>
> TestLoad has a "ParseCygPath" call that doesn't work under Cygwin or without 
> it on Windows.  Removing it actually stabilizes the test under both 
> conditions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2943) DevTests, Refactor Windows checks to use new Util.WINDOWS method for code health

[John Gordon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Gordon updated PIG-2943:
-

Attachment: 0005-One-off-for-TypeCheckingValidator.patch

> DevTests, Refactor Windows checks to use new Util.WINDOWS method for code 
> health
> 
>
> Key: PIG-2943
> URL: https://issues.apache.org/jira/browse/PIG-2943
> Project: Pig
>  Issue Type: Task
>  Components: build
>Affects Versions: 0.10.0
>Reporter: John Gordon
> Attachments: 0004-One-off-for-testParamSubPreproc.patch, 
> 0005-One-off-for-TypeCheckingValidator.patch
>
>
> SOme tests still look for Windows via a search of the OS string.  Since we 
> have a utility function, now, it makes sense to refactor all of those to use 
> that function.  That way, if we ever need to change the logic there is only 
> one place to worry about it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2942) DevTests, TestLoad has a false failure on Windows

[John Gordon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Gordon updated PIG-2942:
-

Attachment: 0004-One-off-for-testParamSubPreproc.patch

> DevTests, TestLoad has a false failure on Windows
> -
>
> Key: PIG-2942
> URL: https://issues.apache.org/jira/browse/PIG-2942
> Project: Pig
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.10.0
>Reporter: John Gordon
> Attachments: 0003-One-off-test-fix-for-TestLoad.patch, 
> 0004-One-off-for-testParamSubPreproc.patch
>
>
> TestLoad has a "ParseCygPath" call that doesn't work under Cygwin or without 
> it on Windows.  Removing it actually stabilizes the test under both 
> conditions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (PIG-2943) DevTests, Refactor Windows checks to use new Util.WINDOWS method for code health

[John Gordon (JIRA)]

John Gordon created PIG-2943:


 Summary: DevTests, Refactor Windows checks to use new Util.WINDOWS 
method for code health
 Key: PIG-2943
 URL: https://issues.apache.org/jira/browse/PIG-2943
 Project: Pig
  Issue Type: Task
  Components: build
Affects Versions: 0.10.0
Reporter: John Gordon


SOme tests still look for Windows via a search of the OS string.  Since we have 
a utility function, now, it makes sense to refactor all of those to use that 
function.  That way, if we ever need to change the logic there is only one 
place to worry about it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (PIG-2942) DevTests, TestLoad has a false failure on Windows

[John Gordon (JIRA)]

John Gordon created PIG-2942:


 Summary: DevTests, TestLoad has a false failure on Windows
 Key: PIG-2942
 URL: https://issues.apache.org/jira/browse/PIG-2942
 Project: Pig
  Issue Type: Bug
  Components: build
Affects Versions: 0.10.0
Reporter: John Gordon
 Attachments: 0003-One-off-test-fix-for-TestLoad.patch

TestLoad has a "ParseCygPath" call that doesn't work under Cygwin or without it 
on Windows.  Removing it actually stabilizes the test under both conditions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2942) DevTests, TestLoad has a false failure on Windows

[John Gordon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Gordon updated PIG-2942:
-

Attachment: 0003-One-off-test-fix-for-TestLoad.patch

> DevTests, TestLoad has a false failure on Windows
> -
>
> Key: PIG-2942
> URL: https://issues.apache.org/jira/browse/PIG-2942
> Project: Pig
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.10.0
>Reporter: John Gordon
> Attachments: 0003-One-off-test-fix-for-TestLoad.patch
>
>
> TestLoad has a "ParseCygPath" call that doesn't work under Cygwin or without 
> it on Windows.  Removing it actually stabilizes the test under both 
> conditions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2941) Ivy resolvers in pig don't have consistent chaining and don't have a kitchen sink option for novices

[John Gordon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Gordon updated PIG-2941:
-

Attachment: 0001-IvySettings.xml-refactor-to-simplify-resolution.patch

Patch to refactor resolvers into 3 cascading chains

> Ivy resolvers in pig don't have consistent chaining and don't have a kitchen 
> sink option for novices
> 
>
> Key: PIG-2941
> URL: https://issues.apache.org/jira/browse/PIG-2941
> Project: Pig
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.10.0
>Reporter: John Gordon
> Attachments: 
> 0001-IvySettings.xml-refactor-to-simplify-resolution.patch
>
>
> The Ivy resolvers in Pig are split into default, external, and internal -- 
> and they are all actually distinct.  There isn't a resolver that rolls over 
> all three, and fallbacks aren't in place.  Ideally, these resolver should 
> chain right through with the default following a best practice fallback for 
> novices.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (PIG-2941) Ivy resolvers in pig don't have consistent chaining and don't have a kitchen sink option for novices

[John Gordon (JIRA)]

John Gordon created PIG-2941:


 Summary: Ivy resolvers in pig don't have consistent chaining and 
don't have a kitchen sink option for novices
 Key: PIG-2941
 URL: https://issues.apache.org/jira/browse/PIG-2941
 Project: Pig
  Issue Type: Bug
  Components: build
Affects Versions: 0.10.0
Reporter: John Gordon


The Ivy resolvers in Pig are split into default, external, and internal -- and 
they are all actually distinct.  There isn't a resolver that rolls over all 
three, and fallbacks aren't in place.  Ideally, these resolver should chain 
right through with the default following a best practice fallback for novices.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2794) Pig test: add utils to simplify testing on Windows

[John Gordon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Gordon updated PIG-2794:
-

Attachment: 0007-Utils-to-support-windows-port.patch

Updated patch.

> Pig test: add utils to simplify testing on Windows
> --
>
> Key: PIG-2794
> URL: https://issues.apache.org/jira/browse/PIG-2794
> Project: Pig
>  Issue Type: Sub-task
>  Components: tools
> Environment: Windows
>Reporter: John Gordon
> Attachments: 0007-Utils-to-support-windows-port.patch, 
> trunkUtilpatch.patch
>
>
> Add facilities to simplify testing on Windows for things like path management 
> and determining if the OS is Windows or not.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2795) Fix test cases that generate pig scripts with "load " + pathStr to encode "\" in the path

[John Gordon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Gordon updated PIG-2795:
-

Attachment: 0006-Test-Changes-For-Path-Separator-on-Windows.patch

Updated test changes for windows path separator "\"

> Fix test cases that generate pig scripts with "load " + pathStr to encode "\" 
> in the path
> -
>
> Key: PIG-2795
> URL: https://issues.apache.org/jira/browse/PIG-2795
> Project: Pig
>  Issue Type: Sub-task
>Reporter: John Gordon
> Attachments: 0006-Test-Changes-For-Path-Separator-on-Windows.patch, 
> trunkEncodeEscape.patch
>
>
> Pig reads "\" as an escape character in pig scripts -- but Windows paths 
> natively contain \.  When we formulate pig queries that should be read 
> through the parser with file paths, we should call encodeEscape first to 
> handle paths generated on Windows.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2796) Local temporary paths are not always valid HDFS path names.

[John Gordon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Gordon updated PIG-2796:
-

Attachment: 0006-Local-Remote-file-mapping-for-tests-with-temps.patch

Test fixes for local file path usage in HDFS.  Largely, it is removing the 
colon in windows paths and minimizing the number of direct File.toString() 
usage to create HDFS paths.

> Local temporary paths are not always valid HDFS path names.
> ---
>
> Key: PIG-2796
> URL: https://issues.apache.org/jira/browse/PIG-2796
> Project: Pig
>  Issue Type: Sub-task
>Reporter: John Gordon
> Attachments: 0006-Local-Remote-file-mapping-for-tests-with-temps.patch
>
>
> A number of pig scripts follow the pattern:
> File tempFile = File.createTempFile("this", ".txt");
> copyFromLocalToCluster (tempFile.to_string(), tempFile.to_string());
> tempFile.delete();
> The goal, here, seems to be to generate a temp filename to avoid issues on 
> the next run if the file doesn't get cleaned up.  The problem is that 
> File.createTempFile on Windows creates files with names like 
> C:\users\myuser\App data\local\temp\file.txt.  The problem is that ":" is not 
> a valid DFS character and so the put fails.
> The easy fix on this is to remove colons from the path before upload.  Then 
> we get something like C\users\myuser\App data\local\temp\file.txt which is a 
> valid DFS pathname with minimal impact to the tests.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2798) pig streaming tests assume interpreters are auto-resolved

[John Gordon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Gordon updated PIG-2798:
-

Attachment: 0008-Updating-test-execution-invocation-model.patch

Patch for streaming tests to specify their interpreters at execution time to 
apply to multiple platforms with a uniform behavior.

> pig streaming tests assume interpreters are auto-resolved
> -
>
> Key: PIG-2798
> URL: https://issues.apache.org/jira/browse/PIG-2798
> Project: Pig
>  Issue Type: Sub-task
>Reporter: John Gordon
> Attachments: 0008-Updating-test-execution-invocation-model.patch
>
>
> many of the pig tests assume the first line (e.g. #!/bin/sh) auto-resolves 
> the interpreter.  This doesn't work on Windows.  To run scripts in a 
> cross-platform context, tests should invoke "perl test.pl" or "sh test.sh" 
> instead.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2801) grunt "sh" command should invoke the shell implicitly instead of calling exec directly with the command tokens

[John Gordon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Gordon updated PIG-2801:
-

Attachment: 0008-Updating-test-execution-invocation-model.patch

Test invocation updates for cross-platform applicability (assuming 
Grunt/Streaming changes) -- work on both Windows and Linux.

> grunt "sh" command should invoke the shell implicitly instead of calling exec 
> directly with the command tokens
> --
>
> Key: PIG-2801
> URL: https://issues.apache.org/jira/browse/PIG-2801
> Project: Pig
>  Issue Type: Sub-task
>Reporter: John Gordon
> Attachments: 0003-Grunt-Streaming-script-executor-changes.patch, 
> 0008-Updating-test-execution-invocation-model.patch
>
>
> The "sh" command in grunt is very sensitive to the behavior of exec.  It 
> requires that you provide an executable filename as the first argument, and 
> those are sort of up for interpretation from platform to platform.  For this 
> to easily handle batch scripts, built-in commands, and executables with one 
> semantic, it should spin up an intermediate command-shell.  This would 
> translate "sh " into "exec("sh" ["-c", ])".  This helps to 
> smooth out some of the operating system differences between Windows and 
> Linux, and also improves usability of grunt as a whole.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2801) grunt "sh" command should invoke the shell implicitly instead of calling exec directly with the command tokens

[John Gordon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Gordon updated PIG-2801:
-

Attachment: 0003-Grunt-Streaming-script-executor-changes.patch

Patch containing fixes to implicitly spawn streaming operations within the OS 
default shell to handle scripts more uniformly across processes.

> grunt "sh" command should invoke the shell implicitly instead of calling exec 
> directly with the command tokens
> --
>
> Key: PIG-2801
> URL: https://issues.apache.org/jira/browse/PIG-2801
> Project: Pig
>  Issue Type: Sub-task
>Reporter: John Gordon
> Attachments: 0003-Grunt-Streaming-script-executor-changes.patch, 
> 0008-Updating-test-execution-invocation-model.patch
>
>
> The "sh" command in grunt is very sensitive to the behavior of exec.  It 
> requires that you provide an executable filename as the first argument, and 
> those are sort of up for interpretation from platform to platform.  For this 
> to easily handle batch scripts, built-in commands, and executables with one 
> semantic, it should spin up an intermediate command-shell.  This would 
> translate "sh " into "exec("sh" ["-c", ])".  This helps to 
> smooth out some of the operating system differences between Windows and 
> Linux, and also improves usability of grunt as a whole.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira