[jira] Subscription: PIG patch available

[jira]

Issue Subscription
Filter: PIG patch available (29 issues)

Subscriber: pigdaily

Key Summary
PIG-4745DataBag should protect content of passed list of tuples
https://issues.apache.org/jira/browse/PIG-4745
PIG-4734TOMAP schema inferring breaks some scripts in type checking for 
bincond
https://issues.apache.org/jira/browse/PIG-4734
PIG-4709Improve performance of GROUPBY operator on Spark
https://issues.apache.org/jira/browse/PIG-4709
PIG-4684Exception should be changed to warning when job diagnostics cannot 
be fetched
https://issues.apache.org/jira/browse/PIG-4684
PIG-4677Display failure information on stop on failure
https://issues.apache.org/jira/browse/PIG-4677
PIG-4656Improve String serialization and comparator performance in 
BinInterSedes
https://issues.apache.org/jira/browse/PIG-4656
PIG-4641Print the instance of Object without using toString()
https://issues.apache.org/jira/browse/PIG-4641
PIG-4598Allow user defined plan optimizer rules
https://issues.apache.org/jira/browse/PIG-4598
PIG-4581thread safe issue in NodeIdGenerator
https://issues.apache.org/jira/browse/PIG-4581
PIG-4551Partition filter is not pushed down in case of SPLIT
https://issues.apache.org/jira/browse/PIG-4551
PIG-4539New PigUnit
https://issues.apache.org/jira/browse/PIG-4539
PIG-4515org.apache.pig.builtin.Distinct throws ClassCastException
https://issues.apache.org/jira/browse/PIG-4515
PIG-4455Should use DependencyOrderWalker instead of DepthFirstWalker in 
MRPrinter
https://issues.apache.org/jira/browse/PIG-4455
PIG-4373Implement PIG-3861 in Tez
https://issues.apache.org/jira/browse/PIG-4373
PIG-4341Add CMX support to pig.tmpfilecompression.codec
https://issues.apache.org/jira/browse/PIG-4341
PIG-4323PackageConverter hanging in Spark
https://issues.apache.org/jira/browse/PIG-4323
PIG-4313StackOverflowError in LIMIT operation on Spark
https://issues.apache.org/jira/browse/PIG-4313
PIG-4251Pig on Storm
https://issues.apache.org/jira/browse/PIG-4251
PIG-4111Make Pig compiles with avro-1.7.7
https://issues.apache.org/jira/browse/PIG-4111
PIG-4002Disable combiner when map-side aggregation is used
https://issues.apache.org/jira/browse/PIG-4002
PIG-3952PigStorage accepts '-tagSplit' to return full split information
https://issues.apache.org/jira/browse/PIG-3952
PIG-3911Define unique fields with @OutputSchema
https://issues.apache.org/jira/browse/PIG-3911
PIG-3877Getting Geo Latitude/Longitude from Address Lines
https://issues.apache.org/jira/browse/PIG-3877
PIG-3873Geo distance calculation using Haversine
https://issues.apache.org/jira/browse/PIG-3873
PIG-3866Create ThreadLocal classloader per PigContext
https://issues.apache.org/jira/browse/PIG-3866
PIG-3864ToDate(userstring, format, timezone) computes DateTime with strange 
handling of Daylight Saving Time with location based timezones
https://issues.apache.org/jira/browse/PIG-3864
PIG-3851Upgrade jline to 2.11
https://issues.apache.org/jira/browse/PIG-3851
PIG-3668COR built-in function when atleast one of the coefficient values is 
NaN
https://issues.apache.org/jira/browse/PIG-3668
PIG-3587add functionality for rolling over dates
https://issues.apache.org/jira/browse/PIG-3587

You may edit this subscription at:
https://issues.apache.org/jira/secure/FilterSubscription!default.jspa?subId=16328&filterId=12322384

[jira] [Updated] (PIG-4764) Make Pig work with Hive 2.0

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated PIG-4764:

Attachment: PIG-4764-3.patch

> Make Pig work with Hive 2.0
> ---
>
> Key: PIG-4764
> URL: https://issues.apache.org/jira/browse/PIG-4764
> Project: Pig
>  Issue Type: Improvement
>  Components: impl
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.16.0
>
> Attachments: PIG-4764-0.patch, PIG-4764-1.patch, PIG-4764-2.patch, 
> PIG-4764-3.patch
>
>
> There are a lot of changes especially around ORC in Hive 2.0. We need to make 
> Pig work with it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : Pig-trunk #1858

[Apache Jenkins Server]

See 

[jira] [Resolved] (PIG-4765) Enable TestPoissonSampleLoader in spark mode

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-4765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang resolved PIG-4765.
--
Resolution: Fixed

Committed to Spark branch. Thanks, Liyun!

> Enable TestPoissonSampleLoader in spark mode
> 
>
> Key: PIG-4765
> URL: https://issues.apache.org/jira/browse/PIG-4765
> Project: Pig
>  Issue Type: Sub-task
>  Components: spark
>Reporter: liyunzhang_intel
>Assignee: liyunzhang_intel
> Fix For: spark-branch
>
> Attachments: PIG-4765.patch
>
>
> in 
> https://builds.apache.org/job/Pig-spark/292/testReport/junit/org.apache.pig.test/,
>  it shows that TestPoissonSampleLoader fails.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PIG-282) Custom Partitioner

[Daniel Dai (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15068619#comment-15068619
 ] 

Daniel Dai commented on PIG-282:


Can you try to declare data type in your load statement?
A = load 'input_data' as (a0:int, a1:chararray..);

> Custom Partitioner
> --
>
> Key: PIG-282
> URL: https://issues.apache.org/jira/browse/PIG-282
> Project: Pig
>  Issue Type: New Feature
>Affects Versions: 0.7.0
>Reporter: Amir Youssefi
>Assignee: Aniket Mokashi
>Priority: Minor
> Fix For: 0.8.0
>
> Attachments: CustomPartitioner.patch, CustomPartitionerFinale.patch, 
> CustomPartitionerTest.patch
>
>
> By adding custom partitioner we can give control over which output partition 
> a key (/value) goes to. We can add keywords to language e.g. 
> PARTITION BY UDF(...)
> or a similar syntax. UDF returns a number between 0 and n-1 where n is number 
> of output partitions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PIG-4761) Add more information to front end error messages

[Daniel Dai (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15068608#comment-15068608
 ] 

Daniel Dai commented on PIG-4761:
-

Looks good. I need to run e2e tests to make sure it does not break existing 
test cases though.

> Add more information to front end error messages
> 
>
> Key: PIG-4761
> URL: https://issues.apache.org/jira/browse/PIG-4761
> Project: Pig
>  Issue Type: Improvement
>Reporter: Eyal Allweil
>Assignee: Eyal Allweil
> Attachments: PIG-4761.patch
>
>
> Some front end errors could include more information - for example, field 
> types when an incompatible join is attempted. Since these errors will fail 
> the job before it is launched, they won't affect the size of actual runtime 
> logs, but help in understanding why a job has failed.
> I'm attaching a patch with a few error description improvements



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PIG-4761) Add more information to front end error messages

[Eyal Allweil (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15068588#comment-15068588
 ] 

Eyal Allweil commented on PIG-4761:
---

Hey guys, can someone look at this? It's very small, but should be helpful.

> Add more information to front end error messages
> 
>
> Key: PIG-4761
> URL: https://issues.apache.org/jira/browse/PIG-4761
> Project: Pig
>  Issue Type: Improvement
>Reporter: Eyal Allweil
>Assignee: Eyal Allweil
> Attachments: PIG-4761.patch
>
>
> Some front end errors could include more information - for example, field 
> types when an incompatible join is attempted. Since these errors will fail 
> the job before it is launched, they won't affect the size of actual runtime 
> logs, but help in understanding why a job has failed.
> I'm attaching a patch with a few error description improvements



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (PIG-4760) TezDAGStats.convertToHadoopCounters is not used, but impose MR counter limit

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-4760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated PIG-4760:

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Patch committed to both trunk and 0.16 branch. Thanks Rohini for review!

> TezDAGStats.convertToHadoopCounters is not used, but impose MR counter limit
> 
>
> Key: PIG-4760
> URL: https://issues.apache.org/jira/browse/PIG-4760
> Project: Pig
>  Issue Type: Bug
>  Components: tez
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.16.0, 0.15.1
>
> Attachments: PIG-4760-1.patch, PIG-4760-2.patch, PIG-4760-3.patch, 
> PIG-4760-4.patch, PIG-4760-5.patch
>
>
> PIG-4529 reveal a MR counter limit issue imposed to Pig. The original fix 
> won't work in Oozie since Oozie launcher already imposed counter limit and 
> Pig cannot change it (there is an internal flag in Limits to make sure it 
> does not init twice).
> Actually the MR counter is completely redundant and should be removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (PIG-4760) TezDAGStats.convertToHadoopCounters is not used, but impose MR counter limit

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-4760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated PIG-4760:

Attachment: PIG-4760-5.patch

Attach the final patch to commit.

> TezDAGStats.convertToHadoopCounters is not used, but impose MR counter limit
> 
>
> Key: PIG-4760
> URL: https://issues.apache.org/jira/browse/PIG-4760
> Project: Pig
>  Issue Type: Bug
>  Components: tez
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.16.0, 0.15.1
>
> Attachments: PIG-4760-1.patch, PIG-4760-2.patch, PIG-4760-3.patch, 
> PIG-4760-4.patch, PIG-4760-5.patch
>
>
> PIG-4529 reveal a MR counter limit issue imposed to Pig. The original fix 
> won't work in Oozie since Oozie launcher already imposed counter limit and 
> Pig cannot change it (there is an internal flag in Limits to make sure it 
> does not init twice).
> Actually the MR counter is completely redundant and should be removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PIG-4764) Make Pig work with Hive 2.0

[Daniel Dai (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15068575#comment-15068575
 ] 

Daniel Dai commented on PIG-4764:
-

Here is RB link: https://reviews.apache.org/r/41655/

> Make Pig work with Hive 2.0
> ---
>
> Key: PIG-4764
> URL: https://issues.apache.org/jira/browse/PIG-4764
> Project: Pig
>  Issue Type: Improvement
>  Components: impl
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.16.0
>
> Attachments: PIG-4764-0.patch, PIG-4764-1.patch, PIG-4764-2.patch
>
>
> There are a lot of changes especially around ORC in Hive 2.0. We need to make 
> Pig work with it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PIG-4764) Make Pig work with Hive 2.0

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15068528#comment-15068528
 ] 

Rohini Palaniswamy commented on PIG-4764:
-

Can you put the patch in reviewboard?

> Make Pig work with Hive 2.0
> ---
>
> Key: PIG-4764
> URL: https://issues.apache.org/jira/browse/PIG-4764
> Project: Pig
>  Issue Type: Improvement
>  Components: impl
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.16.0
>
> Attachments: PIG-4764-0.patch, PIG-4764-1.patch, PIG-4764-2.patch
>
>
> There are a lot of changes especially around ORC in Hive 2.0. We need to make 
> Pig work with it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (PIG-4763) Insufficient check for the number of arguments in runpigmix.pl

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-4763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15068400#comment-15068400
 ] 

Rohini Palaniswamy edited comment on PIG-4763 at 12/22/15 5:05 PM:
---

bq. But if they are optional, wouldn't it be better to set default values just 
like "parallel" or other optional parameters to suppress warnings?
  Agree with that. Looks like it will cause warnings or errors if the 7th and 
8th one are not set. Can you change the patch to add default values based on 
the ones we usually specify in pigmix build.xml ?  We can change to say 8 
arguments are mandatory too, but it is easier for the users if they have to 
specify less arguments as most of the time defaults are good.


was (Author: rohini):
bq. But if they are optional, wouldn't it be better to set default values just 
like "parallel" or other optional parameters to suppress warnings?
  Agree with that. Looks like it will cause warnings or errors if the 7th and 
8th one are not set. Can you change the patch to add default values? 

> Insufficient check for the number of arguments in runpigmix.pl
> --
>
> Key: PIG-4763
> URL: https://issues.apache.org/jira/browse/PIG-4763
> Project: Pig
>  Issue Type: Bug
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Trivial
> Attachments: PIG-4763.1.patch
>
>
> runpigmix.pl first checks the number of arguments as follows:
> {code}
>   3 if(scalar(@ARGV) < 6 )
>   4 {
>   5 print STDERR "Usage: $0
>  
> [parallel] [numruns] [runmapreduce] [cleanup_after_test]\n";
>   6 exit(-1);
>   7 }
> {code}
> but the number of the required parameters is 8, so this check seems 
> insufficient.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (PIG-4763) Insufficient check for the number of arguments in runpigmix.pl

[Rohini Palaniswamy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-4763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohini Palaniswamy reopened PIG-4763:
-

bq. But if they are optional, wouldn't it be better to set default values just 
like "parallel" or other optional parameters to suppress warnings?
  Agree with that. Looks like it will cause warnings or errors if the 7th and 
8th one are not set. Can you change the patch to add default values? 

> Insufficient check for the number of arguments in runpigmix.pl
> --
>
> Key: PIG-4763
> URL: https://issues.apache.org/jira/browse/PIG-4763
> Project: Pig
>  Issue Type: Bug
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Trivial
> Attachments: PIG-4763.1.patch
>
>
> runpigmix.pl first checks the number of arguments as follows:
> {code}
>   3 if(scalar(@ARGV) < 6 )
>   4 {
>   5 print STDERR "Usage: $0
>  
> [parallel] [numruns] [runmapreduce] [cleanup_after_test]\n";
>   6 exit(-1);
>   7 }
> {code}
> but the number of the required parameters is 8, so this check seems 
> insufficient.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)