[jira] Subscription: PIG patch available

2014-08-26 Thread jira 
Issue Subscription
Filter: PIG patch available (15 issues)

Subscriber: pigdaily

Key Summary
PIG-4111Make Pig compiles with avro-1.7.7
https://issues.apache.org/jira/browse/PIG-4111
PIG-4103Fix TestRegisteredJarVisibility(after PIG-4083)
https://issues.apache.org/jira/browse/PIG-4103
PIG-4066An optimization for ROLLUP operation in Pig
https://issues.apache.org/jira/browse/PIG-4066
PIG-4004Upgrade the Pigmix queries from the (old) mapred API to mapreduce
https://issues.apache.org/jira/browse/PIG-4004
PIG-4002Disable combiner when map-side aggregation is used
https://issues.apache.org/jira/browse/PIG-4002
PIG-3952PigStorage accepts '-tagSplit' to return full split information
https://issues.apache.org/jira/browse/PIG-3952
PIG-3911Define unique fields with @OutputSchema
https://issues.apache.org/jira/browse/PIG-3911
PIG-3877Getting Geo Latitude/Longitude from Address Lines
https://issues.apache.org/jira/browse/PIG-3877
PIG-3873Geo distance calculation using Haversine
https://issues.apache.org/jira/browse/PIG-3873
PIG-3866Create ThreadLocal classloader per PigContext
https://issues.apache.org/jira/browse/PIG-3866
PIG-3861duplicate jars get added to distributed cache
https://issues.apache.org/jira/browse/PIG-3861
PIG-3668COR built-in function when atleast one of the coefficient values is 
NaN
https://issues.apache.org/jira/browse/PIG-3668
PIG-3635Fix e2e tests for Hadoop 2.X on Windows
https://issues.apache.org/jira/browse/PIG-3635
PIG-3587add functionality for rolling over dates
https://issues.apache.org/jira/browse/PIG-3587
PIG-3441Allow Pig to use default resources from Configuration objects
https://issues.apache.org/jira/browse/PIG-3441

You may edit this subscription at:
https://issues.apache.org/jira/secure/FilterSubscription!default.jspa?subId=13225&filterId=12322384

[jira] [Resolved] (PIG-4128) New logical optimizer rule: ConstantCalculator

2014-08-26 Thread Daniel Dai (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-4128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai resolved PIG-4128.
-

  Resolution: Fixed
Hadoop Flags: Reviewed

Patch committed to trunk. Thanks Thejas for review!

> New logical optimizer rule: ConstantCalculator
> --
>
> Key: PIG-4128
> URL: https://issues.apache.org/jira/browse/PIG-4128
> Project: Pig
>  Issue Type: New Feature
>  Components: impl
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.14.0
>
> Attachments: PIG-4128-1.patch, PIG-4128-2.patch, PIG-4128-3.patch
>
>
> Pig used to have a LogicExpressionSimplifier to simplify expression which 
> also calculates constant expression. The optimizer rule is buggy and we 
> disable it by default in PIG-2316.
> However, we do need this feature especially in partition/predicate push down, 
> since both does not deal with complex constant expression, we'd like to 
> replace the expression with constant before the actual push down. Yes, user 
> may manually do the calculation and rewrite the query, but even rewrite is 
> sometimes not possible. Consider the case user want to push a datetime 
> predicate, user have to write a ToDate udf since Pig does not have datetime 
> constant.
> In this Jira, I provide a new rule: ConstantCalculator, which is much simpler 
> and much less error prone, to replace LogicExpressionSimplifier.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (PIG-4128) New logical optimizer rule: ConstantCalculator

2014-08-26 Thread Thejas M Nair (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-4128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14111505#comment-14111505
 ] 

Thejas M Nair commented on PIG-4128:


+1

> New logical optimizer rule: ConstantCalculator
> --
>
> Key: PIG-4128
> URL: https://issues.apache.org/jira/browse/PIG-4128
> Project: Pig
>  Issue Type: New Feature
>  Components: impl
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.14.0
>
> Attachments: PIG-4128-1.patch, PIG-4128-2.patch, PIG-4128-3.patch
>
>
> Pig used to have a LogicExpressionSimplifier to simplify expression which 
> also calculates constant expression. The optimizer rule is buggy and we 
> disable it by default in PIG-2316.
> However, we do need this feature especially in partition/predicate push down, 
> since both does not deal with complex constant expression, we'd like to 
> replace the expression with constant before the actual push down. Yes, user 
> may manually do the calculation and rewrite the query, but even rewrite is 
> sometimes not possible. Consider the case user want to push a datetime 
> predicate, user have to write a ToDate udf since Pig does not have datetime 
> constant.
> In this Jira, I provide a new rule: ConstantCalculator, which is much simpler 
> and much less error prone, to replace LogicExpressionSimplifier.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 24789: New logical optimizer rule: ConstantCalculator

2014-08-26 Thread Thejas Nair 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24789/#review51603
---

Ship it!


Ship It!

- Thejas Nair


On Aug. 26, 2014, 10:35 p.m., Daniel Dai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24789/
> ---
> 
> (Updated Aug. 26, 2014, 10:35 p.m.)
> 
> 
> Review request for pig.
> 
> 
> Repository: pig
> 
> 
> Description
> ---
> 
> See PIG-4128
> 
> 
> Diffs
> -
> 
>   trunk/src/org/apache/pig/EvalFunc.java 1618727 
>   trunk/src/org/apache/pig/Main.java 1618727 
>   trunk/src/org/apache/pig/builtin/ABS.java 1618727 
>   trunk/src/org/apache/pig/builtin/ARITY.java 1618727 
>   trunk/src/org/apache/pig/builtin/AddDuration.java 1618727 
>   trunk/src/org/apache/pig/builtin/Assert.java 1618727 
>   trunk/src/org/apache/pig/builtin/BagSize.java 1618727 
>   trunk/src/org/apache/pig/builtin/BagToString.java 1618727 
>   trunk/src/org/apache/pig/builtin/BagToTuple.java 1618727 
>   trunk/src/org/apache/pig/builtin/Base.java 1618727 
>   trunk/src/org/apache/pig/builtin/BigDecimalAbs.java 1618727 
>   trunk/src/org/apache/pig/builtin/BigIntegerAbs.java 1618727 
>   trunk/src/org/apache/pig/builtin/CONCAT.java 1618727 
>   trunk/src/org/apache/pig/builtin/ConstantSize.java 1618727 
>   trunk/src/org/apache/pig/builtin/CubeDimensions.java 1618727 
>   trunk/src/org/apache/pig/builtin/CurrentTime.java 1618727 
>   trunk/src/org/apache/pig/builtin/DIFF.java 1618727 
>   trunk/src/org/apache/pig/builtin/DaysBetween.java 1618727 
>   trunk/src/org/apache/pig/builtin/DoubleRound.java 1618727 
>   trunk/src/org/apache/pig/builtin/DoubleRoundTo.java 1618727 
>   trunk/src/org/apache/pig/builtin/ENDSWITH.java 1618727 
>   trunk/src/org/apache/pig/builtin/EqualsIgnoreCase.java 1618727 
>   trunk/src/org/apache/pig/builtin/FloatAbs.java 1618727 
>   trunk/src/org/apache/pig/builtin/FloatRound.java 1618727 
>   trunk/src/org/apache/pig/builtin/FloatRoundTo.java 1618727 
>   trunk/src/org/apache/pig/builtin/GetDay.java 1618727 
>   trunk/src/org/apache/pig/builtin/GetHour.java 1618727 
>   trunk/src/org/apache/pig/builtin/GetMilliSecond.java 1618727 
>   trunk/src/org/apache/pig/builtin/GetMinute.java 1618727 
>   trunk/src/org/apache/pig/builtin/GetMonth.java 1618727 
>   trunk/src/org/apache/pig/builtin/GetSecond.java 1618727 
>   trunk/src/org/apache/pig/builtin/GetWeek.java 1618727 
>   trunk/src/org/apache/pig/builtin/GetWeekYear.java 1618727 
>   trunk/src/org/apache/pig/builtin/GetYear.java 1618727 
>   trunk/src/org/apache/pig/builtin/HoursBetween.java 1618727 
>   trunk/src/org/apache/pig/builtin/INDEXOF.java 1618727 
>   trunk/src/org/apache/pig/builtin/INVERSEMAP.java 1618727 
>   trunk/src/org/apache/pig/builtin/IntAbs.java 1618727 
>   trunk/src/org/apache/pig/builtin/IsEmpty.java 1618727 
>   trunk/src/org/apache/pig/builtin/KEYSET.java 1618727 
>   trunk/src/org/apache/pig/builtin/LAST_INDEX_OF.java 1618727 
>   trunk/src/org/apache/pig/builtin/LCFIRST.java 1618727 
>   trunk/src/org/apache/pig/builtin/LOWER.java 1618727 
>   trunk/src/org/apache/pig/builtin/LTRIM.java 1618727 
>   trunk/src/org/apache/pig/builtin/LongAbs.java 1618727 
>   trunk/src/org/apache/pig/builtin/MapSize.java 1618727 
>   trunk/src/org/apache/pig/builtin/MilliSecondsBetween.java 1618727 
>   trunk/src/org/apache/pig/builtin/MinutesBetween.java 1618727 
>   trunk/src/org/apache/pig/builtin/MonthsBetween.java 1618727 
>   trunk/src/org/apache/pig/builtin/PluckTuple.java 1618727 
>   trunk/src/org/apache/pig/builtin/REGEX_EXTRACT.java 1618727 
>   trunk/src/org/apache/pig/builtin/REGEX_EXTRACT_ALL.java 1618727 
>   trunk/src/org/apache/pig/builtin/REPLACE.java 1618727 
>   trunk/src/org/apache/pig/builtin/ROUND.java 1618727 
>   trunk/src/org/apache/pig/builtin/ROUND_TO.java 1618727 
>   trunk/src/org/apache/pig/builtin/RTRIM.java 1618727 
>   trunk/src/org/apache/pig/builtin/RollupDimensions.java 1618727 
>   trunk/src/org/apache/pig/builtin/SIZE.java 1618727 
>   trunk/src/org/apache/pig/builtin/SPRINTF.java 1618727 
>   trunk/src/org/apache/pig/builtin/STARTSWITH.java 1618727 
>   trunk/src/org/apache/pig/builtin/STRSPLIT.java 1618727 
>   trunk/src/org/apache/pig/builtin/SUBSTRING.java 1618727 
>   trunk/src/org/apache/pig/builtin/SUBTRACT.java 1618727 
>   trunk/src/org/apache/pig/builtin/SecondsBetween.java 1618727 
>   trunk/src/org/apache/pig/builtin/StringConcat.java 1618727 
>   trunk/src/org/apache/pig/builtin/StringSize.java 1618727 
>   trunk/src/org/apache/pig/builtin/SubtractDuration.java 1618727 
>   trunk/src/org/apache/pig/builtin/TOBAG.java 1618727 
>   trunk/src/org/apache/pig/builtin/TOKENIZE.java 1618727 
>   trunk/src/org/apache/pig/builtin/TOMAP.java 1618727 
>   trunk/src/org/a

[jira] [Updated] (PIG-4128) New logical optimizer rule: ConstantCalculator

2014-08-26 Thread Daniel Dai (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-4128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated PIG-4128:


Attachment: PIG-4128-3.patch

Addressing Thejas' review comments.

> New logical optimizer rule: ConstantCalculator
> --
>
> Key: PIG-4128
> URL: https://issues.apache.org/jira/browse/PIG-4128
> Project: Pig
>  Issue Type: New Feature
>  Components: impl
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.14.0
>
> Attachments: PIG-4128-1.patch, PIG-4128-2.patch, PIG-4128-3.patch
>
>
> Pig used to have a LogicExpressionSimplifier to simplify expression which 
> also calculates constant expression. The optimizer rule is buggy and we 
> disable it by default in PIG-2316.
> However, we do need this feature especially in partition/predicate push down, 
> since both does not deal with complex constant expression, we'd like to 
> replace the expression with constant before the actual push down. Yes, user 
> may manually do the calculation and rewrite the query, but even rewrite is 
> sometimes not possible. Consider the case user want to push a datetime 
> predicate, user have to write a ToDate udf since Pig does not have datetime 
> constant.
> In this Jira, I provide a new rule: ConstantCalculator, which is much simpler 
> and much less error prone, to replace LogicExpressionSimplifier.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 24789: New logical optimizer rule: ConstantCalculator

2014-08-26 Thread Daniel Dai 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24789/
---

(Updated Aug. 26, 2014, 10:35 p.m.)


Review request for pig.


Repository: pig


Description
---

See PIG-4128


Diffs (updated)
-

  trunk/src/org/apache/pig/EvalFunc.java 1618727 
  trunk/src/org/apache/pig/Main.java 1618727 
  trunk/src/org/apache/pig/builtin/ABS.java 1618727 
  trunk/src/org/apache/pig/builtin/ARITY.java 1618727 
  trunk/src/org/apache/pig/builtin/AddDuration.java 1618727 
  trunk/src/org/apache/pig/builtin/Assert.java 1618727 
  trunk/src/org/apache/pig/builtin/BagSize.java 1618727 
  trunk/src/org/apache/pig/builtin/BagToString.java 1618727 
  trunk/src/org/apache/pig/builtin/BagToTuple.java 1618727 
  trunk/src/org/apache/pig/builtin/Base.java 1618727 
  trunk/src/org/apache/pig/builtin/BigDecimalAbs.java 1618727 
  trunk/src/org/apache/pig/builtin/BigIntegerAbs.java 1618727 
  trunk/src/org/apache/pig/builtin/CONCAT.java 1618727 
  trunk/src/org/apache/pig/builtin/ConstantSize.java 1618727 
  trunk/src/org/apache/pig/builtin/CubeDimensions.java 1618727 
  trunk/src/org/apache/pig/builtin/CurrentTime.java 1618727 
  trunk/src/org/apache/pig/builtin/DIFF.java 1618727 
  trunk/src/org/apache/pig/builtin/DaysBetween.java 1618727 
  trunk/src/org/apache/pig/builtin/DoubleRound.java 1618727 
  trunk/src/org/apache/pig/builtin/DoubleRoundTo.java 1618727 
  trunk/src/org/apache/pig/builtin/ENDSWITH.java 1618727 
  trunk/src/org/apache/pig/builtin/EqualsIgnoreCase.java 1618727 
  trunk/src/org/apache/pig/builtin/FloatAbs.java 1618727 
  trunk/src/org/apache/pig/builtin/FloatRound.java 1618727 
  trunk/src/org/apache/pig/builtin/FloatRoundTo.java 1618727 
  trunk/src/org/apache/pig/builtin/GetDay.java 1618727 
  trunk/src/org/apache/pig/builtin/GetHour.java 1618727 
  trunk/src/org/apache/pig/builtin/GetMilliSecond.java 1618727 
  trunk/src/org/apache/pig/builtin/GetMinute.java 1618727 
  trunk/src/org/apache/pig/builtin/GetMonth.java 1618727 
  trunk/src/org/apache/pig/builtin/GetSecond.java 1618727 
  trunk/src/org/apache/pig/builtin/GetWeek.java 1618727 
  trunk/src/org/apache/pig/builtin/GetWeekYear.java 1618727 
  trunk/src/org/apache/pig/builtin/GetYear.java 1618727 
  trunk/src/org/apache/pig/builtin/HoursBetween.java 1618727 
  trunk/src/org/apache/pig/builtin/INDEXOF.java 1618727 
  trunk/src/org/apache/pig/builtin/INVERSEMAP.java 1618727 
  trunk/src/org/apache/pig/builtin/IntAbs.java 1618727 
  trunk/src/org/apache/pig/builtin/IsEmpty.java 1618727 
  trunk/src/org/apache/pig/builtin/KEYSET.java 1618727 
  trunk/src/org/apache/pig/builtin/LAST_INDEX_OF.java 1618727 
  trunk/src/org/apache/pig/builtin/LCFIRST.java 1618727 
  trunk/src/org/apache/pig/builtin/LOWER.java 1618727 
  trunk/src/org/apache/pig/builtin/LTRIM.java 1618727 
  trunk/src/org/apache/pig/builtin/LongAbs.java 1618727 
  trunk/src/org/apache/pig/builtin/MapSize.java 1618727 
  trunk/src/org/apache/pig/builtin/MilliSecondsBetween.java 1618727 
  trunk/src/org/apache/pig/builtin/MinutesBetween.java 1618727 
  trunk/src/org/apache/pig/builtin/MonthsBetween.java 1618727 
  trunk/src/org/apache/pig/builtin/PluckTuple.java 1618727 
  trunk/src/org/apache/pig/builtin/REGEX_EXTRACT.java 1618727 
  trunk/src/org/apache/pig/builtin/REGEX_EXTRACT_ALL.java 1618727 
  trunk/src/org/apache/pig/builtin/REPLACE.java 1618727 
  trunk/src/org/apache/pig/builtin/ROUND.java 1618727 
  trunk/src/org/apache/pig/builtin/ROUND_TO.java 1618727 
  trunk/src/org/apache/pig/builtin/RTRIM.java 1618727 
  trunk/src/org/apache/pig/builtin/RollupDimensions.java 1618727 
  trunk/src/org/apache/pig/builtin/SIZE.java 1618727 
  trunk/src/org/apache/pig/builtin/SPRINTF.java 1618727 
  trunk/src/org/apache/pig/builtin/STARTSWITH.java 1618727 
  trunk/src/org/apache/pig/builtin/STRSPLIT.java 1618727 
  trunk/src/org/apache/pig/builtin/SUBSTRING.java 1618727 
  trunk/src/org/apache/pig/builtin/SUBTRACT.java 1618727 
  trunk/src/org/apache/pig/builtin/SecondsBetween.java 1618727 
  trunk/src/org/apache/pig/builtin/StringConcat.java 1618727 
  trunk/src/org/apache/pig/builtin/StringSize.java 1618727 
  trunk/src/org/apache/pig/builtin/SubtractDuration.java 1618727 
  trunk/src/org/apache/pig/builtin/TOBAG.java 1618727 
  trunk/src/org/apache/pig/builtin/TOKENIZE.java 1618727 
  trunk/src/org/apache/pig/builtin/TOMAP.java 1618727 
  trunk/src/org/apache/pig/builtin/TOTUPLE.java 1618727 
  trunk/src/org/apache/pig/builtin/TRIM.java 1618727 
  trunk/src/org/apache/pig/builtin/ToDate.java 1618727 
  trunk/src/org/apache/pig/builtin/ToDate2ARGS.java 1618727 
  trunk/src/org/apache/pig/builtin/ToDate3ARGS.java 1618727 
  trunk/src/org/apache/pig/builtin/ToDateISO.java 1618727 
  trunk/src/org/apache/pig/builtin/ToMilliSeconds.java 1618727 
  trunk/src/org/apache/pig/builtin/ToString.java 1618727 
  trunk/src/org/apache/pig/builtin/ToUnixT

Re: Review Request 24789: New logical optimizer rule: ConstantCalculator

2014-08-26 Thread Daniel Dai 


> On Aug. 25, 2014, 8:24 p.m., Thejas Nair wrote:
> > trunk/src/org/apache/pig/newplan/logical/rules/ConstantCalculator.java, 
> > line 53
> > 
> >
> > There is no processedOperators.add happening. Is this variable needed ?

Yes, we should add processed operators.


- Daniel


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24789/#review51430
---


On Aug. 19, 2014, 5:41 p.m., Daniel Dai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24789/
> ---
> 
> (Updated Aug. 19, 2014, 5:41 p.m.)
> 
> 
> Review request for pig.
> 
> 
> Repository: pig
> 
> 
> Description
> ---
> 
> See PIG-4128
> 
> 
> Diffs
> -
> 
>   trunk/src/org/apache/pig/EvalFunc.java 1618727 
>   trunk/src/org/apache/pig/Main.java 1618727 
>   trunk/src/org/apache/pig/builtin/ABS.java 1618727 
>   trunk/src/org/apache/pig/builtin/ARITY.java 1618727 
>   trunk/src/org/apache/pig/builtin/AddDuration.java 1618727 
>   trunk/src/org/apache/pig/builtin/Assert.java 1618727 
>   trunk/src/org/apache/pig/builtin/BagSize.java 1618727 
>   trunk/src/org/apache/pig/builtin/BagToString.java 1618727 
>   trunk/src/org/apache/pig/builtin/BagToTuple.java 1618727 
>   trunk/src/org/apache/pig/builtin/Base.java 1618727 
>   trunk/src/org/apache/pig/builtin/BigDecimalAbs.java 1618727 
>   trunk/src/org/apache/pig/builtin/BigIntegerAbs.java 1618727 
>   trunk/src/org/apache/pig/builtin/CONCAT.java 1618727 
>   trunk/src/org/apache/pig/builtin/ConstantSize.java 1618727 
>   trunk/src/org/apache/pig/builtin/CubeDimensions.java 1618727 
>   trunk/src/org/apache/pig/builtin/CurrentTime.java 1618727 
>   trunk/src/org/apache/pig/builtin/DIFF.java 1618727 
>   trunk/src/org/apache/pig/builtin/DaysBetween.java 1618727 
>   trunk/src/org/apache/pig/builtin/DoubleRound.java 1618727 
>   trunk/src/org/apache/pig/builtin/DoubleRoundTo.java 1618727 
>   trunk/src/org/apache/pig/builtin/ENDSWITH.java 1618727 
>   trunk/src/org/apache/pig/builtin/EqualsIgnoreCase.java 1618727 
>   trunk/src/org/apache/pig/builtin/FloatAbs.java 1618727 
>   trunk/src/org/apache/pig/builtin/FloatRound.java 1618727 
>   trunk/src/org/apache/pig/builtin/FloatRoundTo.java 1618727 
>   trunk/src/org/apache/pig/builtin/GetDay.java 1618727 
>   trunk/src/org/apache/pig/builtin/GetHour.java 1618727 
>   trunk/src/org/apache/pig/builtin/GetMilliSecond.java 1618727 
>   trunk/src/org/apache/pig/builtin/GetMinute.java 1618727 
>   trunk/src/org/apache/pig/builtin/GetMonth.java 1618727 
>   trunk/src/org/apache/pig/builtin/GetSecond.java 1618727 
>   trunk/src/org/apache/pig/builtin/GetWeek.java 1618727 
>   trunk/src/org/apache/pig/builtin/GetWeekYear.java 1618727 
>   trunk/src/org/apache/pig/builtin/GetYear.java 1618727 
>   trunk/src/org/apache/pig/builtin/HoursBetween.java 1618727 
>   trunk/src/org/apache/pig/builtin/INDEXOF.java 1618727 
>   trunk/src/org/apache/pig/builtin/INVERSEMAP.java 1618727 
>   trunk/src/org/apache/pig/builtin/IntAbs.java 1618727 
>   trunk/src/org/apache/pig/builtin/IsEmpty.java 1618727 
>   trunk/src/org/apache/pig/builtin/KEYSET.java 1618727 
>   trunk/src/org/apache/pig/builtin/LAST_INDEX_OF.java 1618727 
>   trunk/src/org/apache/pig/builtin/LCFIRST.java 1618727 
>   trunk/src/org/apache/pig/builtin/LOWER.java 1618727 
>   trunk/src/org/apache/pig/builtin/LTRIM.java 1618727 
>   trunk/src/org/apache/pig/builtin/LongAbs.java 1618727 
>   trunk/src/org/apache/pig/builtin/MapSize.java 1618727 
>   trunk/src/org/apache/pig/builtin/MilliSecondsBetween.java 1618727 
>   trunk/src/org/apache/pig/builtin/MinutesBetween.java 1618727 
>   trunk/src/org/apache/pig/builtin/MonthsBetween.java 1618727 
>   trunk/src/org/apache/pig/builtin/PluckTuple.java 1618727 
>   trunk/src/org/apache/pig/builtin/REGEX_EXTRACT.java 1618727 
>   trunk/src/org/apache/pig/builtin/REGEX_EXTRACT_ALL.java 1618727 
>   trunk/src/org/apache/pig/builtin/REPLACE.java 1618727 
>   trunk/src/org/apache/pig/builtin/ROUND.java 1618727 
>   trunk/src/org/apache/pig/builtin/ROUND_TO.java 1618727 
>   trunk/src/org/apache/pig/builtin/RTRIM.java 1618727 
>   trunk/src/org/apache/pig/builtin/RollupDimensions.java 1618727 
>   trunk/src/org/apache/pig/builtin/SIZE.java 1618727 
>   trunk/src/org/apache/pig/builtin/SPRINTF.java 1618727 
>   trunk/src/org/apache/pig/builtin/STARTSWITH.java 1618727 
>   trunk/src/org/apache/pig/builtin/STRSPLIT.java 1618727 
>   trunk/src/org/apache/pig/builtin/SUBSTRING.java 1618727 
>   trunk/src/org/apache/pig/builtin/SUBTRACT.java 1618727 
>   trunk/src/org/apache/pig/builtin/SecondsBetween.java 1618727 
>   trunk/src/org/apache/pig/builtin/StringConcat.java 1618727 
>   t

[jira] [Created] (PIG-4141) Ship UDF/LoadFunc/StoreFunc dependent jar automatically

2014-08-26 Thread Daniel Dai (JIRA) 
Daniel Dai created PIG-4141:
---

 Summary: Ship UDF/LoadFunc/StoreFunc dependent jar automatically
 Key: PIG-4141
 URL: https://issues.apache.org/jira/browse/PIG-4141
 Project: Pig
  Issue Type: New Feature
  Components: impl
Reporter: Daniel Dai
Assignee: Daniel Dai
 Fix For: 0.14.0
 Attachments: PIG-4141-1.patch

When user use AvroStorage/JsonStorage/OrcStorage, they need to register 
dependent jars manually. It would be much convenient if we can provide a 
mechanism for UDF/LoadFunc/StoreFunc to claim the dependency and ship jars 
automatically.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (PIG-4141) Ship UDF/LoadFunc/StoreFunc dependent jar automatically

2014-08-26 Thread Daniel Dai (JIRA) 

 [ 
https://issues.apache.org/jira/browse/PIG-4141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated PIG-4141:


Attachment: PIG-4141-1.patch

> Ship UDF/LoadFunc/StoreFunc dependent jar automatically
> ---
>
> Key: PIG-4141
> URL: https://issues.apache.org/jira/browse/PIG-4141
> Project: Pig
>  Issue Type: New Feature
>  Components: impl
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.14.0
>
> Attachments: PIG-4141-1.patch
>
>
> When user use AvroStorage/JsonStorage/OrcStorage, they need to register 
> dependent jars manually. It would be much convenient if we can provide a 
> mechanism for UDF/LoadFunc/StoreFunc to claim the dependency and ship jars 
> automatically.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (PIG-3087) Refactor TestLogicalPlanBuilder to be meaningful

2014-08-26 Thread Jonathan Coveney (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-3087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14111401#comment-14111401
 ] 

Jonathan Coveney commented on PIG-3087:
---

Please do!


> Refactor TestLogicalPlanBuilder to be meaningful
> 
>
> Key: PIG-3087
> URL: https://issues.apache.org/jira/browse/PIG-3087
> Project: Pig
>  Issue Type: Bug
>Reporter: Jonathan Coveney
>  Labels: newbie
> Fix For: 0.14.0
>
> Attachments: PIG-3087-0.patch
>
>
> I started doing this as part of another patch, but there are some bigger 
> issues, and I don't have the time to dig in atm.
> That said, a lot of the tests as written don't test anything. I used more 
> modern junit patterns, and discovered we had a lot of tests that weren't 
> functioning properly. Making them function properly unveiled that the general 
> buildLp pattern doesn't work properly anymore for many cases where it would 
> throw an error in grunt, but for whatever reason no error is thrown in the 
> tests.
> Any test with _1 is a test that previous failed, that now doesn't. Some, 
> however, don't make sense so I think what really needs to be done is figure 
> out which should be failing, which shouldn't, and then fix buildLp 
> accordingly.
> I will attach my pass at it, but it is incomplete and needs work.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (PIG-3087) Refactor TestLogicalPlanBuilder to be meaningful

2014-08-26 Thread Hao Gao (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-3087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14111354#comment-14111354
 ] 

Hao Gao commented on PIG-3087:
--

Can I try this?

> Refactor TestLogicalPlanBuilder to be meaningful
> 
>
> Key: PIG-3087
> URL: https://issues.apache.org/jira/browse/PIG-3087
> Project: Pig
>  Issue Type: Bug
>Reporter: Jonathan Coveney
>  Labels: newbie
> Fix For: 0.14.0
>
> Attachments: PIG-3087-0.patch
>
>
> I started doing this as part of another patch, but there are some bigger 
> issues, and I don't have the time to dig in atm.
> That said, a lot of the tests as written don't test anything. I used more 
> modern junit patterns, and discovered we had a lot of tests that weren't 
> functioning properly. Making them function properly unveiled that the general 
> buildLp pattern doesn't work properly anymore for many cases where it would 
> throw an error in grunt, but for whatever reason no error is thrown in the 
> tests.
> Any test with _1 is a test that previous failed, that now doesn't. Some, 
> however, don't make sense so I think what really needs to be done is figure 
> out which should be failing, which shouldn't, and then fix buildLp 
> accordingly.
> I will attach my pass at it, but it is incomplete and needs work.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: [DISCUSS] Re: Dropping support for JDK6 in Apache Hadoop

2014-08-26 Thread Alan Gates 
I'm +1 on both of these.  But as a side note Hive actually still 
supports Hadoop 0.20, so you're statement below isn't quite true.


Alan.


Rohini Palaniswamy 
August 26, 2014 at 9:36
Pig has support for jdk7 from Pig 0.10. I think we should drop support for
JDK6 from Pig 0.14 and also publish maven binaries with jdk 1.7 from Pig
0.14.

Also it is high time to drop support for Hadoop 0.20. None of the other
hadoop projects officially support Hadoop 0.20 anymore. I would like 
to get

rid of the reflection in code w.r.t to UGI, be able to add support for
fetching Credentials in UDFs, Load and StoreFunc, etc.

If there are no major objections, will start two separate voting threads
for that.

Regards,
Rohini





--
Sent with Postbox 

--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[DISCUSS] Re: Dropping support for JDK6 in Apache Hadoop

2014-08-26 Thread Rohini Palaniswamy 
Pig has support for jdk7 from Pig 0.10. I think we should drop support for
JDK6 from Pig 0.14 and also publish maven binaries with jdk 1.7 from Pig
0.14.

Also it is high time to drop support for Hadoop 0.20. None of the other
hadoop projects officially support Hadoop 0.20 anymore. I would like to get
rid of the reflection in code w.r.t to UGI, be able to add support for
fetching Credentials in UDFs, Load and StoreFunc, etc.

If there are no major objections, will start two separate voting threads
for that.

Regards,
Rohini


On Tue, Aug 19, 2014 at 10:52 AM, Arun C Murthy  wrote:

> [Apologies for the wide distribution.]
>
> Dear HBase/Hive/Pig/Oozie communities,
>
>  We, over at Hadoop are considering dropping support for JDK6 this year.
>
>  As you maybe aware we just released hadoop-2.5.0 and are now considering
> making the next release i.e. hadoop-2.6.0 the *last* release of Apache
> Hadoop which supports JDK6. This means, from hadoop-2.7.0 onwards we will
> not support JDK6 anymore and we *may* start relying on JDK7-specific apis.
>
>  Now, the above releases a proposal and we do not want to pull the trigger
> without talking to projects downstream - hence the request for you feedback.
>
>  Please feel free to forward this to other communities you might deem to
> be at risk from this too.
>
> thanks,
> Arun
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: Creating a branch for Pig on Spark (PIG-4059)

2014-08-26 Thread Cheolsoo Park 
Ok. I will start a vote today and get back to you in few days.


On Tue, Aug 26, 2014 at 1:28 AM, Jarek Jarcec Cecho 
wrote:

> I would second Mayur - as he and Praveen are most familiar with all the
> changes, I would let them be the only additional branch committers. I’m
> expecting that they will review other contributor's patches similarly as
> normal committers and hence the contributors will start building Karma to
> get the “branch committership” later as well (if the branch will be around
> for that long).
>
> Jarcec
>
> On Aug 26, 2014, at 9:04 AM, Mayur Rustagi 
> wrote:
>
> > Praveen & me should be good enough for us. We will have 1-2 more folks
> > contributing from Sigmoid but we can manage by merging patches from them.
> >
> >Initially it will be good to structure patches through us as we
> > would like to avoid breaking architecture & flow.
> >
> > Regards
> > Mayur
> >
> >
> > On Tue, Aug 26, 2014 at 8:27 AM, Cheolsoo Park 
> wrote:
> >
> >> Hi guys,
> >>
> >> I asked about branch committership to the infra mailing list, and here
> is
> >> the reply-
> >>
> >> Many projects have what they consider 'partial committers' that is
> >> folks who have access to specific parts of a projects svn tree. Some
> >> projects do this for GSoC participants, others as a mechanism for
> >> moving to 'full committership' within the project.
> >>
> >> Do note though that in the eyes of the ASF someone with an ICLA and an
> >> account with any permissions to commit code anywhere in the public svn
> >> tree is a committer. IOW, you would vote, have ICLAs filed, and
> >> request account creation as per normal, and then merely adjust the
> >> karma in asf-authorization-template (and or LDAP)
> >>
> >>
> >> Looks like we need to vote and follow the normal process just like any
> >> other new committer.
> >>
> >> @Praveen, Jacec,
> >> I think Mayur and Praveen from Sigmoid Analytics need branch
> committership.
> >> Will anyone else work on Pig-on-Spark? Please reply.
> >>
> >> Once I have a full list of people, I will open a vote for Pig PMCs.
> >>
> >> Thanks,
> >> Cheolsoo
> >>
> >>
> >> On Mon, Aug 25, 2014 at 11:51 AM, Cheolsoo Park 
> >> wrote:
> >>
> >>> Additionally, I will give "branch-specific" commit permission to people
> >>> who will work on Pig on Spark (assuming it is possible).
> >>>
> >>> Please let me know if you have any objection on this too.
> >>>
> >>>
> >>> On Mon, Aug 25, 2014 at 10:25 AM, Jarek Jarcec Cecho <
> jar...@apache.org>
> >>> wrote:
> >>>
>  No objections from my side, thank you for creating the branch Cheolsoo
>  and kudos to the Sigmoid Analytics team for the great work!
> 
>  Jarcec
> 
>  On Aug 25, 2014, at 7:14 PM, Cheolsoo Park 
> >> wrote:
> 
> > Hi devs,
> >
> > Sigmoid Analytics has been working on Pig-on-Spark (PIG-4059), and
> >> they
>  want to merge their work into Apache.
> >
> > I am going to create a "Spark" branch for them. Please let me know if
>  you have any concerns.
> >
> > Thanks,
> > Cheolsoo
> 
> 
> >>>
> >>
> >
> >
> >
> > --
> > Regards,
> > Mayur Rustagi
> > Ph: +1 (760) 203 3257
> > http://www.sigmoidanalytics.com
> > @mayur_rustagi 
>
>

Re: Creating a branch for Pig on Spark (PIG-4059)

2014-08-26 Thread Jarek Jarcec Cecho 
I would second Mayur - as he and Praveen are most familiar with all the 
changes, I would let them be the only additional branch committers. I’m 
expecting that they will review other contributor's patches similarly as normal 
committers and hence the contributors will start building Karma to get the 
“branch committership” later as well (if the branch will be around for that 
long).

Jarcec

On Aug 26, 2014, at 9:04 AM, Mayur Rustagi  wrote:

> Praveen & me should be good enough for us. We will have 1-2 more folks
> contributing from Sigmoid but we can manage by merging patches from them.
> 
>Initially it will be good to structure patches through us as we
> would like to avoid breaking architecture & flow.
> 
> Regards
> Mayur
> 
> 
> On Tue, Aug 26, 2014 at 8:27 AM, Cheolsoo Park  wrote:
> 
>> Hi guys,
>> 
>> I asked about branch committership to the infra mailing list, and here is
>> the reply-
>> 
>> Many projects have what they consider 'partial committers' that is
>> folks who have access to specific parts of a projects svn tree. Some
>> projects do this for GSoC participants, others as a mechanism for
>> moving to 'full committership' within the project.
>> 
>> Do note though that in the eyes of the ASF someone with an ICLA and an
>> account with any permissions to commit code anywhere in the public svn
>> tree is a committer. IOW, you would vote, have ICLAs filed, and
>> request account creation as per normal, and then merely adjust the
>> karma in asf-authorization-template (and or LDAP)
>> 
>> 
>> Looks like we need to vote and follow the normal process just like any
>> other new committer.
>> 
>> @Praveen, Jacec,
>> I think Mayur and Praveen from Sigmoid Analytics need branch committership.
>> Will anyone else work on Pig-on-Spark? Please reply.
>> 
>> Once I have a full list of people, I will open a vote for Pig PMCs.
>> 
>> Thanks,
>> Cheolsoo
>> 
>> 
>> On Mon, Aug 25, 2014 at 11:51 AM, Cheolsoo Park 
>> wrote:
>> 
>>> Additionally, I will give "branch-specific" commit permission to people
>>> who will work on Pig on Spark (assuming it is possible).
>>> 
>>> Please let me know if you have any objection on this too.
>>> 
>>> 
>>> On Mon, Aug 25, 2014 at 10:25 AM, Jarek Jarcec Cecho 
>>> wrote:
>>> 
 No objections from my side, thank you for creating the branch Cheolsoo
 and kudos to the Sigmoid Analytics team for the great work!
 
 Jarcec
 
 On Aug 25, 2014, at 7:14 PM, Cheolsoo Park 
>> wrote:
 
> Hi devs,
> 
> Sigmoid Analytics has been working on Pig-on-Spark (PIG-4059), and
>> they
 want to merge their work into Apache.
> 
> I am going to create a "Spark" branch for them. Please let me know if
 you have any concerns.
> 
> Thanks,
> Cheolsoo
 
 
>>> 
>> 
> 
> 
> 
> -- 
> Regards,
> Mayur Rustagi
> Ph: +1 (760) 203 3257
> http://www.sigmoidanalytics.com
> @mayur_rustagi

Re: Creating a branch for Pig on Spark (PIG-4059)

2014-08-26 Thread Mayur Rustagi 
Praveen & me should be good enough for us. We will have 1-2 more folks
contributing from Sigmoid but we can manage by merging patches from them.

Initially it will be good to structure patches through us as we
would like to avoid breaking architecture & flow.

Regards
Mayur


On Tue, Aug 26, 2014 at 8:27 AM, Cheolsoo Park  wrote:

> Hi guys,
>
> I asked about branch committership to the infra mailing list, and here is
> the reply-
>
> Many projects have what they consider 'partial committers' that is
> folks who have access to specific parts of a projects svn tree. Some
> projects do this for GSoC participants, others as a mechanism for
> moving to 'full committership' within the project.
>
> Do note though that in the eyes of the ASF someone with an ICLA and an
> account with any permissions to commit code anywhere in the public svn
> tree is a committer. IOW, you would vote, have ICLAs filed, and
> request account creation as per normal, and then merely adjust the
> karma in asf-authorization-template (and or LDAP)
>
>
> Looks like we need to vote and follow the normal process just like any
> other new committer.
>
> @Praveen, Jacec,
> I think Mayur and Praveen from Sigmoid Analytics need branch committership.
> Will anyone else work on Pig-on-Spark? Please reply.
>
> Once I have a full list of people, I will open a vote for Pig PMCs.
>
> Thanks,
> Cheolsoo
>
>
> On Mon, Aug 25, 2014 at 11:51 AM, Cheolsoo Park 
> wrote:
>
> > Additionally, I will give "branch-specific" commit permission to people
> > who will work on Pig on Spark (assuming it is possible).
> >
> > Please let me know if you have any objection on this too.
> >
> >
> > On Mon, Aug 25, 2014 at 10:25 AM, Jarek Jarcec Cecho 
> > wrote:
> >
> >> No objections from my side, thank you for creating the branch Cheolsoo
> >> and kudos to the Sigmoid Analytics team for the great work!
> >>
> >> Jarcec
> >>
> >> On Aug 25, 2014, at 7:14 PM, Cheolsoo Park 
> wrote:
> >>
> >> > Hi devs,
> >> >
> >> > Sigmoid Analytics has been working on Pig-on-Spark (PIG-4059), and
> they
> >> want to merge their work into Apache.
> >> >
> >> > I am going to create a "Spark" branch for them. Please let me know if
> >> you have any concerns.
> >> >
> >> > Thanks,
> >> > Cheolsoo
> >>
> >>
> >
>



-- 
Regards,
Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi