[jira] [Commented] (PIG-4621) Enable Illustrate in spark

2016-02-25 Thread prateek vaishnav (JIRA) 

[ 
https://issues.apache.org/jira/browse/PIG-4621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15168574#comment-15168574
 ] 

prateek vaishnav commented on PIG-4621:
---

After investigating the issue, I have found out the problem. It is not 
different from what [~kellyzly] was pointing out.

If you look at the class ExampleGenerator, execEngine and 
localJobSimulator(does not exist currently) are hard coded.
private MRExecutionEngine execEngine;
private LocalMapReduceSimulator localMRRunner;

If you look further below, in method getData() -
localMRRunner.launchPig(plan, baseData, lineage, attacher, this, pigContext);

This launchPig runs the MR job and illustrates the data at different points.
We need similar classes for spark and tez as well.

To solve the issue for spark, I propose creating following classes -

- LocalJobSimulator
- LocalMRSimulator extends LocalJobSimulator
- LocalSparkSimulator extends LocalJobSimulator

A method getLocalJobSimulator() in HExecutionEngrine class will return 
respective localJobSimulator, which will be used in ExampleGenerator. Since 
LocalMapReduceSimulator is only used in ExampleGenerator, risk is fairly low.

Any thoughts or concerns are welcome. 

> Enable Illustrate in spark
> --
>
> Key: PIG-4621
> URL: https://issues.apache.org/jira/browse/PIG-4621
> Project: Pig
>  Issue Type: Sub-task
>  Components: spark
>Reporter: liyunzhang_intel
>Assignee: prateek vaishnav
> Fix For: spark-branch
>
>
> Current we don't support illustrate in spark mode.
> How illustrate works 
> see:http://pig.apache.org/docs/r0.7.0/piglatin_ref2.html#ILLUSTRATE



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] Subscription: PIG patch available

2016-02-25 Thread jira 
Issue Subscription
Filter: PIG patch available (31 issues)

Subscriber: pigdaily

Key Summary
PIG-4796Authenticate with Kerberos using a keytab file
https://issues.apache.org/jira/browse/PIG-4796
PIG-4788the value BytesRead metric info always returns 0 even the length of 
input file is not 0 in spark engine
https://issues.apache.org/jira/browse/PIG-4788
PIG-4781Fix remaining unit failure about "TestCollectedGroup" for spark 
engine
https://issues.apache.org/jira/browse/PIG-4781
PIG-4745DataBag should protect content of passed list of tuples
https://issues.apache.org/jira/browse/PIG-4745
PIG-4734TOMAP schema inferring breaks some scripts in type checking for 
bincond
https://issues.apache.org/jira/browse/PIG-4734
PIG-4684Exception should be changed to warning when job diagnostics cannot 
be fetched
https://issues.apache.org/jira/browse/PIG-4684
PIG-4656Improve String serialization and comparator performance in 
BinInterSedes
https://issues.apache.org/jira/browse/PIG-4656
PIG-4641Print the instance of Object without using toString()
https://issues.apache.org/jira/browse/PIG-4641
PIG-4598Allow user defined plan optimizer rules
https://issues.apache.org/jira/browse/PIG-4598
PIG-4581thread safe issue in NodeIdGenerator
https://issues.apache.org/jira/browse/PIG-4581
PIG-4551Partition filter is not pushed down in case of SPLIT
https://issues.apache.org/jira/browse/PIG-4551
PIG-4539New PigUnit
https://issues.apache.org/jira/browse/PIG-4539
PIG-4526Make setting up the build environment easier
https://issues.apache.org/jira/browse/PIG-4526
PIG-4515org.apache.pig.builtin.Distinct throws ClassCastException
https://issues.apache.org/jira/browse/PIG-4515
PIG-4455Should use DependencyOrderWalker instead of DepthFirstWalker in 
MRPrinter
https://issues.apache.org/jira/browse/PIG-4455
PIG-4341Add CMX support to pig.tmpfilecompression.codec
https://issues.apache.org/jira/browse/PIG-4341
PIG-4323PackageConverter hanging in Spark
https://issues.apache.org/jira/browse/PIG-4323
PIG-4313StackOverflowError in LIMIT operation on Spark
https://issues.apache.org/jira/browse/PIG-4313
PIG-4251Pig on Storm
https://issues.apache.org/jira/browse/PIG-4251
PIG-4111Make Pig compiles with avro-1.7.7
https://issues.apache.org/jira/browse/PIG-4111
PIG-4002Disable combiner when map-side aggregation is used
https://issues.apache.org/jira/browse/PIG-4002
PIG-3952PigStorage accepts '-tagSplit' to return full split information
https://issues.apache.org/jira/browse/PIG-3952
PIG-3911Define unique fields with @OutputSchema
https://issues.apache.org/jira/browse/PIG-3911
PIG-3906ant site errors out
https://issues.apache.org/jira/browse/PIG-3906
PIG-3877Getting Geo Latitude/Longitude from Address Lines
https://issues.apache.org/jira/browse/PIG-3877
PIG-3873Geo distance calculation using Haversine
https://issues.apache.org/jira/browse/PIG-3873
PIG-3866Create ThreadLocal classloader per PigContext
https://issues.apache.org/jira/browse/PIG-3866
PIG-3864ToDate(userstring, format, timezone) computes DateTime with strange 
handling of Daylight Saving Time with location based timezones
https://issues.apache.org/jira/browse/PIG-3864
PIG-3851Upgrade jline to 2.11
https://issues.apache.org/jira/browse/PIG-3851
PIG-3668COR built-in function when atleast one of the coefficient values is 
NaN
https://issues.apache.org/jira/browse/PIG-3668
PIG-3587add functionality for rolling over dates
https://issues.apache.org/jira/browse/PIG-3587

You may edit this subscription at:
https://issues.apache.org/jira/secure/FilterSubscription!default.jspa?subId=16328=12322384

Re: Welcome to our new Pig PMC member Xuefu Zhang

2016-02-25 Thread Srikanth Sundarrajan 
Congratulations Xuefu

Regards,
Srikanth Sundarrajan

On Thu, Feb 25, 2016 at 10:07 AM Xuefu Zhang  wrote:

> Thank you, Liyun! You did the hard work. I think you well deserve a
> committership once we merge the branch to trunk.
>
> --Xuefu
>
> On Wed, Feb 24, 2016 at 5:18 PM, Zhang, Liyun 
> wrote:
>
> > Congratulations Xuefu!
> >
> >
> > Kelly Zhang/Zhang,Liyun
> > Best Regards
> >
> >
> >
> > -Original Message-
> > From: Jarek Jarcec Cecho [mailto:jar...@gmail.com] On Behalf Of Jarek
> > Jarcec Cecho
> > Sent: Thursday, February 25, 2016 6:36 AM
> > To: dev@pig.apache.org
> > Cc: u...@pig.apache.org
> > Subject: Re: Welcome to our new Pig PMC member Xuefu Zhang
> >
> > Congratulations Xuefu!
> >
> > Jarcec
> >
> > > On Feb 24, 2016, at 1:29 PM, Rohini Palaniswamy <
> rohini.adi...@gmail.com>
> > wrote:
> > >
> > > It is my pleasure to announce that Xuefu Zhang is our newest addition
> > > to the Pig PMC. Xuefu is a long time committer of Pig and has been
> > > actively involved in driving the Pig on Spark effort for the past year.
> > >
> > > Please join me in congratulating Xuefu !!!
> > >
> > > Regards,
> > > Rohini
> >
> >
>