[jira] [Commented] (PIG-4698) Enable dynamic resource allocation/de-allocation on Yarn backends

[ https://issues.apache.org/jira/browse/PIG-4698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14966832#comment-14966832 ] Xuefu Zhang commented on PIG-4698: -- [~sriksun], thanks for looking into this. It's great that "set co

[jira] [Commented] (PIG-4698) Enable dynamic resource allocation/de-allocation on Yarn backends

[ https://issues.apache.org/jira/browse/PIG-4698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14967242#comment-14967242 ] Xuefu Zhang commented on PIG-4698: -- Sounds good. Thanks. > Enable dynamic resource allocation

[jira] [Updated] (PIG-4693) Class conflicts: Kryo bundled in spark vs kryo bundled with pig

[ https://issues.apache.org/jira/browse/PIG-4693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4693: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks

[jira] [Updated] (PIG-4208) Make merge-sparse join work with Spark

[ https://issues.apache.org/jira/browse/PIG-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4208: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks

[jira] [Commented] (PIG-4693) Class conflicts: Kryo bundled in spark vs kryo bundled with pig

[ https://issues.apache.org/jira/browse/PIG-4693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14954315#comment-14954315 ] Xuefu Zhang commented on PIG-4693: -- +1 on the latest patch. > Class conflicts: Kryo bundled in spark

Dependency version on Kryo

Hi all, It was found in PIG-4693 (https://issues.apache.org/jira/browse/PIG-4693) that Pig is currently dependent on Kryo 2.22. However, Spark depends on 2.21. The two versions are not completely compatible. We tried several ways to solve the problem but unfortunately none worked. This is mainly

[jira] [Commented] (PIG-4208) Make merge-sparse join work with Spark

[ https://issues.apache.org/jira/browse/PIG-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950656#comment-14950656 ] Xuefu Zhang commented on PIG-4208: -- Yeah, I saw similar changes in other places. I was just trying

[jira] [Commented] (PIG-4693) Class conflicts: Kryo bundled in spark vs kryo bundled with pig

[ https://issues.apache.org/jira/browse/PIG-4693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14948620#comment-14948620 ] Xuefu Zhang commented on PIG-4693: -- If does it help if the kryo.jar is loaded before spark-core (or spark

[jira] [Commented] (PIG-4693) Class conflicts: Kryo bundled in spark vs kryo bundled with pig

[ https://issues.apache.org/jira/browse/PIG-4693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946808#comment-14946808 ] Xuefu Zhang commented on PIG-4693: -- [~sriksun], I verified that spark-core.jar doesn't contain kryo. From

[jira] [Commented] (PIG-4693) Class conflicts: Kryo bundled in spark vs kryo bundled with pig

[ https://issues.apache.org/jira/browse/PIG-4693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14947183#comment-14947183 ] Xuefu Zhang commented on PIG-4693: -- I was thinking if we exclude chill-java.jar and include our own

[jira] [Commented] (PIG-4693) Class conflicts: Kryo bundled in spark vs kryo bundled with pig

[ https://issues.apache.org/jira/browse/PIG-4693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945697#comment-14945697 ] Xuefu Zhang commented on PIG-4693: -- Instead of depending on spark-assembly.jar, could we only depends

[jira] [Commented] (PIG-4208) Make merge-sparse join work with Spark

[ https://issues.apache.org/jira/browse/PIG-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14944451#comment-14944451 ] Xuefu Zhang commented on PIG-4208: -- Hi [~abhishek.agarwal], thanks for working on this. Could you please

[jira] [Commented] (PIG-4681) Enable Pig on Spark to run on Yarn Cluster mode

[ https://issues.apache.org/jira/browse/PIG-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14910139#comment-14910139 ] Xuefu Zhang commented on PIG-4681: -- Hive solves the problem by creating a remote spark context which runs

[jira] [Resolved] (PIG-4681) Enable Pig on Spark to run on Yarn Cluster mode

[ https://issues.apache.org/jira/browse/PIG-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang resolved PIG-4681. -- Resolution: Won't Fix Release Note: Document the limitation. > Enable Pig on Spark to run on Y

[jira] [Commented] (PIG-4236) Avoid packaging spark specific jars into pig fat jar

[ https://issues.apache.org/jira/browse/PIG-4236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14933127#comment-14933127 ] Xuefu Zhang commented on PIG-4236: -- Yeah. I think this can be closed. > Avoid packaging spark speci

[jira] [Commented] (PIG-4601) Implement Merge CoGroup for Spark engine

[ https://issues.apache.org/jira/browse/PIG-4601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14906474#comment-14906474 ] Xuefu Zhang commented on PIG-4601: -- I don't think so. It's unsigned. Please feel free to take

[jira] [Updated] (PIG-4667) Enable Pig on Spark to run on Yarn Client mode

[ https://issues.apache.org/jira/browse/PIG-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4667: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks

[jira] [Commented] (PIG-4667) Enable Pig on Spark to run on Yarn Client mode

[ https://issues.apache.org/jira/browse/PIG-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14781849#comment-14781849 ] Xuefu Zhang commented on PIG-4667: -- +1 > Enable Pig on Spark to run on Yarn Client m

Re: Review Request 38352: Enable Pig on Spark to run on Yarn Client mode

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/38352/#review99223 --- Ship it! Ship It! - Xuefu Zhang On Sept. 16, 2015, 12:06 p.m

Re: Review Request 38352: Enable Pig on Spark to run on Yarn Client mode

e 71) <https://reviews.apache.org/r/38352/#comment155634> Why do we downgrade? - Xuefu Zhang On Sept. 14, 2015, 11:25 a.m., Srikanth Sundarrajan wrote: > > --- > This is an automatically generated e-mai

[jira] [Commented] (PIG-4667) Enable Pig on Spark to run on Yarn Client mode

[ https://issues.apache.org/jira/browse/PIG-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744259#comment-14744259 ] Xuefu Zhang commented on PIG-4667: -- Patch looks good. Some minor comments/questions on RB. > Enable

[jira] [Commented] (PIG-4059) Pig on Spark

[ https://issues.apache.org/jira/browse/PIG-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741261#comment-14741261 ] Xuefu Zhang commented on PIG-4059: -- Thanks, Peter. I have converted it to a subtask of this JIRA. >

[jira] [Updated] (PIG-4675) Multi Store Statement will fail on the second store statement.

[ https://issues.apache.org/jira/browse/PIG-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4675: - Issue Type: Sub-task (was: Bug) Parent: PIG-4059 > Multi Store Statement will fail on the sec

[jira] [Commented] (PIG-4667) Enable Pig on Spark to run on Yarn Client/Cluster mode

[ https://issues.apache.org/jira/browse/PIG-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737394#comment-14737394 ] Xuefu Zhang commented on PIG-4667: -- [~sriksun], jar problem is indeed very complicated. I'm a little

[jira] [Commented] (PIG-4655) Support InputStats in spark mode

[ https://issues.apache.org/jira/browse/PIG-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14731758#comment-14731758 ] Xuefu Zhang commented on PIG-4655: -- The patch doesn't seem applying. Does it depend on other patches

[jira] [Commented] (PIG-4645) Support hadoop-like Counter using spark accumulator

[ https://issues.apache.org/jira/browse/PIG-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14696346#comment-14696346 ] Xuefu Zhang commented on PIG-4645: -- I guess my previous commit fell apart, probably due

[jira] [Resolved] (PIG-4645) Support hadoop-like Counter using spark accumulator

[ https://issues.apache.org/jira/browse/PIG-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang resolved PIG-4645. -- Resolution: Fixed Committed to Spark branch. Thanks, Xianda/Mohit. Support hadoop-like Counter using

[jira] [Resolved] (PIG-4594) Enable TestMultiQuery in spark mode

[ https://issues.apache.org/jira/browse/PIG-4594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang resolved PIG-4594. -- Resolution: Fixed Committed to Spark branch. Thanks, Liyun and Xianda. Enable TestMultiQuery in spark

[jira] [Commented] (PIG-4634) Fix records count issues in output statistics

[ https://issues.apache.org/jira/browse/PIG-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14634479#comment-14634479 ] Xuefu Zhang commented on PIG-4634: -- Patch looks good. Two questions: 1. In getRecordCount

[jira] [Commented] (PIG-4594) Enable TestMultiQuery in spark mode

[ https://issues.apache.org/jira/browse/PIG-4594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14625832#comment-14625832 ] Xuefu Zhang commented on PIG-4594: -- Patch seems needing a rebase. [~kexianda], could you

[jira] [Resolved] (PIG-4633) Update hadoop version to enable Spark output statistics

[ https://issues.apache.org/jira/browse/PIG-4633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang resolved PIG-4633. -- Resolution: Fixed Committed to Spark branch. Thanks, Xianda. Update hadoop version to enable Spark

[jira] [Resolved] (PIG-4622) Skip TestCubeOperator.testIllustrate and TestMultiQueryLocal.testMultiQueryWithIllustrate

[ https://issues.apache.org/jira/browse/PIG-4622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang resolved PIG-4622. -- Resolution: Fixed Committed to Spark branch. Thanks, Liyun. Skip TestCubeOperator.testIllustrate

[jira] [Updated] (PIG-4619) Cleanup: change the indent size of some files of pig on spark project from 2 to 4 space

[ https://issues.apache.org/jira/browse/PIG-4619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4619: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks, Liyun

[jira] [Resolved] (PIG-4613) Fix unit test failures about TestAssert

[ https://issues.apache.org/jira/browse/PIG-4613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang resolved PIG-4613. -- Resolution: Fixed Committed to Spark branch. Thanks, Xianda. Fix unit test failures about TestAssert

[jira] [Updated] (PIG-4614) Enable TestLocationInPhysicalPlan in spark mode

[ https://issues.apache.org/jira/browse/PIG-4614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4614: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks, Liyun

[jira] [Commented] (PIG-4059) Pig on Spark

[ https://issues.apache.org/jira/browse/PIG-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14608642#comment-14608642 ] Xuefu Zhang commented on PIG-4059: -- No ETA foreseen at this time. Contributions are welcome

[jira] [Updated] (PIG-4607) Enable TestRank1,TestRank3 unit tests in spark mode

[ https://issues.apache.org/jira/browse/PIG-4607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4607: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks

[jira] [Resolved] (PIG-4615) Fix null keys join in SkewedJoin in spark mode

[ https://issues.apache.org/jira/browse/PIG-4615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang resolved PIG-4615. -- Resolution: Fixed Committed to Spark branch. Thanks, Liyun! Fix null keys join in SkewedJoin in spark

[jira] [Updated] (PIG-4610) Enable TestOrcStorage“ unit test in spark mode

[ https://issues.apache.org/jira/browse/PIG-4610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4610: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks, Liyun

[jira] [Updated] (PIG-4606) Enable TestDefaultDateTimeZone unit tests in spark mode

[ https://issues.apache.org/jira/browse/PIG-4606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4606: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks, Liyun

[jira] [Resolved] (PIG-4605) fix a bug when coping Jar to SparkJob working directory

[ https://issues.apache.org/jira/browse/PIG-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang resolved PIG-4605. -- Resolution: Fixed Committed to Spark branch. Thanks, Xianda! fix a bug when coping Jar to SparkJob

[jira] [Resolved] (PIG-4604) Clean up: refactor the package import order in the files under pig/src/org/apache/pig/backend/hadoop/executionengine/spark according to certain rule

[ https://issues.apache.org/jira/browse/PIG-4604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang resolved PIG-4604. -- Resolution: Fixed Committed to Spark branch. Thanks, Liyun! Clean up: refactor the package import order

[jira] [Updated] (PIG-4593) Enable TestMultiQueryLocal in spark mode

[ https://issues.apache.org/jira/browse/PIG-4593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4593: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks, Liyun

[jira] [Updated] (PIG-4585) Use newAPIHadoopRDD instead of newAPIHadoopFile

[ https://issues.apache.org/jira/browse/PIG-4585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4585: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks, Mohit

[jira] [Resolved] (PIG-4597) Enable TestNullConstant unit test in spark mode

[ https://issues.apache.org/jira/browse/PIG-4597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang resolved PIG-4597. -- Resolution: Fixed Committed to Spark branch. Thanks, Xianda! Enable TestNullConstant unit test in spark

[jira] [Updated] (PIG-4595) Fix unit test failures about TestFRJoinNullValue in spark mode

[ https://issues.apache.org/jira/browse/PIG-4595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4595: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks, Liyun

[jira] [Updated] (PIG-4586) Cleanup: Rename POConverter to RDDConverter

[ https://issues.apache.org/jira/browse/PIG-4586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4586: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks, Mohit

[jira] [Updated] (PIG-4588) Move tests under 'test-spark' target

[ https://issues.apache.org/jira/browse/PIG-4588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4588: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks, Mohit

Re: Future of Pig on Spark ?

Hi Bharath, Thanks for your inquiry. A small team is committed to Pig on Spark project and is actively working toward to the direction, while the pace of progression is slower than wished. This is mostly due to the constrained resources and sort of diversion of the initial contributors. We

[jira] [Updated] (PIG-4589) Fix unit test failure in TestCase

[ https://issues.apache.org/jira/browse/PIG-4589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4589: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks

[jira] [Updated] (PIG-4283) Enable unit test TestGrunt for spark

[ https://issues.apache.org/jira/browse/PIG-4283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4283: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks, Liyun

[jira] [Updated] (PIG-4582) Enable TestPruneColumn in spark mode

[ https://issues.apache.org/jira/browse/PIG-4582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4582: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks

[jira] [Updated] (PIG-4577) Use cogroup spark api to implement groupby+secondarysort case in GlobalRearrangeConverter.java

[ https://issues.apache.org/jira/browse/PIG-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4577: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks, Liyun

[jira] [Updated] (PIG-4565) Support custom MR partitioners for Spark engine

[ https://issues.apache.org/jira/browse/PIG-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4565: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks, Mohit

[jira] [Updated] (PIG-4284) Enable unit test TestJoin for spark

[ https://issues.apache.org/jira/browse/PIG-4284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4284: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks, Liyun

[jira] [Resolved] (PIG-4297) Enable unit test org.apache.pig.test.TestProjectRange for spark

[ https://issues.apache.org/jira/browse/PIG-4297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang resolved PIG-4297. -- Resolution: Fixed Committed to spark branch. Thanks, Liyun! Enable unit test

[jira] [Updated] (PIG-4568) Fix unit test failure in TestSecondarySortSpark

[ https://issues.apache.org/jira/browse/PIG-4568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4568: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks, Liyun

[jira] [Updated] (PIG-4549) Set CROSS operation parallelism for Spark engine

[ https://issues.apache.org/jira/browse/PIG-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4549: - Resolution: Fixed Status: Resolved (was: Patch Available) committed to Spark branch. Thanks, Mohit

[jira] [Updated] (PIG-4552) Fix TestForEachNestedPlanLocal for Spark engine

[ https://issues.apache.org/jira/browse/PIG-4552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4552: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks, Mohit

[jira] [Updated] (PIG-4557) Fix POGlobalRearrangeSpark copy constructor for Spark engine

[ https://issues.apache.org/jira/browse/PIG-4557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4557: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks, Mohit

[jira] [Updated] (PIG-4422) Implement MergeJoin (as regular join) for Spark engine

[ https://issues.apache.org/jira/browse/PIG-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4422: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks, Mohit

[jira] [Resolved] (PIG-4295) Enable unit test TestPigContext for spark

[ https://issues.apache.org/jira/browse/PIG-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang resolved PIG-4295. -- Resolution: Fixed Patch #4 is committed to Spark branch. Thanks, Liyun/Mohit. Enable unit test

[jira] [Resolved] (PIG-4558) Modify the test.output value from no to yes to show more error message

[ https://issues.apache.org/jira/browse/PIG-4558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang resolved PIG-4558. -- Resolution: Fixed Committed to Spark branch. Thanks, Liyun. Modify the test.output value from no to yes

[jira] [Reopened] (PIG-4295) Enable unit test TestPigContext for spark

[ https://issues.apache.org/jira/browse/PIG-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang reopened PIG-4295: -- Previous commit was reverted. Reopened this issue. Enable unit test TestPigContext for spark

[jira] [Updated] (PIG-4295) Enable unit test TestPigContext for spark

[ https://issues.apache.org/jira/browse/PIG-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4295: - Affects Version/s: spark-branch Status: Patch Available (was: Open) Enable unit test

[jira] [Updated] (PIG-4504) Enable Secondary key sort feature in spark mode

[ https://issues.apache.org/jira/browse/PIG-4504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4504: - Resolution: Fixed Fix Version/s: spark-branch Status: Resolved (was: Patch Available

[jira] [Updated] (PIG-4295) Enable unit test TestPigContext for spark

[ https://issues.apache.org/jira/browse/PIG-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4295: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks, Liyun

[jira] [Updated] (PIG-4190) Implement replicated join in Spark engine

[ https://issues.apache.org/jira/browse/PIG-4190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4190: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks, Mohit

[jira] [Commented] (PIG-4190) Implement replicated join in Spark engine

[ https://issues.apache.org/jira/browse/PIG-4190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14543026#comment-14543026 ] Xuefu Zhang commented on PIG-4190: -- It seems that the patch needs to be rebased

[jira] [Updated] (PIG-4542) OutputConsumerIterator should flush buffered records

[ https://issues.apache.org/jira/browse/PIG-4542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4542: - Resolution: Pending Closed Status: Resolved (was: Patch Available) Committed to Spark branch

[jira] [Resolved] (PIG-4540) Remove repetitive org.apache.pig.test.Util#isSparkExecType

[ https://issues.apache.org/jira/browse/PIG-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang resolved PIG-4540. -- Resolution: Fixed Committed to Spark branch. Thanks, Liyun! Remove repetitive org.apache.pig.test.Util

[jira] [Updated] (PIG-4276) Fix ordering related failures in TestEvalPipeline for Spark

[ https://issues.apache.org/jira/browse/PIG-4276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4276: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks, Mohit

[jira] [Updated] (PIG-4421) implement visitSkewedJoin in SparkCompiler

[ https://issues.apache.org/jira/browse/PIG-4421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4421: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks, Liyun

[jira] [Commented] (PIG-4295) Enable unit test TestPigContext for spark

[ https://issues.apache.org/jira/browse/PIG-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14520821#comment-14520821 ] Xuefu Zhang commented on PIG-4295: -- I was wondering if we can make these static variables

[jira] [Resolved] (PIG-4518) SparkOperator should correspond to complete Spark job

[ https://issues.apache.org/jira/browse/PIG-4518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang resolved PIG-4518. -- Resolution: Fixed Committed to Spark branch. Thanks, Mohit. SparkOperator should correspond to complete

Re: many users can use one pigserver?

PigServer has state, which isn't meant to be shared by multiple user sessions. On the other hand, PIG-1784 made PigServer thread-safe, so depending on your version, you may choose having multiple instances in your Tomcat, one for each user session. On Tue, Apr 28, 2015 at 1:51 AM, 李运田

[jira] [Commented] (PIG-4295) Enable unit test TestPigContext for spark

[ https://issues.apache.org/jira/browse/PIG-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14517269#comment-14517269 ] Xuefu Zhang commented on PIG-4295: -- I suspect that PIG-1784 might have made the cheap fix

[jira] [Updated] (PIG-4489) Enable local mode tests for Spark engine

[ https://issues.apache.org/jira/browse/PIG-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4489: - Resolution: Fixed Status: Resolved (was: Patch Available) Enable local mode tests for Spark

[jira] [Commented] (PIG-4489) Enable local mode tests for Spark engine

[ https://issues.apache.org/jira/browse/PIG-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506198#comment-14506198 ] Xuefu Zhang commented on PIG-4489: -- Committed to Spark branch. Thanks, Mohit. Enable

[jira] [Commented] (PIG-4276) Fix ordering related failures in TestEvalPipeline for Spark

[ https://issues.apache.org/jira/browse/PIG-4276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504152#comment-14504152 ] Xuefu Zhang commented on PIG-4276: -- Actually if Pig semantics doesn't guarantee an order

Re: Welcome our new Pig PMC chair Rohini Palaniswamy

Congratulations, Rohini! --Xuefu On Wed, Mar 18, 2015 at 6:48 PM, Cheolsoo Park cheol...@apache.org wrote: Hi all, Now it's official that Rohini Palaniswamy is our new Pig PMC chair. Please join me in congratulating Rohini for her new role. Congrats! Thanks! Cheolsoo

[jira] [Commented] (PIG-4193) Make collected group work with Spark

[ https://issues.apache.org/jira/browse/PIG-4193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358627#comment-14358627 ] Xuefu Zhang commented on PIG-4193: -- Hi [~praveenr019], It seems that your patch

[jira] [Updated] (PIG-4300) Enable unit test TestSample for spark

[ https://issues.apache.org/jira/browse/PIG-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4300: - Resolution: Fixed Status: Resolved (was: Patch Available) Fixed via other JIRA as noted above

[jira] [Updated] (PIG-4287) Enable unit test TestLimitVariable for spark

[ https://issues.apache.org/jira/browse/PIG-4287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4287: - Resolution: Fixed Status: Resolved (was: Patch Available) Fixed via other JIRA as noted above

[jira] [Commented] (PIG-4269) Enable unit test TestAccumulator for spark

[ https://issues.apache.org/jira/browse/PIG-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354972#comment-14354972 ] Xuefu Zhang commented on PIG-4269: -- [~kellyzly], it seems that the patch doesn't apply

[jira] [Assigned] (PIG-4269) Enable unit test TestAccumulator for spark

[ https://issues.apache.org/jira/browse/PIG-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang reassigned PIG-4269: Assignee: liyunzhang_intel Enable unit test TestAccumulator for spark

[jira] [Updated] (PIG-4269) Enable unit test TestAccumulator for spark

[ https://issues.apache.org/jira/browse/PIG-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4269: - Status: Patch Available (was: Open) Enable unit test TestAccumulator for spark

[jira] [Updated] (PIG-4269) Enable unit test TestAccumulator for spark

[ https://issues.apache.org/jira/browse/PIG-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4269: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks, Liyun

[jira] [Commented] (PIG-4269) Enable unit test TestAccumulator for spark

[ https://issues.apache.org/jira/browse/PIG-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354966#comment-14354966 ] Xuefu Zhang commented on PIG-4269: -- +1 Enable unit test TestAccumulator for spark

[jira] [Updated] (PIG-4456) Sort the leaves by SparkOperator.operatorKey in SparkLauncher#sparkOperToRDD

[ https://issues.apache.org/jira/browse/PIG-4456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4456: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to Spark branch. Thanks, Liyun

[jira] [Updated] (PIG-4456) Sort the leaves by SparkOperator.operatorKey in SparkLauncher#sparkOperToRDD

[ https://issues.apache.org/jira/browse/PIG-4456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4456: - Fix Version/s: spark-branch Affects Version/s: spark-branch Status: Patch Available

[jira] [Resolved] (PIG-4374) Add SparkPlan in spark package

[ https://issues.apache.org/jira/browse/PIG-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang resolved PIG-4374. -- Resolution: Fixed Committed to Spark branch. Thanks, Liyun! Add SparkPlan in spark package

[jira] [Commented] (PIG-4374) Add SparkPlan in spark package

[ https://issues.apache.org/jira/browse/PIG-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354281#comment-14354281 ] Xuefu Zhang commented on PIG-4374: -- +1 for patch #7. I'm going to commit this patch

[jira] [Commented] (PIG-4456) Sort the leaves by SparkOperator.operatorKey in SparkLauncher#sparkOperToRDD

[ https://issues.apache.org/jira/browse/PIG-4456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354302#comment-14354302 ] Xuefu Zhang commented on PIG-4456: -- FYI, PIG-4374 has been committed. Sort the leaves

Re: about how to implement ship in other mode like spark

You might want to consider SparkContext.addFile() for distributing the file at the client and SparkFiles.get() for retrieving the file at the execution node. --Xuefu On Fri, Jan 2, 2015 at 7:15 PM, Zhang, Liyun liyun.zh...@intel.com wrote: Hi all, I want to ask a question about ship in pig:

[jira] [Updated] (PIG-4361) Fix perl script problem in TestStreaming.java

[ https://issues.apache.org/jira/browse/PIG-4361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4361: - Resolution: Fixed Fix Version/s: 0.15.0 Status: Resolved (was: Patch Available

[jira] [Commented] (PIG-4361) Fix perl script problem in TestStreaming.java

[ https://issues.apache.org/jira/browse/PIG-4361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261204#comment-14261204 ] Xuefu Zhang commented on PIG-4361: -- +1 Fix perl script problem in TestStreaming.java

[jira] [Updated] (PIG-4265) SUM functions returns different value in spark and mapreduce engine

[ https://issues.apache.org/jira/browse/PIG-4265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-4265: - Resolution: Fixed Fix Version/s: 0.15.0 Status: Resolved (was: Patch Available

[jira] [Commented] (PIG-4265) SUM functions returns different value in spark and mapreduce engine

[ https://issues.apache.org/jira/browse/PIG-4265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14198461#comment-14198461 ] Xuefu Zhang commented on PIG-4265: -- The difference could be caused by different compilers

[jira] [Commented] (PIG-4265) AlgebraicDoubleMathBase has Java double precision problems

[ https://issues.apache.org/jira/browse/PIG-4265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14196177#comment-14196177 ] Xuefu Zhang commented on PIG-4265: -- I'm not sure if this is a problem of the algebra. It's

<    1   2   3   4   5   >