git commit: Merge pull request #498 from ScrapCodes/python-api. Closes #498.

2014-02-06 Thread joshrosen
Updated Branches: refs/heads/master 79c95527a -> 084839ba3 Merge pull request #498 from ScrapCodes/python-api. Closes #498. Python api additions Author: Prashant Sharma == Merge branch commits == commit 8b51591f1a7a79a62c13ee66ff8d83040f7eccd8 Author: Prashant Sharma Date: Fri Jan 24 11

[2/2] git commit: Merge pull request #523 from JoshRosen/SPARK-1043

2014-01-28 Thread joshrosen
Merge pull request #523 from JoshRosen/SPARK-1043 Switch from MUTF8 to UTF8 in PySpark serializers. This fixes SPARK-1043, a bug introduced in 0.9.0 where PySpark couldn't serialize strings > 64kB. This fix was written by @tyro89 and @bouk in #512. This commit squashes and rebases th

[1/2] git commit: Switch from MUTF8 to UTF8 in PySpark serializers.

2014-01-28 Thread joshrosen
Updated Branches: refs/heads/master 84670f271 -> f8c742ce2 Switch from MUTF8 to UTF8 in PySpark serializers. This fixes SPARK-1043, a bug introduced in 0.9.0 where PySpark couldn't serialize strings > 64kB. This fix was written by @tyro89 and @bouk in #512. This commit squashes and rebases th

[6/9] git commit: Modifications as suggested in PR feedback-

2014-01-23 Thread joshrosen
Modifications as suggested in PR feedback- - mapPartitions, foreachPartition moved to JavaRDDLike - call scala rdd's setGenerator instead of setting directly in JavaRDD Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-s

[8/9] git commit: fixed ClassTag in mapPartitions

2014-01-23 Thread joshrosen
fixed ClassTag in mapPartitions Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/60e74572 Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/60e74572 Diff: http://git-wip-us.apache.org/repos/a

[9/9] git commit: Merge pull request #406 from eklavya/master

2014-01-23 Thread joshrosen
Merge pull request #406 from eklavya/master Extending Java API coverage Hi, I have added three new methods to JavaRDD. Please review and merge. Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/fad6aacf T

[7/9] git commit: Modifications as suggested in PR feedback-

2014-01-23 Thread joshrosen
Modifications as suggested in PR feedback- - more variants of mapPartitions added to JavaRDDLike - move setGenerator to JavaRDDLike - clean up Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/1442cd5d Tree:

[1/9] git commit: Added setter method setGenerator to JavaRDD.

2014-01-23 Thread joshrosen
Updated Branches: refs/heads/master a2b47dae6 -> fad6aacfb Added setter method setGenerator to JavaRDD. Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/aae8a014 Tree: http://git-wip-us.apache.org/repos/

[5/9] git commit: Remove default param from mapPartitions

2014-01-23 Thread joshrosen
Remove default param from mapPartitions Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/fa42951e Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/fa42951e Diff: http://git-wip-us.apache.org

[4/9] git commit: Remove classtag from mapPartitions.

2014-01-23 Thread joshrosen
Remove classtag from mapPartitions. Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/8fe562c0 Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/8fe562c0 Diff: http://git-wip-us.apache.org/rep

[2/9] git commit: Added mapPartitions method to JavaRDD.

2014-01-23 Thread joshrosen
Added mapPartitions method to JavaRDD. Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/dbadc6b9 Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/dbadc6b9 Diff: http://git-wip-us.apache.org/

[3/9] git commit: Added foreachPartition method to JavaRDD.

2014-01-23 Thread joshrosen
Added foreachPartition method to JavaRDD. Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/6a65feeb Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/6a65feeb Diff: http://git-wip-us.apache.o

[4/4] git commit: Merge pull request #256 from MLnick/master

2013-12-15 Thread joshrosen
Merge pull request #256 from MLnick/master Fix 'IPYTHON=1 ./pyspark' throwing ValueError This fixes an annoying issue where running ```IPYTHON=1 ./pyspark``` resulted in: ``` Welcome to __ / __/__ ___ _/ /__ _\ \/ _ \/ _ `/ __/ '_/ /__ / .__/\_,_/_/ /_/

[1/4] git commit: Fix 'IPYTHON=1 ./pyspark' throwing 'ValueError: Cannot run multiple SparkContexts at once'

2013-12-15 Thread joshrosen
Updated Branches: refs/heads/master c55e69855 -> d2ced6d58 Fix 'IPYTHON=1 ./pyspark' throwing 'ValueError: Cannot run multiple SparkContexts at once' Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/8cd

[3/4] git commit: Making IPython PySpark compatible across versions <1.0.0. Also cleaned up '-i' option and made IPYTHON_OPTS work

2013-12-15 Thread joshrosen
Making IPython PySpark compatible across versions <1.0.0. Also cleaned up '-i' option and made IPYTHON_OPTS work Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/bb5277b1 Tree: http://git-wip-us.apache.org/

[2/4] git commit: Merge remote-tracking branch 'upstream/master'

2013-12-15 Thread joshrosen
Merge remote-tracking branch 'upstream/master' Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/d36ee3b1 Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/d36ee3b1 Diff: http://git-wip-us.apa

[3/5] git commit: Leave default value of numPartitions to Scala code.

2013-12-14 Thread joshrosen
Leave default value of numPartitions to Scala code. Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/e85af507 Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/e85af507 Diff: http://git-wip-u

[2/5] git commit: Use braces to shorten the line.

2013-12-14 Thread joshrosen
Use braces to shorten the line. Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/c82d4f07 Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/c82d4f07 Diff: http://git-wip-us.apache.org/repos/a

[4/5] git commit: Hook directly to Scala API

2013-12-14 Thread joshrosen
Hook directly to Scala API Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/0b494f7d Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/0b494f7d Diff: http://git-wip-us.apache.org/repos/asf/in

[1/5] git commit: Expose numPartitions parameter in JavaPairRDD.sortByKey()

2013-12-14 Thread joshrosen
Updated Branches: refs/heads/master 97ac06018 -> 2fd781d34 Expose numPartitions parameter in JavaPairRDD.sortByKey() This change make Java and Scala API on sortByKey() the same. Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/as

[5/5] git commit: Merge pull request #249 from ngbinh/partitionInJavaSortByKey

2013-12-14 Thread joshrosen
Merge pull request #249 from ngbinh/partitionInJavaSortByKey Expose numPartitions parameter in JavaPairRDD.sortByKey() This change makes Java and Scala API on sortByKey() the same. Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/as