spark git commit: [SPARK-8625] [CORE] Propagate user exceptions in tasks back to driver

2015-08-12 Thread irashid
Repository: spark Updated Branches: refs/heads/branch-1.5 5e6fdc659 - 0579f28df [SPARK-8625] [CORE] Propagate user exceptions in tasks back to driver This allows clients to retrieve the original exception from the cause field of the SparkException that is thrown by the driver. If the original

spark git commit: [SPARK-8625] [CORE] Propagate user exceptions in tasks back to driver

2015-08-12 Thread irashid
Repository: spark Updated Branches: refs/heads/master 3ecb37943 - 2e680668f [SPARK-8625] [CORE] Propagate user exceptions in tasks back to driver This allows clients to retrieve the original exception from the cause field of the SparkException that is thrown by the driver. If the original

spark git commit: [SPARK-9789] [ML] Added logreg threshold param back

2015-08-12 Thread jkbradley
Repository: spark Updated Branches: refs/heads/branch-1.5 65b5b2172 - bdf8dc15d [SPARK-9789] [ML] Added logreg threshold param back Reinstated LogisticRegression.threshold Param for binary compatibility. Param thresholds overrides threshold, if set. CC: mengxr dbtsai feynmanliang Author:

spark git commit: [SPARK-9789] [ML] Added logreg threshold param back

2015-08-12 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master 762bacc16 - 551def5d6 [SPARK-9789] [ML] Added logreg threshold param back Reinstated LogisticRegression.threshold Param for binary compatibility. Param thresholds overrides threshold, if set. CC: mengxr dbtsai feynmanliang Author:

spark git commit: [SPARK-9826] [CORE] Fix cannot use custom classes in log4j.properties

2015-08-12 Thread vanzin
Repository: spark Updated Branches: refs/heads/branch-1.5 8537e51d3 - 74c9dcec3 [SPARK-9826] [CORE] Fix cannot use custom classes in log4j.properties Refactor Utils class and create ShutdownHookManager. NOTE: Wasn't able to run /dev/run-tests on windows machine. Manual tests were conducted

spark git commit: [SPARK-9826] [CORE] Fix cannot use custom classes in log4j.properties

2015-08-12 Thread vanzin
Repository: spark Updated Branches: refs/heads/master 738f35398 - ab7e721cf [SPARK-9826] [CORE] Fix cannot use custom classes in log4j.properties Refactor Utils class and create ShutdownHookManager. NOTE: Wasn't able to run /dev/run-tests on windows machine. Manual tests were conducted

spark git commit: [SPARK-9907] [SQL] Python crc32 is mistakenly calling md5

2015-08-12 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-1.5 6a7582ea2 - b28295fe0 [SPARK-9907] [SQL] Python crc32 is mistakenly calling md5 Author: Reynold Xin r...@databricks.com Closes #8138 from rxin/SPARK-9907. (cherry picked from commit a17384fa343628cec44437da5b80b9403ecd5838)

spark git commit: [SPARK-9894] [SQL] Json writer should handle MapData.

2015-08-12 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master ab7e721cf - 7035d880a [SPARK-9894] [SQL] Json writer should handle MapData. https://issues.apache.org/jira/browse/SPARK-9894 Author: Yin Huai yh...@databricks.com Closes #8137 from yhuai/jsonMapData. Project:

spark git commit: [SPARK-9909] [ML] [TRIVIAL] move weightCol to shared params

2015-08-12 Thread meng
Repository: spark Updated Branches: refs/heads/master caa14d9dc - 6e409bc13 [SPARK-9909] [ML] [TRIVIAL] move weightCol to shared params As per the TODO move weightCol to Shared Params. Author: Holden Karau hol...@pigscanfly.ca Closes #8144 from

spark git commit: [SPARK-9909] [ML] [TRIVIAL] move weightCol to shared params

2015-08-12 Thread meng
Repository: spark Updated Branches: refs/heads/branch-1.5 6aca0cf34 - 2f8793b5f [SPARK-9909] [ML] [TRIVIAL] move weightCol to shared params As per the TODO move weightCol to Shared Params. Author: Holden Karau hol...@pigscanfly.ca Closes #8144 from

spark git commit: [SPARK-9913] [MLLIB] LDAUtils should be private

2015-08-12 Thread meng
Repository: spark Updated Branches: refs/heads/branch-1.5 08f767a1e - 6aca0cf34 [SPARK-9913] [MLLIB] LDAUtils should be private feynmanliang Author: Xiangrui Meng m...@databricks.com Closes #8142 from mengxr/SPARK-9913. (cherry picked from commit caa14d9dc9e2eb1102052b22445b63b0e004e3c7)

spark git commit: [SPARK-9449] [SQL] Include MetastoreRelation's inputFiles

2015-08-12 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-1.5 ed73f5439 - 3298fb69f [SPARK-9449] [SQL] Include MetastoreRelation's inputFiles Author: Michael Armbrust mich...@databricks.com Closes #8119 from marmbrus/metastoreInputFiles. (cherry picked from commit

spark git commit: [SPARK-9826] [CORE] Fix cannot use custom classes in log4j.properties

2015-08-12 Thread vanzin
Repository: spark Updated Branches: refs/heads/branch-1.4 89c8aea94 - 8ce86b23f [SPARK-9826] [CORE] Fix cannot use custom classes in log4j.properties Refactor Utils class and create ShutdownHookManager. NOTE: Wasn't able to run /dev/run-tests on windows machine. Manual tests were conducted

spark git commit: [SPARK-9913] [MLLIB] LDAUtils should be private

2015-08-12 Thread meng
Repository: spark Updated Branches: refs/heads/master 7035d880a - caa14d9dc [SPARK-9913] [MLLIB] LDAUtils should be private feynmanliang Author: Xiangrui Meng m...@databricks.com Closes #8142 from mengxr/SPARK-9913. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [SPARK-9780] [STREAMING] [KAFKA] prevent NPE if KafkaRDD instantiation …

2015-08-12 Thread tdas
Repository: spark Updated Branches: refs/heads/master 660e6dcff - 8ce60963c [SPARK-9780] [STREAMING] [KAFKA] prevent NPE if KafkaRDD instantiation … …fails Author: cody koeninger c...@koeninger.org Closes #8133 from koeninger/SPARK-9780 and squashes the following commits: 406259d [cody

spark git commit: [SPARK-9780] [STREAMING] [KAFKA] prevent NPE if KafkaRDD instantiation …

2015-08-12 Thread tdas
Repository: spark Updated Branches: refs/heads/branch-1.5 3298fb69f - 62ab2a4c6 [SPARK-9780] [STREAMING] [KAFKA] prevent NPE if KafkaRDD instantiation … …fails Author: cody koeninger c...@koeninger.org Closes #8133 from koeninger/SPARK-9780 and squashes the following commits: 406259d

spark git commit: [SPARK-9915] [ML] stopWords should use StringArrayParam

2015-08-12 Thread meng
Repository: spark Updated Branches: refs/heads/master e6aef5576 - fc1c7fd66 [SPARK-9915] [ML] stopWords should use StringArrayParam hhbyyh Author: Xiangrui Meng m...@databricks.com Closes #8141 from mengxr/SPARK-9915. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [SPARK-9092] Fixed incompatibility when both num-executors and dynamic...

2015-08-12 Thread vanzin
Repository: spark Updated Branches: refs/heads/master a17384fa3 - 738f35398 [SPARK-9092] Fixed incompatibility when both num-executors and dynamic... … allocation are set. Now, dynamic allocation is set to false when num-executors is explicitly specified as an argument. Consequently,

spark git commit: [SPARK-9092] Fixed incompatibility when both num-executors and dynamic...

2015-08-12 Thread vanzin
Repository: spark Updated Branches: refs/heads/branch-1.5 b28295fe0 - 8537e51d3 [SPARK-9092] Fixed incompatibility when both num-executors and dynamic... … allocation are set. Now, dynamic allocation is set to false when num-executors is explicitly specified as an argument. Consequently,

spark git commit: [SPARK-9766] [ML] [PySpark] check and add miss docs for PySpark ML

2015-08-12 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master 60103ecd3 - 762bacc16 [SPARK-9766] [ML] [PySpark] check and add miss docs for PySpark ML Check and add miss docs for PySpark ML (this issue only check miss docs for o.a.s.ml not o.a.s.mllib). Author: Yanbo Liang yblia...@gmail.com

spark git commit: [SPARK-8967] [DOC] add Since annotation

2015-08-12 Thread meng
Repository: spark Updated Branches: refs/heads/branch-1.5 bdf8dc15d - 6a7582ea2 [SPARK-8967] [DOC] add Since annotation Add `Since` as a Scala annotation. The benefit is that we can use it without having explicit JavaDoc. This is useful for inherited methods. The limitation is that is

spark git commit: [SPARK-9894] [SQL] Json writer should handle MapData.

2015-08-12 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-1.5 74c9dcec3 - 08f767a1e [SPARK-9894] [SQL] Json writer should handle MapData. https://issues.apache.org/jira/browse/SPARK-9894 Author: Yin Huai yh...@databricks.com Closes #8137 from yhuai/jsonMapData. (cherry picked from commit

spark git commit: [SPARK-9912] [MLLIB] QRDecomposition should use QType and RType for type names instead of UType and VType

2015-08-12 Thread meng
Repository: spark Updated Branches: refs/heads/master 6e409bc13 - e6aef5576 [SPARK-9912] [MLLIB] QRDecomposition should use QType and RType for type names instead of UType and VType hhbyyh Author: Xiangrui Meng m...@databricks.com Closes #8140 from mengxr/SPARK-9912. Project:

spark git commit: [SPARK-9912] [MLLIB] QRDecomposition should use QType and RType for type names instead of UType and VType

2015-08-12 Thread meng
Repository: spark Updated Branches: refs/heads/branch-1.5 2f8793b5f - 31b7fdc06 [SPARK-9912] [MLLIB] QRDecomposition should use QType and RType for type names instead of UType and VType hhbyyh Author: Xiangrui Meng m...@databricks.com Closes #8140 from mengxr/SPARK-9912. (cherry picked

spark git commit: [SPARK-9449] [SQL] Include MetastoreRelation's inputFiles

2015-08-12 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master fc1c7fd66 - 660e6dcff [SPARK-9449] [SQL] Include MetastoreRelation's inputFiles Author: Michael Armbrust mich...@databricks.com Closes #8119 from marmbrus/metastoreInputFiles. Project: http://git-wip-us.apache.org/repos/asf/spark/repo

spark git commit: [SPARK-9915] [ML] stopWords should use StringArrayParam

2015-08-12 Thread meng
Repository: spark Updated Branches: refs/heads/branch-1.5 31b7fdc06 - ed73f5439 [SPARK-9915] [ML] stopWords should use StringArrayParam hhbyyh Author: Xiangrui Meng m...@databricks.com Closes #8141 from mengxr/SPARK-9915. (cherry picked from commit fc1c7fd66e64ccea53b31cd2fbb98bc6d307329c)

spark git commit: [SPARK-9724] [WEB UI] Avoid unnecessary redirects in the Spark Web UI.

2015-08-12 Thread vanzin
Repository: spark Updated Branches: refs/heads/master 8ce60963c - 0d1d146c2 [SPARK-9724] [WEB UI] Avoid unnecessary redirects in the Spark Web UI. Author: Rohit Agarwal roh...@qubole.com Closes #8014 from mindprince/SPARK-9724 and squashes the following commits: a7af5ff [Rohit Agarwal]

spark git commit: [SPARK-9827] [SQL] fix fd leak in UnsafeRowSerializer

2015-08-12 Thread rxin
Repository: spark Updated Branches: refs/heads/master 7b13ed27c - 7c35746c9 [SPARK-9827] [SQL] fix fd leak in UnsafeRowSerializer Currently, UnsafeRowSerializer does not close the InputStream, will cause fd leak if the InputStream has an open fd in it. TODO: the fd could still be leaked, if

spark git commit: [SPARK-9827] [SQL] fix fd leak in UnsafeRowSerializer

2015-08-12 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-1.5 4b547b91d - eebb3f945 [SPARK-9827] [SQL] fix fd leak in UnsafeRowSerializer Currently, UnsafeRowSerializer does not close the InputStream, will cause fd leak if the InputStream has an open fd in it. TODO: the fd could still be

spark git commit: [SPARK-9908] [SQL] When spark.sql.tungsten.enabled is false, broadcast join does not work

2015-08-12 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-1.5 eebb3f945 - 71ea61f90 [SPARK-9908] [SQL] When spark.sql.tungsten.enabled is false, broadcast join does not work https://issues.apache.org/jira/browse/SPARK-9908 Author: Yin Huai yh...@databricks.com Closes #8149 from

spark git commit: [SPARK-9908] [SQL] When spark.sql.tungsten.enabled is false, broadcast join does not work

2015-08-12 Thread rxin
Repository: spark Updated Branches: refs/heads/master 7c35746c9 - 4413d0855 [SPARK-9908] [SQL] When spark.sql.tungsten.enabled is false, broadcast join does not work https://issues.apache.org/jira/browse/SPARK-9908 Author: Yin Huai yh...@databricks.com Closes #8149 from yhuai/SPARK-9908.

spark git commit: [SPARK-9855] [SPARKR] Add expression functions into SparkR whose params are simple

2015-08-12 Thread shivaram
Repository: spark Updated Branches: refs/heads/master 0d1d146c2 - f4bc01f1f [SPARK-9855] [SPARKR] Add expression functions into SparkR whose params are simple I added lots of expression functions for SparkR. This PR includes only functions whose params are only `(Column)` or `(Column,

spark git commit: [SPARK-9855] [SPARKR] Add expression functions into SparkR whose params are simple

2015-08-12 Thread shivaram
Repository: spark Updated Branches: refs/heads/branch-1.5 62ab2a4c6 - ca39c9e91 [SPARK-9855] [SPARKR] Add expression functions into SparkR whose params are simple I added lots of expression functions for SparkR. This PR includes only functions whose params are only `(Column)` or `(Column,

spark git commit: [SPARK-9832] [SQL] add a thread-safe lookup for BytesToBytseMap

2015-08-12 Thread rxin
Repository: spark Updated Branches: refs/heads/master 227821905 - a8ab2634c [SPARK-9832] [SQL] add a thread-safe lookup for BytesToBytseMap This patch add a thread-safe lookup for BytesToBytseMap, and use that in broadcasted HashedRelation. Author: Davies Liu dav...@databricks.com Closes

spark git commit: [SPARK-9832] [SQL] add a thread-safe lookup for BytesToBytseMap

2015-08-12 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-1.5 3b1b8ea3e - 8229437c3 [SPARK-9832] [SQL] add a thread-safe lookup for BytesToBytseMap This patch add a thread-safe lookup for BytesToBytseMap, and use that in broadcasted HashedRelation. Author: Davies Liu dav...@databricks.com

spark git commit: [SPARK-9927] [SQL] Revert 8049 since it's pushing wrong filter down

2015-08-12 Thread lian
Repository: spark Updated Branches: refs/heads/master d7eb371eb - d0b18919d [SPARK-9927] [SQL] Revert 8049 since it's pushing wrong filter down I made a mistake in #8049 by casting literal value to attribute's data type, which would cause simply truncate the literal value and push a wrong

spark git commit: [SPARK-9903] [MLLIB] skip local processing in PrefixSpan if there are no small prefixes

2015-08-12 Thread meng
Repository: spark Updated Branches: refs/heads/branch-1.5 a06860c2f - af470a757 [SPARK-9903] [MLLIB] skip local processing in PrefixSpan if there are no small prefixes There exists a chance that the prefixes keep growing to the maximum pattern length. Then the final local processing step

spark git commit: [SPARK-9920] [SQL] The simpleString of TungstenAggregate does not show its output

2015-08-12 Thread rxin
Repository: spark Updated Branches: refs/heads/master 2fb4901b7 - 227821905 [SPARK-9920] [SQL] The simpleString of TungstenAggregate does not show its output https://issues.apache.org/jira/browse/SPARK-9920 Taking `sqlContext.sql(select i, sum(j1) as sum from testAgg group by i).explain()`

spark git commit: [SPARK-9917] [ML] add getMin/getMax and doc for originalMin/origianlMax in MinMaxScaler

2015-08-12 Thread meng
Repository: spark Updated Branches: refs/heads/branch-1.5 8229437c3 - 16f4bf4ca [SPARK-9917] [ML] add getMin/getMax and doc for originalMin/origianlMax in MinMaxScaler hhbyyh Author: Xiangrui Meng m...@databricks.com Closes #8145 from mengxr/SPARK-9917. (cherry picked from commit

spark git commit: [SPARK-9917] [ML] add getMin/getMax and doc for originalMin/origianlMax in MinMaxScaler

2015-08-12 Thread meng
Repository: spark Updated Branches: refs/heads/master a8ab2634c - 5fc058a1f [SPARK-9917] [ML] add getMin/getMax and doc for originalMin/origianlMax in MinMaxScaler hhbyyh Author: Xiangrui Meng m...@databricks.com Closes #8145 from mengxr/SPARK-9917. Project:

spark git commit: [SPARK-8922] [DOCUMENTATION, MLLIB] Add @since tags to mllib.evaluation

2015-08-12 Thread meng
Repository: spark Updated Branches: refs/heads/master 5fc058a1f - df5438921 [SPARK-8922] [DOCUMENTATION, MLLIB] Add @since tags to mllib.evaluation Author: shikai.tang tar.sk...@gmail.com Closes #7429 from mosessky/master. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [SPARK-8922] [DOCUMENTATION, MLLIB] Add @since tags to mllib.evaluation

2015-08-12 Thread meng
Repository: spark Updated Branches: refs/heads/branch-1.5 8f055e595 - 690284037 [SPARK-8922] [DOCUMENTATION, MLLIB] Add @since tags to mllib.evaluation Author: shikai.tang tar.sk...@gmail.com Closes #7429 from mosessky/master. (cherry picked from commit

spark git commit: [SPARK-9903] [MLLIB] skip local processing in PrefixSpan if there are no small prefixes

2015-08-12 Thread meng
Repository: spark Updated Branches: refs/heads/master d2d5e7fe2 - d7053bea9 [SPARK-9903] [MLLIB] skip local processing in PrefixSpan if there are no small prefixes There exists a chance that the prefixes keep growing to the maximum pattern length. Then the final local processing step

spark git commit: [SPARK-9916] [BUILD] [SPARKR] removed left-over sparkr.zip copy/create commands from codebase

2015-08-12 Thread shivaram
Repository: spark Updated Branches: refs/heads/master d7053bea9 - 2fb4901b7 [SPARK-9916] [BUILD] [SPARKR] removed left-over sparkr.zip copy/create commands from codebase sparkr.zip is now built by SparkSubmit on a need-to-build basis. cc shivaram Author: Burak Yavuz brk...@gmail.com

spark git commit: [SPARK-9916] [BUILD] [SPARKR] removed left-over sparkr.zip copy/create commands from codebase

2015-08-12 Thread shivaram
Repository: spark Updated Branches: refs/heads/branch-1.5 af470a757 - 3d1b9f007 [SPARK-9916] [BUILD] [SPARKR] removed left-over sparkr.zip copy/create commands from codebase sparkr.zip is now built by SparkSubmit on a need-to-build basis. cc shivaram Author: Burak Yavuz brk...@gmail.com

spark git commit: [SPARK-9920] [SQL] The simpleString of TungstenAggregate does not show its output

2015-08-12 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-1.5 3d1b9f007 - 3b1b8ea3e [SPARK-9920] [SQL] The simpleString of TungstenAggregate does not show its output https://issues.apache.org/jira/browse/SPARK-9920 Taking `sqlContext.sql(select i, sum(j1) as sum from testAgg group by

spark git commit: [SPARK-9914] [ML] define setters explicitly for Java and use setParam group in RFormula

2015-08-12 Thread meng
Repository: spark Updated Branches: refs/heads/master df5438921 - d7eb371eb [SPARK-9914] [ML] define setters explicitly for Java and use setParam group in RFormula The problem with defining setters in the base class is that it doesn't return the correct type in Java. ericl Author:

spark git commit: [SPARK-9870] Disable driver UI and Master REST server in SparkSubmitSuite

2015-08-12 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master f4bc01f1f - 7b13ed27c [SPARK-9870] Disable driver UI and Master REST server in SparkSubmitSuite I think that we should pass additional configuration flags to disable the driver UI and Master REST server in SparkSubmitSuite and

spark git commit: [SPARK-9870] Disable driver UI and Master REST server in SparkSubmitSuite

2015-08-12 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.5 ca39c9e91 - 4b547b91d [SPARK-9870] Disable driver UI and Master REST server in SparkSubmitSuite I think that we should pass additional configuration flags to disable the driver UI and Master REST server in SparkSubmitSuite and

[2/2] spark git commit: Preparing development version 1.5.0-SNAPSHOT

2015-08-12 Thread pwendell
Preparing development version 1.5.0-SNAPSHOT Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/8f055e59 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/8f055e59 Diff:

Git Push Summary

2015-08-12 Thread pwendell
Repository: spark Updated Tags: refs/tags/v1.5.0-preview-20150812 [created] cedce9bdb - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org

[1/2] spark git commit: Preparing Spark release v1.5.0-preview-20150812

2015-08-12 Thread pwendell
Repository: spark Updated Branches: refs/heads/branch-1.5 16f4bf4ca - 8f055e595 Preparing Spark release v1.5.0-preview-20150812 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/cedce9bd Tree: http://git-wip-us.apache.org

spark git commit: [SPARK-9927] [SQL] Revert 8049 since it's pushing wrong filter down

2015-08-12 Thread lian
Repository: spark Updated Branches: refs/heads/branch-1.5 690284037 - 694e7a3c4 [SPARK-9927] [SQL] Revert 8049 since it's pushing wrong filter down I made a mistake in #8049 by casting literal value to attribute's data type, which would cause simply truncate the literal value and push a

spark git commit: [SPARK-9806] [WEB UI] Don't share ReplayListenerBus between multiple applications

2015-08-12 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master b85f9a242 - a807fcbe5 [SPARK-9806] [WEB UI] Don't share ReplayListenerBus between multiple applications Author: Rohit Agarwal roh...@qubole.com Closes #8088 from mindprince/SPARK-9806. Project:

spark git commit: [SPARK-9806] [WEB UI] Don't share ReplayListenerBus between multiple applications

2015-08-12 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.5 2f909184e - 402c0ca9a [SPARK-9806] [WEB UI] Don't share ReplayListenerBus between multiple applications Author: Rohit Agarwal roh...@qubole.com Closes #8088 from mindprince/SPARK-9806. (cherry picked from commit

spark git commit: [SPARK-9829] [WEBUI] Display the update value for peak execution memory

2015-08-12 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master a807fcbe5 - 4e3f4b934 [SPARK-9829] [WEBUI] Display the update value for peak execution memory The peak execution memory is not correct because it shows the sum of finished tasks' values when a task finishes. This PR fixes it by using the

spark git commit: [SPARK-9829] [WEBUI] Display the update value for peak execution memory

2015-08-12 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.5 402c0ca9a - d9d4bdea2 [SPARK-9829] [WEBUI] Display the update value for peak execution memory The peak execution memory is not correct because it shows the sum of finished tasks' values when a task finishes. This PR fixes it by using

spark git commit: [SPARK-9575] [MESOS] Add docuemntation around Mesos shuffle service.

2015-08-12 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 5c99d8bf9 - 741a29f98 [SPARK-9575] [MESOS] Add docuemntation around Mesos shuffle service. andrewor14 Author: Timothy Chen tnac...@gmail.com Closes #7907 from tnachen/mesos_shuffle. Project:

spark git commit: [SPARK-8366] maxNumExecutorsNeeded should properly handle failed tasks

2015-08-12 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.5 b994f8928 - 2f909184e [SPARK-8366] maxNumExecutorsNeeded should properly handle failed tasks Author: xutingjun xuting...@huawei.com Author: meiyoula 1039320...@qq.com Closes #6817 from XuTingjun/SPARK-8366. (cherry picked from commit

spark git commit: [SPARK-8366] maxNumExecutorsNeeded should properly handle failed tasks

2015-08-12 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master b1581ac28 - b85f9a242 [SPARK-8366] maxNumExecutorsNeeded should properly handle failed tasks Author: xutingjun xuting...@huawei.com Author: meiyoula 1039320...@qq.com Closes #6817 from XuTingjun/SPARK-8366. Project:

spark git commit: [SPARK-8798] [MESOS] Allow additional uris to be fetched with mesos

2015-08-12 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.5 93fc95934 - a2f805729 [SPARK-8798] [MESOS] Allow additional uris to be fetched with mesos Some users like to download additional files in their sandbox that they can refer to from their spark program, or even later mount these files

spark git commit: [SPARK-8798] [MESOS] Allow additional uris to be fetched with mesos

2015-08-12 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master bab892328 - 5c99d8bf9 [SPARK-8798] [MESOS] Allow additional uris to be fetched with mesos Some users like to download additional files in their sandbox that they can refer to from their spark program, or even later mount these files to

spark git commit: [SPARK-9806] [WEB UI] Don't share ReplayListenerBus between multiple applications

2015-08-12 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.4 6dde38026 - 89c8aea94 [SPARK-9806] [WEB UI] Don't share ReplayListenerBus between multiple applications Author: Rohit Agarwal roh...@qubole.com Closes #8088 from mindprince/SPARK-9806. (cherry picked from commit

spark git commit: [SPARK-9426] [WEBUI] Job page DAG visualization is not shown

2015-08-12 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 4e3f4b934 - bab892328 [SPARK-9426] [WEBUI] Job page DAG visualization is not shown To reproduce the issue, go to the stage page and click DAG Visualization once, then go to the job page to show the job DAG visualization. You will only see

spark git commit: [SPARK-9426] [WEBUI] Job page DAG visualization is not shown

2015-08-12 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.5 d9d4bdea2 - 93fc95934 [SPARK-9426] [WEBUI] Job page DAG visualization is not shown To reproduce the issue, go to the stage page and click DAG Visualization once, then go to the job page to show the job DAG visualization. You will only

spark git commit: [SPARK-9795] Dynamic allocation: avoid double counting when killing same executor twice

2015-08-12 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.5 0579f28df - bc4ac65d4 [SPARK-9795] Dynamic allocation: avoid double counting when killing same executor twice This is based on KaiXinXiaoLei's changes in #7716. The issue is that when someone calls `sc.killExecutor(1)` on the same

spark git commit: [SPARK-7583] [MLLIB] User guide update for RegexTokenizer

2015-08-12 Thread meng
Repository: spark Updated Branches: refs/heads/branch-1.5 bc4ac65d4 - 2d86faddd [SPARK-7583] [MLLIB] User guide update for RegexTokenizer jira: https://issues.apache.org/jira/browse/SPARK-7583 User guide update for RegexTokenizer Author: Yuhao Yang hhb...@gmail.com Closes #7828 from

spark git commit: [SPARK-7583] [MLLIB] User guide update for RegexTokenizer

2015-08-12 Thread meng
Repository: spark Updated Branches: refs/heads/master be5d19120 - 66d87c1d7 [SPARK-7583] [MLLIB] User guide update for RegexTokenizer jira: https://issues.apache.org/jira/browse/SPARK-7583 User guide update for RegexTokenizer Author: Yuhao Yang hhb...@gmail.com Closes #7828 from

spark git commit: [SPARK-9747] [SQL] Avoid starving an unsafe operator in aggregation

2015-08-12 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-1.5 2d86faddd - 4c6b1296d [SPARK-9747] [SQL] Avoid starving an unsafe operator in aggregation This is the sister patch to #8011, but for aggregation. In a nutshell: create the `TungstenAggregationIterator` before computing the parent

spark git commit: [SPARK-9747] [SQL] Avoid starving an unsafe operator in aggregation

2015-08-12 Thread rxin
Repository: spark Updated Branches: refs/heads/master 66d87c1d7 - e0110792e [SPARK-9747] [SQL] Avoid starving an unsafe operator in aggregation This is the sister patch to #8011, but for aggregation. In a nutshell: create the `TungstenAggregationIterator` before computing the parent

spark git commit: [SPARK-9407] [SQL] Relaxes Parquet ValidTypeMap to allow ENUM predicates to be pushed down

2015-08-12 Thread lian
Repository: spark Updated Branches: refs/heads/master 9d0822455 - 3ecb37943 [SPARK-9407] [SQL] Relaxes Parquet ValidTypeMap to allow ENUM predicates to be pushed down This PR adds a hacky workaround for PARQUET-201, and should be removed once we upgrade to parquet-mr 1.8.1 or higher

spark git commit: [SPARK-9182] [SQL] Filters are not passed through to jdbc source

2015-08-12 Thread lian
Repository: spark Updated Branches: refs/heads/master 741a29f98 - 9d0822455 [SPARK-9182] [SQL] Filters are not passed through to jdbc source This PR fixes unable to push filter down to JDBC source caused by `Cast` during pattern matching. While we are comparing columns of different type,

spark git commit: [SPARK-9182] [SQL] Filters are not passed through to jdbc source

2015-08-12 Thread lian
Repository: spark Updated Branches: refs/heads/branch-1.5 5dd0c5cd6 - 8e32db9a5 [SPARK-9182] [SQL] Filters are not passed through to jdbc source This PR fixes unable to push filter down to JDBC source caused by `Cast` during pattern matching. While we are comparing columns of different

spark git commit: [SPARK-9407] [SQL] Relaxes Parquet ValidTypeMap to allow ENUM predicates to be pushed down

2015-08-12 Thread lian
Repository: spark Updated Branches: refs/heads/branch-1.5 8e32db9a5 - 5e6fdc659 [SPARK-9407] [SQL] Relaxes Parquet ValidTypeMap to allow ENUM predicates to be pushed down This PR adds a hacky workaround for PARQUET-201, and should be removed once we upgrade to parquet-mr 1.8.1 or higher

spark git commit: [SPARK-9847] [ML] Modified copyValues to distinguish between default, explicit param values

2015-08-12 Thread meng
Repository: spark Updated Branches: refs/heads/master 57ec27dd7 - 70fe55886 [SPARK-9847] [ML] Modified copyValues to distinguish between default, explicit param values From JIRA: Currently, Params.copyValues copies default parameter values to the paramMap of the target instance, rather than

spark git commit: [SPARK-9726] [PYTHON] PySpark DF join no longer accepts on=None

2015-08-12 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-1.5 b515f890d - 8629c33b6 [SPARK-9726] [PYTHON] PySpark DF join no longer accepts on=None rxin First pull request for Spark so let me know if I am missing anything The contribution is my original work and I license the work to the project

spark git commit: [SPARK-9726] [PYTHON] PySpark DF join no longer accepts on=None

2015-08-12 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 70fe55886 - 60103ecd3 [SPARK-9726] [PYTHON] PySpark DF join no longer accepts on=None rxin First pull request for Spark so let me know if I am missing anything The contribution is my original work and I license the work to the project

spark git commit: [SPARK-9804] [HIVE] Use correct value for isSrcLocal parameter.

2015-08-12 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-1.5 4c6b1296d - e9641f192 [SPARK-9804] [HIVE] Use correct value for isSrcLocal parameter. If the correct parameter is not provided, Hive will run into an error because it calls methods that are specific to the local filesystem to copy the