spark git commit: [SPARK-4920][UI] add version on master and worker page for standalone mode

2014-12-30 Thread joshrosen
Repository: spark Updated Branches: refs/heads/master 040d6f2d1 - 9077e721c [SPARK-4920][UI] add version on master and worker page for standalone mode Author: Zhang, Liye liye.zh...@intel.com Closes #3769 from liyezhang556520/spark-4920_WebVersion and squashes the following commits:

spark git commit: [SPARK-4882] Register PythonBroadcast with Kryo so that PySpark works with KryoSerializer

2014-12-30 Thread joshrosen
Repository: spark Updated Branches: refs/heads/master 9077e721c - efa80a531 [SPARK-4882] Register PythonBroadcast with Kryo so that PySpark works with KryoSerializer This PR fixes an issue where PySpark broadcast variables caused NullPointerExceptions if KryoSerializer was used. The fix is

spark git commit: [SPARK-4882] Register PythonBroadcast with Kryo so that PySpark works with KryoSerializer

2014-12-30 Thread joshrosen
Repository: spark Updated Branches: refs/heads/branch-1.1 d5e0a45ed - 822a0b42f [SPARK-4882] Register PythonBroadcast with Kryo so that PySpark works with KryoSerializer This PR fixes an issue where PySpark broadcast variables caused NullPointerExceptions if KryoSerializer was used. The

spark git commit: [SPARK-4882] Register PythonBroadcast with Kryo so that PySpark works with KryoSerializer

2014-12-30 Thread joshrosen
Repository: spark Updated Branches: refs/heads/branch-1.2 e20d63242 - 42809db9b [SPARK-4882] Register PythonBroadcast with Kryo so that PySpark works with KryoSerializer This PR fixes an issue where PySpark broadcast variables caused NullPointerExceptions if KryoSerializer was used. The

spark git commit: Revert [SPARK-4882] Register PythonBroadcast with Kryo so that PySpark works with KryoSerializer

2014-12-30 Thread joshrosen
Repository: spark Updated Branches: refs/heads/branch-1.1 822a0b42f - d6b8d2c03 Revert [SPARK-4882] Register PythonBroadcast with Kryo so that PySpark works with KryoSerializer This reverts commit 822a0b42f79acea2771d0b298e803c11c37aab81. This fix does not apply to branch-1.1 or branch-1.0,

spark git commit: [SPARK-4908][SQL] Prevent multiple concurrent hive native commands

2014-12-30 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master efa80a531 - 480bd1d2e [SPARK-4908][SQL] Prevent multiple concurrent hive native commands This is just a quick fix that locks when calling `runHive`. If we can find a way to avoid the error without a global lock that would be better.

spark git commit: [SPARK-4908][SQL] Prevent multiple concurrent hive native commands

2014-12-30 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-1.2 42809db9b - cde8a310a [SPARK-4908][SQL] Prevent multiple concurrent hive native commands This is just a quick fix that locks when calling `runHive`. If we can find a way to avoid the error without a global lock that would be better.

spark git commit: [SQL] enable view test

2014-12-30 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 480bd1d2e - 94d60b702 [SQL] enable view test This is a follow up of #3396 , just add a test to white list. Author: Daoyuan Wang daoyuan.w...@intel.com Closes #3826 from adrian-wang/viewtest and squashes the following commits: f105f68

spark git commit: [SPARK-4975][SQL] Fix HiveInspectorSuite test failure

2014-12-30 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 94d60b702 - 65357f11c [SPARK-4975][SQL] Fix HiveInspectorSuite test failure HiveInspectorSuite test failure： [info] - wrap / unwrap null, constant null and writables *** FAILED *** (21 milliseconds) [info] 1 did not equal 0

spark git commit: [SPARK-4959] [SQL] Attributes are case sensitive when using a select query from a projection

2014-12-30 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 65357f11c - 5595eaa74 [SPARK-4959] [SQL] Attributes are case sensitive when using a select query from a projection Author: Cheng Hao hao.ch...@intel.com Closes #3796 from chenghao-intel/spark_4959 and squashes the following commits:

spark git commit: [SPARK-4904] [SQL] Remove the unnecessary code change in Generic UDF

2014-12-30 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 5595eaa74 - 63b84b7d6 [SPARK-4904] [SQL] Remove the unnecessary code change in Generic UDF Since #3429 has been merged, the bug of wrapping to Writable for HiveGenericUDF is resolved, we can safely remove the foldable checking in

spark git commit: [SPARK-5002][SQL] Using ascending by default when not specify order in order by

2014-12-30 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 63b84b7d6 - daac22130 [SPARK-5002][SQL] Using ascending by default when not specify order in order by spark sql does not support ```SELECT a, b FROM testData2 ORDER BY a desc, b```. Author: wangfei wangf...@huawei.com Closes #3838 from

spark git commit: [Spark-4512] [SQL] Unresolved Attribute Exception in Sort By

2014-12-30 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master daac22130 - 53f0a00b6 [Spark-4512] [SQL] Unresolved Attribute Exception in Sort By It will cause exception while do query like: SELECT key+key FROM src sort by value; Author: Cheng Hao hao.ch...@intel.com Closes #3386 from

spark git commit: [SPARK-4493][SQL] Tests for IsNull / IsNotNull in the ParquetFilterSuite

2014-12-30 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 53f0a00b6 - 19a8802e7 [SPARK-4493][SQL] Tests for IsNull / IsNotNull in the ParquetFilterSuite This is a follow-up of #3367 and #3644. At the time #3644 was written, #3367 hadn't been merged yet, thus `IsNull` and `IsNotNull` filters are

spark git commit: [SPARK-4916][SQL][DOCS]Update SQL programming guide about cache section

2014-12-30 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 19a8802e7 - f7a41a0e7 [SPARK-4916][SQL][DOCS]Update SQL programming guide about cache section `SchemeRDD.cache()` now uses in-memory columnar storage. Author: luogankun luogan...@gmail.com Closes #3759 from luogankun/SPARK-4916 and

spark git commit: [SPARK-4930][SQL][DOCS]Update SQL programming guide, CACHE TABLE is eager

2014-12-30 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master f7a41a0e7 - 2deac748b [SPARK-4930][SQL][DOCS]Update SQL programming guide, CACHE TABLE is eager `CACHE TABLE tbl` is now __eager__ by default not __lazy__ Author: luogankun luogan...@gmail.com Closes #3773 from luogankun/SPARK-4930 and

spark git commit: [SPARK-4928][SQL] Fix: Operator ', , =, =' with decimal between different precision report error

2014-12-30 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 2deac748b - a75dd83b7 [SPARK-4928][SQL] Fix: Operator ',,=,=' with decimal between different precision report error case operator with decimal between different precision, we need change them to unlimited Author: guowei2

spark git commit: [SPARK-4937][SQL] Normalizes conjunctions and disjunctions to eliminate common predicates

2014-12-30 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master a75dd83b7 - 61a99f6a1 [SPARK-4937][SQL] Normalizes conjunctions and disjunctions to eliminate common predicates This PR is a simplified version of several filter optimization rules introduced in #3778 authored by scwf. Newly introduced

spark git commit: [SPARK-4386] Improve performance when writing Parquet files

2014-12-30 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 61a99f6a1 - 7425bec32 [SPARK-4386] Improve performance when writing Parquet files Convert type of RowWriteSupport.attributes to Array. Analysis of performance for writing very wide tables shows that time is spent predominantly in apply

spark git commit: [SPARK-4386] Improve performance when writing Parquet files

2014-12-30 Thread marmbrus
Repository: spark Updated Branches: refs/heads/branch-1.2 cde8a310a - 7a245412f [SPARK-4386] Improve performance when writing Parquet files Convert type of RowWriteSupport.attributes to Array. Analysis of performance for writing very wide tables shows that time is spent predominantly in

spark git commit: [SPARK-4935][SQL] When hive.cli.print.header configured, spark-sql aborted if passed in a invalid sql

2014-12-30 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 7425bec32 - 8f29b7caf [SPARK-4935][SQL] When hive.cli.print.header configured, spark-sql aborted if passed in a invalid sql If we passed in a wrong sql like ```abdcdfsfs```, the spark-sql script aborted. Author: wangfei

spark git commit: [SPARK-4570][SQL]add BroadcastLeftSemiJoinHash

2014-12-30 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 8f29b7caf - 07fa1910d [SPARK-4570][SQL]add BroadcastLeftSemiJoinHash JIRA issue: [SPARK-4570](https://issues.apache.org/jira/browse/SPARK-4570) We are planning to create a `BroadcastLeftSemiJoinHash` to implement the broadcast join for

svn commit: r1648609 - in /spark: _layouts/ site/ site/graphx/ site/mllib/ site/news/ site/releases/ site/screencasts/ site/sql/ site/streaming/

2014-12-30 Thread pwendell
Author: pwendell Date: Tue Dec 30 22:18:56 2014 New Revision: 1648609 URL: http://svn.apache.org/r1648609 Log: Adding external package index to website Modified: spark/_layouts/global.html spark/site/community.html spark/site/documentation.html spark/site/downloads.html

spark git commit: [Spark-4995] Replace Vector.toBreeze.activeIterator with foreachActive

2014-12-30 Thread meng
Repository: spark Updated Branches: refs/heads/master b239ea1c3 - 0f31992c6 [Spark-4995] Replace Vector.toBreeze.activeIterator with foreachActive New foreachActive method of vector was introduced by SPARK-4431 as more efficient alternative to vector.toBreeze.activeIterator. There are some

spark git commit: [SPARK-4813][Streaming] Fix the issue that ContextWaiter didn't handle 'spurious wakeup'

2014-12-30 Thread tdas
Repository: spark Updated Branches: refs/heads/branch-1.2 7a245412f - edc96d81d [SPARK-4813][Streaming] Fix the issue that ContextWaiter didn't handle 'spurious wakeup' Used `Condition` to rewrite `ContextWaiter` because it provides a convenient API `awaitNanos` for timeout. Author:

spark git commit: [SPARK-4813][Streaming] Fix the issue that ContextWaiter didn't handle 'spurious wakeup'

2014-12-30 Thread tdas
Repository: spark Updated Branches: refs/heads/master 0f31992c6 - 6a8978294 [SPARK-4813][Streaming] Fix the issue that ContextWaiter didn't handle 'spurious wakeup' Used `Condition` to rewrite `ContextWaiter` because it provides a convenient API `awaitNanos` for timeout. Author: zsxwing

spark git commit: [SPARK-4813][Streaming] Fix the issue that ContextWaiter didn't handle 'spurious wakeup'

2014-12-30 Thread tdas
Repository: spark Updated Branches: refs/heads/branch-1.0 78157d494 - f47e162b9 [SPARK-4813][Streaming] Fix the issue that ContextWaiter didn't handle 'spurious wakeup' Used `Condition` to rewrite `ContextWaiter` because it provides a convenient API `awaitNanos` for timeout. Author:

spark git commit: [SPARK-4813][Streaming] Fix the issue that ContextWaiter didn't handle 'spurious wakeup'

2014-12-30 Thread tdas
Repository: spark Updated Branches: refs/heads/branch-1.1 d6b8d2c03 - eac740e9a [SPARK-4813][Streaming] Fix the issue that ContextWaiter didn't handle 'spurious wakeup' Used `Condition` to rewrite `ContextWaiter` because it provides a convenient API `awaitNanos` for timeout. Author:

spark git commit: [SPARK-4998][MLlib]delete the train function

2014-12-30 Thread meng
Repository: spark Updated Branches: refs/heads/master 6a8978294 - 035bac88c [SPARK-4998][MLlib]delete the train function To make the functions with the same in object effective, specially when using java reflection. As the train function defined in class DecisionTree will hide the functions

spark git commit: [SPARK-1010] Clean up uses of System.setProperty in unit tests

2014-12-30 Thread joshrosen
Repository: spark Updated Branches: refs/heads/master 035bac88c - 352ed6bbe [SPARK-1010] Clean up uses of System.setProperty in unit tests Several of our tests call System.setProperty (or test code which implicitly sets system properties) and don't always reset/clear the modified properties,