spark git commit: [SPARK-5402] log executor ID at executor-construction time

2015-01-25 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.2 1f8b71868 - c573af4f1 [SPARK-5402] log executor ID at executor-construction time also rename slaveHostname to executorHostname Author: Ryan Williams ryan.blake.willi...@gmail.com Closes #4195 from ryan-williams/exec and squashes the

spark git commit: Revert [SPARK-5344][WebUI] HistoryServer cannot recognize that inprogress file was renamed to completed file

2015-01-25 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.2 8f55beeb5 - 2a2da4271 Revert [SPARK-5344][WebUI] HistoryServer cannot recognize that inprogress file was renamed to completed file This reverts commit 8f55beeb51e6ea72e63af3f276497f61dd24d09b. Project:

spark git commit: [SPARK-5326] Show fetch wait time as optional metric in the UI

2015-01-25 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 8f5c827b0 - fc2168f04 [SPARK-5326] Show fetch wait time as optional metric in the UI With this change, here's what the UI looks like:

spark git commit: [CORE] The descriptionof jobHistory config should be spark.history.fs.logDirectory

2015-04-02 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.3 0c1c0fb90 - 17ab6b0cf [CORE] The descriptionof jobHistory config should be spark.history.fs.logDirectory The config option is spark.history.fs.logDirectory, not spark.fs.history.logDirectory. So the descriptionof should be changed.

spark git commit: [CORE] The descriptionof jobHistory config should be spark.history.fs.logDirectory

2015-04-02 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.2 2991dd042 - f4a9c41b1 [CORE] The descriptionof jobHistory config should be spark.history.fs.logDirectory The config option is spark.history.fs.logDirectory, not spark.fs.history.logDirectory. So the descriptionof should be changed.

spark git commit: [CORE] The descriptionof jobHistory config should be spark.history.fs.logDirectory

2015-04-02 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 4b82bd730 - 8a0aa81ca [CORE] The descriptionof jobHistory config should be spark.history.fs.logDirectory The config option is spark.history.fs.logDirectory, not spark.fs.history.logDirectory. So the descriptionof should be changed.

spark git commit: [SPARK-6650] [core] Stop ExecutorAllocationManager when context stops.

2015-04-02 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 052dee070 - 45134ec92 [SPARK-6650] [core] Stop ExecutorAllocationManager when context stops. This fixes the thread leak. I also changed the unit test to keep track of allocated contexts and make sure they're closed after tests are run;

spark git commit: [SPARK-6650] [core] Stop ExecutorAllocationManager when context stops.

2015-04-02 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.3 2927af102 - 0ef46b2d8 [SPARK-6650] [core] Stop ExecutorAllocationManager when context stops. This fixes the thread leak. I also changed the unit test to keep track of allocated contexts and make sure they're closed after tests are run;

spark git commit: [SPARK-6688] [core] Always use resolved URIs in EventLoggingListener.

2015-04-03 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master ffe8cc9a2 - 14632b794 [SPARK-6688] [core] Always use resolved URIs in EventLoggingListener. Author: Marcelo Vanzin van...@cloudera.com Closes #5340 from vanzin/SPARK-6688 and squashes the following commits: ccfddd9 [Marcelo Vanzin]

spark git commit: [SPARK-6688] [core] Always use resolved URIs in EventLoggingListener.

2015-04-03 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.3 0c1b78b72 - f17a2fe9b [SPARK-6688] [core] Always use resolved URIs in EventLoggingListener. Author: Marcelo Vanzin van...@cloudera.com Closes #5340 from vanzin/SPARK-6688 and squashes the following commits: ccfddd9 [Marcelo Vanzin]

spark git commit: [Spark-4848] Allow different Worker configurations in standalone cluster

2015-04-13 Thread andrewor14
' into feature/spark-4848 044ca6f [Nathan Kronenfeld] Documentation and formatting as requested by by andrewor14 d739640 [Nathan Kronenfeld] Move looping through instances from the master to the workers, so that each worker respects its own number of instances and web-ui port Project: http://git-wip

spark git commit: [SPARK-5931][CORE] Use consistent naming for time properties

2015-04-13 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master c5602bdc3 - c4ab255e9 [SPARK-5931][CORE] Use consistent naming for time properties I've added new utility methods to do the conversion from times specified as e.g. 120s, 240ms, 360us to convert to a consistent internal representation.

spark git commit: [MINOR] [CORE] Warn users who try to cache RDDs with dynamic allocation on.

2015-04-28 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.2 059c39003 - d2080477d [MINOR] [CORE] Warn users who try to cache RDDs with dynamic allocation on. Author: Marcelo Vanzin van...@cloudera.com Closes #5751 from vanzin/cached-rdd-warning and squashes the following commits: 554cc07

spark git commit: [MINOR] [CORE] Warn users who try to cache RDDs with dynamic allocation on.

2015-04-28 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.3 37fb5a968 - 7e210260e [MINOR] [CORE] Warn users who try to cache RDDs with dynamic allocation on. Author: Marcelo Vanzin van...@cloudera.com Closes #5751 from vanzin/cached-rdd-warning and squashes the following commits: 554cc07

spark git commit: [MINOR] [CORE] Warn users who try to cache RDDs with dynamic allocation on.

2015-04-28 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 53befacce - 28b1af742 [MINOR] [CORE] Warn users who try to cache RDDs with dynamic allocation on. Author: Marcelo Vanzin van...@cloudera.com Closes #5751 from vanzin/cached-rdd-warning and squashes the following commits: 554cc07 [Marcelo

[2/3] spark git commit: [SPARK-5338] [MESOS] Add cluster mode support for Mesos

2015-04-28 Thread andrewor14
http://git-wip-us.apache.org/repos/asf/spark/blob/53befacc/core/src/main/scala/org/apache/spark/deploy/rest/StandaloneRestServer.scala -- diff --git a/core/src/main/scala/org/apache/spark/deploy/rest/StandaloneRestServer.scala

[3/3] spark git commit: [SPARK-5338] [MESOS] Add cluster mode support for Mesos

2015-04-28 Thread andrewor14
[SPARK-5338] [MESOS] Add cluster mode support for Mesos This patch adds the support for cluster mode to run on Mesos. It introduces a new Mesos framework dedicated to launch new apps/drivers, and can be called with the spark-submit script and specifying --master flag to the cluster mode REST

spark git commit: Revert [SPARK-6314] [CORE] handle JsonParseException for history server

2015-04-29 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.3 2254576e1 - bf4ca12f7 Revert [SPARK-6314] [CORE] handle JsonParseException for history server This reverts commit 37fb5a96812d9a8b2fb4b9492feb4c91f4f64f91. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [SPARK-2691] [MESOS] Support for Mesos DockerInfo

2015-05-01 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master b4b43df8a - 8f50a07d2 [SPARK-2691] [MESOS] Support for Mesos DockerInfo This patch adds partial support for running spark on mesos inside of a docker container. Only fine-grained mode is presently supported, and there is no checking done

spark git commit: [SPARK-7216] [MESOS] Add driver details page to Mesos cluster UI.

2015-05-01 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 099327d53 - 202219341 [SPARK-7216] [MESOS] Add driver details page to Mesos cluster UI. Add a details page that displays Mesos driver in the Mesos cluster UI Author: Timothy Chen tnac...@gmail.com Closes #5763 from

spark git commit: [SPARK-6443] [SPARK SUBMIT] Could not submit app in standalone cluster mode when HA is enabled

2015-05-01 Thread andrewor14
an exception), we use akka gateway to submit apps. I have tested simply on standalone HA cluster(with two masters alive and one alive/one dead), it worked. There might remains some issues on style or message print, but we can check the solution then fix them together. /cc srowen andrewor14 Author

spark git commit: [SPARK-6954] [YARN] ExecutorAllocationManager can end up requesting a negative n...

2015-05-01 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master ae98eec73 - 099327d53 [SPARK-6954] [YARN] ExecutorAllocationManager can end up requesting a negative n... ...umber of executors Author: Sandy Ryza sa...@cloudera.com Closes #5704 from sryza/sandy-spark-6954 and squashes the following

spark git commit: [SPARK-4286] Add an external shuffle service that can be run as a daemon.

2015-04-28 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 52ccf1d37 - 8aab94d89 [SPARK-4286] Add an external shuffle service that can be run as a daemon. This allows Mesos deployments to use the shuffle service (and implicitly dynamic allocation). It does so by adding a new main class and two

spark git commit: [SPARK-5932] [CORE] Use consistent naming for size properties

2015-04-28 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 8aab94d89 - 2d222fb39 [SPARK-5932] [CORE] Use consistent naming for size properties I've added an interface to JavaUtils to do byte conversion and added hooks within Utils.scala to handle conversion within Spark code (like for time

spark git commit: [SPARK-6314] [CORE] handle JsonParseException for history server

2015-04-28 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 2d222fb39 - 80098109d [SPARK-6314] [CORE] handle JsonParseException for history server This is handled in the same way with [SPARK-6197](https://issues.apache.org/jira/browse/SPARK-6197). The result of this PR is that exception showed in

spark git commit: [SPARK-6314] [CORE] handle JsonParseException for history server

2015-04-28 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.3 80ba82e8e - 37fb5a968 [SPARK-6314] [CORE] handle JsonParseException for history server This is handled in the same way with [SPARK-6197](https://issues.apache.org/jira/browse/SPARK-6197). The result of this PR is that exception

spark git commit: [SPARK-7373] [MESOS] Add docker support for launching drivers in mesos cluster mode.

2015-05-07 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 0c33bf817 - 4eecf550a [SPARK-7373] [MESOS] Add docker support for launching drivers in mesos cluster mode. Using the existing docker support for mesos, also enabling the mesos cluster mode scheduler to launch Spark drivers in docker

spark git commit: [SPARK-7347] DAG visualization: add tooltips to RDDs

2015-05-07 Thread andrewor14
: Andrew Or and...@databricks.com Closes #5957 from andrewor14/viz-hover2 and squashes the following commits: 60e3758 [Andrew Or] Add tooltips for RDDs on job page Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/88717ee4 Tree

spark git commit: [SPARK-7347] DAG visualization: add tooltips to RDDs

2015-05-07 Thread andrewor14
Author: Andrew Or and...@databricks.com Closes #5957 from andrewor14/viz-hover2 and squashes the following commits: 60e3758 [Andrew Or] Add tooltips for RDDs on job page (cherry picked from commit 88717ee4e7542ac8d5d2e5756c912dd390b37e88) Signed-off-by: Andrew Or and...@databricks.com Project

spark git commit: [SPARK-7399] [SPARK CORE] Fixed compilation error in scala 2.11

2015-05-07 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.4 2b0c42385 - d4e31bfcd [SPARK-7399] [SPARK CORE] Fixed compilation error in scala 2.11 scala has deterministic naming-scheme for the generated methods which return default arguments . here one of the default argument of overloaded

spark git commit: [HOT FIX] [SPARK-7418] Ignore flaky SparkSubmitUtilsSuite test

2015-05-06 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 1ad04dae0 - fbf1f342a [HOT FIX] [SPARK-7418] Ignore flaky SparkSubmitUtilsSuite test Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/fbf1f342 Tree:

spark git commit: [HOT FIX] [SPARK-7418] Ignore flaky SparkSubmitUtilsSuite test

2015-05-06 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.4 b681b9312 - c0ec20a51 [HOT FIX] [SPARK-7418] Ignore flaky SparkSubmitUtilsSuite test Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c0ec20a5 Tree:

[1/3] spark git commit: [SPARK-7371] [SPARK-7377] [SPARK-7408] DAG visualization addendum (#5729)

2015-05-06 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.4 ba24dfae7 - 76e8344f2 http://git-wip-us.apache.org/repos/asf/spark/blob/76e8344f/core/src/main/resources/org/apache/spark/ui/static/spark-dag-viz.css -- diff --git

[1/3] spark git commit: [SPARK-7371] [SPARK-7377] [SPARK-7408] DAG visualization addendum (#5729)

2015-05-06 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 316a5c042 - 8fa6829f5 http://git-wip-us.apache.org/repos/asf/spark/blob/8fa6829f/core/src/main/resources/org/apache/spark/ui/static/spark-dag-viz.css -- diff --git

[3/3] spark git commit: [SPARK-7371] [SPARK-7377] [SPARK-7408] DAG visualization addendum (#5729)

2015-05-06 Thread andrewor14
on stage page as requested by mateiz pwendell This patch also includes general code clean up. img src=https://issues.apache.org/jira/secure/attachment/12730992/before-after.png; width=500px/img Author: Andrew Or and...@databricks.com Closes #5954 from andrewor14/viz-emphasize-rdd and squashes

[6/7] spark git commit: [SPARK-6943] [SPARK-6944] DAG visualization on SparkUI

2015-05-04 Thread andrewor14
http://git-wip-us.apache.org/repos/asf/spark/blob/fc8b5819/core/src/main/resources/org/apache/spark/ui/static/d3.min.js -- diff --git a/core/src/main/resources/org/apache/spark/ui/static/d3.min.js

[5/7] spark git commit: [SPARK-6943] [SPARK-6944] DAG visualization on SparkUI

2015-05-04 Thread andrewor14
/resources/org/apache/spark/ui/static/dagre-d3.min.js new file mode 100644 index 000..6d2da25 --- /dev/null +++ b/core/src/main/resources/org/apache/spark/ui/static/dagre-d3.min.js @@ -0,0 +1,29 @@ +/*v0.4.3 with 1 additional commit (see http://github.com/andrewor14/dagre-d3)*/(function(f){if(typeof

[1/7] spark git commit: [SPARK-6943] [SPARK-6944] DAG visualization on SparkUI

2015-05-04 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master f32e69ecc - fc8b58195 http://git-wip-us.apache.org/repos/asf/spark/blob/fc8b5819/core/src/main/scala/org/apache/spark/util/JsonProtocol.scala -- diff --git

[4/7] spark git commit: [SPARK-6943] [SPARK-6944] DAG visualization on SparkUI

2015-05-04 Thread andrewor14
http://git-wip-us.apache.org/repos/asf/spark/blob/863ec0cb/core/src/main/resources/org/apache/spark/ui/static/graphlib-dot.min.js -- diff --git a/core/src/main/resources/org/apache/spark/ui/static/graphlib-dot.min.js

[5/7] spark git commit: [SPARK-6943] [SPARK-6944] DAG visualization on SparkUI

2015-05-04 Thread andrewor14
/resources/org/apache/spark/ui/static/dagre-d3.min.js new file mode 100644 index 000..6d2da25 --- /dev/null +++ b/core/src/main/resources/org/apache/spark/ui/static/dagre-d3.min.js @@ -0,0 +1,29 @@ +/*v0.4.3 with 1 additional commit (see http://github.com/andrewor14/dagre-d3)*/(function(f){if(typeof

[7/7] spark git commit: [SPARK-6943] [SPARK-6944] DAG visualization on SparkUI

2015-05-04 Thread andrewor14
Or and...@databricks.com Closes #5729 from andrewor14/viz2 and squashes the following commits: 666c03b [Andrew Or] Round corners of RDD boxes on stage page (minor) 01ba336 [Andrew Or] Change RDD cache color to red (minor) 6f9574a [Andrew Or] Add tests for RDDOperationScope 1c310e4 [Andrew Or] Wrap a few more RDD

[3/7] spark git commit: [SPARK-6943] [SPARK-6944] DAG visualization on SparkUI

2015-05-04 Thread andrewor14
, + * and its input data is expected to be populated in div#dag-viz-metadata + * by Spark's UI code. This is currently used only on the stage page and on + * the job page. + * + * This requires jQuery, d3, and dagre-d3. Note that we use a custom release + * of dagre-d3 (http://github.com/andrewor14/dagre

[1/7] spark git commit: [SPARK-6943] [SPARK-6944] DAG visualization on SparkUI

2015-05-04 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.4 34edaa8ac - 863ec0cb4 http://git-wip-us.apache.org/repos/asf/spark/blob/863ec0cb/core/src/main/scala/org/apache/spark/util/JsonProtocol.scala -- diff --git

[6/7] spark git commit: [SPARK-6943] [SPARK-6944] DAG visualization on SparkUI

2015-05-04 Thread andrewor14
http://git-wip-us.apache.org/repos/asf/spark/blob/863ec0cb/core/src/main/resources/org/apache/spark/ui/static/d3.min.js -- diff --git a/core/src/main/resources/org/apache/spark/ui/static/d3.min.js

[7/7] spark git commit: [SPARK-6943] [SPARK-6944] DAG visualization on SparkUI

2015-05-04 Thread andrewor14
Or and...@databricks.com Closes #5729 from andrewor14/viz2 and squashes the following commits: 666c03b [Andrew Or] Round corners of RDD boxes on stage page (minor) 01ba336 [Andrew Or] Change RDD cache color to red (minor) 6f9574a [Andrew Or] Add tests for RDDOperationScope 1c310e4 [Andrew Or] Wrap a few more RDD

[2/7] spark git commit: [SPARK-6943] [SPARK-6944] DAG visualization on SparkUI

2015-05-04 Thread andrewor14
http://git-wip-us.apache.org/repos/asf/spark/blob/fc8b5819/core/src/main/scala/org/apache/spark/rdd/RDD.scala -- diff --git a/core/src/main/scala/org/apache/spark/rdd/RDD.scala b/core/src/main/scala/org/apache/spark/rdd/RDD.scala

[4/7] spark git commit: [SPARK-6943] [SPARK-6944] DAG visualization on SparkUI

2015-05-04 Thread andrewor14
http://git-wip-us.apache.org/repos/asf/spark/blob/fc8b5819/core/src/main/resources/org/apache/spark/ui/static/graphlib-dot.min.js -- diff --git a/core/src/main/resources/org/apache/spark/ui/static/graphlib-dot.min.js

spark git commit: [SPARK-7007] [CORE] Add a metric source for ExecutorAllocationManager

2015-05-05 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.4 acc877a98 - 29350eef3 [SPARK-7007] [CORE] Add a metric source for ExecutorAllocationManager Add a metric source to expose the internal status of ExecutorAllocationManager to better monitoring the resource usage of executors when

spark git commit: [SPARK-7318] [STREAMING] DStream cleans objects that are not closures

2015-05-05 Thread andrewor14
#5860 from andrewor14/streaming-closure-cleaner and squashes the following commits: 8e971d7 [Andrew Or] Do not throw exception if object to clean is not closure 5ee4e25 [Andrew Or] Fix tests eed3390 [Andrew Or] Merge branch 'master' of github.com:apache/spark into streaming-closure-cleaner 67eeff4

spark git commit: [SPARK-7318] [STREAMING] DStream cleans objects that are not closures

2015-05-05 Thread andrewor14
Closes #5860 from andrewor14/streaming-closure-cleaner and squashes the following commits: 8e971d7 [Andrew Or] Do not throw exception if object to clean is not closure 5ee4e25 [Andrew Or] Fix tests eed3390 [Andrew Or] Merge branch 'master' of github.com:apache/spark into streaming-closure-cleaner

spark git commit: [SPARK-7237] Many user provided closures are not actually cleaned

2015-05-05 Thread andrewor14
Or and...@databricks.com Closes #5787 from andrewor14/clean-more and squashes the following commits: 2f1f476 [Andrew Or] Merge branch 'master' of github.com:apache/spark into clean-more 7265865 [Andrew Or] Merge branch 'master' of github.com:apache/spark into clean-more df3caa3 [Andrew Or] Address comments

spark git commit: [SPARK-7237] Many user provided closures are not actually cleaned

2015-05-05 Thread andrewor14
Or and...@databricks.com Closes #5787 from andrewor14/clean-more and squashes the following commits: 2f1f476 [Andrew Or] Merge branch 'master' of github.com:apache/spark into clean-more 7265865 [Andrew Or] Merge branch 'master' of github.com:apache/spark into clean-more df3caa3 [Andrew Or] Address comments

spark git commit: [SPARK-7007] [CORE] Add a metric source for ExecutorAllocationManager

2015-05-05 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 57e9f29e1 - 9f1f9b103 [SPARK-7007] [CORE] Add a metric source for ExecutorAllocationManager Add a metric source to expose the internal status of ExecutorAllocationManager to better monitoring the resource usage of executors when dynamic

spark git commit: [SPARK-7466] DAG visualization: fix orphan nodes

2015-05-08 Thread andrewor14
=https://issues.apache.org/jira/secure/attachment/12731384/after.png; width=250px/ Author: Andrew Or and...@databricks.com Closes #6002 from andrewor14/dag-viz-orphan-nodes and squashes the following commits: a1468dc [Andrew Or] Fix null check Project: http://git-wip-us.apache.org/repos/asf

spark git commit: [MINOR] Defeat early garbage collection of test suite variable

2015-05-08 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 4e7360e12 - 31da40dfe [MINOR] Defeat early garbage collection of test suite variable The JVM is free to collect references to variables that no longer participate in a computation. This simple patch adds an operation to the variable

spark git commit: [SPARK-7378] [CORE] Handle deep links to unloaded apps.

2015-05-08 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.4 3da5f8b71 - 3024f6b01 [SPARK-7378] [CORE] Handle deep links to unloaded apps. The code was treating deep links as if they were attempt IDs, so for example if you tried to load /history/app1/jobs directly, that would fail because the

spark git commit: [SPARK-7378] [CORE] Handle deep links to unloaded apps.

2015-05-08 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 9042f8f37 - 5467c34c3 [SPARK-7378] [CORE] Handle deep links to unloaded apps. The code was treating deep links as if they were attempt IDs, so for example if you tried to load /history/app1/jobs directly, that would fail because the code

[2/3] spark git commit: [WEBUI] Remove debug feature for vis.js

2015-05-08 Thread andrewor14
http://git-wip-us.apache.org/repos/asf/spark/blob/1dde3b36/core/src/main/resources/org/apache/spark/ui/static/vis.map -- diff --git a/core/src/main/resources/org/apache/spark/ui/static/vis.map

[3/3] spark git commit: [WEBUI] Remove debug feature for vis.js

2015-05-08 Thread andrewor14
[WEBUI] Remove debug feature for vis.js `vis.min.js` refers `vis.map` and this even refers `vis.js` which is used for debug `vis.js` but this debug feature is not needed for Spark itself. This issue is really minor so I don't file this in JIRA. /CC andrewor14 Author: Kousuke Saruta saru

spark git commit: [MINOR] Ignore python/lib/pyspark.zip

2015-05-08 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 25889d8d9 - dc71e47f0 [MINOR] Ignore python/lib/pyspark.zip Add `python/lib/pyspark.zip` to `.gitignore`. After merging #5580, `python/lib/pyspark.zip` will be generated when building Spark. Author: zsxwing zsxw...@gmail.com Closes

[1/3] spark git commit: [WEBUI] Remove debug feature for vis.js

2015-05-08 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.4 ab0caa060 - 1dde3b36b http://git-wip-us.apache.org/repos/asf/spark/blob/1dde3b36/core/src/main/resources/org/apache/spark/ui/static/vis.min.js -- diff --git

spark git commit: [MINOR] [CORE] Allow History Server to read kerberos opts from config file.

2015-05-08 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 3b0c5e71f - 9042f8f37 [MINOR] [CORE] Allow History Server to read kerberos opts from config file. Order of initialization code was wrong. Author: Marcelo Vanzin van...@cloudera.com Closes #5998 from vanzin/hs-conf-fix and squashes the

spark git commit: [SPARK-7466] DAG visualization: fix orphan nodes

2015-05-08 Thread andrewor14
src=https://issues.apache.org/jira/secure/attachment/12731384/after.png; width=250px/ Author: Andrew Or and...@databricks.com Closes #6002 from andrewor14/dag-viz-orphan-nodes and squashes the following commits: a1468dc [Andrew Or] Fix null check (cherry picked from commit

spark git commit: [MINOR] [CORE] Allow History Server to read kerberos opts from config file.

2015-05-08 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.4 ca2f1c56c - 3da5f8b71 [MINOR] [CORE] Allow History Server to read kerberos opts from config file. Order of initialization code was wrong. Author: Marcelo Vanzin van...@cloudera.com Closes #5998 from vanzin/hs-conf-fix and squashes

spark git commit: Revert [SPARK-7490] [CORE] [Minor] MapOutputTracker.deserializeMapStatuses: close input streams

2015-05-08 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.3 ef4a0ea7b - 6f0171130 Revert [SPARK-7490] [CORE] [Minor] MapOutputTracker.deserializeMapStatuses: close input streams This reverts commit ef4a0ea7bab19d5ec8ecad3ff4f8556361abeebe. Project:

spark git commit: [SPARK-7469] [SQL] DAG visualization: show SQL query operators

2015-05-08 Thread andrewor14
to the words) img src=https://issues.apache.org/jira/secure/attachment/12731587/after.png; width=600px/ - Author: Andrew Or and...@databricks.com Closes #5999 from andrewor14/dag-viz-sql and squashes the following commits: 0db23a4 [Andrew Or] Merge branch 'master' of github.com:apache

[1/3] spark git commit: [WEBUI] Remove debug feature for vis.js

2015-05-08 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master dc71e47f0 - c45c09b01 http://git-wip-us.apache.org/repos/asf/spark/blob/c45c09b0/core/src/main/resources/org/apache/spark/ui/static/vis.min.js -- diff --git

[3/3] spark git commit: [WEBUI] Remove debug feature for vis.js

2015-05-08 Thread andrewor14
[WEBUI] Remove debug feature for vis.js `vis.min.js` refers `vis.map` and this even refers `vis.js` which is used for debug `vis.js` but this debug feature is not needed for Spark itself. This issue is really minor so I don't file this in JIRA. /CC andrewor14 Author: Kousuke Saruta saru

spark git commit: [SPARK-7489] [SPARK SHELL] Spark shell crashes when compiled with scala 2.11

2015-05-08 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.4 1dde3b36b - 3b7fb7aaa [SPARK-7489] [SPARK SHELL] Spark shell crashes when compiled with scala 2.11 Spark shell crashes when compiled with scala 2.11 and SPARK_PREPEND_CLASSES=true There is a similar Resolved JIRA issue -SPARK-7470

spark git commit: [SPARK-7489] [SPARK SHELL] Spark shell crashes when compiled with scala 2.11

2015-05-08 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master c45c09b01 - 4e7360e12 [SPARK-7489] [SPARK SHELL] Spark shell crashes when compiled with scala 2.11 Spark shell crashes when compiled with scala 2.11 and SPARK_PREPEND_CLASSES=true There is a similar Resolved JIRA issue -SPARK-7470 and a

spark git commit: [MINOR] Defeat early garbage collection of test suite variable

2015-05-08 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.4 3b7fb7aaa - f734c5895 [MINOR] Defeat early garbage collection of test suite variable The JVM is free to collect references to variables that no longer participate in a computation. This simple patch adds an operation to the variable

spark git commit: [SPARK-7485] [BUILD] Remove pyspark files from assembly.

2015-05-12 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 984787526 - 82e890fb1 [SPARK-7485] [BUILD] Remove pyspark files from assembly. The sbt part of the build is hacky; it basically tricks sbt into generating the zip by using a generator, but returns an empty list for the generated files so

spark git commit: [SPARK-7467] Dag visualization: treat checkpoint as an RDD operation

2015-05-12 Thread andrewor14
Or and...@databricks.com Closes #6004 from andrewor14/dag-viz-checkpoint and squashes the following commits: 9217439 [Andrew Or] Fix checkpoints 4ae8806 [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-checkpoint 19bc07b [Andrew Or] Treat checkpoint as an RDD operation Project: http

spark git commit: [MINOR] [PYSPARK] Set PYTHONPATH to python/lib/pyspark.zip rather than python/pyspark

2015-05-12 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.4 af374ed26 - 4092a2e85 [MINOR] [PYSPARK] Set PYTHONPATH to python/lib/pyspark.zip rather than python/pyspark As PR #5580 we have created pyspark.zip on building and set PYTHONPATH to python/lib/pyspark.zip, so to keep consistence

spark git commit: [MINOR] [PYSPARK] Set PYTHONPATH to python/lib/pyspark.zip rather than python/pyspark

2015-05-12 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 8a4edecc4 - 984787526 [MINOR] [PYSPARK] Set PYTHONPATH to python/lib/pyspark.zip rather than python/pyspark As PR #5580 we have created pyspark.zip on building and set PYTHONPATH to python/lib/pyspark.zip, so to keep consistence update

spark git commit: [SPARK-7467] Dag visualization: treat checkpoint as an RDD operation

2015-05-12 Thread andrewor14
Or and...@databricks.com Closes #6004 from andrewor14/dag-viz-checkpoint and squashes the following commits: 9217439 [Andrew Or] Fix checkpoints 4ae8806 [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-checkpoint 19bc07b [Andrew Or] Treat checkpoint as an RDD operation (cherry picked

spark git commit: [SPARK-7485] [BUILD] Remove pyspark files from assembly.

2015-05-12 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.4 4092a2e85 - afe54b76a [SPARK-7485] [BUILD] Remove pyspark files from assembly. The sbt part of the build is hacky; it basically tricks sbt into generating the zip by using a generator, but returns an empty list for the generated files

spark git commit: [SPARK-7534] [CORE] [WEBUI] Fix the Stage table when a stage is missing

2015-05-12 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.4 6523fb81b - af374ed26 [SPARK-7534] [CORE] [WEBUI] Fix the Stage table when a stage is missing Just improved the Stage table when a stage is missing. Before: ![screen shot 2015-05-11 at 10 11 51

spark git commit: [SPARK-7534] [CORE] [WEBUI] Fix the Stage table when a stage is missing

2015-05-12 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 640f63b95 - 8a4edecc4 [SPARK-7534] [CORE] [WEBUI] Fix the Stage table when a stage is missing Just improved the Stage table when a stage is missing. Before: ![screen shot 2015-05-11 at 10 11 51

spark git commit: [HOT FIX #6076] DAG visualization: curve the edges

2015-05-12 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 4e290522c - b9b01f44f [HOT FIX #6076] DAG visualization: curve the edges Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b9b01f44 Tree:

[1/3] spark git commit: [SPARK-7500] DAG visualization: move cluster labeling to dagre-d3

2015-05-12 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master bfcaf8adc - 65697bbea http://git-wip-us.apache.org/repos/asf/spark/blob/65697bbe/core/src/main/resources/org/apache/spark/ui/static/spark-dag-viz.js -- diff --git

[2/3] spark git commit: [SPARK-7500] DAG visualization: move cluster labeling to dagre-d3

2015-05-12 Thread andrewor14
}var i=typeof require==functionrequire;for(var o=0;or.length;o++)s(r[o]);return s}({1:[function(require,module,exports){/** +/* This is a custom version of dagre-d3 on top of v0.4.3. The full list of commits can be found at http://github.com/andrewor14/dagre-d3/ */!function(e){if(object==typeof

[2/3] spark git commit: [SPARK-7500] DAG visualization: move cluster labeling to dagre-d3

2015-05-12 Thread andrewor14
}var i=typeof require==functionrequire;for(var o=0;or.length;o++)s(r[o]);return s}({1:[function(require,module,exports){/** +/* This is a custom version of dagre-d3 on top of v0.4.3. The full list of commits can be found at http://github.com/andrewor14/dagre-d3/ */!function(e){if(object==typeof

[3/3] spark git commit: [SPARK-7500] DAG visualization: move cluster labeling to dagre-d3

2015-05-12 Thread andrewor14
. For the other half of the changes, visit http://github.com/andrewor14/dagre-d3. --- **Before.** img src=https://cloud.githubusercontent.com/assets/2133137/7582769/b1423440-f845-11e4-8248-b3446a01bf79.png; width=300px/ --- **After.** img src=https

[3/3] spark git commit: [SPARK-7500] DAG visualization: move cluster labeling to dagre-d3

2015-05-12 Thread andrewor14
. For the other half of the changes, visit http://github.com/andrewor14/dagre-d3. --- **Before.** img src=https://cloud.githubusercontent.com/assets/2133137/7582769/b1423440-f845-11e4-8248-b3446a01bf79.png; width=300px/ --- **After.** img src=https

spark git commit: [CORE] Remove unreachable Heartbeat message from Worker

2015-05-15 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 7da33ce50 - daf4ae72f [CORE] Remove unreachable Heartbeat message from Worker It doesn't look to me Heartbeat is sent to Worker from anyone. Author: Kan Zhang kzh...@apache.org Closes #6163 from kanzhang/deadwood and squashes the

spark git commit: [CORE] Remove unreachable Heartbeat message from Worker

2015-05-15 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.4 1206a5597 - 6742b4ecc [CORE] Remove unreachable Heartbeat message from Worker It doesn't look to me Heartbeat is sent to Worker from anyone. Author: Kan Zhang kzh...@apache.org Closes #6163 from kanzhang/deadwood and squashes the

spark git commit: [HOT FIX #6125] Do not wait for all stages to start rendering

2015-05-13 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.4 d518c0369 - 2d4a961f8 [HOT FIX #6125] Do not wait for all stages to start rendering zsxwing Author: Andrew Or and...@databricks.com Closes #6138 from andrewor14/dag-viz-clean-properly and squashes the following commits: 19d4e98

spark git commit: [STREAMING] [MINOR] Keep streaming.UIUtils private

2015-05-13 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.4 895d46a24 - e499a1e61 [STREAMING] [MINOR] Keep streaming.UIUtils private zsxwing Author: Andrew Or and...@databricks.com Closes #6134 from andrewor14/private-streaming-uiutils and squashes the following commits: 225df94 [Andrew

spark git commit: [STREAMING] [MINOR] Keep streaming.UIUtils private

2015-05-13 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master aa1837875 - bb6dec3b1 [STREAMING] [MINOR] Keep streaming.UIUtils private zsxwing Author: Andrew Or and...@databricks.com Closes #6134 from andrewor14/private-streaming-uiutils and squashes the following commits: 225df94 [Andrew

spark git commit: [SPARK-7356] [STREAMING] Fix flakey tests in FlumePollingStreamSuite using SparkSink's batch CountDownLatch.

2015-05-13 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.4 e499a1e61 - 6c0644ae2 [SPARK-7356] [STREAMING] Fix flakey tests in FlumePollingStreamSuite using SparkSink's batch CountDownLatch. This is meant to make the FlumePollingStreamSuite deterministic. Now we basically count the number of

spark git commit: [SPARK-7399] Spark compilation error for scala 2.11

2015-05-13 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master f6e18388d - f88ac7015 [SPARK-7399] Spark compilation error for scala 2.11 Subsequent fix following #5966. I tried this out locally. Author: Andrew Or and...@databricks.com Closes #6129 from andrewor14/211-compilation and squashes

spark git commit: [SPARK-7608] Clean up old state in RDDOperationGraphListener

2015-05-13 Thread andrewor14
andrewor14/viz-listener-leak and squashes the following commits: 8660949 [Andrew Or] Fix thing + add tests 33c0843 [Andrew Or] Clean up old job state (cherry picked from commit f6e18388d993d99f768c6d547327e0720ec64224) Signed-off-by: Andrew Or and...@databricks.com Project: http://git-wip-us.apache.org

[1/3] spark git commit: [SPARK-7464] DAG visualization: highlight the same RDDs on hover

2015-05-13 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.4 e6b8cef51 - 4b4f10bc9 http://git-wip-us.apache.org/repos/asf/spark/blob/4b4f10bc/core/src/main/resources/org/apache/spark/ui/static/spark-dag-viz.css -- diff --git

[2/3] spark git commit: [SPARK-7464] DAG visualization: highlight the same RDDs on hover

2015-05-13 Thread andrewor14
http://git-wip-us.apache.org/repos/asf/spark/blob/44403414/core/src/main/resources/org/apache/spark/ui/static/dagre-d3.min.js -- diff --git a/core/src/main/resources/org/apache/spark/ui/static/dagre-d3.min.js

[2/3] spark git commit: [SPARK-7464] DAG visualization: highlight the same RDDs on hover

2015-05-13 Thread andrewor14
http://git-wip-us.apache.org/repos/asf/spark/blob/4b4f10bc/core/src/main/resources/org/apache/spark/ui/static/dagre-d3.min.js -- diff --git a/core/src/main/resources/org/apache/spark/ui/static/dagre-d3.min.js

spark git commit: [SPARK-7502] DAG visualization: gracefully handle removed stages

2015-05-13 Thread andrewor14
: Andrew Or and...@databricks.com Closes #6132 from andrewor14/dag-viz-remove-gracefully and squashes the following commits: 43175cd [Andrew Or] Handle removed jobs and stages gracefully Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark

spark git commit: [SPARK-7399] Spark compilation error for scala 2.11

2015-05-13 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.4 ec342308a - e6b8cef51 [SPARK-7399] Spark compilation error for scala 2.11 Subsequent fix following #5966. I tried this out locally. Author: Andrew Or and...@databricks.com Closes #6129 from andrewor14/211-compilation and squashes

spark git commit: [SPARK-7598] [DEPLOY] Add aliveWorkers metrics in Master

2015-05-14 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 11a1a135d - 93dbb3ad8 [SPARK-7598] [DEPLOY] Add aliveWorkers metrics in Master In Spark Standalone setup, when some workers are DEAD, they will stay in master worker list for a while. master.workers metrics for master is only showing the

spark git commit: [SPARK-7649] [STREAMING] [WEBUI] Use window.localStorage to store the status rather than the url

2015-05-14 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.4 8d8876d3b - 335848577 [SPARK-7649] [STREAMING] [WEBUI] Use window.localStorage to store the status rather than the url Use window.localStorage to store the status rather than the url so that the url won't be changed. cc tdas

<    1   2   3   4   5   6   7   8   9   10   >