spark git commit: [SPARK-6286][Mesos][minor] Handle missing Mesos case TASK_ERROR

2015-03-20 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-1.3 03e263f5b - db812d99e [SPARK-6286][Mesos][minor] Handle missing Mesos case TASK_ERROR - Made TaskState.isFailed for handling TASK_LOST and TASK_ERROR and synchronizing CoarseMesosSchedulerBackend and MesosSchedulerBackend - This is

spark git commit: SPARK-5134 [BUILD] Bump default Hadoop version to 2+

2015-03-20 Thread srowen
: Sean Owen so...@cloudera.com Closes #5027 from srowen/SPARK-5134 and squashes the following commits: acbee14 [Sean Owen] Bump default Hadoop version to 2.2.0. (This is already the dependency version reported by published Maven artifacts.) Project: http://git-wip-us.apache.org/repos/asf/spark/repo

spark git commit: [SPARK-6286][Mesos][minor] Handle missing Mesos case TASK_ERROR

2015-03-20 Thread srowen
Repository: spark Updated Branches: refs/heads/master 0745a305f - 116c553fd [SPARK-6286][Mesos][minor] Handle missing Mesos case TASK_ERROR - Made TaskState.isFailed for handling TASK_LOST and TASK_ERROR and synchronizing CoarseMesosSchedulerBackend and MesosSchedulerBackend - This is

spark git commit: SPARK-6338 [CORE] Use standard temp dir mechanisms in tests to avoid orphaned temp files

2015-03-20 Thread srowen
, and simplify Author: Sean Owen so...@cloudera.com Closes #5029 from srowen/SPARK-6338 and squashes the following commits: 27b740a [Sean Owen] Fix hive-thriftserver tests that don't expect an existing dir 4a212fa [Sean Owen] Standardize a bit more temp dir management 9004081 [Sean Owen] Revert

spark git commit: [SPARK-6291] [MLLIB] GLM toString toDebugString

2015-03-19 Thread srowen
Repository: spark Updated Branches: refs/heads/master 3c4e486b9 - dda4dedca [SPARK-6291] [MLLIB] GLM toString toDebugString GLM toString prints out intercept, numFeatures. For LogisticRegression and SVM model, toString also prints out numClasses, threshold. GLM toDebugString prints out the

spark git commit: [Core][minor] remove unused `visitedStages` in `DAGScheduler.stageDependsOn`

2015-03-19 Thread srowen
Repository: spark Updated Branches: refs/heads/master 8cb23a1f9 - 3b5aaa6a5 [Core][minor] remove unused `visitedStages` in `DAGScheduler.stageDependsOn` We define and update `visitedStages` in `DAGScheduler.stageDependsOn`, but never read it. So we can safely remove it. Author: Wenchen Fan

spark git commit: [SPARK-6370][core] Documentation: Improve all 3 docs for RDD.sample

2015-03-20 Thread srowen
Repository: spark Updated Branches: refs/heads/master db4d317cc - 28bcb9e9e [SPARK-6370][core] Documentation: Improve all 3 docs for RDD.sample The docs for the `sample` method were insufficient, now less so. Author: mbonaci mbon...@gmail.com Closes #5097 from mbonaci/master and squashes

spark git commit: [SPARK-6426][Doc]User could also point the yarn cluster config directory via YARN_CONF_DI...

2015-03-20 Thread srowen
Repository: spark Updated Branches: refs/heads/master 28bcb9e9e - 385b2ff10 [SPARK-6426][Doc]User could also point the yarn cluster config directory via YARN_CONF_DI... ...R https://issues.apache.org/jira/browse/SPARK-6426 Author: WangTaoTheTonic wangtao...@huawei.com Closes #5103 from

spark git commit: [SPARK-6371] [build] Update version to 1.4.0-SNAPSHOT.

2015-03-20 Thread srowen
Repository: spark Updated Branches: refs/heads/master 385b2ff10 - a74564591 [SPARK-6371] [build] Update version to 1.4.0-SNAPSHOT. Author: Marcelo Vanzin van...@cloudera.com Closes #5056 from vanzin/SPARK-6371 and squashes the following commits: 63220df [Marcelo Vanzin] Merge branch

spark git commit: [SPARK-6423][Mesos] MemoryUtils should use memoryOverhead if it's set

2015-03-20 Thread srowen
Repository: spark Updated Branches: refs/heads/master 6b36470c6 - 49a01c7ea [SPARK-6423][Mesos] MemoryUtils should use memoryOverhead if it's set - Fixed calculateTotalMemory to use spark.mesos.executor.memoryOverhead - Added testCase Author: Jongyoul Lee jongy...@gmail.com Closes #5099

spark git commit: SPARK-6245 [SQL] jsonRDD() of empty RDD results in exception

2015-03-16 Thread srowen
), but it seems like this case can come up if the input is down-sampled so much that nothing is sampled. Now stuff like this: ``` sqlContext.jsonRDD(sc.parallelize(List[String]())) ``` just results in ``` org.apache.spark.sql.DataFrame = [] ``` Author: Sean Owen so...@cloudera.com Closes #4971 from srowen

spark git commit: Fix python typo (+ Scala, Java typos)

2015-03-06 Thread srowen
Repository: spark Updated Branches: refs/heads/master dba0b2ead - 48a723c98 Fix python typo (+ Scala, Java typos) Author: RobertZK technoguy...@gmail.com Author: Robert Krzyzanowski technoguy...@gmail.com Closes #4840 from robertzk/patch-1 and squashes the following commits: d286215

spark git commit: [EC2] [SPARK-6188] Instance types can be mislabeled when re-starting cluster with default arguments

2015-03-09 Thread srowen
Repository: spark Updated Branches: refs/heads/master 55b1b32dc - f7c799204 [EC2] [SPARK-6188] Instance types can be mislabeled when re-starting cluster with default arguments As described in https://issues.apache.org/jira/browse/SPARK-6188 and discovered in

spark git commit: [SPARK-6177][MLlib]Add note in LDA example to remind possible coalesce

2015-03-10 Thread srowen
Repository: spark Updated Branches: refs/heads/master 8767565ce - 9a0272fbb [SPARK-6177][MLlib]Add note in LDA example to remind possible coalesce JIRA: https://issues.apache.org/jira/browse/SPARK-6177 Add comment to introduce coalesce to LDA example to avoid the possible massive partitions

spark git commit: [SPARK-6087][CORE] Provide actionable exception if Kryo buffer is not large enough

2015-03-10 Thread srowen
Repository: spark Updated Branches: refs/heads/master 9a0272fbb - c4c4b07bf [SPARK-6087][CORE] Provide actionable exception if Kryo buffer is not large enough A simple try-catch wrapping KryoException to be more informative. Author: Lev Khomich levkhom...@gmail.com Closes #4947 from

spark git commit: [SPARK-6191] [EC2] Generalize ability to download libs

2015-03-10 Thread srowen
Repository: spark Updated Branches: refs/heads/master c4c4b07bf - d14df06c0 [SPARK-6191] [EC2] Generalize ability to download libs Right now we have a method to specifically download boto. This PR generalizes it so it's easy to download additional libraries if we want. For example, adding

spark git commit: [SPARK-6186] [EC2] Make Tachyon version configurable in EC2 deployment script

2015-03-10 Thread srowen
Repository: spark Updated Branches: refs/heads/master d14df06c0 - 7c7d2d5e0 [SPARK-6186] [EC2] Make Tachyon version configurable in EC2 deployment script This PR comes from Tachyon community to solve the issue: https://tachyon.atlassian.net/browse/TACHYON-11 An accompanying PR is in

spark git commit: [SPARK-6178][Shuffle] Removed unused imports

2015-03-06 Thread srowen
Repository: spark Updated Branches: refs/heads/master 05cb6b34d - dba0b2ead [SPARK-6178][Shuffle] Removed unused imports Author: Vinod K C vinod.kchuawei.com Author: Vinod K C vinod...@huawei.com Closes #4900 from vinodkc/unused_imports and squashes the following commits: 5373456 [Vinod K

spark git commit: [Minor] Resolve sbt warnings: postfix operator second should be enabled

2015-03-06 Thread srowen
Repository: spark Updated Branches: refs/heads/master cd7594ca6 - 05cb6b34d [Minor] Resolve sbt warnings: postfix operator second should be enabled Resolve sbt warnings: ``` [warn] spark/streaming/src/main/scala/org/apache/spark/streaming/util/WriteAheadLogManager.scala:155: postfix

spark git commit: [CORE, DEPLOY][minor] align arguments order with docs of worker

2015-03-06 Thread srowen
Repository: spark Updated Branches: refs/heads/master eb48fd6e9 - d8b3da9dd [CORE, DEPLOY][minor] align arguments order with docs of worker The help message for starting `worker` is `Usage: Worker [options] master`. While in `start-slaves.sh`, the format is not align with that, it is

spark git commit: [GraphX] Improve LiveJournalPageRank example

2015-03-08 Thread srowen
Repository: spark Updated Branches: refs/heads/master f16b7b031 - 55b1b32dc [GraphX] Improve LiveJournalPageRank example 1. Removed unnecessary import 2. Modified usage print since user must specify the --numEPart parameter as it is required in Analytics.main Author: Jacky Li

spark git commit: [SPARK-5641] [EC2] Allow spark_ec2.py to copy arbitrary files to cluster

2015-03-07 Thread srowen
Repository: spark Updated Branches: refs/heads/master 729c05bda - 334c5bd1a [SPARK-5641] [EC2] Allow spark_ec2.py to copy arbitrary files to cluster Give users an easy way to rcp a directory structure to the master's / as part of the cluster launch, at a useful point in the workflow (before

spark git commit: [Minor]fix the wrong description

2015-03-07 Thread srowen
Repository: spark Updated Branches: refs/heads/master 2646794ff - 729c05bda [Minor]fix the wrong description Found it by accident. I'm not gonna file jira for this as it is a very tiny fix. Author: WangTaoTheTonic wangtao...@huawei.com Closes #4936 from WangTaoTheTonic/wrongdesc and

spark git commit: [EC2] Reorder print statements on termination

2015-03-07 Thread srowen
Repository: spark Updated Branches: refs/heads/master 48a723c98 - 2646794ff [EC2] Reorder print statements on termination The PR reorders some print statements slightly on cluster termination so that they read better. For example, from this: ``` Are you sure you want to destroy the cluster

spark git commit: [SPARK-6193] [EC2] Push group filter up to EC2

2015-03-08 Thread srowen
Repository: spark Updated Branches: refs/heads/master 334c5bd1a - 52ed7da12 [SPARK-6193] [EC2] Push group filter up to EC2 When looking for a cluster, spark-ec2 currently pulls down [info for all

spark git commit: SPARK-6205 [CORE] UISeleniumSuite fails for Hadoop 2.x test with NoClassDefFoundError

2015-03-08 Thread srowen
from srowen/SPARK-6205 and squashes the following commits: ddd4d32 [Sean Owen] Add xml-apis to core test deps to work aroudn UISeleniumSuite classpath issue Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/f16b7b03 Tree: http

spark git commit: SPARK-4044 [CORE] Thriftserver fails to start when JAVA_HOME points to JRE instead of JDK

2015-03-13 Thread srowen
JAR_CMD. Follow up to https://github.com/apache/spark/pull/4873 for branch 1.3. Author: Sean Owen so...@cloudera.com Closes #4981 from srowen/SPARK-4044.2 and squashes the following commits: 3aafc76 [Sean Owen] Don't use JAR_CMD unless present in archive check. Add datanucleus always if present

spark git commit: [SPARK-6036][CORE] avoid race condition between eventlogListener and akka actor system

2015-03-13 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-1.3 4aa41327d - f81611dca [SPARK-6036][CORE] avoid race condition between eventlogListener and akka actor system For detail description, pls refer to [SPARK-6036](https://issues.apache.org/jira/browse/SPARK-6036). Author: Zhang, Liye

spark git commit: [SPARK-6087][CORE] Provide actionable exception if Kryo buffer is not large enough

2015-03-13 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-1.3 f81611dca - 9846790f4 [SPARK-6087][CORE] Provide actionable exception if Kryo buffer is not large enough A simple try-catch wrapping KryoException to be more informative. Author: Lev Khomich levkhom...@gmail.com Closes #4947 from

spark git commit: [SPARK-4600][GraphX]: org.apache.spark.graphx.VertexRDD.diff does not work

2015-03-13 Thread srowen
Repository: spark Updated Branches: refs/heads/master 7f13434a5 - b943f5d90 [SPARK-4600][GraphX]: org.apache.spark.graphx.VertexRDD.diff does not work Turns out, per the [convo on the JIRA](https://issues.apache.org/jira/browse/SPARK-4600), `diff` is acting exactly as should. It became a

spark git commit: [SPARK-6133] Make sc.stop() idempotent

2015-03-13 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-1.3 338bea7b3 - a08588c7e [SPARK-6133] Make sc.stop() idempotent Before we would get the following (benign) error if we called `sc.stop()` twice. This is because the listener bus would try to post the end event again even after it has

spark git commit: SPARK-4300 [CORE] Race condition during SparkWorker shutdown

2015-03-13 Thread srowen
`waitFor()` although it was harmless) CC tdas since I think you wrote this method. Author: Sean Owen so...@cloudera.com Closes #4787 from srowen/SPARK-4300 and squashes the following commits: e0cdabf [Sean Owen] Close appender saving stdout/stderr before destroying process to avoid exception

spark git commit: [SPARK-6275][Documentation]Miss toDF() function in docs/sql-programming-guide.md

2015-03-13 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-1.3 a08588c7e - 301278126 [SPARK-6275][Documentation]Miss toDF() function in docs/sql-programming-guide.md Miss `toDF()` function in docs/sql-programming-guide.md Author: zzcclp xm_...@sina.com Closes #4977 from zzcclp/SPARK-6275 and

spark git commit: SPARK-4704 [CORE] SparkSubmitDriverBootstrap doesn't flush output

2015-03-13 Thread srowen
created this section of code Author: Sean Owen so...@cloudera.com Closes #4788 from srowen/SPARK-4704 and squashes the following commits: ad7114e [Sean Owen] Join on output threads to make sure any lingering output from process reaches stdout, stderr before exiting Project: http://git-wip

[1/2] spark git commit: [SPARK-6132] ContextCleaner race condition across SparkContexts

2015-03-13 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-1.3 9846790f4 - 338bea7b3 [SPARK-6132] ContextCleaner race condition across SparkContexts The problem is that `ContextCleaner` may clean variables that belong to a different `SparkContext`. This can happen if the `SparkContext` to which

[2/2] spark git commit: [SPARK-6132][HOTFIX] ContextCleaner InterruptedException should be quiet

2015-03-13 Thread srowen
[SPARK-6132][HOTFIX] ContextCleaner InterruptedException should be quiet If the cleaner is stopped, we shouldn't print a huge stack trace when the cleaner thread is interrupted because we purposefully did this. Author: Andrew Or and...@databricks.com Closes #4882 from

spark git commit: [SQL]Delete some dupliate code in HiveThriftServer2

2015-03-14 Thread srowen
Repository: spark Updated Branches: refs/heads/master b38e073fe - ee15404a2 [SQL]Delete some dupliate code in HiveThriftServer2 Author: ArcherShao archers...@users.noreply.github.com Author: ArcherShao shaoch...@huawei.com Closes #5007 from ArcherShao/20150313 and squashes the following

spark git commit: [SPARK-1210] Prevent ContextClassLoader of Actor from becoming ClassLoader of Executo...

2015-03-14 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-0.9 6665df6b7 - 2f03fc17c [SPARK-1210] Prevent ContextClassLoader of Actor from becoming ClassLoader of Executo... ...r. Constructor of `org.apache.spark.executor.Executor` should not set context class loader of current thread, which is

spark git commit: [SPARK-6329][Docs]: Minor doc changes for Mesos and TOC

2015-03-14 Thread srowen
Repository: spark Updated Branches: refs/heads/master 5be6b0e4f - 127268bc3 [SPARK-6329][Docs]: Minor doc changes for Mesos and TOC Updated the configuration docs from the minor items that Reynold had left over from SPARK-1182; specifically I updated the `running-on-mesos` link to point

spark git commit: [SPARK-5790][GraphX]: VertexRDD's won't zip properly for `diff` capability (added tests)

2015-03-14 Thread srowen
Repository: spark Updated Branches: refs/heads/master 127268bc3 - c49d15662 [SPARK-5790][GraphX]: VertexRDD's won't zip properly for `diff` capability (added tests) Added tests that maropu

spark git commit: [SPARK-4423] Improve foreach() documentation to avoid confusion between local- and cluster-mode behavior

2015-03-11 Thread srowen
Repository: spark Updated Branches: refs/heads/master 5b335bdda - 548643a9e [SPARK-4423] Improve foreach() documentation to avoid confusion between local- and cluster-mode behavior Hi all - I've added a writeup on how closures work within Spark to help clarify the general case for this

spark git commit: SPARK-6225 [CORE] [SQL] [STREAMING] Resolve most build warnings, 1.3.0 edition

2015-03-11 Thread srowen
Closes #4950 from srowen/SPARK-6225 and squashes the following commits: 3080972 [Sean Owen] Ordered imports: Java, Scala, 3rd party, Spark c67985b [Sean Owen] Resolve javac, scalac warnings of various types -- deprecations, Scala lang, unchecked cast, etc. Project: http://git-wip-us.apache.org

spark git commit: [SPARK-6228] [network] Move SASL classes from network/shuffle to network...

2015-03-11 Thread srowen
Repository: spark Updated Branches: refs/heads/master 6e94c4ead - 5b335bdda [SPARK-6228] [network] Move SASL classes from network/shuffle to network... .../common. No code changes. Left the shuffle-related files in the shuffle module. Author: Marcelo Vanzin van...@cloudera.com Closes #4953

spark git commit: SPARK-3642. Document the nuances of shared variables.

2015-03-11 Thread srowen
Repository: spark Updated Branches: refs/heads/master 548643a9e - 2d87a415f SPARK-3642. Document the nuances of shared variables. Author: Sandy Ryza sa...@cloudera.com Closes #2490 from sryza/sandy-spark-3642 and squashes the following commits: aae3340 [Sandy Ryza] SPARK-3642. Document the

spark git commit: [SPARK-6279][Streaming]In KafkaRDD.scala, Miss expressions flag s at logging string

2015-03-11 Thread srowen
Repository: spark Updated Branches: refs/heads/master 40f49795e - ec30c1782 [SPARK-6279][Streaming]In KafkaRDD.scala, Miss expressions flag s at logging string In KafkaRDD.scala, Miss expressions flag s at logging string In logging file, it print `Beginning offset $ {part.fromOffset} is the

spark git commit: [MINOR] [DOCS] Fix map - mapToPair in Streaming Java example

2015-03-11 Thread srowen
Repository: spark Updated Branches: refs/heads/master 517975d89 - 35b25640a [MINOR] [DOCS] Fix map - mapToPair in Streaming Java example Fix map - mapToPair in Java example. (And zap some unneeded throws Exception while here) Author: Sean Owen so...@cloudera.com Closes #4967 from srowen

spark git commit: [SQL][Minor] fix typo in comments

2015-03-11 Thread srowen
Repository: spark Updated Branches: refs/heads/master 35b25640a - 40f49795e [SQL][Minor] fix typo in comments Removed an repeated from in the comments. Author: Hongbo Liu liuh...@gmail.com Closes #4976 from liuhb86/mine and squashes the following commits: e280e7c [Hongbo Liu] [SQL][Minor]

spark git commit: [docs] [SPARK-6306] Readme points to dead link

2015-03-12 Thread srowen
Repository: spark Updated Branches: refs/heads/master 0cba802ad - 4e47d54be [docs] [SPARK-6306] Readme points to dead link The link to Specifying the Hadoop Version currently points to http://spark.apache.org/docs/latest/building-with-maven.html#specifying-the-hadoop-version. The correct

spark git commit: [SPARK-6275][Documentation]Miss toDF() function in docs/sql-programming-guide.md

2015-03-12 Thread srowen
Repository: spark Updated Branches: refs/heads/master 4e47d54be - 304366c46 [SPARK-6275][Documentation]Miss toDF() function in docs/sql-programming-guide.md Miss `toDF()` function in docs/sql-programming-guide.md Author: zzcclp xm_...@sina.com Closes #4977 from zzcclp/SPARK-6275 and

spark git commit: [SPARK-5845][Shuffle] Time to cleanup spilled shuffle files not included in shuffle write time

2015-03-13 Thread srowen
Repository: spark Updated Branches: refs/heads/master 3980ebdf1 - 0af9ea74a [SPARK-5845][Shuffle] Time to cleanup spilled shuffle files not included in shuffle write time I've added a timer in the right place to fix this inaccuracy. Author: Ilya Ganelin ilya.gane...@capitalone.com Closes

spark git commit: [CORE][minor] remove unnecessary ClassTag in `DAGScheduler`

2015-03-13 Thread srowen
Repository: spark Updated Branches: refs/heads/master 9048e8102 - ea3d2eed9 [CORE][minor] remove unnecessary ClassTag in `DAGScheduler` This existed at the very beginning, but became unnecessary after [this

spark git commit: [SPARK-6197][CORE] handle json exception when hisotry file not finished writing

2015-03-13 Thread srowen
Repository: spark Updated Branches: refs/heads/master 69ff8e8cf - 9048e8102 [SPARK-6197][CORE] handle json exception when hisotry file not finished writing For details, please refer to [SPARK-6197](https://issues.apache.org/jira/browse/SPARK-6197) Author: Zhang, Liye liye.zh...@intel.com

spark git commit: [build] [hotfix] Fix make-distribution.sh for Scala 2.11.

2015-03-12 Thread srowen
Repository: spark Updated Branches: refs/heads/master 304366c46 - 8f1bc7989 [build] [hotfix] Fix make-distribution.sh for Scala 2.11. Author: Marcelo Vanzin van...@cloudera.com Closes #5002 from vanzin/mkdist-hotfix and squashes the following commits: ced65f7 [Marcelo Vanzin] [build]

spark git commit: [SPARK-6477][Build]: Run MIMA tests before the Spark test suite

2015-03-24 Thread srowen
Repository: spark Updated Branches: refs/heads/master 1afcf773d - 37fac1dcd [SPARK-6477][Build]: Run MIMA tests before the Spark test suite This moves the MIMA checks to before the full Spark test suite such that, if new PR's fail the MIMA check, they will return much faster having not run

spark git commit: [SPARK-6473] [core] Do not try to figure out Scala version if not needed...

2015-03-24 Thread srowen
Repository: spark Updated Branches: refs/heads/master c12312f8b - b293afc42 [SPARK-6473] [core] Do not try to figure out Scala version if not needed... Author: Marcelo Vanzin van...@cloudera.com Closes #5143 from vanzin/SPARK-6473 and squashes the following commits: a2e5e2d [Marcelo

spark git commit: Update the command to use IPython notebook

2015-03-24 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-1.3 6f10142de - e5451432e Update the command to use IPython notebook As for notebook --pylab inline is not supported any more, update the related documentation for this. Author: Cong Yue yuecong1...@gmail.com Closes #5111 from

spark git commit: [SPARK-5559] [Streaming] [Test] Remove oppotunity we met flakiness when running FlumeStreamSuite

2015-03-24 Thread srowen
Repository: spark Updated Branches: refs/heads/master b293afc42 - 85cf06368 [SPARK-5559] [Streaming] [Test] Remove oppotunity we met flakiness when running FlumeStreamSuite When we run FlumeStreamSuite on Jenkins, sometimes we get error like as follows. sbt.ForkMain$ForkError: The code

spark git commit: [SPARK-5559] [Streaming] [Test] Remove oppotunity we met flakiness when running FlumeStreamSuite

2015-03-24 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-1.3 e5451432e - 8722369c2 [SPARK-5559] [Streaming] [Test] Remove oppotunity we met flakiness when running FlumeStreamSuite When we run FlumeStreamSuite on Jenkins, sometimes we get error like as follows. sbt.ForkMain$ForkError: The

spark git commit: [ML][docs][minor] Define LabeledDocument/Document classes in CV example

2015-03-24 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-1.3 8722369c2 - 4ff577160 [ML][docs][minor] Define LabeledDocument/Document classes in CV example To easier copy/paste Cross-Validation example code snippet need to define LabeledDocument/Document in it, since they difined in a previous

spark git commit: [ML][docs][minor] Define LabeledDocument/Document classes in CV example

2015-03-24 Thread srowen
Repository: spark Updated Branches: refs/heads/master 85cf06368 - 08d452801 [ML][docs][minor] Define LabeledDocument/Document classes in CV example To easier copy/paste Cross-Validation example code snippet need to define LabeledDocument/Document in it, since they difined in a previous

spark git commit: [SPARK-5559] [Streaming] [Test] Remove oppotunity we met flakiness when running FlumeStreamSuite

2015-03-24 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-1.2 e080cc3e5 - 8ef69957f [SPARK-5559] [Streaming] [Test] Remove oppotunity we met flakiness when running FlumeStreamSuite When we run FlumeStreamSuite on Jenkins, sometimes we get error like as follows. sbt.ForkMain$ForkError: The

spark git commit: [SPARK-6496] [MLLIB] GeneralizedLinearAlgorithm.run(input, initialWeights) should initialize numFeatures

2015-03-25 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-1.3 8e4e2e3f8 - 2be4255a0 [SPARK-6496] [MLLIB] GeneralizedLinearAlgorithm.run(input, initialWeights) should initialize numFeatures In GeneralizedLinearAlgorithm ```numFeatures``` is default to -1, we need to update it to correct value

spark git commit: SPARK-6063 MLlib doesn't pass mvn scalastyle check due to UTF chars in LDAModel.scala

2015-03-25 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-1.3 2be4255a0 - 6791f425d SPARK-6063 MLlib doesn't pass mvn scalastyle check due to UTF chars in LDAModel.scala Remove unicode characters from MLlib file. Author: Michael Griffiths msjgriffi...@gmail.com Author: Griffiths, Michael

spark git commit: [SPARK-6496] [MLLIB] GeneralizedLinearAlgorithm.run(input, initialWeights) should initialize numFeatures

2015-03-25 Thread srowen
Repository: spark Updated Branches: refs/heads/master 64262ed99 - 10c78607b [SPARK-6496] [MLLIB] GeneralizedLinearAlgorithm.run(input, initialWeights) should initialize numFeatures In GeneralizedLinearAlgorithm ```numFeatures``` is default to -1, we need to update it to correct value when

spark git commit: [MLlib]remove unused import

2015-03-26 Thread srowen
Repository: spark Updated Branches: refs/heads/master 1c05027a1 - 3ddb975fa [MLlib]remove unused import minor thing. Let me know if jira is required. Author: Yuhao Yang hhb...@gmail.com Closes #5207 from hhbyyh/adjustImport and squashes the following commits: 2240121 [Yuhao Yang] remove

spark git commit: [SPARK-6491] Spark will put the current working dir to the CLASSPATH

2015-03-26 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-1.3 836c92165 - 5b5f0e2b0 [SPARK-6491] Spark will put the current working dir to the CLASSPATH When running bin/computer-classpath.sh, the output will be:

spark git commit: [SPARK-6468][Block Manager] Fix the race condition of subDirs in DiskBlockManager

2015-03-26 Thread srowen
Repository: spark Updated Branches: refs/heads/master f88f51bbd - 0c88ce541 [SPARK-6468][Block Manager] Fix the race condition of subDirs in DiskBlockManager There are two race conditions of `subDirs` in `DiskBlockManager`: 1. `getAllFiles` does not use correct locks to read the contents in

spark git commit: [SPARK-6406] Launch Spark using assembly jar instead of a separate launcher jar

2015-03-29 Thread srowen
Repository: spark Updated Branches: refs/heads/master 55153f5c1 - e3eb39396 [SPARK-6406] Launch Spark using assembly jar instead of a separate launcher jar Author: Nishkam Ravi nr...@cloudera.com Author: nishkamravi2 nishkamr...@gmail.com Author: nravi nr...@c1704.halxg.cloudera.com Closes

spark git commit: [SPARK-4655][Core] Split Stage into ShuffleMapStage and ResultStage subclasses

2015-04-01 Thread srowen
Repository: spark Updated Branches: refs/heads/master 305abe1e5 - ff1915e12 [SPARK-4655][Core] Split Stage into ShuffleMapStage and ResultStage subclasses Hi all - this patch changes the Stage class to an abstract class and introduces two new classes that extend it: ShuffleMapStage and

spark git commit: [EC2] [SPARK-6600] Open ports in ec2/spark_ec2.py to allow HDFS NFS gateway

2015-04-01 Thread srowen
Repository: spark Updated Branches: refs/heads/master ff1915e12 - 412262346 [EC2] [SPARK-6600] Open ports in ec2/spark_ec2.py to allow HDFS NFS gateway Authorizes incoming access to master on the ports required to use the hadoop hdfs nfs gateway from outside the cluster. Author: Florian

spark git commit: SPARK-6626 [DOCS]: Corrected Scala:TwitterUtils parameters

2015-04-01 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-1.3 e527b3590 - 7d029cb1e SPARK-6626 [DOCS]: Corrected Scala:TwitterUtils parameters Per Sean Owen's request, here is the update call for TwitterUtils using Scala :) Author: jayson jay...@ziprecruiter.com Closes #5295 from

spark git commit: SPARK-6626 [DOCS]: Corrected Scala:TwitterUtils parameters

2015-04-01 Thread srowen
Repository: spark Updated Branches: refs/heads/master d824c11c9 - 0358b08db SPARK-6626 [DOCS]: Corrected Scala:TwitterUtils parameters Per Sean Owen's request, here is the update call for TwitterUtils using Scala :) Author: jayson jay...@ziprecruiter.com Closes #5295 from

spark git commit: [SPARK-6597][Minor] Replace `input:checkbox` with `input[type=checkbox]` in additional-metrics.js

2015-04-01 Thread srowen
Repository: spark Updated Branches: refs/heads/master 412262346 - d824c11c9 [SPARK-6597][Minor] Replace `input:checkbox` with `input[type=checkbox]` in additional-metrics.js In additional-metrics.js, there are some selector notation like `input:checkbox` but JQuery's official document says

spark git commit: [SPARK-6556][Core] Fix wrong parsing logic of executorTimeoutMs and checkTimeoutIntervalMs in HeartbeatReceiver

2015-03-27 Thread srowen
Repository: spark Updated Branches: refs/heads/master f43a61031 - da546b7ba [SPARK-6556][Core] Fix wrong parsing logic of executorTimeoutMs and checkTimeoutIntervalMs in HeartbeatReceiver The current reading logic of `executorTimeoutMs` is: ```Scala private val executorTimeoutMs =

spark git commit: [SPARK-6544][build] Increment Avro version from 1.7.6 to 1.7.7

2015-03-27 Thread srowen
Repository: spark Updated Branches: refs/heads/master da546b7ba - aa2b99174 [SPARK-6544][build] Increment Avro version from 1.7.6 to 1.7.7 Fixes bug causing Kryo serialization to fail with Avro files in between stages.

spark git commit: [SPARK-4123][Project Infra]: Show new dependencies added in pull requests

2015-03-29 Thread srowen
Repository: spark Updated Branches: refs/heads/master 5eef00d0c - 55153f5c1 [SPARK-4123][Project Infra]: Show new dependencies added in pull requests Starting work on this, but need to find a way to ensure that, after doing a checkout from `apache/master`, we can successfully return to the

spark git commit: [SPARK-6552][Deploy][Doc]expose start-slave.sh to user and update outdated doc

2015-03-28 Thread srowen
Repository: spark Updated Branches: refs/heads/master 5909f0973 - 99631438c [SPARK-6552][Deploy][Doc]expose start-slave.sh to user and update outdated doc https://issues.apache.org/jira/browse/SPARK-6552 /cc srowen Author: WangTaoTheTonic wangtao...@huawei.com Closes #5205 from

spark git commit: SPARK-6433 hive tests to import spark-sql test JAR for QueryTest access

2015-04-01 Thread srowen
Repository: spark Updated Branches: refs/heads/master d36c5fca7 - ee11be258 SPARK-6433 hive tests to import spark-sql test JAR for QueryTest access 1. Test JARs are built published 1. log4j.resources is explicitly excluded. Without this, downstream test run logging depends on the order the

svn commit: r1663444 - in /spark/site: examples.html index.html

2015-03-02 Thread srowen
Author: srowen Date: Mon Mar 2 22:06:52 2015 New Revision: 1663444 URL: http://svn.apache.org/r1663444 Log: SPARK-4992 Improve var naming in Python example; change other var names to match Modified: spark/site/examples.html spark/site/index.html Modified: spark/site/examples.html URL

svn commit: r1663449 - /spark/site/index.html

2015-03-02 Thread srowen
Author: srowen Date: Mon Mar 2 22:18:42 2015 New Revision: 1663449 URL: http://svn.apache.org/r1663449 Log: SPARK-4992 follow-up to inhibit undesired text wrapping in Python example Modified: spark/site/index.html Modified: spark/site/index.html URL: http://svn.apache.org/viewvc/spark

spark git commit: SPARK-5390 [DOCS] Encourage users to post on Stack Overflow in Community Docs

2015-03-02 Thread srowen
, under Mailing Lists: StackOverflow also has an apache-spark tag for Spark QA. or similar. Author: Sean Owen so...@cloudera.com Closes #4843 from srowen/SPARK-5390 and squashes the following commits: 3508ac6 [Sean Owen] Point Community to main Spark Community page; mention SO tag apache-spark

spark git commit: SPARK-5390 [DOCS] Encourage users to post on Stack Overflow in Community Docs

2015-03-02 Thread srowen
Mailing Lists: StackOverflow also has an apache-spark tag for Spark QA. or similar. Author: Sean Owen so...@cloudera.com Closes #4843 from srowen/SPARK-5390 and squashes the following commits: 3508ac6 [Sean Owen] Point Community to main Spark Community page; mention SO tag apache-spark Project

svn commit: r1663693 - /spark/site/downloads.html

2015-03-03 Thread srowen
Author: srowen Date: Tue Mar 3 13:04:20 2015 New Revision: 1663693 URL: http://svn.apache.org/r1663693 Log: SPARK-5055: fix Chose - Choose typo Modified: spark/site/downloads.html Modified: spark/site/downloads.html URL: http://svn.apache.org/viewvc/spark/site/downloads.html?rev=1663693r1

spark git commit: SPARK-5983 [WEBUI] Don't respond to HTTP TRACE in HTTP-based UIs

2015-02-28 Thread srowen
Repository: spark Updated Branches: refs/heads/master b36b1bc22 - f91298e2c SPARK-5983 [WEBUI] Don't respond to HTTP TRACE in HTTP-based UIs Disallow TRACE HTTP method in servlets Author: Sean Owen so...@cloudera.com Closes #4765 from srowen/SPARK-5983 and squashes the following commits

spark git commit: SPARK-1965 [WEBUI] Spark UI throws NPE on trying to load the app page for non-existent app

2015-02-28 Thread srowen
just render empty content instead Author: Sean Owen so...@cloudera.com Closes #4777 from srowen/SPARK-1965 and squashes the following commits: 7e16590 [Sean Owen] Update app not found message cb878d6 [Sean Owen] Return basic not found page for unknown appId d8270da [Sean Owen] Don't throw NPE

spark git commit: SPARK-6063 MLlib doesn't pass mvn scalastyle check due to UTF chars in LDAModel.scala

2015-02-28 Thread srowen
Repository: spark Updated Branches: refs/heads/master e6003f0a5 - b36b1bc22 SPARK-6063 MLlib doesn't pass mvn scalastyle check due to UTF chars in LDAModel.scala Remove unicode characters from MLlib file. Author: Michael Griffiths msjgriffi...@gmail.com Author: Griffiths, Michael (NYC-RPM)

spark git commit: [Streaming][Minor]Fix some error docs in streaming examples

2015-03-02 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-1.3 6a2fc85e0 - 1fe677a36 [Streaming][Minor]Fix some error docs in streaming examples Small changes, please help to review, thanks a lot. Author: Saisai Shao saisai.s...@intel.com Closes #4837 from jerryshao/doc-fix and squashes the

spark git commit: [Streaming][Minor]Fix some error docs in streaming examples

2015-03-02 Thread srowen
Repository: spark Updated Branches: refs/heads/master 3f00bb3ef - d8fb40ede [Streaming][Minor]Fix some error docs in streaming examples Small changes, please help to review, thanks a lot. Author: Saisai Shao saisai.s...@intel.com Closes #4837 from jerryshao/doc-fix and squashes the

spark git commit: SPARK-3357 [CORE] Internal log messages should be set at DEBUG level instead of INFO

2015-03-02 Thread srowen
suggested some of this. Author: Sean Owen so...@cloudera.com Closes #4838 from srowen/SPARK-3357 and squashes the following commits: dce75c1 [Sean Owen] Back out some debug level changes d9b784d [Sean Owen] Demote some 'noisy' log messages to debug level Project: http://git-wip-us.apache.org/repos

spark git commit: [SPARK-6103][Graphx]remove unused class to import in EdgeRDDImpl

2015-03-02 Thread srowen
Repository: spark Updated Branches: refs/heads/master 948c2390a - 49c7a8f6f [SPARK-6103][Graphx]remove unused class to import in EdgeRDDImpl Class TaskContext is unused in EdgeRDDImpl, so we need to remove it from import list. Author: Lianhui Wang lianhuiwan...@gmail.com Closes #4846 from

spark git commit: [SPARK-6111] Fixed usage string in documentation.

2015-03-02 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-1.3 a3fef2c02 - b2b7f011e [SPARK-6111] Fixed usage string in documentation. Usage info in documentation does not match actual usage info. Doc string usage says ```Usage: network_wordcount.py zk topic``` whereas the actual usage is

spark git commit: [SPARK-5434] [EC2] Preserve spaces in EC2 path

2015-02-27 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-1.2 d4ce702c4 - 6b227413d [SPARK-5434] [EC2] Preserve spaces in EC2 path Fixes [SPARK-5434](https://issues.apache.org/jira/browse/SPARK-5434). Simple demonstration of the problem and the fix: ``` $ spacey_path=/path/with some/spaces $

spark git commit: [SPARK-6059][Yarn] Add volatile to ApplicationMaster's reporterThread and allocator

2015-02-27 Thread srowen
Repository: spark Updated Branches: refs/heads/master e747e9849 - 57566d0af [SPARK-6059][Yarn] Add volatile to ApplicationMaster's reporterThread and allocator `ApplicationMaster.reporterThread` and `ApplicationMaster.allocator` are accessed in multiple threads, so they should be marked as

spark git commit: [SPARK-6058][Yarn] Log the user class exception in ApplicationMaster

2015-02-27 Thread srowen
Repository: spark Updated Branches: refs/heads/master 8cd1692c9 - e747e9849 [SPARK-6058][Yarn] Log the user class exception in ApplicationMaster Because ApplicationMaster doesn't set SparkUncaughtExceptionHandler, the exception in the user class won't be logged. This PR added a `logError`

spark git commit: [SPARK-6058][Yarn] Log the user class exception in ApplicationMaster

2015-02-27 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-1.3 b8db84c5b - bff808894 [SPARK-6058][Yarn] Log the user class exception in ApplicationMaster Because ApplicationMaster doesn't set SparkUncaughtExceptionHandler, the exception in the user class won't be logged. This PR added a

spark git commit: [SPARK-5417] Remove redundant executor-id set() call

2015-02-27 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-1.2 6b227413d - 17b7cc733 [SPARK-5417] Remove redundant executor-id set() call This happens inside SparkEnv initialization as of #4194 Author: Ryan Williams ryan.blake.willi...@gmail.com Closes #4213 from ryan-williams/exec-id-set and

spark git commit: [EXAMPLES] fix typo.

2015-03-04 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-1.2 77a8c0622 - e753f9c9b [EXAMPLES] fix typo. Author: Makoto Fukuhara fuku...@gmail.com Closes #4724 from fukuo33/fix-typo and squashes the following commits: 8c806b9 [Makoto Fukuhara] fix typo. Project:

spark git commit: [EXAMPLES] fix typo.

2015-03-04 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-1.1 d70754df0 - c5836816f [EXAMPLES] fix typo. Author: Makoto Fukuhara fuku...@gmail.com Closes #4724 from fukuo33/fix-typo and squashes the following commits: 8c806b9 [Makoto Fukuhara] fix typo. Project:

spark git commit: SPARK-6085 Increase default value for memory overhead

2015-03-04 Thread srowen
Repository: spark Updated Branches: refs/heads/master 76e20a0a0 - 8d3e2414d SPARK-6085 Increase default value for memory overhead Author: tedyu yuzhih...@gmail.com Closes #4836 from tedyu/master and squashes the following commits: d65b495 [tedyu] SPARK-6085 Increase default value for memory

spark git commit: [SPARK-6107][CORE] Display inprogress application information for event log history for standalone mode

2015-03-04 Thread srowen
Repository: spark Updated Branches: refs/heads/master aef8a84e4 - f6773edce [SPARK-6107][CORE] Display inprogress application information for event log history for standalone mode when application is finished running abnormally (Ctrl + c for example), the history event log file is still

<    1   2   3   4   5   6   7   8   9   10   >