spark git commit: [SPARK-17927][SQL] Remove dead code in WriterContainer.

2016-10-13 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 44cbb61b3 -> 8543996c3 [SPARK-17927][SQL] Remove dead code in WriterContainer. ## What changes were proposed in this pull request? speculationEnabled and DATASOURCE_OUTPUTPATH seem like just dead code. ## How was this patch tested? Tests

spark git commit: [SPARK-15957][FOLLOW-UP][ML][PYSPARK] Add Python API for RFormula forceIndexLabel.

2016-10-13 Thread yliang
Repository: spark Updated Branches: refs/heads/master 9dc0ca060 -> 44cbb61b3 [SPARK-15957][FOLLOW-UP][ML][PYSPARK] Add Python API for RFormula forceIndexLabel. ## What changes were proposed in this pull request? Follow-up work of #13675, add Python API for ```RFormula forceIndexLabel```. ##

spark git commit: [SPARK-17368][SQL] Add support for value class serialization and deserialization

2016-10-13 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master adc112429 -> 9dc0ca060 [SPARK-17368][SQL] Add support for value class serialization and deserialization ## What changes were proposed in this pull request? Value classes were unsupported because catalyst data types were obtained through

spark git commit: [SPARK-17678][REPL][BRANCH-1.6] Honor spark.replClassServer.port in scala-2.11 repl

2016-10-13 Thread zsxwing
Repository: spark Updated Branches: refs/heads/branch-1.6 585c5657f -> 18b173cfc [SPARK-17678][REPL][BRANCH-1.6] Honor spark.replClassServer.port in scala-2.11 repl ## What changes were proposed in this pull request? Spark 1.6 Scala-2.11 repl doesn't honor "spark.replClassServer.port"

spark git commit: [SPARK-17661][SQL] Consolidate various listLeafFiles implementations

2016-10-13 Thread rxin
Repository: spark Updated Branches: refs/heads/master 7106866c2 -> adc112429 [SPARK-17661][SQL] Consolidate various listLeafFiles implementations ## What changes were proposed in this pull request? There are 4 listLeafFiles-related functions in Spark: - ListingFileCatalog.listLeafFiles

[2/2] spark git commit: [SPARK-17731][SQL][STREAMING] Metrics for structured streaming

2016-10-13 Thread tdas
[SPARK-17731][SQL][STREAMING] Metrics for structured streaming ## What changes were proposed in this pull request? Metrics are needed for monitoring structured streaming apps. Here is the design doc for implementing the necessary metrics.

[1/2] spark git commit: [SPARK-17731][SQL][STREAMING] Metrics for structured streaming

2016-10-13 Thread tdas
Repository: spark Updated Branches: refs/heads/master 08eac3560 -> 7106866c2 http://git-wip-us.apache.org/repos/asf/spark/blob/7106866c/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreSuite.scala

spark git commit: [SPARK-17834][SQL] Fetch the earliest offsets manually in KafkaSource instead of counting on KafkaConsumer

2016-10-13 Thread zsxwing
Repository: spark Updated Branches: refs/heads/branch-2.0 d38f38a09 -> d7fa3e324 [SPARK-17834][SQL] Fetch the earliest offsets manually in KafkaSource instead of counting on KafkaConsumer ## What changes were proposed in this pull request? Because `KafkaConsumer.poll(0)` may update the

spark git commit: [SPARK-17827][SQL] maxColLength type should be Int for String and Binary

2016-10-13 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/master 04d417a7c -> 84f149e41 [SPARK-17827][SQL] maxColLength type should be Int for String and Binary ## What changes were proposed in this pull request? correct the expected type from Length function to be Int ## How was this patch tested?

spark git commit: [SPARK-17830][SQL] Annotate remaining SQL APIs with InterfaceStability

2016-10-13 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/master 0a8e51a5e -> 04d417a7c [SPARK-17830][SQL] Annotate remaining SQL APIs with InterfaceStability ## What changes were proposed in this pull request? This patch annotates all the remaining APIs in SQL (excluding streaming) with

spark git commit: [SPARK-17657][SQL] Disallow Users to Change Table Type

2016-10-13 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 7bf8a4049 -> 0a8e51a5e [SPARK-17657][SQL] Disallow Users to Change Table Type ### What changes were proposed in this pull request? Hive allows users to change the table type from `Managed` to `External` or from `External` to `Managed` by

spark git commit: [SPARK-17850][CORE] Add a flag to ignore corrupt files (branch 1.6)

2016-10-13 Thread mridulm80
Repository: spark Updated Branches: refs/heads/branch-1.6 d3890deb7 -> 585c5657f [SPARK-17850][CORE] Add a flag to ignore corrupt files (branch 1.6) ## What changes were proposed in this pull request? This is the patch for 1.6. It only adds Spark conf `spark.files.ignoreCorruptFiles`

spark git commit: [SPARK-17686][CORE] Support printing out scala and java version with spark-submit --version command

2016-10-13 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master db8784fea -> 7bf8a4049 [SPARK-17686][CORE] Support printing out scala and java version with spark-submit --version command ## What changes were proposed in this pull request? In our universal gateway service we need to specify different

spark git commit: [SPARK-17899][SQL] add a debug mode to keep raw table properties in HiveExternalCatalog

2016-10-13 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 6f2fa6c54 -> db8784fea [SPARK-17899][SQL] add a debug mode to keep raw table properties in HiveExternalCatalog ## What changes were proposed in this pull request? Currently `HiveExternalCatalog` will filter out the Spark SQL internal

spark git commit: [SPARK-11272][WEB UI] Add support for downloading event logs from HistoryServer UI

2016-10-13 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 7222a25a1 -> 6f2fa6c54 [SPARK-11272][WEB UI] Add support for downloading event logs from HistoryServer UI ## What changes were proposed in this pull request? This is a reworked PR based on feedback in #9238 after it was closed and not