spark git commit: [SPARK-8420] [SQL] Fix comparision of timestamps/dates with strings

Repository: spark Updated Branches: refs/heads/master 9814b971f - a333a72e0 [SPARK-8420] [SQL] Fix comparision of timestamps/dates with strings In earlier versions of Spark SQL we casted `TimestampType` and `DataType` to `StringType` when it was involved in a binary comparison with a

spark git commit: [SPARK-8452] [SPARKR] expose jobGroup API in SparkR

Repository: spark Updated Branches: refs/heads/master 54976e55e - 1fa29c2df [SPARK-8452] [SPARKR] expose jobGroup API in SparkR This pull request adds following methods to SparkR: ```R setJobGroup() cancelJobGroup() clearJobGroup() ``` For each method, the spark context is passed as the

spark git commit: [SPARK-8093] [SQL] Remove empty structs inferred from JSON documents

Repository: spark Updated Branches: refs/heads/branch-1.4 1a6b51078 - 0131142d9 [SPARK-8093] [SQL] Remove empty structs inferred from JSON documents Author: Nathan Howell nhow...@godaddy.com Closes #6799 from NathanHowell/spark-8093 and squashes the following commits: 76ac3e8 [Nathan

spark git commit: [SPARK-8093] [SQL] Remove empty structs inferred from JSON documents

Repository: spark Updated Branches: refs/heads/master 1fa29c2df - 9814b971f [SPARK-8093] [SQL] Remove empty structs inferred from JSON documents Author: Nathan Howell nhow...@godaddy.com Closes #6799 from NathanHowell/spark-8093 and squashes the following commits: 76ac3e8 [Nathan Howell]

spark git commit: [SPARK-8390] [STREAMING] [KAFKA] fix docs related to HasOffsetRanges

Repository: spark Updated Branches: refs/heads/master a333a72e0 - b305e377f [SPARK-8390] [STREAMING] [KAFKA] fix docs related to HasOffsetRanges Author: cody koeninger c...@koeninger.org Closes #6863 from koeninger/SPARK-8390 and squashes the following commits: 26a06bd [cody koeninger]

spark git commit: [HOTFIX] [SPARK-8489] Correct JIRA number in previous commit

Repository: spark Updated Branches: refs/heads/master 093c34838 - bec40e52b [HOTFIX] [SPARK-8489] Correct JIRA number in previous commit It should be SPARK-8489, not SPARK-8498. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [HOTFIX] [SPARK-8489] Correct JIRA number in previous commit

Repository: spark Updated Branches: refs/heads/branch-1.4 a7b773a8b - 9b16508d2 [HOTFIX] [SPARK-8489] Correct JIRA number in previous commit It should be SPARK-8489, not SPARK-8498. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [SPARK-8127] [STREAMING] [KAFKA] KafkaRDD optimize count() take() isEmpty()

Repository: spark Updated Branches: refs/heads/master bec40e52b - 1b6fe9b1a [SPARK-8127] [STREAMING] [KAFKA] KafkaRDD optimize count() take() isEmpty() …ed KafkaRDD methods. Possible fix for [SPARK-7122], but probably a worthwhile optimization regardless. Author: cody koeninger

[2/2] spark git commit: [SPARK-8390] [STREAMING] [KAFKA] fix docs related to HasOffsetRanges

[SPARK-8390] [STREAMING] [KAFKA] fix docs related to HasOffsetRanges Author: cody koeninger c...@koeninger.org Closes #6863 from koeninger/SPARK-8390 and squashes the following commits: 26a06bd [cody koeninger] Merge branch 'master' into SPARK-8390 3744492 [cody koeninger]

[1/2] spark git commit: [SPARK-8389] [STREAMING] [KAFKA] Example of getting offset ranges out o…

Repository: spark Updated Branches: refs/heads/branch-1.4 2248ad8b7 - a7b773a8b [SPARK-8389] [STREAMING] [KAFKA] Example of getting offset ranges out o… …f the existing java direct stream api Author: cody koeninger c...@koeninger.org Closes #6846 from koeninger/SPARK-8389 and squashes

spark git commit: [HOT-FIX] Fix compilation (caused by 0131142d98b191f6cc112d383aa10582a3ac35bf)

Repository: spark Updated Branches: refs/heads/branch-1.4 0131142d9 - 2510365fa [HOT-FIX] Fix compilation (caused by 0131142d98b191f6cc112d383aa10582a3ac35bf) Author: Yin Huai yh...@databricks.com Closes #6913 from yhuai/branch-1.4-hotfix and squashes the following commits: 7f91fa0 [Yin

spark git commit: [SPARK-8498] [SQL] Add regression test for SPARK-8470

Repository: spark Updated Branches: refs/heads/branch-1.4 2510365fa - 2248ad8b7 [SPARK-8498] [SQL] Add regression test for SPARK-8470 **Summary of the problem in SPARK-8470.** When using `HiveContext` to create a data frame of a user case class, Spark throws

spark git commit: [SPARK-8498] [SQL] Add regression test for SPARK-8470

Repository: spark Updated Branches: refs/heads/master b305e377f - 093c34838 [SPARK-8498] [SQL] Add regression test for SPARK-8470 **Summary of the problem in SPARK-8470.** When using `HiveContext` to create a data frame of a user case class, Spark throws

spark git commit: [SPARK-8339] [PYSPARK] integer division for python 3

Repository: spark Updated Branches: refs/heads/branch-1.4 f48f3a2e2 - 164b9d32e [SPARK-8339] [PYSPARK] integer division for python 3 Itertools islice requires an integer for the stop argument. Switching to integer division here prevents a ValueError when vs is evaluated above. davies This

spark git commit: [SPARK-8339] [PYSPARK] integer division for python 3

Repository: spark Updated Branches: refs/heads/master a2016b4bc - fdf63f124 [SPARK-8339] [PYSPARK] integer division for python 3 Itertools islice requires an integer for the stop argument. Switching to integer division here prevents a ValueError when vs is evaluated above. davies This is

spark git commit: [SPARK-8348][SQL] Add in operator to DataFrame Column

Repository: spark Updated Branches: refs/heads/master a71cbbdea - 754929b15 [SPARK-8348][SQL] Add in operator to DataFrame Column I have added it for only Scala. TODO: we should also support `in` operator in Python. Author: Yu ISHIKAWA yuu.ishik...@gmail.com Closes #6824 from

spark git commit: [SPARK-8444] [STREAMING] Adding Python streaming example for queueStream

Repository: spark Updated Branches: refs/heads/master 754929b15 - a2016b4bc [SPARK-8444] [STREAMING] Adding Python streaming example for queueStream A Python example similar to the existing one for Scala. Author: Bryan Cutler bjcut...@us.ibm.com Closes #6884 from

spark git commit: [SPARK-7913] [CORE] Make AppendOnlyMap use the same growth strategy of OpenHashSet and consistent exception message

Repository: spark Updated Branches: refs/heads/master 54557f353 - 93360dc3c [SPARK-7913] [CORE] Make AppendOnlyMap use the same growth strategy of OpenHashSet and consistent exception message This is a follow up PR for #6456 to make AppendOnlyMap consistent with OpenHashSet. /cc srowen

spark git commit: [SPARK-8389] [STREAMING] [KAFKA] Example of getting offset ranges out o…

Repository: spark Updated Branches: refs/heads/master ebd363aec - 47af7c1eb [SPARK-8389] [STREAMING] [KAFKA] Example of getting offset ranges out o… …f the existing java direct stream api Author: cody koeninger c...@koeninger.org Closes #6846 from koeninger/SPARK-8389 and squashes the

spark git commit: [SPARK-7265] Improving documentation for Spark SQL Hive support

Repository: spark Updated Branches: refs/heads/master 93360dc3c - ebd363aec [SPARK-7265] Improving documentation for Spark SQL Hive support Please review this pull request. Author: Jihong MA linlin200...@gmail.com Closes #5933 from JihongMA/SPARK-7265 and squashes the following commits:

spark git commit: [SPARK-8387] [FOLLOWUP ] [WEBUI] Update driver log URL to show only 4096 bytes

Repository: spark Updated Branches: refs/heads/master fdf63f124 - 54557f353 [SPARK-8387] [FOLLOWUP ] [WEBUI] Update driver log URL to show only 4096 bytes This is to follow up #6834 , update the driver log URL as well for consistency. Author: Carson Wang carson.w...@intel.com Closes #6878

spark git commit: [SPARK-8151] [MLLIB] pipeline components should correctly implement copy

Repository: spark Updated Branches: refs/heads/branch-1.4 164b9d32e - 1f2dafb77 [SPARK-8151] [MLLIB] pipeline components should correctly implement copy Otherwise, extra params get ignored in `PipelineModel.transform`. jkbradley Author: Xiangrui Meng m...@databricks.com Closes #6622 from

spark git commit: [SPARK-8207] [SQL] Add math function bin

Repository: spark Updated Branches: refs/heads/master 43c7ec638 - 2c59d5c12 [SPARK-8207] [SQL] Add math function bin JIRA: https://issues.apache.org/jira/browse/SPARK-8207 Author: Liang-Chi Hsieh vii...@gmail.com Closes #6721 from viirya/expr_bin and squashes the following commits: 07e1c8f

spark git commit: [SPARK-8151] [MLLIB] pipeline components should correctly implement copy

Repository: spark Updated Branches: refs/heads/master 47af7c1eb - 43c7ec638 [SPARK-8151] [MLLIB] pipeline components should correctly implement copy Otherwise, extra params get ignored in `PipelineModel.transform`. jkbradley Author: Xiangrui Meng m...@databricks.com Closes #6622 from

spark git commit: [SPARK-8476] [CORE] Setters inc/decDiskBytesSpilled in TaskMetrics should also be private.

Repository: spark Updated Branches: refs/heads/master 9baf09301 - fe08561e2 [SPARK-8476] [CORE] Setters inc/decDiskBytesSpilled in TaskMetrics should also be private. This is a follow-up of [SPARK-3288](https://issues.apache.org/jira/browse/SPARK-3288). Author: Takuya UESHIN

spark git commit: Add example that reads a local file, writes to a DFS path provided by th...

Repository: spark Updated Branches: refs/heads/master 0c32fc125 - a9858036b Add example that reads a local file, writes to a DFS path provided by th... ...e user, reads the file back from the DFS, and compares word counts on the local and DFS versions. Useful for verifying DFS correctness.

spark git commit: [SPARK-8451] [SPARK-7287] SparkSubmitSuite should check exit code

Repository: spark Updated Branches: refs/heads/branch-1.4 6f2e41108 - aedd893b4 [SPARK-8451] [SPARK-7287] SparkSubmitSuite should check exit code This patch also reenables the tests. Now that we have access to the log4j logs it should be easier to debug the flakiness. yhuai brkyvz Author:

spark git commit: [SPARK-8451] [SPARK-7287] SparkSubmitSuite should check exit code

Repository: spark Updated Branches: refs/heads/master 866816eb9 - 68a2dca29 [SPARK-8451] [SPARK-7287] SparkSubmitSuite should check exit code This patch also reenables the tests. Now that we have access to the log4j logs it should be easier to debug the flakiness. yhuai brkyvz Author:

spark git commit: [SPARK-5836] [DOCS] [STREAMING] Clarify what may cause long-running Spark apps to preserve shuffle files

Repository: spark Updated Branches: refs/heads/master 68a2dca29 - 4be53d039 [SPARK-5836] [DOCS] [STREAMING] Clarify what may cause long-running Spark apps to preserve shuffle files Clarify what may cause long-running Spark apps to preserve shuffle files Author: Sean Owen so...@cloudera.com

spark git commit: [SPARK-8234][SQL] misc function: md5

Repository: spark Updated Branches: refs/heads/master fe08561e2 - 0c32fc125 [SPARK-8234][SQL] misc function: md5 Author: Shilei shilei.q...@intel.com Closes #6779 from qiansl127/MD5 and squashes the following commits: 11fcdb2 [Shilei] Fix the indent 04bd27b [Shilei] Add codegen da60eb3

spark git commit: [SPARK-8451] [SPARK-7287] SparkSubmitSuite should check exit code

Repository: spark Updated Branches: refs/heads/branch-1.3 cf232f0ee - 1d441475d [SPARK-8451] [SPARK-7287] SparkSubmitSuite should check exit code This patch also reenables the tests. Now that we have access to the log4j logs it should be easier to debug the flakiness. yhuai brkyvz Author:

spark git commit: [SPARK-5836] [DOCS] [STREAMING] Clarify what may cause long-running Spark apps to preserve shuffle files

Repository: spark Updated Branches: refs/heads/branch-1.4 aedd893b4 - 3415fb978 [SPARK-5836] [DOCS] [STREAMING] Clarify what may cause long-running Spark apps to preserve shuffle files Clarify what may cause long-running Spark apps to preserve shuffle files Author: Sean Owen

spark git commit: [SPARK-5836] [DOCS] [STREAMING] Clarify what may cause long-running Spark apps to preserve shuffle files

Repository: spark Updated Branches: refs/heads/branch-1.3 1d441475d - 0b8dce0c0 [SPARK-5836] [DOCS] [STREAMING] Clarify what may cause long-running Spark apps to preserve shuffle files Clarify what may cause long-running Spark apps to preserve shuffle files Author: Sean Owen

spark git commit: [SPARK-8368] [SPARK-8058] [SQL] HiveContext may override the context class loader of the current thread

Repository: spark Updated Branches: refs/heads/master 4be53d039 - c5876e529 [SPARK-8368] [SPARK-8058] [SQL] HiveContext may override the context class loader of the current thread https://issues.apache.org/jira/browse/SPARK-8368 Also, I add tests according

spark git commit: [SPARK-8430] ExternalShuffleBlockResolver of shuffle service should support UnsafeShuffleManager

Repository: spark Updated Branches: refs/heads/master 2c59d5c12 - 9baf09301 [SPARK-8430] ExternalShuffleBlockResolver of shuffle service should support UnsafeShuffleManager andrewor14 can you take a look?thanks Author: Lianhui Wang lianhuiwan...@gmail.com Closes #6873 from

spark git commit: [SPARK-8430] ExternalShuffleBlockResolver of shuffle service should support UnsafeShuffleManager

Repository: spark Updated Branches: refs/heads/branch-1.4 1f2dafb77 - 6f2e41108 [SPARK-8430] ExternalShuffleBlockResolver of shuffle service should support UnsafeShuffleManager andrewor14 can you take a look?thanks Author: Lianhui Wang lianhuiwan...@gmail.com Closes #6873 from

spark git commit: [SPARK-8368] [SPARK-8058] [SQL] HiveContext may override the context class loader of the current thread (branch 1.4)

Repository: spark Updated Branches: refs/heads/branch-1.4 4b2c793a2 - 9ac839366 [SPARK-8368] [SPARK-8058] [SQL] HiveContext may override the context class loader of the current thread (branch 1.4) This is for 1.4 branch (based on https://github.com/apache/spark/pull/6891). Author: Yin Huai

spark git commit: [HOTFIX] Fix scala style in DFSReadWriteTest that causes tests failed

Repository: spark Updated Branches: refs/heads/master c5876e529 - 4a462c282 [HOTFIX] Fix scala style in DFSReadWriteTest that causes tests failed This scala style problem causes tested failed. Author: Liang-Chi Hsieh vii...@gmail.com Closes #6907 from viirya/hotfix_style and squashes the

spark git commit: [SPARK-8461] [SQL] fix codegen with REPL class loader

Repository: spark Updated Branches: refs/heads/master 4a462c282 - e41e2fd6c [SPARK-8461] [SQL] fix codegen with REPL class loader The ExecutorClassLoader for REPL will cause Janino failed to find class for those in java.lang, so switch to use default class loader for Janino, which will also

spark git commit: [SPARK-4118] [MLLIB] [PYSPARK] Python bindings for StreamingKMeans

Repository: spark Updated Branches: refs/heads/master e41e2fd6c - 54976e55e [SPARK-4118] [MLLIB] [PYSPARK] Python bindings for StreamingKMeans Python bindings for StreamingKMeans Will change status to MRG once docs, tests and examples are updated. Author: MechCoder