spark git commit: [SPARK-9335] [STREAMING] [TESTS] Make sure the test stream is deleted in KinesisBackedBlockRDDSuite

Repository: spark Updated Branches: refs/heads/master 9c5612f4e -> d93ab93d6 [SPARK-9335] [STREAMING] [TESTS] Make sure the test stream is deleted in KinesisBackedBlockRDDSuite KinesisBackedBlockRDDSuite should make sure delete the stream. Author: zsxwing Closes #7663 from zsxwing/fix-SPAR

spark git commit: [MINOR] [SQL] Support mutable expression unit test with codegen projection

Repository: spark Updated Branches: refs/heads/master 60f08c7c8 -> 9c5612f4e [MINOR] [SQL] Support mutable expression unit test with codegen projection This is actually contains 3 minor issues: 1) Enable the unit test(codegen) for mutable expressions (FormatNumber, Regexp_Replace/Regexp_Extra

spark git commit: [SPARK-9373][SQL] Support StructType in Tungsten projection

Repository: spark Updated Branches: refs/heads/master 63a492b93 -> 60f08c7c8 [SPARK-9373][SQL] Support StructType in Tungsten projection This pull request updates GenerateUnsafeProjection to support StructType. If an input struct type is backed already by an UnsafeRow, GenerateUnsafeProjectio

spark git commit: [SPARK-8828] [SQL] Revert SPARK-5680

Repository: spark Updated Branches: refs/heads/master 3bc7055e2 -> 63a492b93 [SPARK-8828] [SQL] Revert SPARK-5680 JIRA: https://issues.apache.org/jira/browse/SPARK-8828 Author: Yijie Shen Closes #7667 from yjshen/revert_combinesum_2 and squashes the following commits: c37ccb1 [Yijie Shen]

spark git commit: Fixed a test failure.

Repository: spark Updated Branches: refs/heads/master 84da8792e -> 3bc7055e2 Fixed a test failure. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/3bc7055e Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/3bc7055e D

spark git commit: [SPARK-9395][SQL] Create a SpecializedGetters interface to track all the specialized getters.

Repository: spark Updated Branches: refs/heads/master 2e7f99a00 -> 84da8792e [SPARK-9395][SQL] Create a SpecializedGetters interface to track all the specialized getters. As we are adding more and more specialized getters to more classes (coming soon ArrayData), this interface can help us pr

spark git commit: [SPARK-8195] [SPARK-8196] [SQL] udf next_day last_day

Repository: spark Updated Branches: refs/heads/master daa1964b6 -> 2e7f99a00 [SPARK-8195] [SPARK-8196] [SQL] udf next_day last_day next_day, returns next certain dayofweek. last_day, returns the last day of the month which given date belongs to. Author: Daoyuan Wang Closes #6986 from adrian

spark git commit: [SPARK-8882] [STREAMING] Add a new Receiver scheduling mechanism

Repository: spark Updated Branches: refs/heads/master ce89ff477 -> daa1964b6 [SPARK-8882] [STREAMING] Add a new Receiver scheduling mechanism The design doc: https://docs.google.com/document/d/1ZsoRvHjpISPrDmSjsGzuSu8UjwgbtmoCTzmhgTurHJw/edit?usp=sharing Author: zsxwing Closes #7276 from z

spark git commit: [SPARK-9352] [SPARK-9353] Add tests for standalone scheduling code

Repository: spark Updated Branches: refs/heads/branch-1.4 43035b4b4 -> c103c99d2 [SPARK-9352] [SPARK-9353] Add tests for standalone scheduling code This also fixes a small issue in the standalone Master that was uncovered by the new tests. For more detail, read the description of SPARK-9353.

spark git commit: [SPARK-9386] [SQL] Feature flag for metastore partition pruning

Repository: spark Updated Branches: refs/heads/master 8ddfa52c2 -> ce89ff477 [SPARK-9386] [SQL] Feature flag for metastore partition pruning Since we have been seeing a lot of failures related to this new feature, lets put it behind a flag and turn it off by default. Author: Michael Armbrust

spark git commit: [SPARK-9230] [ML] Support StringType features in RFormula

Repository: spark Updated Branches: refs/heads/master dafe8d857 -> 8ddfa52c2 [SPARK-9230] [ML] Support StringType features in RFormula This adds StringType feature support via OneHotEncoder. As part of this task it was necessary to change RFormula to an Estimator, so that factor levels could

spark git commit: [SPARK-9385] [PYSPARK] Enable PEP8 but disable installing pylint.

Repository: spark Updated Branches: refs/heads/master ab6259566 -> dafe8d857 [SPARK-9385] [PYSPARK] Enable PEP8 but disable installing pylint. Instead of disabling all python style check, we should enable PEP8. So, this PR just comments out the part installing pylint. Author: Yin Huai Clos

spark git commit: [SPARK-4352] [YARN] [WIP] Incorporate locality preferences in dynamic allocation requests

Repository: spark Updated Branches: refs/heads/master 2104931d7 -> ab6259566 [SPARK-4352] [YARN] [WIP] Incorporate locality preferences in dynamic allocation requests Currently there's no locality preference for container request in YARN mode, this will affect the performance if fetching dat

spark git commit: [SPARK-9385] [HOT-FIX] [PYSPARK] Comment out Python style check

Repository: spark Updated Branches: refs/heads/master c1be9f309 -> 2104931d7 [SPARK-9385] [HOT-FIX] [PYSPARK] Comment out Python style check https://issues.apache.org/jira/browse/SPARK-9385 Comment out Python style check because of error shown in https://amplab.cs.berkeley.edu/jenkins/job/Sp

spark git commit: [SPARK-8988] [YARN] Make sure driver log links appear in secure cluste…

Repository: spark Updated Branches: refs/heads/master 3ab7525dc -> c1be9f309 [SPARK-8988] [YARN] Make sure driver log links appear in secure cluste… …r mode. The NodeReports API currently used does not work in secure mode since we do not get RM tokens. Instead this patch just uses enviro

spark git commit: [SPARK-9355][SQL] Remove InternalRow.get generic getter call in columnar cache code

Repository: spark Updated Branches: refs/heads/master 8e7d2bee2 -> 3ab7525dc [SPARK-9355][SQL] Remove InternalRow.get generic getter call in columnar cache code Author: Wenchen Fan Closes #7673 from cloud-fan/row-generic-getter-columnar and squashes the following commits: 88b1170 [Wenchen

spark git commit: [SPARK-9378] [SQL] Fixes test case "CTAS with serde"

Repository: spark Updated Branches: refs/heads/master 55946e76f -> 8e7d2bee2 [SPARK-9378] [SQL] Fixes test case "CTAS with serde" This is a proper version of PR #7693 authored by viirya The reason why "CTAS with serde" fails is that the `MetastoreRelation` gets converted to a Parquet data so

spark git commit: [SPARK-9349] [SQL] UDAF cleanup

Repository: spark Updated Branches: refs/heads/master fa84e4a7b -> 55946e76f [SPARK-9349] [SQL] UDAF cleanup https://issues.apache.org/jira/browse/SPARK-9349 With this PR, we only expose `UserDefinedAggregateFunction` (an abstract class) and `MutableAggregationBuffer` (an interface). Other i

spark git commit: Closes #7690 since it has been merged into branch-1.4.

Repository: spark Updated Branches: refs/heads/master 85a50a635 -> fa84e4a7b Closes #7690 since it has been merged into branch-1.4. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/fa84e4a7 Tree: http://git-wip-us.apache.or

spark git commit: [HOTFIX] Disable pylint since it is failing master.

Repository: spark Updated Branches: refs/heads/master 75438422c -> 85a50a635 [HOTFIX] Disable pylint since it is failing master. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/85a50a63 Tree: http://git-wip-us.apache.org/r

spark git commit: [SPARK-9369][SQL] Support IntervalType in UnsafeRow

Repository: spark Updated Branches: refs/heads/master dd9ae7945 -> 75438422c [SPARK-9369][SQL] Support IntervalType in UnsafeRow Author: Wenchen Fan Closes #7688 from cloud-fan/interval and squashes the following commits: 5b36b17 [Wenchen Fan] fix codegen a99ed50 [Wenchen Fan] address comme

spark git commit: [SPARK-9351] [SQL] remove literals from grouping expressions in Aggregate

Repository: spark Updated Branches: refs/heads/master 1f7b3d9dc -> dd9ae7945 [SPARK-9351] [SQL] remove literals from grouping expressions in Aggregate literals in grouping expressions have no effect at all, only make our grouping key bigger, so we should remove them in Optimizer. I also make

spark git commit: [SPARK-7423] [MLLIB] Modify ClassificationModel and Probabalistic model to use Vector.argmax

Repository: spark Updated Branches: refs/heads/master e2f38167f -> 1f7b3d9dc [SPARK-7423] [MLLIB] Modify ClassificationModel and Probabalistic model to use Vector.argmax Use Vector.argmax call instead of converting to dense vector before calculating predictions. Author: George Dittmar Clo

spark git commit: [SPARK-9376] [SQL] use a seed in RandomDataGeneratorSuite

Repository: spark Updated Branches: refs/heads/master c0b7df68f -> e2f38167f [SPARK-9376] [SQL] use a seed in RandomDataGeneratorSuite Make this test deterministic, i.e. make sure this test can be passed no matter how many times we run it. The origin implementation uses a random seed and giv

spark git commit: [SPARK-9366] use task's stageAttemptId in TaskEnd event

Repository: spark Updated Branches: refs/heads/master ecad9d434 -> c0b7df68f [SPARK-9366] use task's stageAttemptId in TaskEnd event Author: Ryan Williams Closes #7681 from ryan-williams/task-stage-attempt and squashes the following commits: d6d5f0f [Ryan Williams] use task's stageAttemptI

spark git commit: Pregel example type fix

Repository: spark Updated Branches: refs/heads/branch-1.4 a671dad62 -> cfca1c5af Pregel example type fix Pregel example to express single source shortest path from https://spark.apache.org/docs/latest/graphx-programming-guide.html#pregel-api does not work due to incorrect type. The reason is

spark git commit: [SPARK-9364] Fix array out of bounds and use-after-free bugs in UnsafeExternalSorter

Repository: spark Updated Branches: refs/heads/master 90006f3c5 -> ecad9d434 [SPARK-9364] Fix array out of bounds and use-after-free bugs in UnsafeExternalSorter This patch fixes two bugs in UnsafeExternalSorter and UnsafeExternalRowSorter: - UnsafeExternalSorter does not properly update fre

spark git commit: [SPARK-9371][SQL] fix SPARK-9371 for branch 1.4

Repository: spark Updated Branches: refs/heads/branch-1.4 cfca1c5af -> 43035b4b4 [SPARK-9371][SQL] fix SPARK-9371 for branch 1.4 a follow up of https://github.com/apache/spark/pull/7684 Author: Wenchen Fan Closes #7690 from cloud-fan/branch-1.4 and squashes the following commits: 450904d [

spark git commit: Pregel example type fix

Repository: spark Updated Branches: refs/heads/master aa19c696e -> 90006f3c5 Pregel example type fix Pregel example to express single source shortest path from https://spark.apache.org/docs/latest/graphx-programming-guide.html#pregel-api does not work due to incorrect type. The reason is tha

spark git commit: [SPARK-4176] [SQL] Supports decimal types with precision > 18 in Parquet

Repository: spark Updated Branches: refs/heads/master 622838165 -> aa19c696e [SPARK-4176] [SQL] Supports decimal types with precision > 18 in Parquet This PR is based on #6796 authored by rtreffer. To support large decimal precisions (> 18), we do the following things in this PR: 1. Making

spark git commit: [SPARK-8405] [DOC] Add how to view logs on Web UI when yarn log aggregation is enabled

Repository: spark Updated Branches: refs/heads/branch-1.4 2b1973dd2 -> a671dad62 [SPARK-8405] [DOC] Add how to view logs on Web UI when yarn log aggregation is enabled Some users may not be aware that the logs are available on Web UI even if Yarn log aggregation is enabled. Update the doc to

spark git commit: [SPARK-8405] [DOC] Add how to view logs on Web UI when yarn log aggregation is enabled

Repository: spark Updated Branches: refs/heads/master 72981bc8f -> 622838165 [SPARK-8405] [DOC] Add how to view logs on Web UI when yarn log aggregation is enabled Some users may not be aware that the logs are available on Web UI even if Yarn log aggregation is enabled. Update the doc to mak

spark git commit: [SPARK-7943] [SPARK-8105] [SPARK-8435] [SPARK-8714] [SPARK-8561] Fixes multi-database support

Repository: spark Updated Branches: refs/heads/master 4ffd3a1db -> 72981bc8f [SPARK-7943] [SPARK-8105] [SPARK-8435] [SPARK-8714] [SPARK-8561] Fixes multi-database support This PR fixes a set of issues related to multi-database. A new data structure `TableIdentifier` is introduced to identify