spark git commit: [SPARK-15756][SQL] Support command 'create table stored as orcfile/parquetfile/avrofile'

2016-06-03 Thread rxin
Repository: spark Updated Branches: refs/heads/master 61d729abd -> 2ca563cc4 [SPARK-15756][SQL] Support command 'create table stored as orcfile/parquetfile/avrofile' ## What changes were proposed in this pull request? Now Spark SQL can support 'create table src stored as orc/parquet/avro'

spark git commit: [SPARK-15756][SQL] Support command 'create table stored as orcfile/parquetfile/avrofile'

2016-06-03 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 a2540b936 -> cf8782116 [SPARK-15756][SQL] Support command 'create table stored as orcfile/parquetfile/avrofile' ## What changes were proposed in this pull request? Now Spark SQL can support 'create table src stored as

spark git commit: [SPARK-15754][YARN] Not letting the credentials containing hdfs delegation tokens to be added in current user credential.

2016-06-03 Thread vanzin
Repository: spark Updated Branches: refs/heads/master 3074f575a -> 61d729abd [SPARK-15754][YARN] Not letting the credentials containing hdfs delegation tokens to be added in current user credential. ## What changes were proposed in this pull request? The credentials are not added to the

spark git commit: [SPARK-15754][YARN] Not letting the credentials containing hdfs delegation tokens to be added in current user credential.

2016-06-03 Thread vanzin
Repository: spark Updated Branches: refs/heads/branch-1.6 4259a2858 -> a0cf7d0b2 [SPARK-15754][YARN] Not letting the credentials containing hdfs delegation tokens to be added in current user credential. ## What changes were proposed in this pull request? The credentials are not added to the

spark git commit: [SPARK-15754][YARN] Not letting the credentials containing hdfs delegation tokens to be added in current user credential.

2016-06-03 Thread vanzin
Repository: spark Updated Branches: refs/heads/branch-2.0 849fe3534 -> a2540b936 [SPARK-15754][YARN] Not letting the credentials containing hdfs delegation tokens to be added in current user credential. ## What changes were proposed in this pull request? The credentials are not added to the

spark git commit: [SPARK-15391] [SQL] manage the temporary memory of timsort

2016-06-03 Thread davies
Repository: spark Updated Branches: refs/heads/branch-2.0 f702e9941 -> 849fe3534 [SPARK-15391] [SQL] manage the temporary memory of timsort ## What changes were proposed in this pull request? Currently, the memory for temporary buffer used by TimSort is always allocated as on-heap without

spark git commit: [SPARK-15391] [SQL] manage the temporary memory of timsort

2016-06-03 Thread davies
Repository: spark Updated Branches: refs/heads/master 67cc89ff0 -> 3074f575a [SPARK-15391] [SQL] manage the temporary memory of timsort ## What changes were proposed in this pull request? Currently, the memory for temporary buffer used by TimSort is always allocated as on-heap without

spark git commit: [SPARK-15168][PYSPARK][ML] Add missing params to MultilayerPerceptronClassifier

2016-06-03 Thread mlnick
Repository: spark Updated Branches: refs/heads/branch-2.0 3670b2c64 -> f702e9941 [SPARK-15168][PYSPARK][ML] Add missing params to MultilayerPerceptronClassifier ## What changes were proposed in this pull request? MultilayerPerceptronClassifier is missing step size, solver, and weights. Add

spark git commit: [SPARK-15168][PYSPARK][ML] Add missing params to MultilayerPerceptronClassifier

2016-06-03 Thread mlnick
Repository: spark Updated Branches: refs/heads/master b1cc7da3e -> 67cc89ff0 [SPARK-15168][PYSPARK][ML] Add missing params to MultilayerPerceptronClassifier ## What changes were proposed in this pull request? MultilayerPerceptronClassifier is missing step size, solver, and weights. Add

spark git commit: [SPARK-15722][SQL] Disallow specifying schema in CTAS statement

2016-06-03 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.0 7315acf89 -> 3670b2c64 [SPARK-15722][SQL] Disallow specifying schema in CTAS statement ## What changes were proposed in this pull request? As of this patch, the following throws an exception because the schemas may not match: ```

spark git commit: [SPARK-15722][SQL] Disallow specifying schema in CTAS statement

2016-06-03 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 11c83f83d -> b1cc7da3e [SPARK-15722][SQL] Disallow specifying schema in CTAS statement ## What changes were proposed in this pull request? As of this patch, the following throws an exception because the schemas may not match: ``` CREATE

spark git commit: [SPARK-15140][SQL] make the semantics of null input object for encoder clear

2016-06-03 Thread lian
Repository: spark Updated Branches: refs/heads/branch-2.0 52376e067 -> 7315acf89 [SPARK-15140][SQL] make the semantics of null input object for encoder clear ## What changes were proposed in this pull request? For input object of non-flat type, we can't encode it to row if it's null, as

spark git commit: [SPARK-15140][SQL] make the semantics of null input object for encoder clear

2016-06-03 Thread lian
Repository: spark Updated Branches: refs/heads/master 28ad0f7b0 -> 11c83f83d [SPARK-15140][SQL] make the semantics of null input object for encoder clear ## What changes were proposed in this pull request? For input object of non-flat type, we can't encode it to row if it's null, as Spark

spark git commit: [SPARK-15681][CORE] allow lowercase or mixed case log level string when calling sc.setLogLevel

2016-06-03 Thread vanzin
Repository: spark Updated Branches: refs/heads/branch-2.0 35765a0e9 -> 52376e067 [SPARK-15681][CORE] allow lowercase or mixed case log level string when calling sc.setLogLevel ## What changes were proposed in this pull request? Currently `SparkContext API setLogLevel(level: String) `can not

spark git commit: [SPARK-15681][CORE] allow lowercase or mixed case log level string when calling sc.setLogLevel

2016-06-03 Thread vanzin
Repository: spark Updated Branches: refs/heads/master 61b80d552 -> 28ad0f7b0 [SPARK-15681][CORE] allow lowercase or mixed case log level string when calling sc.setLogLevel ## What changes were proposed in this pull request? Currently `SparkContext API setLogLevel(level: String) `can not

spark git commit: [SPARK-15547][SQL] nested case class in encoder can have different number of fields from the real schema

2016-06-03 Thread lian
Repository: spark Updated Branches: refs/heads/master eb10b481c -> 61b80d552 [SPARK-15547][SQL] nested case class in encoder can have different number of fields from the real schema ## What changes were proposed in this pull request? There are 2 kinds of `GetStructField`: 1. resolved from

spark git commit: [SPARK-15286][SQL] Make the output readable for EXPLAIN CREATE TABLE and DESC EXTENDED

2016-06-03 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.0 4018c4800 -> 35765a0e9 [SPARK-15286][SQL] Make the output readable for EXPLAIN CREATE TABLE and DESC EXTENDED What changes were proposed in this pull request? Before this PR, the output of EXPLAIN of following SQL is like ```SQL

spark git commit: [SPARK-15286][SQL] Make the output readable for EXPLAIN CREATE TABLE and DESC EXTENDED

2016-06-03 Thread wenchen
Repository: spark Updated Branches: refs/heads/master e52691398 -> eb10b481c [SPARK-15286][SQL] Make the output readable for EXPLAIN CREATE TABLE and DESC EXTENDED What changes were proposed in this pull request? Before this PR, the output of EXPLAIN of following SQL is like ```SQL

spark git commit: [SPARK-15742][SQL] Reduce temp collections allocations in TreeNode transform methods

2016-06-03 Thread joshrosen
Repository: spark Updated Branches: refs/heads/branch-2.0 bcf0c51b6 -> 4018c4800 [SPARK-15742][SQL] Reduce temp collections allocations in TreeNode transform methods In Catalyst's TreeNode transform methods we end up calling `productIterator.map(...).toArray` in a number of places, which is

spark git commit: [SPARK-15742][SQL] Reduce temp collections allocations in TreeNode transform methods

2016-06-03 Thread joshrosen
Repository: spark Updated Branches: refs/heads/master efd3b11a4 -> e52691398 [SPARK-15742][SQL] Reduce temp collections allocations in TreeNode transform methods In Catalyst's TreeNode transform methods we end up calling `productIterator.map(...).toArray` in a number of places, which is

spark git commit: [SPARK-15665][CORE] spark-submit --kill and --status are not working

2016-06-03 Thread vanzin
Repository: spark Updated Branches: refs/heads/branch-2.0 396be560d -> bcf0c51b6 [SPARK-15665][CORE] spark-submit --kill and --status are not working ## What changes were proposed in this pull request? --kill and --status were not considered while handling in OptionParser and due to that it

spark git commit: [SPARK-15665][CORE] spark-submit --kill and --status are not working

2016-06-03 Thread vanzin
Repository: spark Updated Branches: refs/heads/master 9e2eb13ca -> efd3b11a4 [SPARK-15665][CORE] spark-submit --kill and --status are not working ## What changes were proposed in this pull request? --kill and --status were not considered while handling in OptionParser and due to that it was

spark git commit: [SPARK-15677][SQL] Query with scalar sub-query in the SELECT list throws UnsupportedOperationException

2016-06-03 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 8fa00dd05 -> 9e2eb13ca [SPARK-15677][SQL] Query with scalar sub-query in the SELECT list throws UnsupportedOperationException ## What changes were proposed in this pull request? Queries with scalar sub-query in the SELECT list run against

spark git commit: [SPARK-15677][SQL] Query with scalar sub-query in the SELECT list throws UnsupportedOperationException

2016-06-03 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.0 da29762f1 -> 396be560d [SPARK-15677][SQL] Query with scalar sub-query in the SELECT list throws UnsupportedOperationException ## What changes were proposed in this pull request? Queries with scalar sub-query in the SELECT list run

spark git commit: [SPARK-15737][CORE] fix jetty warning

2016-06-03 Thread vanzin
Repository: spark Updated Branches: refs/heads/branch-2.0 deda53a9e -> da29762f1 [SPARK-15737][CORE] fix jetty warning ## What changes were proposed in this pull request? After upgrading Jetty to 9.2, we always see "WARN org.eclipse.jetty.server.handler.AbstractHandler: No Server set for

spark git commit: [SPARK-15737][CORE] fix jetty warning

2016-06-03 Thread vanzin
Repository: spark Updated Branches: refs/heads/master c2f0cb4f6 -> 8fa00dd05 [SPARK-15737][CORE] fix jetty warning ## What changes were proposed in this pull request? After upgrading Jetty to 9.2, we always see "WARN org.eclipse.jetty.server.handler.AbstractHandler: No Server set for

spark git commit: [SPARK-15714][CORE] Fix flaky o.a.s.scheduler.BlacklistIntegrationSuite

2016-06-03 Thread irashid
Repository: spark Updated Branches: refs/heads/master 190ff274f -> c2f0cb4f6 [SPARK-15714][CORE] Fix flaky o.a.s.scheduler.BlacklistIntegrationSuite ## What changes were proposed in this pull request? BlacklistIntegrationSuite (introduced by SPARK-10372) is a bit flaky because of some race

spark git commit: [SPARK-15494][SQL] encoder code cleanup

2016-06-03 Thread lian
Repository: spark Updated Branches: refs/heads/branch-2.0 3a9ee549c -> deda53a9e [SPARK-15494][SQL] encoder code cleanup ## What changes were proposed in this pull request? Our encoder framework has been evolved a lot, this PR tries to clean up the code to make it more readable and

spark git commit: [SPARK-15494][SQL] encoder code cleanup

2016-06-03 Thread lian
Repository: spark Updated Branches: refs/heads/master b9fcfb3bd -> 190ff274f [SPARK-15494][SQL] encoder code cleanup ## What changes were proposed in this pull request? Our encoder framework has been evolved a lot, this PR tries to clean up the code to make it more readable and emphasise

spark git commit: [SPARK-15744][SQL] Rename two TungstenAggregation*Suites and update codgen/error messages/comments

2016-06-03 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 306601282 -> 3a9ee549c [SPARK-15744][SQL] Rename two TungstenAggregation*Suites and update codgen/error messages/comments ## What changes were proposed in this pull request? For consistency, this PR updates some remaining

spark git commit: [SPARK-15744][SQL] Rename two TungstenAggregation*Suites and update codgen/error messages/comments

2016-06-03 Thread rxin
Repository: spark Updated Branches: refs/heads/master f7288e166 -> b9fcfb3bd [SPARK-15744][SQL] Rename two TungstenAggregation*Suites and update codgen/error messages/comments ## What changes were proposed in this pull request? For consistency, this PR updates some remaining

spark git commit: [SPARK-15745][SQL] Use classloader's getResource() for reading resource files in HiveTests

2016-06-03 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 1e13d09c5 -> 306601282 [SPARK-15745][SQL] Use classloader's getResource() for reading resource files in HiveTests ## What changes were proposed in this pull request? This is a cleaner approach in general but my motivation behind this

spark git commit: [SPARK-15745][SQL] Use classloader's getResource() for reading resource files in HiveTests

2016-06-03 Thread rxin
Repository: spark Updated Branches: refs/heads/master 76aa45d35 -> f7288e166 [SPARK-15745][SQL] Use classloader's getResource() for reading resource files in HiveTests ## What changes were proposed in this pull request? This is a cleaner approach in general but my motivation behind this