[GitHub] spark issue #17312: [SPARK-19973] Display num of executors for the stage.

2017-03-17 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17312 Can you put a screenshot here? Might actually be useful to have. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #17318: [SPARK-19896][SQL] Throw an exception if case classes ha...

2017-03-17 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17318 Can you put the after exception in the pr description as well? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #17337: [SQL][MINOR] Fix scaladoc for UDFRegistration

2017-03-17 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17337 Merging in master/branch-2.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #17330: [SPARK-19993][SQL][WIP] Caching logical plans con...

2017-03-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17330#discussion_r106758290 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/subquery.scala --- @@ -61,6 +63,36 @@ abstract class SubqueryExpression

[GitHub] spark issue #16483: [SPARK-18847][GraphX] PageRank gives incorrect results f...

2017-03-17 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16483 Merging in master. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #17322: [SPARK-19987][SQL] Pass all filters into FileInde...

2017-03-16 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/17322 [SPARK-19987][SQL] Pass all filters into FileIndex ## What changes were proposed in this pull request? This is a tiny teeny refactoring to pass data filters also to the FileIndex, so FileIndex

[GitHub] spark issue #17191: [SPARK-14471][SQL] Aliases in SELECT could be used in GR...

2017-03-16 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17191 I personally have run into this issue and was surprised that we didn't support it ... it's pretty verbose to retype everything. If Postgres and MySQL both support it, I think we should do

[GitHub] spark issue #17303: [SPARK-19112][CORE] add codec for ZStandard

2017-03-15 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17303 Yes it'd be nice to have some benchmark on this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #17273: [MINOR][CORE] Fix a info message of `prunePartitions` in...

2017-03-15 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17273 Merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #17304: [SPARK-19960][CORE] Move `SparkHadoopWriter` to `interna...

2017-03-15 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17304 Merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #17301: [SPARK-19944][SQL] Move SQLConf from sql/core to ...

2017-03-15 Thread rxin
Github user rxin closed the pull request at: https://github.com/apache/spark/pull/17301 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #17166: [SPARK-19820] [core] Allow reason to be specified for ta...

2017-03-15 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17166 hm it might be useful to have details, but it'd also be useful to have this in the overview page without having to drill down. iiuc, the pr already has the information in task list page, doesn't

[GitHub] spark pull request #17301: [SPARK-19944][SQL] Move SQLConf from sql/core to ...

2017-03-15 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/17301 [SPARK-19944][SQL] Move SQLConf from sql/core to sql/catalyst (branch-2.1) ## What changes were proposed in this pull request? This patch moves SQLConf from sql/core to sql/catalyst. To minimize

[GitHub] spark issue #17273: [MINOR][CORE] No need to call `prunePartitions` in case ...

2017-03-15 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17273 I'd fix the log msg instead. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #17292: DebugFilesystem.assertNoOpenStreams should report...

2017-03-15 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17292#discussion_r106093910 --- Diff: core/src/test/scala/org/apache/spark/SparkContextSuite.scala --- @@ -537,6 +539,21 @@ class SparkContextSuite extends SparkFunSuite

[GitHub] spark issue #17264: [SPARK-19923][SQL] Remove unnecessary type conversions p...

2017-03-14 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17264 In the future can we put the perf result in PR descriptions? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request #17285: [SPARK-19944][SQL] Move SQLConf from sql/core to ...

2017-03-14 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17285#discussion_r105976759 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SimpleCatalystConf.scala --- @@ -0,0 +1,48 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #17285: [SPARK-19944][SQL] Move SQLConf from sql/core to ...

2017-03-13 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/17285 [SPARK-19944][SQL] Move SQLConf from sql/core to sql/catalyst ## What changes were proposed in this pull request? This patch moves SQLConf from sql/core to sql/catalyst. To minimize the changes

[GitHub] spark issue #16541: [SPARK-19088][SQL] Optimize sequence type deserializatio...

2017-03-10 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16541 I didn't look into the details here, but very often scanning data twice doesn't necessarily slow things down, especially in the case of sequential scan. --- If your project is set up for it, you

[GitHub] spark pull request #16826: [SPARK-19540][SQL] Add ability to clone SparkSess...

2017-03-10 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16826#discussion_r105506911 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/internal/SessionState.scala --- @@ -17,43 +17,70 @@ package org.apache.spark.sql.internal

[GitHub] spark pull request #17241: [SPARK-19877][SQL] Restrict the nested level of a...

2017-03-10 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17241#discussion_r105453191 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -595,6 +594,11 @@ class Analyzer( case view

[GitHub] spark issue #17241: [SPARK-19877][SQL] Restrict the nested level of a view

2017-03-10 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17241 SGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #17244: [SPARK-19889][SQL] Make TaskContext callbacks thread saf...

2017-03-10 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17244 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #17220: [SPARK-19862] 'tungsten-sort' should be deleted in Spark...

2017-03-09 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17220 I don't think you understand this. This value is here so if at some point some user picked tungsten-sort, we won't break it. In recent versions of Spark the default sort manager accomplishes the thing

[GitHub] spark issue #17220: [SPARK-19862] 'tungsten-sort' should be deleted in Spark...

2017-03-09 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17220 If anything, we should just update the file to add a line of comment to make sure people don't delete this in the future. --- If your project is set up for it, you can reply to this email and have

[GitHub] spark issue #17220: [SPARK-19862] 'tungsten-sort' should be deleted in Spark...

2017-03-09 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17220 Is this change even correct? This is here for backward compatibility. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request #17202: [SPARK-19861][SS] watermark should not be a negat...

2017-03-08 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17202#discussion_r104983300 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -576,6 +576,11 @@ class Dataset[T] private[sql]( val parsedDelay

[GitHub] spark pull request #17202: [SPARK-19861][SS] watermark should not be a negat...

2017-03-08 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17202#discussion_r104983221 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -563,7 +563,7 @@ class Dataset[T] private[sql]( * @param eventTime the name

[GitHub] spark issue #17205: [SPARK-19843] [SQL] [Followup] Classdoc for `IntWrapper`...

2017-03-08 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17205 Merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #17205: [SPARK-19843] [SQL] [Followup] Classdoc for `IntWrapper`...

2017-03-07 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17205 LGTM too --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #17184: [SPARK-19843] [SQL] UTF8String => (int / long) co...

2017-03-07 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17184#discussion_r104845661 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java --- @@ -897,41 +898,52 @@ public long toLong() { break

[GitHub] spark issue #17184: [SPARK-19843] [SQL] UTF8String => (int / long) conversio...

2017-03-07 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17184 I believe IBM J9 actually improved this specific case (their JIT handles tons of exceptions better). Oh well -- if only JIT is perfect. --- If your project is set up for it, you can reply

[GitHub] spark pull request #17184: [SPARK-19843] [SQL] UTF8String => (int / long) co...

2017-03-07 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17184#discussion_r104841789 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java --- @@ -897,41 +898,52 @@ public long toLong() { break

[GitHub] spark pull request #17184: [SPARK-19843] [SQL] UTF8String => (int / long) co...

2017-03-07 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17184#discussion_r104841761 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java --- @@ -897,41 +898,52 @@ public long toLong() { break

[GitHub] spark pull request #17184: [SPARK-19843] [SQL] UTF8String => (int / long) co...

2017-03-07 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17184#discussion_r104841735 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java --- @@ -850,26 +850,27 @@ public UTF8String translate(Map<Charac

[GitHub] spark pull request #17196: [SPARK-19855][SQL] Create an internal FilePartiti...

2017-03-07 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17196#discussion_r104804384 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FilePartitionStrategy.scala --- @@ -0,0 +1,156 @@ +/* + * Licensed

[GitHub] spark pull request #16958: [SPARK-13721][SQL] Make GeneratorOuter unresolved...

2017-03-07 Thread rxin
Github user rxin closed the pull request at: https://github.com/apache/spark/pull/16958 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request #17196: [SPARK-19855][SQL] Create an internal FilePartiti...

2017-03-07 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17196#discussion_r104798525 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FilePartitionStrategy.scala --- @@ -0,0 +1,156 @@ +/* + * Licensed

[GitHub] spark pull request #17196: [SPARK-19855][SQL] Create an internal FilePartiti...

2017-03-07 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/17196 [SPARK-19855][SQL] Create an internal FilePartitionStrategy interface ## What changes were proposed in this pull request? The way we currently do file partitioning strategy is hard coded

[GitHub] spark pull request #17166: [SPARK-19820] [core] Allow reason to be specified...

2017-03-06 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17166#discussion_r104595706 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -2250,6 +2250,25 @@ class SparkContext(config: SparkConf) extends Logging

[GitHub] spark pull request #17166: [SPARK-19820] [core] Allow reason to be specified...

2017-03-06 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17166#discussion_r104593920 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedClusterMessage.scala --- @@ -40,7 +40,8 @@ private[spark] object

[GitHub] spark pull request #17166: [SPARK-19820] [core] Allow reason to be specified...

2017-03-06 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17166#discussion_r104593825 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -732,6 +732,13 @@ class DAGScheduler

[GitHub] spark pull request #17166: [SPARK-19820] [core] Allow reason to be specified...

2017-03-06 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17166#discussion_r104593790 --- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala --- @@ -158,7 +158,8 @@ private[spark] class Executor( threadPool.execute(tr

[GitHub] spark pull request #17166: [SPARK-19820] [core] Allow reason to be specified...

2017-03-06 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17166#discussion_r104593710 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -2250,6 +2250,25 @@ class SparkContext(config: SparkConf) extends Logging

[GitHub] spark pull request #17166: [SPARK-19820] [core] Allow reason to be specified...

2017-03-06 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17166#discussion_r104593724 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -2250,6 +2250,25 @@ class SparkContext(config: SparkConf) extends Logging

[GitHub] spark issue #15928: [SPARK-18478][SQL] Support codegen'd Hive UDFs

2017-03-02 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15928 What do you mean? The improvement was small? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17114: [SPARK-19758][SQL] Resolving timezone aware expressions ...

2017-02-28 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17114 Put the test case in a sql file? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #17099: [SPARK-19766][SQL] Constant alias columns in INNE...

2017-02-28 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17099#discussion_r103501851 --- Diff: sql/core/src/test/resources/sql-tests/inputs/inner-join.sql --- @@ -0,0 +1,25 @@ +CREATE TEMPORARY VIEW t1 AS SELECT * FROM VALUES (1

[GitHub] spark issue #17049: [SPARK-17495] [SQL] Add more tests for hive hash

2017-02-24 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17049 Merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #17053: [SPARK-18939][SQL] Timezone support in partition ...

2017-02-23 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17053#discussion_r102889140 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalog.scala --- @@ -251,7 +251,8 @@ abstract class ExternalCatalog

[GitHub] spark issue #17049: [SPARK-17495] [SQL] Add more tests for hive hash

2017-02-23 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17049 Looks good except that comment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #17049: [SPARK-17495] [SQL] Add more tests for hive hash

2017-02-23 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17049#discussion_r102881054 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/HashExpressionsSuite.scala --- @@ -71,6 +75,242 @@ class HashExpressionsSuite

[GitHub] spark issue #17002: [SPARK-19669][SQL] Open up visibility for sharedState, s...

2017-02-21 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17002 Yea @gatorsmile be careful in the future and check the commit hash. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request #17002: [SPARK-19669][SQL] Open up visibility for sharedS...

2017-02-20 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17002#discussion_r102070142 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala --- @@ -95,16 +95,26 @@ class SparkSession private( /** * State

[GitHub] spark pull request #17002: [SPARK-19669][SQL] Open up visibility for sharedS...

2017-02-20 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/17002 [SPARK-19669][SQL] Open up visibility for sharedState, sessionState, and a few other functions ## What changes were proposed in this pull request? To ease debugging, most of Spark SQL internals

[GitHub] spark issue #16977: [SPARK-19651][CORE] ParallelCollectionRDD.collect should...

2017-02-19 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16977 Are tests flaky right now? Otherwise it seems like this has introduced legitimate issue with the test timing out. Three times in a row. --- If your project is set up for it, you can reply

[GitHub] spark issue #16960: [SPARK-19447] Make Range operator generate "recordsRead"...

2017-02-18 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16960 Merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #16960: [SPARK-19447] Make Range operator generate "recordsRead"...

2017-02-16 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16960 cc @hvanhovell if you have a min to review this ... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request #16960: [SPARK-19447] Make Range operator generate "recor...

2017-02-16 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16960#discussion_r101575264 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsSuite.scala --- @@ -309,4 +314,84 @@ class SQLMetricsSuite extends

[GitHub] spark pull request #16960: [SPARK-19447] Make Range operator generate "recor...

2017-02-16 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16960#discussion_r101575199 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsSuite.scala --- @@ -309,4 +314,84 @@ class SQLMetricsSuite extends

[GitHub] spark issue #16958: [SPARK-13721][SQL] Make GeneratorOuter unresolved.

2017-02-16 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16958 So nice when I got two LGTMs and then Jenkins disagreed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #16826: [WIP][SPARK-19540][SQL] Add ability to clone SparkSessio...

2017-02-16 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16826 What's WIP about this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #16611: [SPARK-17967][SPARK-17878][SQL][PYTHON] Support for arra...

2017-02-16 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16611 For SQL, rather than "array", can we follow Python, e.g. ``` CREATE TEMPORARY TABLE tableA USING csv OPTIONS (nullValue ['NA', 'null'], ...) ``` --- If your project

[GitHub] spark pull request #16611: [SPARK-17967][SPARK-17878][SQL][PYTHON] Support f...

2017-02-16 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16611#discussion_r101553890 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala --- @@ -97,6 +99,15 @@ class DataFrameReader private[sql](sparkSession

[GitHub] spark issue #16534: [SPARK-19161][PYTHON][SQL] Improving UDF Docstrings

2017-02-16 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16534 Change looks good to me but I didn't look super carefully. @holdenk can you take a look at this? --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark issue #16958: [SPARK-13721][SQL] Make GeneratorOuter unresolved.

2017-02-16 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16958 cc @hvanhovell @bogdanrdc --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark pull request #16958: [SPARK-13721][SQL] Make GeneratorOuter unresolved...

2017-02-16 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/16958 [SPARK-13721][SQL] Make GeneratorOuter unresolved. ## What changes were proposed in this pull request? This is a small change to make GeneratorOuter always unresolved. It is mostly no-op change

[GitHub] spark pull request #16956: [SPARK-19598][SQL]Remove the alias parameter in U...

2017-02-16 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16956#discussion_r101530187 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveHints.scala --- @@ -54,10 +54,6 @@ object ResolveHints

[GitHub] spark issue #16943: [SPARK-19607][HOTFIX] Finding QueryExecution that matche...

2017-02-15 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16943 Merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #16941: [SPARK-16475][SQL] broadcast hint for SQL queries - disa...

2017-02-15 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16941 Merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #16941: [SPARK-16475][SQL] broadcast hint for SQL queries...

2017-02-15 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16941#discussion_r101329235 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala --- @@ -524,7 +530,7 @@ class PlanParserSuite extends

[GitHub] spark issue #16940: [SPARK-19607] Finding QueryExecution that matches provid...

2017-02-15 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16940 Merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #16925: [SPARK-16475][SQL] Broadcast hint for SQL Queries

2017-02-15 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16925#discussion_r101289645 --- Diff: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 --- @@ -374,6 +374,16 @@ querySpecification windows

[GitHub] spark pull request #16925: [SPARK-16475][SQL] Broadcast hint for SQL Queries

2017-02-15 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16925#discussion_r101289574 --- Diff: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 --- @@ -374,6 +374,16 @@ querySpecification windows

[GitHub] spark pull request #16925: [SPARK-16475][SQL] Broadcast hint for SQL Queries

2017-02-15 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16925#discussion_r101288304 --- Diff: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 --- @@ -374,6 +374,16 @@ querySpecification windows

[GitHub] spark issue #16920: [MINOR][DOCS] Add jira url in pull request description

2017-02-15 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16920 Yea the only issue is that it requires another manual update. Why not use the chrome plugin I sent? --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark issue #16940: [SPARK-19607] Finding QueryExecution that matches provid...

2017-02-15 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16940 LGTM (pending Jenkins). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #16920: [MINOR][DOCS] Add jira url in pull request description

2017-02-15 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16920 Why is this necessary? It seems like an extra step needed and doesn't provide any real information. I suggest you use this: https://chrome.google.com/webstore/detail/jirafy

[GitHub] spark pull request #16939: [SPARK-16475][SQL] broadcast hint for SQL queries...

2017-02-15 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/16939 [SPARK-16475][SQL] broadcast hint for SQL queries - follow up ## What changes were proposed in this pull request? A small update to https://github.com/apache/spark/pull/16925 1. Rename

[GitHub] spark issue #16925: [SPARK-16475][SQL] Broadcast hint for SQL Queries

2017-02-14 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16925 the latest commit hasn't finished running tests yet ... but probably fine given the small change. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request #16925: [SPARK-16475][SQL] Broadcast hint for SQL Queries

2017-02-14 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16925#discussion_r101137229 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/SubstituteHintsSuite.scala --- @@ -0,0 +1,123 @@ +/* + * Licensed

[GitHub] spark pull request #16925: [SPARK-16475][SQL] Broadcast hint for SQL Queries

2017-02-14 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16925#discussion_r101129634 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/SubstituteHints.scala --- @@ -0,0 +1,103 @@ +/* + * Licensed

[GitHub] spark pull request #16925: [SPARK-16475][SQL] Broadcast hint for SQL Queries

2017-02-14 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16925#discussion_r101129594 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/SubstituteHints.scala --- @@ -0,0 +1,103 @@ +/* + * Licensed

[GitHub] spark pull request #16925: [SPARK-16475][SQL] Broadcast hint for SQL Queries

2017-02-14 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16925#discussion_r101129453 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/SubstituteHintsSuite.scala --- @@ -0,0 +1,123 @@ +/* + * Licensed

[GitHub] spark issue #16925: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

2017-02-14 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16925 cc @dongjoon-hyun, @cloud-fan, @gatorsmile and @hvanhovell This should be ready for review. Note that the semantics is different from the earlier versions. --- If your project is set up for it, you

[GitHub] spark pull request #16925: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

2017-02-14 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16925#discussion_r101088496 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/SubstituteHints.scala --- @@ -0,0 +1,85 @@ +/* + * Licensed

[GitHub] spark issue #16925: [SPARK-16475][SQL] Broadcast Hint for SQL Queries - WIP

2017-02-14 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16925 Actually I'm going to completely rewrite this. I don't think the current implementation makes sense. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request #16925: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

2017-02-14 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/16925 [SPARK-16475][SQL] Broadcast Hint for SQL Queries ## What changes were proposed in this pull request? This PR aims to achieve the following two goals in Spark SQL. 1. Generic Hint

[GitHub] spark issue #14426: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

2017-02-14 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/14426 Actually I have some time. I will submit a pr based on this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #14426: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

2017-02-14 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/14426 @dongjoon-hyun do you have time to update the pull request now the view canonicalization work is done? Basically we can remove all the SQL generation stuff. --- If your project is set up

[GitHub] spark issue #16914: [SPARK-19514] Enhancing the test for Range interruption.

2017-02-13 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16914 Merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #16914: [SPARK-19514] Enhancing the test for Range interruption.

2017-02-13 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16914 LGTM pending jenkins. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #16872: [SPARK-19514] Making range interruptible.

2017-02-13 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16872#discussion_r100789955 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameRangeSuite.scala --- @@ -127,4 +133,28 @@ class DataFrameRangeSuite extends QueryTest

[GitHub] spark issue #16888: [WIP] [SPARK-19552] [BUILD] Upgrade Netty version to 4.1...

2017-02-13 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16888 Are there specific benefits brought by updating to 4.1 of Netty? Netty is so core to Spark that any bug in it would be extremely difficult to debug (yes we have founds bugs in Netty and helped fix

[GitHub] spark pull request #16386: [SPARK-18352][SQL] Support parsing multiline json...

2017-02-12 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16386#discussion_r100687458 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala --- @@ -48,69 +47,110 @@ class JacksonParser

[GitHub] spark issue #16887: [SPARK-19549] Allow providing reason for stage/job cance...

2017-02-10 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16887 Merging in master! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #16888: [SPARK-19552] [BUILD] Upgrade Netty version to 4.1.8 fin...

2017-02-10 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16888 BTW for Netty we shouldn't just bump to the highest version. We should use the maintenance branches. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark issue #16888: [SPARK-19552] [BUILD] Upgrade Netty version to 4.1.8 fin...

2017-02-10 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16888 Shouldn't we use netty-4.0.44.Final rather than 4.1.x? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #16664: [SPARK-18120 ][SQL] Call QueryExecutionListener callback...

2017-02-10 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16664 Yea we should fix that. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #16664: [SPARK-18120 ][SQL] Call QueryExecutionListener callback...

2017-02-10 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16664 Actually @cloud-fan are you sure it is a problem right now? DataSOurce.write itself creates the commands, and if the information are propagated correctly, the QueryExecution object should have

<    4   5   6   7   8   9   10   11   12   13   >