[GitHub] spark pull request #22889: [SPARK-25882][SQL] Added a function to join two d...

2018-11-03 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22889#discussion_r230554727 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -883,6 +883,31 @@ class Dataset[T] private[sql]( join(right, Seq

[GitHub] spark pull request #19796: [SPARK-22581][SQL] Catalog api does not allow to ...

2018-11-03 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/19796#discussion_r230554452 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/internal/CatalogSuite.scala --- @@ -519,6 +518,28 @@ class CatalogSuite

[GitHub] spark pull request #22693: [SPARK-25701][SQL] Supports calculation of table ...

2018-11-03 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22693#discussion_r230554142 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala --- @@ -115,26 +116,45 @@ class ResolveHiveSerdeTable(session

[GitHub] spark issue #22930: [SPARK-24869][SQL] Fix SaveIntoDataSourceCommand's input...

2018-11-03 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22930 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22930: [SPARK-24869][SQL] Fix SaveIntoDataSourceCommand's input...

2018-11-03 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22930 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22930: [SPARK-24869][SQL] Fix SaveIntoDataSourceCommand'...

2018-11-02 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22930#discussion_r230471897 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala --- @@ -56,10 +62,6 @@ class DefaultSource

[GitHub] spark pull request #22930: [SPARK-24869][SQL] Fix SaveIntoDataSourceCommand'...

2018-11-02 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22930#discussion_r230471768 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/test/TestSQLContext.scala --- @@ -39,7 +39,10 @@ private[spark] class TestSparkSession(sc

[GitHub] spark pull request #22930: [SPARK-24869][SQL] Fix SaveIntoDataSourceCommand'...

2018-11-02 Thread wangyum
GitHub user wangyum opened a pull request: https://github.com/apache/spark/pull/22930 [SPARK-24869][SQL] Fix SaveIntoDataSourceCommand's input Dataset does not use cached data ## What changes were proposed in this pull request? This PR fix 2 things: 1. Fix

[GitHub] spark pull request #22530: [SPARK-24869][SQL] Fix SaveIntoDataSourceCommand'...

2018-11-02 Thread wangyum
Github user wangyum closed the pull request at: https://github.com/apache/spark/pull/22530 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22908: [MONOR][SQL] Replace all TreeNode's node name in the sim...

2018-11-02 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22908 Thanks @srowen I'd like to be consistent with other `simpleString`. So when I see a `simpleString`, I know the class that corresponds

[GitHub] spark pull request #22530: [SPARK-24869][SQL] Fix SaveIntoDataSourceCommand'...

2018-11-02 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22530#discussion_r230357101 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala --- @@ -56,10 +61,6 @@ class DefaultSource

[GitHub] spark pull request #22530: [SPARK-24869][SQL] Fix SaveIntoDataSourceCommand'...

2018-11-02 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22530#discussion_r230355700 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/test/TestSQLContext.scala --- @@ -39,7 +39,10 @@ private[spark] class TestSparkSession(sc

[GitHub] spark pull request #22908: [MONOR][SQL] Replace all TreeNode's node name in ...

2018-10-31 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22908#discussion_r229761443 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala --- @@ -932,12 +932,12 @@ case class HashAggregateExec

[GitHub] spark pull request #22908: [MONOR][SQL] Replace all TreeNode's node name in ...

2018-10-31 Thread wangyum
GitHub user wangyum opened a pull request: https://github.com/apache/spark/pull/22908 [MONOR][SQL] Replace all TreeNode's node name in the simpleString as their nodeName ## What changes were proposed in this pull request? This pr replace all `TreeNode`'s node name

[GitHub] spark pull request #22906: [SPARK-25895][Core]Adding testcase to compare Lz4...

2018-10-31 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22906#discussion_r229723135 --- Diff: core/src/test/scala/org/apache/spark/io/CompressionCodecSuite.scala --- @@ -128,6 +130,69 @@ class CompressionCodecSuite extends SparkFunSuite

[GitHub] spark pull request #22903: [SPARK-24196][SQL] Implement Spark's own GetSchem...

2018-10-31 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22903#discussion_r229649065 --- Diff: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/SparkMetadataOperationSuite.scala --- @@ -0,0 +1,103

[GitHub] spark pull request #22903: [SPARK-24196][SQL] Implement Spark's own GetSchem...

2018-10-31 Thread wangyum
GitHub user wangyum opened a pull request: https://github.com/apache/spark/pull/22903 [SPARK-24196][SQL] Implement Spark's own GetSchemasOperation ## What changes were proposed in this pull request? This PR fix SQL Client tools can't show DBs by implementing Spark's own

[GitHub] spark issue #22879: [SPARK-25872][SQL][TEST] Add an optimizer tracker for TP...

2018-10-31 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22879 @yucai make sense. Maybe we can also reuse `setupTables` with `TPCDSQuerySuite`. --- - To unsubscribe, e-mail: reviews

[GitHub] spark issue #22778: [SPARK-25784][SQL] Infer filters from constraints after ...

2018-10-29 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22778 We also need to add the `CombineFilters` based on https://github.com/apache/spark/pull/22879. --- - To unsubscribe, e-mail

[GitHub] spark pull request #22879: [SPARK-25872][SQL][TEST] Add an optimizer tracker...

2018-10-29 Thread wangyum
GitHub user wangyum opened a pull request: https://github.com/apache/spark/pull/22879 [SPARK-25872][SQL][TEST] Add an optimizer tracker for TPC-DS queries ## What changes were proposed in this pull request? Add an optimizer tracker for TPC-DS queries. For example

[GitHub] spark pull request #22861: [SPARK-25663][SPARK-25661][SQL][TEST] Refactor Bu...

2018-10-29 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22861#discussion_r228992409 --- Diff: external/avro/src/test/scala/org/apache/spark/sql/execution/benchmark/AvroWriteBenchmark.scala --- @@ -19,22 +19,17 @@ package

[GitHub] spark pull request #22861: [SPARK-25663][SPARK-25661][SQL][TEST] Refactor Bu...

2018-10-29 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22861#discussion_r228991112 --- Diff: external/avro/src/test/scala/org/apache/spark/sql/execution/benchmark/AvroWriteBenchmark.scala --- @@ -19,22 +19,17 @@ package

[GitHub] spark pull request #22844: [SPARK-25847][SQL][TEST] Refactor JSONBenchmarks ...

2018-10-29 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22844#discussion_r228841276 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonBenchmarks.scala --- @@ -16,32 +16,33 @@ */ package

[GitHub] spark pull request #22844: [SPARK-25847][SQL][TEST] Refactor JSONBenchmarks ...

2018-10-29 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22844#discussion_r228832524 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonBenchmarks.scala --- @@ -16,32 +16,30 @@ */ package

[GitHub] spark pull request #22844: [SPARK-25847][SQL][TEST] Refactor JSONBenchmarks ...

2018-10-29 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22844#discussion_r228832419 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonBenchmarks.scala --- @@ -16,32 +16,30 @@ */ package

[GitHub] spark pull request #22845: [SPARK-25848][SQL][TEST] Refactor CSVBenchmarks t...

2018-10-29 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22845#discussion_r228831997 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVBenchmarks.scala --- @@ -16,30 +16,30 @@ */ package

[GitHub] spark pull request #22845: [SPARK-25848][SQL][TEST] Refactor CSVBenchmarks t...

2018-10-29 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22845#discussion_r228831306 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVBenchmarks.scala --- @@ -16,30 +16,30 @@ */ package

[GitHub] spark issue #22721: [SPARK-25403][SQL] Refreshes the table after inserting t...

2018-10-29 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22721 Retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark issue #22850: [MINOR][DOC] Fix comment error of HiveUtils

2018-10-26 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22850 cc @dongjoon-hyun --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22794: [SPARK-24570][SQL] Implement Spark own GetTablesO...

2018-10-26 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22794#discussion_r228422431 --- Diff: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/SparkMetadataOperationSuite.scala --- @@ -0,0 +1,117

[GitHub] spark pull request #22842: [SPARK-25846][SQL][TEST] Refactor ExternalAppendO...

2018-10-25 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22842#discussion_r228392611 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/ExternalAppendOnlyUnsafeRowArrayBenchmark.scala --- @@ -20,13 +20,27 @@ package

[GitHub] spark pull request #22842: [SPARK-25846][SQL][TEST] Refactor ExternalAppendO...

2018-10-25 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22842#discussion_r228391545 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/ExternalAppendOnlyUnsafeRowArrayBenchmark.scala --- @@ -20,13 +20,27 @@ package

[GitHub] spark pull request #22823: [SPARK-25676][SQL][TEST] Refactor BenchmarkWideTa...

2018-10-25 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22823#discussion_r228377255 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -910,12 +910,14 @@ class CodegenContext

[GitHub] spark issue #22794: [SPARK-24570][SQL] Implement Spark own GetTablesOperatio...

2018-10-25 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22794 @mgaido91 You are right. But may be we only override `newExecuteStatementOperation`, `newGetSchemasOperation`, `newGetTablesOperation`, `newGetTableTypesOperation`,`newGetColumnsOperation

[GitHub] spark pull request #22823: [SPARK-25676][SQL][TEST] Refactor BenchmarkWideTa...

2018-10-25 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22823#discussion_r228163049 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/BenchmarkWideTable.scala --- @@ -21,32 +21,30 @@ import

[GitHub] spark pull request #22823: [SPARK-25676][SQL][TEST] Refactor BenchmarkWideTa...

2018-10-25 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22823#discussion_r228162450 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/BenchmarkWideTable.scala --- @@ -21,32 +21,30 @@ import

[GitHub] spark pull request #22823: [SPARK-25676][SQL][TEST] Refactor BenchmarkWideTa...

2018-10-25 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22823#discussion_r228161805 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -910,12 +910,14 @@ class CodegenContext

[GitHub] spark pull request #22823: [SPARK-25676][SQL][TEST] Refactor BenchmarkWideTa...

2018-10-25 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22823#discussion_r228123807 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/BenchmarkWideTable.scala --- @@ -21,32 +21,30 @@ import

[GitHub] spark issue #22803: [SPARK-25808][BUILD] Upgrade jsr305 version from 1.3.9 t...

2018-10-25 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22803 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22778: [SPARK-25784][SQL] Infer filters from constraints after ...

2018-10-25 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22778 @maropu This is optimizer statistics before and after this patch. ``` === Metrics of Analyzer/Optimizer Rules before this patch

[GitHub] spark pull request #22794: [SPARK-24570][SQL] Implement Spark own GetTablesO...

2018-10-25 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22794#discussion_r228058051 --- Diff: sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetTablesOperation.scala --- @@ -0,0 +1,121

[GitHub] spark pull request #22794: [SPARK-24570][SQL] Implement Spark own GetTablesO...

2018-10-25 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22794#discussion_r228057957 --- Diff: sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetTablesOperation.scala --- @@ -0,0 +1,121

[GitHub] spark pull request #22794: [SPARK-24570][SQL] Implement Spark own GetTablesO...

2018-10-25 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22794#discussion_r228058028 --- Diff: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala --- @@ -644,6 +644,47 @@ class

[GitHub] spark issue #22803: [SPARK-25808][BUILD] Upgrade jsr305 version from 1.3.9 t...

2018-10-24 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22803 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22819: [BUILD] Close stale PRs

2018-10-24 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22819 How about adding https://github.com/apache/spark/pull/21933 because we have upgraded netty to 4.1.30? --- - To unsubscribe, e

[GitHub] spark issue #22794: [SPARK-24570][SQL] Implement Spark own GetTablesOperatio...

2018-10-24 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22794 cc @gatorsmile @srowen @dongjoon-hyun --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands

[GitHub] spark issue #22794: [SPARK-24570][SQL] Implement Spark own GetTablesOperatio...

2018-10-24 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22794 Thanks @mgaido91 Changed to `sqlContext.sessionState.catalog` to obtain table names now. --- - To unsubscribe, e-mail: reviews

[GitHub] spark pull request #22804: [SPARK-25665][SQL][TEST] Refactor ObjectHashAggre...

2018-10-23 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22804#discussion_r227330755 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/execution/benchmark/ObjectHashAggregateExecBenchmark.scala --- @@ -21,207 +21,212 @@ import

[GitHub] spark issue #22803: change jsr305 version from 1.3.9 to 3.0.0

2018-10-23 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22803 Thank you for your first contribution, @daviddingly Could you [create issues ticket](https://issues.apache.org/jira/projects/SPARK) first and change the pr title to `[SPARK-x][BUILD

[GitHub] spark pull request #22778: [SPARK-25784][SQL] Infer filters from constraints...

2018-10-23 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22778#discussion_r227236021 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/RewriteSubquerySuite.scala --- @@ -33,23 +34,44 @@ class

[GitHub] spark pull request #22778: [SPARK-25784][SQL] Infer filters from constraints...

2018-10-22 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22778#discussion_r227208719 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/RewriteSubquerySuite.scala --- @@ -33,23 +34,44 @@ class

[GitHub] spark issue #22778: [SPARK-25784][SQL] Infer filters from constraints after ...

2018-10-20 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22778 cc @gatorsmile @cloud-fan @maropu --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #22778: [SPARK-25784][SQL] Infer filters from constraints after ...

2018-10-20 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22778 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22721: [SPARK-25403][SQL] Refreshes the table after inse...

2018-10-19 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22721#discussion_r226812885 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoHadoopFsRelationCommand.scala --- @@ -189,6 +189,7 @@ case class

[GitHub] spark pull request #22778: [SPARK-25784][SQL] Infer filters from constraints...

2018-10-19 Thread wangyum
GitHub user wangyum opened a pull request: https://github.com/apache/spark/pull/22778 [SPARK-25784][SQL] Infer filters from constraints after rewriting predicate subquery ## What changes were proposed in this pull request? Infer filters from constraints after rewriting

[GitHub] spark issue #22743: [SPARK-25740][SQL] Refactor DetermineTableStats to inval...

2018-10-19 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22743 Yes. you are right, if datasource table stats is empty, `DetermineTableStats` doesn't set stats for it, so it's only a problem for hive tables

[GitHub] spark issue #22743: [SPARK-25740][SQL] Refactor DetermineTableStats to inval...

2018-10-19 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22743 Datasource table will not cache in [tableRelationCache](https://github.com/apache/spark/blob/01c3dfab158d40653f8ce5d96f57220297545d5b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst

[GitHub] spark issue #22263: [SPARK-25269][SQL] SQL interface support specify Storage...

2018-10-19 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22263 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22263: [SPARK-25269][SQL] SQL interface support specify Storage...

2018-10-19 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22263 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22721: [SPARK-25403][SQL] Refreshes the table after inserting t...

2018-10-19 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22721 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22758: [SPARK-25332][SQL] Instead of broadcast hash join ,Sort ...

2018-10-18 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22758 I think the cost of get the stats from `HadoopFileSystem` may be quite high. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #22743: [SPARK-25740][SQL] Refactor DetermineTableStats to inval...

2018-10-18 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22743 This happens when a table `LogicalRelation` has been cached, then we change `spark.sql.statistics.fallBackToHdfs` or `spark.sql.defaultSizeInBytes` will not have any effect to stats, it always uses

[GitHub] spark pull request #22501: [SPARK-25492][TEST] Refactor WideSchemaBenchmark ...

2018-10-18 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22501#discussion_r226520120 --- Diff: sql/core/benchmarks/WideSchemaBenchmark-results.txt --- @@ -1,117 +1,145 @@ -Java HotSpot(TM) 64-Bit Server VM 1.8.0_92-b14 on Mac OS X

[GitHub] spark pull request #22263: [SPARK-25269][SQL] SQL interface support specify ...

2018-10-18 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22263#discussion_r226514429 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala --- @@ -325,6 +325,21 @@ class CachedTableSuite extends QueryTest

[GitHub] spark issue #22743: [SPARK-25740][SQL] Refactor DetermineTableStats to inval...

2018-10-18 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22743 cc @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21871: [SPARK-24916][SQL] Fix type coercion for IN expression w...

2018-10-18 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/21871 workaround: ```sql SELECT * FROM t4 WHERE (t4a, t4b, t4c) IN (SELECT t5a, t5b, t5c FROM t5); -> SELECT * FROM t4 WHERE (t4a, t4b, t4c) IN (SELECT CAST(

[GitHub] spark pull request #22721: [SPARK-25403][SQL] Refreshes the table after inse...

2018-10-18 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22721#discussion_r226205949 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoHadoopFsRelationCommand.scala --- @@ -189,6 +189,7 @@ case class

[GitHub] spark issue #22721: [SPARK-25403][SQL] Refreshes the table after inserting t...

2018-10-18 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22721 The answer is here: https://github.com/apache/spark/pull/22758#discussion_r226203589 --- - To unsubscribe, e-mail: reviews

[GitHub] spark pull request #22758: [SPARK-25332][SQL] Instead of broadcast hash join...

2018-10-18 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22758#discussion_r226203589 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -193,6 +193,16 @@ private[hive] class HiveMetastoreCatalog

[GitHub] spark pull request #22758: [SPARK-25332][SQL] Instead of broadcast hash join...

2018-10-18 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22758#discussion_r226199284 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -193,6 +193,16 @@ private[hive] class HiveMetastoreCatalog

[GitHub] spark issue #22743: [SPARK-25740][SQL] Refactor DetermineTableStats to inval...

2018-10-18 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22743 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22263: [SPARK-25269][SQL] SQL interface support specify ...

2018-10-18 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22263#discussion_r226187423 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala --- @@ -288,6 +297,65 @@ class CachedTableSuite extends QueryTest

[GitHub] spark pull request #22263: [SPARK-25269][SQL] SQL interface support specify ...

2018-10-18 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22263#discussion_r226182333 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala --- @@ -288,6 +297,65 @@ class CachedTableSuite extends QueryTest

[GitHub] spark issue #22721: [SPARK-25403][SQL] Refreshes the table after inserting t...

2018-10-17 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22721 cc @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22263: [SPARK-25269][SQL] SQL interface support specify ...

2018-10-16 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22263#discussion_r225762035 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala --- @@ -288,6 +297,65 @@ class CachedTableSuite extends QueryTest

[GitHub] spark issue #21588: [SPARK-24590][BUILD] Make Jenkins tests passed with hado...

2018-10-16 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/21588 Thanks @HyukjinKwon Upgrade Hive to 2.3.2 can fix [SPARK-12014](https://issues.apache.org/jira/browse/SPARK-12014), [SPARK-18673](https://issues.apache.org/jira/browse/SPARK-18673), [SPARK

[GitHub] spark pull request #22707: [SPARK-25717][SQL] Insert overwrite a recreated e...

2018-10-16 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22707#discussion_r225590134 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala --- @@ -227,18 +227,22 @@ case class InsertIntoHiveTable

[GitHub] spark pull request #22707: [SPARK-25717][SQL] Insert overwrite a recreated e...

2018-10-16 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22707#discussion_r225589876 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala --- @@ -227,18 +227,22 @@ case class InsertIntoHiveTable

[GitHub] spark issue #22747: [MINOR][SQL] Set AddJarCommand return empty

2018-10-16 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22747 Hive will not return 0: ```sql hive> add jar /Users/yumwang/spark/sql/hive/src/test/resources/TestUDTF.jar; Added [/Users/yumwang/spark/sql/hive/src/test/resources/TestUDTF.jar] to cl

[GitHub] spark pull request #22747: [MINOR][SQL] Set AddJarCommand return empty

2018-10-16 Thread wangyum
GitHub user wangyum opened a pull request: https://github.com/apache/spark/pull/22747 [MINOR][SQL] Set AddJarCommand return empty ## What changes were proposed in this pull request? Only `AddJarCommand` return `0`, the user will be confused about what it means. This PR

[GitHub] spark pull request #22743: [WIP][SPARK-25740][SQL] Set some configuration ne...

2018-10-16 Thread wangyum
GitHub user wangyum opened a pull request: https://github.com/apache/spark/pull/22743 [WIP][SPARK-25740][SQL] Set some configuration need invalidateStatsCache ## What changes were proposed in this pull request? How to reproduce: ```sql # spark-sql create table t1

[GitHub] spark issue #22263: [SPARK-25269][SQL] SQL interface support specify Storage...

2018-10-15 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22263 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22721: [SPARK-25403][SQL] Refreshes the table after inse...

2018-10-15 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22721#discussion_r225092331 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala --- @@ -2370,4 +2370,26 @@ class HiveDDLSuite

[GitHub] spark issue #22721: [SPARK-25403][SQL] Refreshes the table after inserting t...

2018-10-15 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22721 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22721: [SPARK-25403][SQL] Refreshes the table after inserting t...

2018-10-15 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22721 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22721: [SPARK-25403][SQL] Refreshes the table after inse...

2018-10-14 Thread wangyum
GitHub user wangyum opened a pull request: https://github.com/apache/spark/pull/22721 [SPARK-25403][SQL] Refreshes the table after inserting the table ## What changes were proposed in this pull request? Refreshes the table after inserting the table, Otherwise, we will encounter

[GitHub] spark pull request #22636: [SPARK-25629][TEST] Reduce ParquetFilterSuite: fi...

2018-10-14 Thread wangyum
GitHub user wangyum reopened a pull request: https://github.com/apache/spark/pull/22636 [SPARK-25629][TEST] Reduce ParquetFilterSuite: filter pushdown test time costs in Jenkins ## What changes were proposed in this pull request? Only test these 4 cases is enough

[GitHub] spark issue #22263: [SPARK-25269][SQL] SQL interface support specify Storage...

2018-10-14 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22263 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22707: [SPARK-25717][SQL] Insert overwrite a recreated external...

2018-10-12 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22707 Could you add test case for this change? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22661: [SPARK-25664][SQL][TEST] Refactor JoinBenchmark t...

2018-10-12 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22661#discussion_r224714936 --- Diff: core/src/test/scala/org/apache/spark/benchmark/Benchmark.scala --- @@ -200,11 +200,12 @@ private[spark] object Benchmark { def

[GitHub] spark pull request #22636: [SPARK-25629][TEST] Reduce ParquetFilterSuite: fi...

2018-10-12 Thread wangyum
Github user wangyum closed the pull request at: https://github.com/apache/spark/pull/22636 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22661: [SPARK-25664][SQL][TEST] Refactor JoinBenchmark to use m...

2018-10-11 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22661 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22698: [SPARK-25710][SQL] range should report metrics co...

2018-10-11 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22698#discussion_r224563916 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/RangeBenchmark.scala --- @@ -0,0 +1,64 @@ +/* + * Licensed

[GitHub] spark pull request #22661: [SPARK-25664][SQL][TEST] Refactor JoinBenchmark t...

2018-10-11 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22661#discussion_r224396901 --- Diff: sql/core/benchmarks/JoinBenchmark-results.txt --- @@ -0,0 +1,80 @@ +Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6

[GitHub] spark pull request #22661: [SPARK-25664][SQL][TEST] Refactor JoinBenchmark t...

2018-10-10 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22661#discussion_r224300031 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/JoinBenchmark.scala --- @@ -19,229 +19,165 @@ package

[GitHub] spark issue #22661: [SPARK-25664][SQL][TEST] Refactor JoinBenchmark to use m...

2018-10-08 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22661 cc @dongjoon-hyun --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22669: [SPARK-25677] [Core] spark.io.compression.codec = org.ap...

2018-10-08 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22669 cc @HyukjinKwon --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22669: [SPARK-25677] [Core] spark.io.compression.codec = org.ap...

2018-10-08 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22669 Please update title to: ``` [SPARK-25677][DOC] ... ``` --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #22661: [SPARK-25664][SQL][TEST] Refactor JoinBenchmark t...

2018-10-07 Thread wangyum
GitHub user wangyum opened a pull request: https://github.com/apache/spark/pull/22661 [SPARK-25664][SQL][TEST] Refactor JoinBenchmark to use main method ## What changes were proposed in this pull request? Refactor `JoinBenchmark` to use main method. 1. use `spark-submit

[GitHub] spark issue #22651: [SPARK-25657][SQL][TEST] Refactor HashBenchmark to use m...

2018-10-07 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22651 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

<    1   2   3   4   5   6   7   8   9   10   >