Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14537#discussion_r80640012
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -237,21 +237,24 @@ private[hive] class
HiveMetastoreCatalog
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14537#discussion_r80639483
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcQuerySuite.scala ---
@@ -372,6 +373,40 @@ class OrcQuerySuite extends QueryTest
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14537#discussion_r80639203
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -237,21 +237,24 @@ private[hive] class
HiveMetastoreCatalog
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14897#discussion_r80280321
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
---
@@ -424,17 +496,24 @@ class SessionCatalog
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14897#discussion_r80280674
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
---
@@ -453,7 +532,11 @@ class SessionCatalog
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14897#discussion_r80282210
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala ---
@@ -60,20 +90,21 @@ case class CreateViewCommand(
child
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14897#discussion_r80080673
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
---
@@ -47,6 +50,8 @@ object SessionCatalog
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14897#discussion_r80279937
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
---
@@ -371,16 +431,24 @@ class SessionCatalog
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14897#discussion_r80080872
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
---
@@ -142,8 +149,12 @@ class SessionCatalog
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14897#discussion_r80281859
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala ---
@@ -19,13 +19,46 @@ package
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14897
Nit: All occurrences of `_global_temp` in the PR description should be
replaced to `global_temp`.
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/15095
LGTM. Merging to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/15082
My understanding of the main concern of closing this PR is that:
1. Although this issue can be potentially dangerous, the current code work
fine without fixing this issue.
1. We still
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14868
LGTM except for a few minor issues. The failed test should be a typo in
[this line].
[1]:
https://github.com/apache/spark/pull/14868/files#diff-9b242f568b320219b1e2cfa2ee10137bR173
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14868#discussion_r76803012
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentileSuite.scala
---
@@ -0,0 +1,318
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14868#discussion_r76761771
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
---
@@ -0,0 +1,304
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14868#discussion_r76761692
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
---
@@ -0,0 +1,304
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14868#discussion_r76761387
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
---
@@ -0,0 +1,304
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14868#discussion_r76761445
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
---
@@ -0,0 +1,304
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14865
@maropu Discussed with @clockfly and @cloud-fan offline. @cloud-fan
proposed a simpler alternative of #10896. Please refer to [this comment][1] for
details.
This PR still LGTM and I'm
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/10896
+1 for @cloud-fan's proposal. Instead of creating a performant plan using
tricky code, it's clearer to create a naive but correct physical plan first and
then optimize it.
---
If your project
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14749
@rxin It doesn't fail any tests. Found this issue while working on related
code path.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14749
@cloud-fan I noticed this issue because I saw `First` and `Last` are marked
as non-deterministic. Although we are not checking this property for now, it's
still good to set the right values
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14749#discussion_r75783595
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/collect.scala
---
@@ -54,6 +54,10 @@ abstract class Collect
GitHub user liancheng opened a pull request:
https://github.com/apache/spark/pull/14749
[SPARK-17182][SQL] Mark Collect as non-deterministic
## What changes were proposed in this pull request?
This PR marks the abstract class `Collect` as non-deterministic since
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14723#discussion_r75632667
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala
---
@@ -389,3 +389,126 @@ abstract class
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14585
OK, resolved the conflict manually and got it merged into branch-2.0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14585
LGTM, merging to master and branch-2.0. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14477
@srowen Sorry for the late reply, this LGTM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14476
LGTM, merging to master. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14480
Thanks, merging to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
GitHub user liancheng opened a pull request:
https://github.com/apache/spark/pull/14480
[MINOR][SQL] Fix minor formatting issue of SortAggregateExec.toString
## What changes were proposed in this pull request?
This PR fixes a minor formatting issue (missing space after
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14458
LGTM, merging to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/13893#discussion_r73156772
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala
---
@@ -64,10 +64,17 @@ object PhysicalOperation extends
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/11443#discussion_r73107764
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala ---
@@ -745,6 +825,80 @@ class DataFrame private[sql
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14368#discussion_r73076628
--- Diff: examples/src/main/r/RSparkSQLExample.R ---
@@ -18,31 +18,43 @@
library(SparkR)
# $example on:init_session$
-sparkR.session
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14368#discussion_r72417903
--- Diff: examples/src/main/r/RSparkSQLExample.R ---
@@ -18,31 +18,43 @@
library(SparkR)
# $example on:init_session$
-sparkR.session
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14368#discussion_r72387899
--- Diff: examples/src/main/r/RSparkSQLExample.R ---
@@ -18,31 +18,43 @@
library(SparkR)
# $example on:init_session$
-sparkR.session
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14368#discussion_r72387754
--- Diff: examples/src/main/r/RSparkSQLExample.R ---
@@ -18,31 +18,43 @@
library(SparkR)
# $example on:init_session$
-sparkR.session
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14257
LGTM, merging to master. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14358#discussion_r72243943
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
---
@@ -228,18 +229,12 @@ case class Cast(child: Expression
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14368
cc @yhuai @cloud-fan @JoshRosen @felixcheung
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14302
LGTM, merging to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
GitHub user liancheng opened a pull request:
https://github.com/apache/spark/pull/14368
[SPARK-16734][EXAMPLES][SQL] Revise examples of all language bindings
## What changes were proposed in this pull request?
This PR makes various minor updates to examples of all language
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14344
LGTM, merging to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14257
LGTM except for one minor comment. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14257#discussion_r72199634
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/catalyst/SQLBuilder.scala ---
@@ -38,14 +39,20 @@ import org.apache.spark.sql.types.{ByteType
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/13704#discussion_r72080255
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -1441,6 +1441,12 @@ object PushPredicateThroughJoin
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/13704#discussion_r72077898
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -1441,6 +1441,12 @@ object PushPredicateThroughJoin
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14339
LGTM, merging to master and branch-2.0. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14304
I'm merging this to master. Thanks for fixing this!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14302#discussion_r72071947
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -520,7 +522,7 @@ case class DescribeTableCommand(table
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14302#discussion_r72070878
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -436,11 +436,13 @@ case class DescribeTableCommand(table
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14302#discussion_r72069708
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -520,7 +522,7 @@ case class DescribeTableCommand(table
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14331
Merging to master, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14297
Merging to master, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14331#discussion_r72047779
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
---
@@ -173,8 +190,18 @@ case class CatalogTable
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14344#discussion_r72045939
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
---
@@ -501,6 +501,143 @@ case class MapObjects
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14344#discussion_r72036250
--- Diff:
sql/core/src/test/java/test/org/apache/spark/sql/JavaDatasetSuite.java ---
@@ -673,21 +694,24 @@ public void testJavaBeanEncoder
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14344#discussion_r72035518
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
---
@@ -501,6 +501,143 @@ case class MapObjects
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14344#discussion_r72035534
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
---
@@ -501,6 +501,143 @@ case class MapObjects
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14297
LGTM except for one minor issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14297#discussion_r72032682
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala ---
@@ -58,10 +68,8 @@ case class CreateViewCommand
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14297#discussion_r72032392
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala ---
@@ -44,7 +50,11 @@ import
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14334
@rxin PR title updated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14297#discussion_r72029905
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala ---
@@ -44,7 +50,11 @@ import
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14297#discussion_r72014267
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala ---
@@ -44,7 +50,11 @@ import
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14295
Tried to make `ignoreNullsExpr` also a child node of `First` and `Last` in
the last commit as @cloud-fan suggested. Hopefully this is a cleaner fix.
---
If your project is set up for it, you can
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14295#discussion_r72013449
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/First.scala
---
@@ -45,6 +45,17 @@ case class First(child
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14334
@rxin @dongjoon-hyun Thanks! JIRA ticket added and updated the golden files.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14331
Overall LGTM, a few minor comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14331#discussion_r72009811
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala ---
@@ -20,10 +20,11 @@ package org.apache.spark.sql.sources
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14331#discussion_r72009802
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
---
@@ -110,6 +110,25 @@ case class CatalogTablePartition
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14331#discussion_r72009804
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
---
@@ -173,8 +190,18 @@ case class CatalogTable
GitHub user liancheng opened a pull request:
https://github.com/apache/spark/pull/14334
[MINOR][SQL] Fixes window specification SQL format
## What changes were proposed in this pull request?
This PR fixes a minor formatting issue of `WindowSpecDefinition.sql` when
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14317#discussion_r71981344
--- Diff: examples/src/main/python/sql/datasource.py ---
@@ -0,0 +1,154 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14317#discussion_r71981335
--- Diff: examples/src/main/python/sql/basic.py ---
@@ -0,0 +1,194 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14317#discussion_r71981312
--- Diff: docs/sql-programming-guide.md ---
@@ -79,7 +79,7 @@ The entry point into all functionality in Spark is the
[`SparkSession`](api/java
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14295
Oh, that's a good point, should have realized both of them are affected.
Updated. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14317
@JoshRosen Would you mind to have a look at this? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14295#discussion_r71920301
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Last.scala
---
@@ -42,6 +42,17 @@ case class Last(child
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14304#discussion_r71903616
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetEncodingSuite.scala
---
@@ -78,4 +78,30 @@ class
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14304
Two minor issues, otherwise LGTM. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14304#discussion_r71902498
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetEncodingSuite.scala
---
@@ -78,4 +78,30 @@ class
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14304#discussion_r71901937
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetEncodingSuite.scala
---
@@ -78,4 +78,30 @@ class
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14304#discussion_r71901576
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetEncodingSuite.scala
---
@@ -78,4 +78,30 @@ class
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14295#discussion_r71896054
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Last.scala
---
@@ -42,6 +42,17 @@ case class Last(child
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14098
@wangmiao1981 Thanks for working on this. For the Hive example, I guess you
probably forgot to call `enableHiveSupport()` over the `SparkSession` object.
And I made a mistake about
GitHub user liancheng opened a pull request:
https://github.com/apache/spark/pull/14317
[SPARK-16380][EXAMPLES] Update SQL examples and programming guide for
Python language binding
This PR is based on PR #14098 authored by @wangmiao1981.
## What changes were proposed
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14300
Merging to master and branch-2.0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
GitHub user liancheng opened a pull request:
https://github.com/apache/spark/pull/14300
[SPARK-16632][SQL] Revert PR #14272: Respect Hive schema when merging
parquet schema
## What changes were proposed in this pull request?
PR #14278 is a more general and simpler fix
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14278
Thanks for the review! I'm merging this to master and branch-2.0. Will send
PRs to revert #14272 since this one is a more general fix of the same issue.
---
If your project is set up for it, you
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14278#discussion_r71671818
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
---
@@ -136,7 +137,9 @@ public
GitHub user liancheng opened a pull request:
https://github.com/apache/spark/pull/14295
[SPARK-16648][SQL] Overrides TreeNode.withNewChildren in Last
## What changes were proposed in this pull request?
Default `TreeNode.withNewChildren` implementation doesn't work for `Last
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14014
@yhuai All comments addressed. PR description is also updated. Thanks for
the review!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14013
I'm closing this one since we decided to fix this in master only.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user liancheng closed the pull request at:
https://github.com/apache/spark/pull/14013
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14045
@viirya Thanks for your work! This would be very useful. I'll help review
this one soon after finishing my 2.0 tasks at hand!
---
If your project is set up for it, you can reply to this email
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14278
@viirya Basically we are mapping the logic in `ParquetRowConverter` for the
non-vectorized Parquet reader. It's just implemented at a lower lever in the
case of vectorized reader.
---
If your
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14278
@viirya The updated schema field in this PR is only used to guide the
vectorized reader to interpret basic Parquet types into logical types (e.g.
Parquet `int32` to Spark `ByteType`, and Parquet
201 - 300 of 5176 matches
Mail list logo
