Github user DaimonPl commented on the issue:
https://github.com/apache/spark/pull/22357
@mallman regarding
"But how do we know that pushing down IsNotNull(employer) does not negate
that instruction? "
isn't it pretty obvious that when you read 'e
Github user DaimonPl commented on a diff in the pull request:
https://github.com/apache/spark/pull/21320#discussion_r195704995
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -1288,8 +1288,18 @@ object SQLConf {
"issues.
Github user DaimonPl commented on the issue:
https://github.com/apache/spark/pull/16578
So if it's not going to be included in `2.3.0` maybe we could change
`spark.sql.nestedSchemaPruning.enabled` to default `true` ? I hope that this
time this PR could be finalized at the early stage
Github user DaimonPl commented on the issue:
https://github.com/apache/spark/pull/16578
New year, new review? ;)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user DaimonPl commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r151917066
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -961,6 +961,15 @@ object SQLConf {
.booleanConf
Github user DaimonPl commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r151485646
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -961,6 +961,15 @@ object SQLConf {
.booleanConf
Github user DaimonPl commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r151476679
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -961,6 +961,15 @@ object SQLConf {
.booleanConf
Github user DaimonPl commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r151344821
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -961,6 +961,15 @@ object SQLConf {
.booleanConf
Github user DaimonPl commented on the issue:
https://github.com/apache/spark/pull/16578
@mallman how about finalizing it as is? IMHO performance improvements are
worth more than (possibly) redundant workaround - it could be cleaned later
Github user DaimonPl commented on the issue:
https://github.com/apache/spark/pull/16578
@mallman @viirya from my understanding current workaround is for case when
reading columns which are not in file schema
> Parquet-mr will throw an exception if we try to read a super
Github user DaimonPl commented on the issue:
https://github.com/apache/spark/pull/16578
@mallman how about adding comment explaining why such workaround was done +
bug number in parquet-mr ? So in future once that bug is fixed, code can be
cleaned.
Also maybe it's time
Github user DaimonPl commented on the issue:
https://github.com/apache/spark/pull/16578
@mallman (just pure curiosity :) ) how is it possible that this NPE was not
found by automated tests? There's no single test case in entire spark suite
which verifies scenarios like that described
12 matches
Mail list logo
