Github user LuciferYang commented on the issue:
https://github.com/apache/spark/pull/23204
ok~ already close #23214
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user LuciferYang closed the pull request at:
https://github.com/apache/spark/pull/23214
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user LuciferYang commented on the issue:
https://github.com/apache/spark/pull/23214
As @cloud-fan said `the hash join metrics is wrongly implemented`, we will
partial revert # SPARK-21052, no longer need this patch, close
Github user LuciferYang commented on the issue:
https://github.com/apache/spark/pull/23204
@cloud-fan If we decide to partial revert SPARK-21052 and no need for
#23214, I will close it.
---
-
To unsubscribe, e
Github user LuciferYang commented on the issue:
https://github.com/apache/spark/pull/23214
On the other hand, if is only a `multi-thread problem`, may not affect
performance because there is no synchronized code part
Github user LuciferYang commented on the issue:
https://github.com/apache/spark/pull/23214
```
For broadcast hash join, we will copy the broadcasted hash relation to
avoid multi-thread problem, via HashedRelation.asReadOnlyCopy. However, this is
a shallow copy
Github user LuciferYang commented on a diff in the pull request:
https://github.com/apache/spark/pull/23214#discussion_r238558889
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
---
@@ -398,8 +399,8 @@ private[execution] final class
Github user LuciferYang commented on the issue:
https://github.com/apache/spark/pull/23214
@adrian-wang ok~ I will add some comments to explain the reason
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user LuciferYang commented on the issue:
https://github.com/apache/spark/pull/23214
@JkSelf thx~
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user LuciferYang commented on the issue:
https://github.com/apache/spark/pull/23204
@cloud-fan @viirya #23214 maybe reslove this problem and we needn't revert
this patch.
---
-
To unsubscribe, e-mail
Github user LuciferYang commented on the issue:
https://github.com/apache/spark/pull/23214
ping @viirya
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user LuciferYang commented on the issue:
https://github.com/apache/spark/pull/23214
cc @cloud-fan , help to review this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user LuciferYang commented on the issue:
https://github.com/apache/spark/pull/23214
cc @JkSelf help to check this patch.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
GitHub user LuciferYang opened a pull request:
https://github.com/apache/spark/pull/23214
[SPARK-26155] Optimizing the performance of LongToUnsafeRowMap
## What changes were proposed in this pull request?
To slove @JkSelf report problem at
[SPARK-26155](https
Github user LuciferYang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22149#discussion_r235340787
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/ScriptTransformationSuite.scala
---
@@ -49,6 +52,23 @@ class
Github user LuciferYang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22149#discussion_r235337042
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/ScriptTransformationSuite.scala
---
@@ -170,3 +198,17 @@ private case class
Github user LuciferYang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22149#discussion_r235336280
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformationExec.scala
---
@@ -308,12 +308,15 @@ private class
Github user LuciferYang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22149#discussion_r235276659
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformationExec.scala
---
@@ -308,6 +308,12 @@ private class
Github user LuciferYang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22149#discussion_r235257700
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformationExec.scala
---
@@ -308,6 +308,12 @@ private class
Github user LuciferYang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22149#discussion_r235248435
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformationExec.scala
---
@@ -308,6 +308,12 @@ private class
Github user LuciferYang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22149#discussion_r235247176
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformationExec.scala
---
@@ -308,6 +308,12 @@ private class
Github user LuciferYang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22149#discussion_r235246544
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformationExec.scala
---
@@ -308,6 +308,12 @@ private class
Github user LuciferYang commented on the issue:
https://github.com/apache/spark/pull/22149
@gatorsmile should we fix this problem?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user LuciferYang commented on the issue:
https://github.com/apache/spark/pull/22149
> Is that possible to add a test case?
Thanks for your reply Xiao, I tried to add a test case to reproduce bad
case, but it only reproduce in small probabilityï¼we need to ensure that T
Github user LuciferYang commented on the issue:
https://github.com/apache/spark/pull/22149
ping @xuanyuanking
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
GitHub user LuciferYang opened a pull request:
https://github.com/apache/spark/pull/22149
[SPARK-25158][SQL]Executor accidentally exit because
ScriptTransformationWriterThread throws TaskKilledException.
## What changes were proposed in this pull request?
Run Spark-Sql job
26 matches
Mail list logo
