Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/23228
Please update the title `[MINOR][DOC] Update the condition description of
serialized shuffle`
---
-
To unsubscribe, e-mail
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/23222
Shall we add a SQL tag to the title?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/23046
I searched the code and didn't find similar issues, so this is the only one
shall be fixed.
---
-
To unsubscribe, e-mail
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22912
Thanks, merging to master!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22723#discussion_r229717747
--- Diff:
core/src/main/scala/org/apache/spark/input/WholeTextFileInputFormat.scala ---
@@ -48,11 +50,11 @@ private[spark] class
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22723#discussion_r229717581
--- Diff:
core/src/main/scala/org/apache/spark/input/WholeTextFileInputFormat.scala ---
@@ -48,11 +50,11 @@ private[spark] class
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22849
It may happen that a busy executor is marked as lost and later it
re-register to the driver, in that case currently we call `makeOffers()` and
that will add the executor
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22849
What do you mean by "better performance" ? If that means we can spend less
time on `TaskSchedulerImpl.resourceOffers()` then I agree it's true, but AFAIK
it's never reporte
Github user jiangxb1987 closed the pull request at:
https://github.com/apache/spark/pull/22853
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22853
Merging to master, I can open another PR against 2.4 if required in the
future.
---
-
To unsubscribe, e-mail: reviews
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22853
Also cc @gatorsmile @cloud-fan @hvanhovell
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
GitHub user jiangxb1987 opened a pull request:
https://github.com/apache/spark/pull/22853
[SPARK-25845][SQL] Fix MatchError for calendar interval type in range frame
left boundary
## What changes were proposed in this pull request?
WindowSpecDefinition checks start < l
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22813
IIUC it's not expected to share the SPARK_WORK_DIR with any other usage.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22771#discussion_r227459990
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -1364,6 +1385,16 @@ private[spark] class DAGScheduler
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22674
LGTM, do you have any other concerns @hvanhovell @brkyvz @dongjoon-hyun ?
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22677
Sounds good!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22699
Let's also update the title to include the deprecation changes.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22699#discussion_r224508691
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/HistoryServerArguments.scala
---
@@ -34,26 +34,25 @@ private[history] class
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22699#discussion_r224508223
--- Diff: sbin/start-history-server.sh ---
@@ -28,7 +28,22 @@ if [ -z "${SPARK_HOME}" ]; then
export SPARK_HOME="$(cd "`di
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22699#discussion_r224507524
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/HistoryServerArguments.scala
---
@@ -34,26 +34,25 @@ private[history] class
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22699#discussion_r224504103
--- Diff: sbin/start-history-server.sh ---
@@ -28,7 +28,22 @@ if [ -z "${SPARK_HOME}" ]; then
export SPARK_HOME="$(cd "`di
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22699#discussion_r224504246
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/HistoryServerArguments.scala
---
@@ -34,26 +34,25 @@ private[history] class
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22165
Actually my original thinking was like this:
```
val state = new ContextBarrierState(barrierId, numTasks)
val requester = mockRequester()
val request
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22677
Though it looks a little strange, the log content is actually right, I
don't think we want to make the last report timestamp to current time (that can
confuse users what happened before
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22674#discussion_r223729445
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/util/QueryExecutionListener.scala
---
@@ -75,95 +76,69 @@ trait QueryExecutionListener
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22674
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22325
LGTM
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22165#discussion_r220589416
--- Diff: core/src/main/scala/org/apache/spark/BarrierCoordinator.scala ---
@@ -187,6 +191,12 @@ private[spark] class BarrierCoordinator
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22165#discussion_r220590215
--- Diff: core/src/main/scala/org/apache/spark/BarrierCoordinator.scala ---
@@ -187,6 +191,12 @@ private[spark] class BarrierCoordinator
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22165#discussion_r220591706
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/BarrierCoordinatorSuite.scala ---
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22458#discussion_r220410022
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
---
@@ -2348,4 +2348,17 @@ class HiveDDLSuite
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22526#discussion_r219891354
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -685,7 +685,7 @@ private[ui] class TaskDataSource(
private
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22458#discussion_r219370221
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
---
@@ -467,9 +467,9 @@ private[hive] class HiveClientImpl
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22325#discussion_r218873184
--- Diff:
core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
---
@@ -444,36 +444,34 @@ final class
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22192#discussion_r218861435
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -136,6 +136,26 @@ private[spark] class Executor(
// for fetching
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22192#discussion_r218865220
--- Diff: core/src/test/java/org/apache/spark/ExecutorPluginSuite.java ---
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22325#discussion_r218857081
--- Diff:
core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
---
@@ -444,36 +444,34 @@ final class
Github user jiangxb1987 closed the pull request at:
https://github.com/apache/spark/pull/20414
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22351
Just confirmed if the view is created and retrieved both at Spark side then
there will be no exception thrown
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22351
This is actually read some view created by Hive, so I don't think it shall
be a problem with view write side
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22165
I think it should be fine to make `ContextBarrierState` private[spark] to
test it, WDYT @mengxr ?
---
-
To unsubscribe, e
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22277
You can have `select * from (select a, a as c from table1 where a > 10) t
where a > c`
---
-
To unsubscribe,
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22277
Thank you for interest in this issue, however, I don't think the changes
proposed in this PR is valid, consider you have another predicate like `a > z`,
it is surely desired to infer a
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22240
LGTM
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
GitHub user jiangxb1987 opened a pull request:
https://github.com/apache/spark/pull/22330
[SPARK-19355][SQL][FOLLOWUP][TEST] Properly recycle SparkSession on
TakeOrderedAndProjectSuite finishes
## What changes were proposed in this pull request?
Previously
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22240
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22112
ping @tgravescs @mridulm @squito @markhamstra
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22240#discussion_r213754911
--- Diff: core/src/main/scala/org/apache/spark/scheduler/Task.scala ---
@@ -82,31 +82,22 @@ private[spark] abstract class Task[T
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22112
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22112
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22112
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22247#discussion_r213378931
--- Diff: python/pyspark/taskcontext.py ---
@@ -108,38 +108,12 @@ def _load_from_socket(port, auth_secret):
"""
Load da
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22240
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22112
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21976#discussion_r213176636
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ---
@@ -2474,19 +2478,21 @@ class DAGSchedulerSuite extends
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22085#discussion_r213050049
--- Diff:
core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala ---
@@ -180,7 +188,73 @@ private[spark] abstract class BasePythonRunner
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21976#discussion_r213042176
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ---
@@ -2474,19 +2478,21 @@ class DAGSchedulerSuite extends
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22112
The changes looks good from my side, it summarizes the current insight we
have towards the data correctness issue caused by input order aware operators
and inconsistent shuffle output order
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/21698
Thanks everyone! I closed this in favor of #22112
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user jiangxb1987 closed the pull request at:
https://github.com/apache/spark/pull/21698
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22112#discussion_r212653282
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -1502,6 +1502,53 @@ private[spark] class DAGScheduler
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22112#discussion_r212651948
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala
---
@@ -305,17 +306,19 @@ object
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22211
LGTM
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22112#discussion_r212383406
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala
---
@@ -305,17 +306,19 @@ object
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22112#discussion_r212379326
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -1502,6 +1502,53 @@ private[spark] class DAGScheduler
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22112#discussion_r212381036
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -1502,6 +1502,53 @@ private[spark] class DAGScheduler
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22112#discussion_r212368000
--- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
@@ -812,11 +813,13 @@ abstract class RDD[T: ClassTag](
*/
private[spark
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22112#discussion_r212376990
--- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
@@ -1865,6 +1876,39 @@ abstract class RDD[T: ClassTag](
// RDD chain
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22165
One general idea is that we don't need to rely on the RPC framework to test
`ContextBarrierState`, just mock `RpcCallContext`s should be enough (haven't go
into detail so correct me if I'm
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22165
I'll make one pass of this later today :) Thanks for taking this task!
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22079
LGTM, thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22085#discussion_r211683369
--- Diff: python/pyspark/taskcontext.py ---
@@ -95,3 +99,126 @@ def getLocalProperty(self, key):
Get a local property set upstream
Github user jiangxb1987 closed the pull request at:
https://github.com/apache/spark/pull/22166
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
GitHub user jiangxb1987 opened a pull request:
https://github.com/apache/spark/pull/22166
[2.3][SPARK-25114][Core][FOLLOWUP] Fix RecordBinaryComparatorSuite build
failure
## What changes were proposed in this pull request?
Fix RecordBinaryComparatorSuite build failure
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22158
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22158
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
GitHub user jiangxb1987 opened a pull request:
https://github.com/apache/spark/pull/22158
[SPARK-25161][Core] Fix several bugs in failure handling of barrier
execution mode
## What changes were proposed in this pull request?
Fix several bugs in failure handling of barrier
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22101
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22101
Thanks @squito I've added another test case to cover when the last byte
differs.
---
-
To unsubscribe, e-mail: reviews
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22085#discussion_r211182337
--- Diff: python/pyspark/taskcontext.py ---
@@ -95,3 +99,124 @@ def getLocalProperty(self, key):
Get a local property set upstream
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22085#discussion_r210963511
--- Diff: python/pyspark/taskcontext.py ---
@@ -95,3 +99,124 @@ def getLocalProperty(self, key):
Get a local property set upstream
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22085#discussion_r210963181
--- Diff:
core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala ---
@@ -381,6 +421,45 @@ private[spark] abstract class BasePythonRunner
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22112
> IMO we should traverse the dependency graph and rely on how ShuffledRDD
is configured
A trivial point here - Since `ShuffleDependency` is also a DeveloperAPI,
it's possible for us
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22101
ping @gatorsmile @mridulm @squito
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22112#discussion_r210450123
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -1441,6 +1441,18 @@ class DAGScheduler
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22112#discussion_r210449640
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -1441,6 +1441,18 @@ class DAGScheduler
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/21698
Thanks @cloud-fan your summary above is super useful, and I think it's
clear enough.
> So when we see fetch failure and rerun map tasks, we should track which
reducers have its shuf
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22085#discussion_r209974729
--- Diff:
core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala ---
@@ -180,7 +183,42 @@ private[spark] abstract class BasePythonRunner
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22101
@squito I've created a new JIRA task and updated the title, thanks for
reminding!
---
-
To unsubscribe, e-mail: reviews
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22085#discussion_r209853276
--- Diff: python/pyspark/taskcontext.py ---
@@ -95,3 +95,92 @@ def getLocalProperty(self, key):
Get a local property set upstream
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22101
cc @mridulm @squito
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
GitHub user jiangxb1987 opened a pull request:
https://github.com/apache/spark/pull/22101
[SPARK-23207][Core][FOLLOWUP] Fix RecordBinaryComparator when subtraction
between two words is divisible by Integer.MAX_VALUE.
## What changes were proposed in this pull request
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22079#discussion_r209822194
--- Diff:
sql/catalyst/src/main/java/org/apache/spark/sql/execution/RecordBinaryComparator.java
---
@@ -0,0 +1,70 @@
+/*
+ * Licensed
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22001
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/21698
@tgravescs I'm still working on this but I would be glad if you can also
work on the "sort the serialized bytes of T" approach, actually the
retry-all-tasks approach seems more comp
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/21698
We fixed the DataFrame repartition correctness issue by inserting a local
sort before repartition, and feedback for this approach is generally negative
because the performance of repartition
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22001
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22085#discussion_r209490553
--- Diff:
core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala ---
@@ -180,7 +183,42 @@ private[spark] abstract class BasePythonRunner
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/22079
Both seems fine to me, it's just a minor improvement. Normally we don't
backport a improvement, but since it's a simple and small change I'm confident
it is safe to also include the change
GitHub user jiangxb1987 opened a pull request:
https://github.com/apache/spark/pull/22085
[SPARK-25095][PySpark] Python support for BarrierTaskContext
## What changes were proposed in this pull request?
Add method `barrier()` and `getTaskInfos()` in python TaskContext
1 - 100 of 1796 matches
Mail list logo
