Github user f7753 commented on the issue:
https://github.com/apache/spark/pull/14239
@tgravescs Sorry for so late to reply, I am really sorry for that.
I haven't use external shuffle, I need to rebase the code to recent
release and retest it. Iâll rebase it on exe
Github user f7753 commented on the issue:
https://github.com/apache/spark/pull/14239
@zdove please refer to `org.apache.spark.SparkConf`, spark load all
parameter you set on the file spark-defaults by scripts when JVM process start
and filter those contains "spark", then a
Github user f7753 commented on the issue:
https://github.com/apache/spark/pull/16591
Nice work
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
Github user f7753 commented on the issue:
https://github.com/apache/spark/pull/14239
@tgravescs To make it more readable and answer the question above.
**1. Are you saying that you are loading all the data for all the maps from
disk into memory and caching it waiting for the
Github user f7753 commented on the issue:
https://github.com/apache/spark/pull/14239
@tgravescs Thank you.
Currently, I'm not load all the data into memory, I use parameters
`spark.shuffle.prepare.open ` to switch this mechanism off/on and
`spark.shuffle.prepare.coun
Github user f7753 closed the pull request at:
https://github.com/apache/spark/pull/14232
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
GitHub user f7753 opened a pull request:
https://github.com/apache/spark/pull/14239
[SPARK-16593] [CORE] Provide a pre-fetch mechanism to accelerate shuffle
stage.
## What changes were proposed in this pull request?
Added a pre-fetch mechanism for shuffle stage.
The
Github user f7753 commented on a diff in the pull request:
https://github.com/apache/spark/pull/14232#discussion_r71064072
--- Diff:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/protocol/BlockTransferMessage.java
---
@@ -35,6 +35,9 @@
* shuffle
GitHub user f7753 opened a pull request:
https://github.com/apache/spark/pull/14232
[SPARK-16587] The annotation for the abstract Class 'BlockTransferMessage'
missing several kind of messages.
**# What changes were proposed in this pull request?**
Currently, the Mes
Github user f7753 closed the pull request at:
https://github.com/apache/spark/pull/9616
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
GitHub user f7753 opened a pull request:
https://github.com/apache/spark/pull/9616
Branch 1.4
Has any one meet the following situation:
We observed that during the shuffle period of a sort job, the tasks
distributed among several nodes are running after by after, not
11 matches
Mail list logo
