[
https://issues.apache.org/jira/browse/SPARK-38062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17484013#comment-17484013
]
Erik Krogen edited comment on SPARK-38062 at 1/28/22, 11:10 PM:
----------------------------------------------------------------
Sorry, I should have clarified -- the test issues/failures I described were
observed on 3.1.1.
I did confirm that on the current {{master}} branch, these tests are entirely
skipped in our environment, so there are no failures because the tests are
never executed.
After the changes in the PR I put up, all FallbackStorageSuite tests run
successfully on master and 3.1.1
was (Author: xkrogen):
Sorry, I should have clarified -- the test issues/failures I described were
observed on 3.1.1.
I did confirm that on the current {{master}} branch, these tests are entirely
skipped in our environment, so there are no failures.
After the changes in the PR I put up, all FallbackStorageSuite tests run
successfully on master and 3.1.1
> FallbackStorage shouldn't attempt to resolve arbitrary "remote" hostname
> ------------------------------------------------------------------------
>
> Key: SPARK-38062
> URL: https://issues.apache.org/jira/browse/SPARK-38062
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.1.1
> Reporter: Erik Krogen
> Priority: Major
>
> {{FallbackStorage}} uses a placeholder block manager ID:
> {code:scala}
> private[spark] object FallbackStorage extends Logging {
> /** We use one block manager id as a place holder. */
> val FALLBACK_BLOCK_MANAGER_ID: BlockManagerId = BlockManagerId("fallback",
> "remote", 7337)
> {code}
> That second argument is normally interpreted as a hostname, but is passed as
> the string "remote" in this case.
> {{BlockManager}} will consider this placeholder as one of the peers in some
> cases:
> {code:language=scala|title=BlockManager.scala}
> private[storage] def getPeers(forceFetch: Boolean): Seq[BlockManagerId] = {
> peerFetchLock.synchronized {
> ...
> if (cachedPeers.isEmpty &&
>
> conf.get(config.STORAGE_DECOMMISSION_FALLBACK_STORAGE_PATH).isDefined) {
> Seq(FallbackStorage.FALLBACK_BLOCK_MANAGER_ID)
> } else {
> cachedPeers
> }
> }
> }
> {code}
> {{BlockManagerDecommissioner.ShuffleMigrationRunnable}} will then attempt to
> perform an upload to this placeholder ID:
> {code:scala}
> try {
> blocks.foreach { case (blockId, buffer) =>
> logDebug(s"Migrating sub-block ${blockId}")
> bm.blockTransferService.uploadBlockSync(
> peer.host,
> peer.port,
> peer.executorId,
> blockId,
> buffer,
> StorageLevel.DISK_ONLY,
> null) // class tag, we don't need for shuffle
> logDebug(s"Migrated sub-block $blockId")
> }
> logInfo(s"Migrated $shuffleBlockInfo to $peer")
> } catch {
> case e: IOException =>
> ...
> if
> (bm.migratableResolver.getMigrationBlocks(shuffleBlockInfo).size <
> blocks.size) {
> logWarning(s"Skipping block $shuffleBlockInfo, block
> deleted.")
> } else if (fallbackStorage.isDefined) {
> fallbackStorage.foreach(_.copy(shuffleBlockInfo, bm))
> } else {
> logError(s"Error occurred during migrating
> $shuffleBlockInfo", e)
> keepRunning = false
> }
> {code}
> Since "remote" is not expected to be a resolvable hostname, an
> {{IOException}} occurs, and {{fallbackStorage}} is used. But, we shouldn't
> try to resolve this. First off, it's completely unnecessary and strange to be
> treating the placeholder ID as a resolvable hostname, relying on an exception
> to realize that we need to use the {{fallbackStorage}}.
> To make matters worse, in some network environments, "remote" may be a
> resolvable hostname, completely breaking this functionality. In the
> particular environment that I use for running automated tests, there is a DNS
> entry for "remote" which, when you attempt to connect to it, will hang for a
> long period of time. This essentially hangs the executor decommission
> process, and in the case of unit tests, breaks {{FallbackStorageSuite}} as it
> exceeds its timeouts. I'm not sure, but it's possible this is related to
> SPARK-35584 as well (if sometimes in the GA environment, it takes a long time
> for the OS to decide that "remote" is not a valid hostname).
> We shouldn't attempt to treat this placeholder ID as a real hostname.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
