[
https://issues.apache.org/jira/browse/HIVE-19562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16479965#comment-16479965
]
Hive QA commented on HIVE-19562:
--------------------------------
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
13s{color} | {color:green} The patch has no ill-formed XML file. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
54s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11037/dev-support/hive-personality.sh
|
| git revision | master / 0b6bea8 |
| modules | C: . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11037/yetus.txt |
| Powered by | Apache Yetus http://yetus.apache.org |
This message was automatically generated.
> Flaky test: TestMiniSparkOnYarn FileNotFoundException in spark-submit
> ---------------------------------------------------------------------
>
> Key: HIVE-19562
> URL: https://issues.apache.org/jira/browse/HIVE-19562
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Priority: Major
> Attachments: HIVE-19562.1.patch, HIVE-19562.3.patch,
> HIVE-19562.4.patch
>
>
> Seeing sporadic failures during test setup. Specifically, when spark-submit
> runs this error (or a similar error) gets thrown:
> {code}
> 2018-05-15T10:55:02,112 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: Exception in thread "main"
> java.io.FileNotFoundException: File
> file:/tmp/spark-56e217f7-b8a5-4c63-9a6b-d737a64f2820/__spark_libs__7371510645900072447.zip
> does not exist
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:867)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:365)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:316)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:356)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.spark.deploy.yarn.Client.org$apache$spark$deploy$yarn$Client$$distribute$1(Client.scala:478)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:565)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:863)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:169)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.spark.deploy.yarn.Client.run(Client.scala:1146)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1518)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> {code}
> Essentially, Spark is writing some files for container localization to a tmp
> dir, and that tmp dir is getting deleted. We have seen a lot of issues with
> writing files to {{/tmp/}} in the past, so its probably best to write these
> files to a test-specific dir.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
