[ https://issues.apache.org/jira/browse/SPARK-19718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shixiong Zhu resolved SPARK-19718. ---------------------------------- Resolution: Fixed Assignee: Shixiong Zhu Fix Version/s: 2.2.0 > Fix flaky test: > org.apache.spark.sql.kafka010.KafkaSourceStressForDontFailOnDataLossSuite: > stress test for failOnDataLoss=false > ------------------------------------------------------------------------------------------------------------------------------- > > Key: SPARK-19718 > URL: https://issues.apache.org/jira/browse/SPARK-19718 > Project: Spark > Issue Type: Bug > Components: Structured Streaming > Affects Versions: 2.2.0 > Reporter: Shixiong Zhu > Assignee: Shixiong Zhu > Fix For: 2.2.0 > > > SPARK-19617 changed HDFSMetadataLog to enable interrupts when using the local > file system. However, now we hit HADOOP-12074: `Shell.runCommand` converts > `InterruptedException` to `new IOException(ie.toString())` before Hadoop 2.8. > Test failure: > https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.6/2504/consoleFull > {code} > [info] - stress test for failOnDataLoss=false *** FAILED *** (1 minute, 1 > second) > [info] org.apache.spark.sql.streaming.StreamingQueryException: Query [id = > 27d45f4f-14dc-4c74-8b52-4bbd4f2b9bec, runId = > 23b8c1ea-4da9-4096-967a-692933e4b319] terminated with exception: > java.lang.InterruptedException > [info] at > org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runBatches(StreamExecution.scala:304) > [info] at > org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:190) > [info] Cause: java.io.IOException: java.lang.InterruptedException > [info] at org.apache.hadoop.util.Shell.runCommand(Shell.java:578) > [info] at org.apache.hadoop.util.Shell.run(Shell.java:478) > [info] at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:766) > [info] at org.apache.hadoop.util.Shell.execCommand(Shell.java:859) > [info] at org.apache.hadoop.util.Shell.execCommand(Shell.java:842) > [info] at > org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:661) > [info] at > org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:300) > [info] at > org.apache.hadoop.fs.FileSystem.primitiveCreate(FileSystem.java:1014) > [info] at > org.apache.hadoop.fs.DelegateToFileSystem.createInternal(DelegateToFileSystem.java:85) > [info] at > org.apache.hadoop.fs.ChecksumFs$ChecksumFSOutputSummer.<init>(ChecksumFs.java:354) > [info] at > org.apache.hadoop.fs.ChecksumFs.createInternal(ChecksumFs.java:394) > [info] at > org.apache.hadoop.fs.AbstractFileSystem.create(AbstractFileSystem.java:577) > [info] at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:680) > [info] at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:676) > [info] at > org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) > [info] at org.apache.hadoop.fs.FileContext.create(FileContext.java:676) > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org