[ https://issues.apache.org/jira/browse/SPARK-19909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968092#comment-15968092 ]
Rob commented on SPARK-19909: ----------------------------- Is there an easy way to avoid this issue while waiting for it to be resolved? Perhaps tweaking a setting? > Batches will fail in case that temporary checkpoint dir is on local file > system while metadata dir is on HDFS > ------------------------------------------------------------------------------------------------------------- > > Key: SPARK-19909 > URL: https://issues.apache.org/jira/browse/SPARK-19909 > Project: Spark > Issue Type: Bug > Components: Structured Streaming > Affects Versions: 2.2.0 > Reporter: Kousuke Saruta > Priority: Minor > > When we try to run Structured Streaming in local mode but use HDFS for the > storage, batches will be fail because of error like as follows. > {code} > val handle = stream.writeStream.format("console").start() > 17/03/09 16:54:45 ERROR StreamMetadata: Error writing stream metadata > StreamMetadata(fc07a0b1-5423-483e-a59d-b2206a49491e) to > /private/var/folders/4y/tmspvv353y59p3w4lknrf7cc0000gn/T/temporary-79d4fe05-4301-4b6d-a902-dff642d0ddca/metadata > org.apache.hadoop.security.AccessControlException: Permission denied: > user=kou, access=WRITE, > inode="/private/var/folders/4y/tmspvv353y59p3w4lknrf7cc0000gn/T/temporary-79d4fe05-4301-4b6d-a902-dff642d0ddca/metadata":hdfs:supergroup:drwxr-xr-x > {code} > It's because that a temporary checkpoint directory is created on local file > system but metadata whose path is based on the checkpoint directory will be > created on HDFS. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org