[ https://issues.apache.org/jira/browse/OOZIE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16894407#comment-16894407 ]
Steve Loughran commented on OOZIE-3529: --------------------------------------- Creating a real S3A filesystem instance kicks off a HEAD request to the bucket; will require the caller to be online and inevitably some creds (unless the AnonymousCredentialProvider is used). You could register a fake S3 client underneath, but it'd be brittle as to how bits of the AWS SDK work with things. If you do create a real S3A connection, then you can actually call toString() on it and look for blockFactory=<something>. I'd recommend you just do that for asserts, e.g. assertTrue("wrong block factory in " + fs, fs.getConf(...)), so it stops the test being brittle there, but does at least dump enough information to start debugging things > Oozie not supported for s3 as filesystem > ---------------------------------------- > > Key: OOZIE-3529 > URL: https://issues.apache.org/jira/browse/OOZIE-3529 > Project: Oozie > Issue Type: Bug > Components: core > Affects Versions: 4.3.1, 5.1.0 > Reporter: Denes Bodo > Assignee: Denes Bodo > Priority: Critical > Labels: S3 > Attachments: OOZIE-3529.001.patch, id.pig, job.properties, > workflow.xml > > > Many customer who uses s3 file system as secondary one experiences the > following error when Oozie tries to submit the Yarn application: > {noformat} > 2019-04-29 13:02:53,770 WARN ForkedActionStartXCommand:523 - > SERVER[hwnode1.puretec.purestorage.com] USER[hrt_qa] GROUP[-] TOKEN[] > APP[demo-wf] JOB[0000001-190423141707256-oozie-oozi-W] > ACTION[0000001-190423141707256-oozie-oozi-W@streaming-node] Error starting > action [streaming-node]. ErrorType [ERROR], ErrorCode > [UnsupportedOperationException], Message [UnsupportedOperationException: > Accessing local file system is not allowed] > org.apache.oozie.action.ActionExecutorException: > UnsupportedOperationException: Accessing local file system is not allowed > at > org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446) > at > org.apache.oozie.action.hadoop.JavaActionExecutor.createLauncherConf(JavaActionExecutor.java:1092) > at > org.apache.oozie.action.hadoop.MapReduceActionExecutor.createLauncherConf(MapReduceActionExecutor.java:309) > at > org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1197) > at > org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1472) > at > org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234) > at > org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:41) > at > org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:30) > at org.apache.oozie.command.XCommand.call(XCommand.java:287) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.UnsupportedOperationException: Accessing local file > system is not allowed > at > org.apache.hadoop.fs.RawLocalFileSystem.initialize(RawLocalFileSystem.java:48) > at > org.apache.hadoop.fs.LocalFileSystem.initialize(LocalFileSystem.java:47) > at > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354) > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124) > at > org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477) > at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:433) > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:301) > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:378) > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createTmpFileForWrite(LocalDirAllocator.java:461) > at > org.apache.hadoop.fs.LocalDirAllocator.createTmpFileForWrite(LocalDirAllocator.java:200) > at > org.apache.hadoop.fs.s3a.S3AFileSystem.createTmpFileForWrite(S3AFileSystem.java:572) > at > org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory.create(S3ADataBlocks.java:811) > at > org.apache.hadoop.fs.s3a.S3ABlockOutputStream.createBlockIfNeeded(S3ABlockOutputStream.java:190) > at > org.apache.hadoop.fs.s3a.S3ABlockOutputStream.<init>(S3ABlockOutputStream.java:168) > at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:778) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1169) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1038) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1026) > at > org.apache.oozie.action.hadoop.LauncherMapperHelper.setupLauncherInfo(LauncherMapperHelper.java:156) > at > org.apache.oozie.action.hadoop.JavaActionExecutor.createLauncherConf(JavaActionExecutor.java:1033) > ... 12 more > {noformat} > Does anybody has any idea how could we modify the RawLocalFileSystem class to > make it a bit less strict? > Thank you for the repro wf to Soumitra Sulav. -- This message was sent by Atlassian JIRA (v7.6.14#76016)