Do you have multiple supervisors running on the same box? If so then that is the problem. I don't know of any other way to hit this issues, as from the code https://github.com/apache/storm/blob/v1.1.0/storm-core/src/jvm/org/apache/storm/localizer/AsyncLocalizer.java?utf8=%E2%9C%93#L147 Right above the line that is getting the exception it is checking to see if stormRoot already exists and if so it either cleans it up or just goes on assuming everything is fine, but then just a few lines later it is getting an error because there is already a fine in stormRoot that it is not able to overwrite.
If you don't have multiple supervisors sharing the same storm root location then something very wrong is happening, and we need to debug it because synchronization is off within the localization code. - Bobby On Friday, June 30, 2017, 5:04:42 AM CDT, Etienne Orliac <[email protected]> wrote: Dear all, I have a systematic failure when I try to launch storm 1.1.0 in production (non-local) mode. Can anyone guess what's going on there? In local mode, it works perfectly. Thank you in advance for your help! Etienne 2017-06-30 11:46:50.521 o.a.s.l.AsyncLocalizer main [INFO] Cleaning up unused topologies in /disk/storm/data/supervisor/stormdist 2017-06-30 11:46:50.526 o.a.s.d.s.Supervisor main [INFO] Starting supervisor with id b7672575-e969-4f57-b602-5d7fc4188739 at host xxx. 2017-06-30 11:46:50.527 o.a.s.d.m.MetricsUtils main [INFO] Using statistics reporter plugin:org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter 2017-06-30 11:46:50.528 o.a.s.d.m.r.JmxPreparableReporter main [INFO] Preparing... 2017-06-30 11:46:50.532 o.a.s.m.StormMetricsRegistry main [INFO] Started statistics report plugin... 2017-06-30 11:46:52.529 o.a.s.d.s.Slot SLOT_6720 [INFO] STATE EMPTY msInState: 2013 -> WAITING_FOR_BASIC_LOCALIZATION msInState: 0 2017-06-30 11:46:52.578 o.a.s.u.NimbusClient Async Localizer [INFO] Found leader nimbus : xxx:6627 2017-06-30 11:47:03.902 o.a.s.l.AsyncLocalizer Async Localizer [WARN] Failed to download basic resources for topology-id ttt-2-1498815928 2017-06-30 11:47:03.902 o.a.s.d.s.AdvancedFSOps Async Localizer [INFO] Deleting path /disk/storm/data/supervisor/tmp/bee206fd-01c8-41d0-be28-d734ee239daa 2017-06-30 11:47:04.023 o.a.s.d.s.AdvancedFSOps Async Localizer [INFO] Deleting path /disk/storm/data/supervisor/stormdist/ttt-2-1498815928 2017-06-30 11:47:04.023 o.a.s.l.AsyncLocalizer Async Localizer [WARN] Caught Exception While Downloading (rethrowing)... java.nio.file.FileAlreadyExistsException: /disk/storm/data/supervisor/tmp/bee206fd-01c8-41d0-be28-d734ee239daa -> /disk/storm/data/supervisor/stormdist/ttt-2-1498815928 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:88) ~[?:1.8.0_131] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_131] at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:396) ~[?:1.8.0_131] at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) ~[?:1.8.0_131] at java.nio.file.Files.move(Files.java:1395) ~[?:1.8.0_131] at org.apache.storm.daemon.supervisor.AdvancedFSOps.moveDirectoryPreferAtomic(AdvancedFSOps.java:176) ~[storm-core-1.1.0.jar:1.1.0] at org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:147) ~[storm-core-1.1.0.jar:1.1.0] at org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:101) ~[storm-core-1.1.0.jar:1.1.0] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] 2017-06-30 11:47:04.024 o.a.s.d.s.Slot SLOT_6720 [ERROR] Error when processing event java.util.concurrent.ExecutionException: java.nio.file.FileAlreadyExistsException: /disk/storm/data/supervisor/tmp/bee206fd-01c8-41d0-be28-d734ee239daa -> /disk/storm/data/supervisor/stormdist/ttt-2-1498815928
