[
https://issues.apache.org/jira/browse/HADOOP-12239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14636022#comment-14636022
]
Chris Nauroth commented on HADOOP-12239:
----------------------------------------
Hello [~onpduo]. The patch looks good to me. I also looked around for other
bare calls to {{updateFolderLastModifiedTime}} that aren't guarded by the
lease, and I didn't find any. Hopefully that means these issues are completely
fixed now.
bq. The patch doesn't appear to include any new or modified tests. Please
justify why no new tests are needed for this patch. Also please list what
manual steps were performed to verify this patch.
I understand that it would be extremely complex to try to cover this in a unit
test. Can you please describe any manual testing that you've done? Thanks!
> StorageException complaining " no lease ID" when updating
> FolderLastModifiedTime in WASB
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-12239
> URL: https://issues.apache.org/jira/browse/HADOOP-12239
> Project: Hadoop Common
> Issue Type: Bug
> Components: azure, tools
> Affects Versions: 2.7.0
> Environment: Windows Azure
> Reporter: Duo Xu
> Assignee: Duo Xu
> Attachments: HADOOP-12239.01.patch
>
>
> This is a similar issue as HADOOP-11523 and HADOOP-12089, which I found in a
> customer's HBase cluster logs, but the piece of code is in a different place.
> {code}
> 2015-07-09 13:38:57,388 INFO org.apache.hadoop.hbase.master.SplitLogManager:
> dead splitlog workers
> [workernode3.xxx.b6.internal.cloudapp.net,60020,1436448555180]
> 2015-07-09 13:38:57,466 ERROR org.apache.hadoop.hbase.executor.EventHandler:
> Caught throwable while processing event M_SERVER_SHUTDOWN
> java.io.IOException: failed log splitting for
> workernode12.xxx.b6.internal.cloudapp.net,60020,1436 448566374, will retry
> at
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:343)
> at
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:211)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Unable to write RenamePending file for folder
> rename from
> hbase/WALs/workernode12.xxx.b6.internal.cloudapp.net,60020,1436448566374 to
> hbase/WALs/workernode12.xxx.b6.internal.cloudapp.net,60020,1436448566374-splitting
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.writeFile(NativeAzureFileSystem.java:258)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.prepareAtomicFolderRename(NativeAzureFileSystem.java:2110)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1998)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.getLogDirs(MasterFileSystem.java:325)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:412)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:390)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:288)
> at
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:204)
> ... 4 more
> Caused by: org.apache.hadoop.fs.azure.AzureException:
> com.microsoft.azure.storage.StorageException: There is currently a lease on
> the blob and no lease ID was specified in the request.
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2598)
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2609)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1366)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1195)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:775)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.writeFile(NativeAzureFileSystem.java:255)
> ... 11 more
> Caused by: com.microsoft.azure.storage.StorageException: There is currently a
> lease on the blob and no lease ID was specified in the request.
> at
> com.microsoft.azure.storage.StorageException.translateException(StorageException.java:89)
> at
> com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:307)
> at
> com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:182)
> at
> com.microsoft.azure.storage.blob.CloudBlob.uploadProperties(CloudBlob.java:2892)
> at
> org.apache.hadoop.fs.azure.StorageInterfaceImpl$CloudBlobWrapperImpl.uploadProperties(StorageInterfaceImpl.java:372)
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2593)
> ... 19 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
