[
https://issues.apache.org/jira/browse/HADOOP-16948?focusedWorklogId=551370&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-551370
]
ASF GitHub Bot logged work on HADOOP-16948:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 11/Feb/21 16:14
Start Date: 11/Feb/21 16:14
Worklog Time Spent: 10m
Work Description: billierinaldi commented on a change in pull request
#1925:
URL: https://github.com/apache/hadoop/pull/1925#discussion_r574631046
##########
File path: hadoop-tools/hadoop-azure/src/site/markdown/abfs.md
##########
@@ -877,6 +877,21 @@ enabled for your Azure Storage account."
The directories can be specified as comma separated values. By default the
value
is "/hbase"
+### <a name="singlewriteroptions"></a> Single Writer Options
+`fs.azure.singlewriter.directories`: Directories for single writer support
+can be specified comma separated in this config. By default, multiple
+clients will be able to write to the same file simultaneously. When writing
+to files contained within the directories specified in this config, the
+client will obtain a lease on the file that will prevent any other clients
+from writing to the file. The lease will be renewed by the client until the
+output stream is closed, after which it will be released. To revoke a client's
+write access for a file, the AzureBlobFilesystem breakLease method may be
+ called.
+
+`fs.azure.lease.threads`: This is the size of the thread pool that will be
+used for lease operations for single writer directories. By default the value
+is 0, so it must be set to at least 1 to support single writer directories.
Review comment:
> is there any validation here, that if a path in the local FS is to be
leased then the executor count must be >1?
Yes, in SelfRenewingLease it throws an exception if there are < 1 lease
threads.
> what if I'm working with >1 FS? Will this configuration be per-fs? Or does
it take a list of paths which can be full URIs to paths in a store?
I believe the single writer dirs config accepts a list of full URIs -- I
will double check -- but they all share the same pool of lease threads.
I am also looking into whether it makes sense to make the lease duration
configurable. This would allow configuration of a finite or infinite lease
duration, and in the infinite lease case we could avoid frequent calls to the
Azure API to renew the lease. (For an infinite lease, if the client stopped
without releasing the lease, the lease would have to be explicitly broken for a
different writer to obtain a new lease on the file. It's a tradeoff, and I
could imagine both finite and infinite lease options being useful.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 551370)
Time Spent: 3.5h (was: 3h 20m)
> ABFS: Support single writer dirs
> --------------------------------
>
> Key: HADOOP-16948
> URL: https://issues.apache.org/jira/browse/HADOOP-16948
> Project: Hadoop Common
> Issue Type: Sub-task
> Reporter: Billie Rinaldi
> Assignee: Billie Rinaldi
> Priority: Minor
> Labels: abfsactive, pull-request-available
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> This would allow some directories to be configured as single writer
> directories. The ABFS driver would obtain a lease when creating or opening a
> file for writing and would automatically renew the lease and release the
> lease when closing the file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
