[jira] [Updated] (HDFS-14431) RBF: Rename with multiple subclusters should fail if no eligible locations
[ https://issues.apache.org/jira/browse/HDFS-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-14431: Parent Issue: HDFS-14603 (was: HDFS-13891) > RBF: Rename with multiple subclusters should fail if no eligible locations > -- > > Key: HDFS-14431 > URL: https://issues.apache.org/jira/browse/HDFS-14431 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-14431-HDFS-13891.001.patch, > HDFS-14431-HDFS-13891.002.patch, HDFS-14431-HDFS-13891.003.patch, > HDFS-14431-HDFS-13891.004.patch, HDFS-14431-HDFS-13891.005.patch, > HDFS-14431-HDFS-13891.006.patch, HDFS-14431-HDFS-13891.007.patch > > > Currently, the rename will fail with FileNotFoundException which is not clear > to the user. > The operation should fail stating the reason is that there are no eligible > destinations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14257) RBF : NPE when given the Invalid path to create target dir
[
https://issues.apache.org/jira/browse/HDFS-14257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-14257:
Parent Issue: HDFS-14603 (was: HDFS-13891)
> RBF : NPE when given the Invalid path to create target dir
> --
>
> Key: HDFS-14257
> URL: https://issues.apache.org/jira/browse/HDFS-14257
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Harshakiran Reddy
>Assignee: venkata ramkumar
>Priority: Major
> Labels: RBF
>
> bin> ./hdfs dfs -mkdir hdfs://{color:red}hacluster2 /hacluster1{color}dest2/
> {noformat}
> -mkdir: Fatal internal error
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.FileSystem.fixRelativePart(FileSystem.java:2714)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.fixRelativePart(DistributedFileSystem.java:3229)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1618)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1742)
> at
> org.apache.hadoop.fs.shell.Mkdir.processNonexistentPath(Mkdir.java:74)
> at
> org.apache.hadoop.fs.shell.Command.processArgument(Command.java:287)
> at
> org.apache.hadoop.fs.shell.Command.processArguments(Command.java:269)
> at
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:121)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:391)
> bin>
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14430) RBF: Fix MockNamenode bug about mocking RPC getListing and mkdir
[ https://issues.apache.org/jira/browse/HDFS-14430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-14430: Parent Issue: HDFS-14603 (was: HDFS-13891) > RBF: Fix MockNamenode bug about mocking RPC getListing and mkdir > > > Key: HDFS-14430 > URL: https://issues.apache.org/jira/browse/HDFS-14430 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Reporter: He Xiaoqiao >Assignee: He Xiaoqiao >Priority: Major > Attachments: HDFS-14430-HDFS-13891.001.patch > > > Some unexpected result when invoke mocking #getListing and #mkdirs in current > MockNamenode implement. > * for mock mkdirs, we do not check if parent directory exists. > * for mock getListing, some child dirs/files are not listing. > It may be cause some unexpected result and cause some unit test fail. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14284) RBF: Log Router identifier when reporting exceptions
[ https://issues.apache.org/jira/browse/HDFS-14284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-14284: Parent Issue: HDFS-14603 (was: HDFS-13891) > RBF: Log Router identifier when reporting exceptions > > > Key: HDFS-14284 > URL: https://issues.apache.org/jira/browse/HDFS-14284 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Priority: Major > > The typical setup is to use multiple Routers through > ConfiguredFailoverProxyProvider. > In a regular HA Namenode setup, it is easy to know which NN was used. > However, in RBF, any Router can be the one reporting the exception and it is > hard to know which was the one. > We should have a way to identify which Router/Namenode was the one triggering > the exception. > This would also apply with Observer Namenodes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14427) RBF: Optimize some testing set up logic in MiniRouterDFSCluster
[ https://issues.apache.org/jira/browse/HDFS-14427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-14427: Parent Issue: HDFS-14603 (was: HDFS-13891) > RBF: Optimize some testing set up logic in MiniRouterDFSCluster > --- > > Key: HDFS-14427 > URL: https://issues.apache.org/jira/browse/HDFS-14427 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Major > Attachments: HDFS-14427-HDFS-13891.001.patch > > > [https://github.com/apache/hadoop/blob/HDFS-13891/hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/MiniRouterDFSCluster.java#L808] > the comment says one router is created per name service, while in the code > one router is created per namenode in each nameservice. > There are a couple of things that might need to consider optimization: > # make the code as the the comment > # add some ways to specify the number of routers -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14254) RBF: Getfacl gives a wrong acl entries when the order of the mount table set to HASH_ALL or RANDOM
[ https://issues.apache.org/jira/browse/HDFS-14254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-14254: Parent Issue: HDFS-14603 (was: HDFS-13891) > RBF: Getfacl gives a wrong acl entries when the order of the mount table set > to HASH_ALL or RANDOM > -- > > Key: HDFS-14254 > URL: https://issues.apache.org/jira/browse/HDFS-14254 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Shubham Dewan >Assignee: Ranith Sardar >Priority: Major > Attachments: HDFS-14254-HDFS-13891.000.patch, > HDFS-14254-HDFS-13891.001.patch, HDFS-14254-HDFS-13891.002.patch, > HDFS-14254-HDFS-13891.003.patch > > > ACL entries are missing when Order is set to HASH_ALL or RANDOM -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14385) RBF: Optimize MiniRouterDFSCluster with optional light weight MiniDFSCluster
[ https://issues.apache.org/jira/browse/HDFS-14385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-14385: Parent Issue: HDFS-14603 (was: HDFS-13891) > RBF: Optimize MiniRouterDFSCluster with optional light weight MiniDFSCluster > > > Key: HDFS-14385 > URL: https://issues.apache.org/jira/browse/HDFS-14385 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Reporter: He Xiaoqiao >Assignee: He Xiaoqiao >Priority: Major > Attachments: HDFS-14385-HDFS-13891.001.patch > > > MiniRouterDFSCluster mimic federated HDFS cluster with routers to support RBF > test, In MiniRouterDFSCluster, it starts MiniDFSCluster with complete roles > of HDFS which have significant time cost. As HDFS-14351 discussed, it is > better to provide mock MiniDFSCluster/Namenodes as one option to support some > test case and reduce time cost. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13896) RBF: Web UI not displaying clearly which target path is pointing to which name service in mount table
[ https://issues.apache.org/jira/browse/HDFS-13896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13896: Parent Issue: HDFS-14603 (was: HDFS-13891) > RBF: Web UI not displaying clearly which target path is pointing to which > name service in mount table > -- > > Key: HDFS-13896 > URL: https://issues.apache.org/jira/browse/HDFS-13896 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: venkata ramkumar >Assignee: venkata ramkumar >Priority: Minor > Attachments: HDFS-13896.001.patch, HDFS-13896.002.patch, > HDFS-13896_after.png, HDFS-13896_before.png > > > Commands:/HDFS/hadoop/bin> ./hdfs dfsrouteradmin -add /apps hacluster1 /opt > 18/09/05 12:32:44 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Successfully added mount point /apps > Command: ./hdfs dfsrouteradmin -add /apps hacluster2 /opt1 > 18/09/05 12:33:13 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Successfully added mount point /apps > Command: /HDFS/hadoop/bin> ./hdfs dfsrouteradmin -add /apps hacluster1 /opt2 > 18/09/05 14:21:12 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Successfully added mount point /apps > Command :/HDFS/hadoop/bin> ./hdfs dfsrouteradmin -ls > 18/09/05 12:34:16 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Mount Table Entries: > Source Destinations Owner Group Mode Quota/Usage > /apps hacluster1->/opt,hacluster2->/opt1,hacluster1->/opt2 securedn users > rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] > WebUI : Mount Table > ||Global path||Target nameservice||Target path||Order||Read > only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created|| > |/apps|hacluster1,hacluster2|/opt,/opt1,/opt2|HASH| > |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota: -/-]|2018/09/05 > 16:50:54|2018/09/05 15:02:25| > > > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14169) RBF: Correct the returned value in case of IOException in NamenodeBeanMetrics#getFederationMetrics
[ https://issues.apache.org/jira/browse/HDFS-14169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-14169: Parent Issue: HDFS-14603 (was: HDFS-13891) > RBF: Correct the returned value in case of IOException in > NamenodeBeanMetrics#getFederationMetrics > -- > > Key: HDFS-14169 > URL: https://issues.apache.org/jira/browse/HDFS-14169 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-14169-HDFS-13891-01.patch > > > Presently in case of IOException the the metrics value returned is 0 which is > a legal entry.Better to change to a value which could indicate that the value > hasn't been actually fetched. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[
https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-14117:
Parent Issue: HDFS-14603 (was: HDFS-13891)
> RBF: We can only delete the files or dirs of one subcluster in a cluster with
> multiple subclusters when trash is enabled
>
>
> Key: HDFS-14117
> URL: https://issues.apache.org/jira/browse/HDFS-14117
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: venkata ramkumar
>Priority: Major
> Labels: RBF
> Attachments: HDFS-14117-HDFS-13891.001.patch,
> HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch,
> HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch,
> HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch,
> HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch,
> HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch,
> HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch,
> HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch,
> HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch,
> HDFS-14117-HDFS-13891.018.patch, HDFS-14117-HDFS-13891.019.patch,
> HDFS-14117-HDFS-13891.020.patch, HDFS-14117.001.patch, HDFS-14117.002.patch,
> HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch
>
>
> When we delete files or dirs in hdfs, it will move the deleted files or dirs
> to trash by default.
> But in the global path we can only mount one trash dir /user. So we mount
> trash dir /user of the subcluster ns1 to the global path /user. Then we can
> delete files or dirs of ns1, but when we delete the files or dirs of another
> subcluser, such as hacluster, it will be failed.
> h1. Mount Table
> ||Global path||Target nameservice||Target path||Order||Read
> only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created||
> |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota:
> -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42|
> |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota:
> -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05|
> |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota:
> -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20|
> commands:
> {noformat}
> 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/.
> 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 1 items
> -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd
> 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/.
> 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 1 items
> -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd
> 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm
> /tmp/mapred.cmd
> 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination
> parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found.
> 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd
> 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved:
> 'hdfs://router/test/hdfs.cmd' to trash at:
> hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13811) RBF: Race condition between router admin quota update and periodic quota update service
[ https://issues.apache.org/jira/browse/HDFS-13811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13811: Parent Issue: HDFS-14603 (was: HDFS-13891) > RBF: Race condition between router admin quota update and periodic quota > update service > --- > > Key: HDFS-13811 > URL: https://issues.apache.org/jira/browse/HDFS-13811 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Dibyendu Karmakar >Assignee: Dibyendu Karmakar >Priority: Major > Attachments: HDFS-13811-000.patch, HDFS-13811-HDFS-13891-000.patch > > > If we try to update quota of an existing mount entry and at the same time > periodic quota update service is running on the same mount entry, it is > leading the mount table to _inconsistent state._ > Here transactions are: > A - Quota update service is fetching mount table entries. > B - Quota update service is updating the mount table with current usage. > A' - User is trying to update quota using admin cmd. > and the transaction sequence is [ A A' B ] > quota update service is updating the mount table with old quota value. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14079) RBF: RouterAdmin should have failover concept for router
[
https://issues.apache.org/jira/browse/HDFS-14079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16872034#comment-16872034
]
Hadoop QA commented on HDFS-14079:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} HDFS-14079 does not apply to HDFS-13891. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-14079 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949647/HDFS-14079-HDFS-13891.02.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/27070/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF: RouterAdmin should have failover concept for router
>
>
> Key: HDFS-14079
> URL: https://issues.apache.org/jira/browse/HDFS-14079
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Major
> Attachments: HDFS-14079-HDFS-13891.01.patch,
> HDFS-14079-HDFS-13891.02.patch
>
>
> Currenlty {{RouterAdmin}} connect with only one router for admin operation,
> if the configured router is down then router admin command is failing. It
> should allow to configure all the router admin address.
> {code}
> // Initialize RouterClient
> try {
> String address = getConf().getTrimmed(
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_KEY,
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_DEFAULT);
> InetSocketAddress routerSocket = NetUtils.createSocketAddr(address);
> client = new RouterClient(routerSocket, getConf());
> } catch (RPC.VersionMismatch v) {
> System.err.println(
> "Version mismatch between client and server... command aborted");
> return exitCode;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14090) RBF: Improved isolation for downstream name nodes.
[ https://issues.apache.org/jira/browse/HDFS-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-14090: Parent Issue: HDFS-14603 (was: HDFS-13891) > RBF: Improved isolation for downstream name nodes. > -- > > Key: HDFS-14090 > URL: https://issues.apache.org/jira/browse/HDFS-14090 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: CR Hota >Assignee: CR Hota >Priority: Major > Attachments: HDFS-14090-HDFS-13891.001.patch, > HDFS-14090-HDFS-13891.002.patch, HDFS-14090-HDFS-13891.003.patch, > HDFS-14090-HDFS-13891.004.patch, HDFS-14090-HDFS-13891.005.patch, RBF_ > Isolation design.pdf > > > Router is a gateway to underlying name nodes. Gateway architectures, should > help minimize impact of clients connecting to healthy clusters vs unhealthy > clusters. > For example - If there are 2 name nodes downstream, and one of them is > heavily loaded with calls spiking rpc queue times, due to back pressure the > same with start reflecting on the router. As a result of this, clients > connecting to healthy/faster name nodes will also slow down as same rpc queue > is maintained for all calls at the router layer. Essentially the same IPC > thread pool is used by router to connect to all name nodes. > Currently router uses one single rpc queue for all calls. Lets discuss how we > can change the architecture and add some throttling logic for > unhealthy/slow/overloaded name nodes. > One way could be to read from current call queue, immediately identify > downstream name node and maintain a separate queue for each underlying name > node. Another simpler way is to maintain some sort of rate limiter configured > for each name node and let routers drop/reject/send error requests after > certain threshold. > This won’t be a simple change as router’s ‘Server’ layer would need redesign > and implementation. Currently this layer is the same as name node. > Opening this ticket to discuss, design and implement this feature. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14079) RBF: RouterAdmin should have failover concept for router
[
https://issues.apache.org/jira/browse/HDFS-14079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-14079:
Parent Issue: HDFS-14603 (was: HDFS-13891)
> RBF: RouterAdmin should have failover concept for router
>
>
> Key: HDFS-14079
> URL: https://issues.apache.org/jira/browse/HDFS-14079
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Major
> Attachments: HDFS-14079-HDFS-13891.01.patch,
> HDFS-14079-HDFS-13891.02.patch
>
>
> Currenlty {{RouterAdmin}} connect with only one router for admin operation,
> if the configured router is down then router admin command is failing. It
> should allow to configure all the router admin address.
> {code}
> // Initialize RouterClient
> try {
> String address = getConf().getTrimmed(
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_KEY,
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_DEFAULT);
> InetSocketAddress routerSocket = NetUtils.createSocketAddr(address);
> client = new RouterClient(routerSocket, getConf());
> } catch (RPC.VersionMismatch v) {
> System.err.println(
> "Version mismatch between client and server... command aborted");
> return exitCode;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14602) EC:RESTAPI Support for set/unset REPLICATE EcPolicy via WebHdfs
[ https://issues.apache.org/jira/browse/HDFS-14602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16872030#comment-16872030 ] Souryakanta Dwivedy commented on HDFS-14602: Yes it is working with policy name as replication [~ayushtkn].As in ec document and hdfs shell command the configuration option is provided as replicate not replication,which create confusion for me. But after configuration of replicate , if you do get policy via webHdfs it is returning value as NULL. > EC:RESTAPI Support for set/unset REPLICATE EcPolicy via WebHdfs > > > Key: HDFS-14602 > URL: https://issues.apache.org/jira/browse/HDFS-14602 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.1.1 >Reporter: Souryakanta Dwivedy >Assignee: Ayush Saxena >Priority: Minor > > EC:RESTAPI Support for set/unset REPLICATE EcPolicy via WebHdfs -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13694) Making md5 computing being in parallel with image loading
[
https://issues.apache.org/jira/browse/HDFS-13694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16872008#comment-16872008
]
Hadoop QA commented on HDFS-13694:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 19s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 35s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 28 unchanged - 0 fixed = 29 total (was 28) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 21s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
18s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}121m 0s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}179m 34s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Should
org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader$DigestThread
be a _static_ inner class? At FSImageFormatProtobuf.java:inner class? At
FSImageFormatProtobuf.java:[lines 176-203] |
| Failed junit tests | hadoop.hdfs.server.diskbalancer.TestDiskBalancer |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-13694 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12972805/HDFS-13694-003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux a4e9d697a2b6 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Work logged] (HDDS-1617) Restructure the code layout for Ozone Manager
[ https://issues.apache.org/jira/browse/HDDS-1617?focusedWorklogId=266334=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266334 ] ASF GitHub Bot logged work on HDDS-1617: Author: ASF GitHub Bot Created on: 25/Jun/19 05:09 Start Date: 25/Jun/19 05:09 Worklog Time Spent: 10m Work Description: anuengineer commented on pull request #880: HDDS-1617. Restructure the code layout for Ozone Manager URL: https://github.com/apache/hadoop/pull/880 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266334) Time Spent: 2.5h (was: 2h 20m) > Restructure the code layout for Ozone Manager > - > > Key: HDDS-1617 > URL: https://issues.apache.org/jira/browse/HDDS-1617 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Manager >Reporter: Anu Engineer >Assignee: Anu Engineer >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > The Ozone Manager has a flat structure that deals with lot of specific > functions. This Jira proposes to refactor ozone managers code base and move > function specific packages. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1617) Restructure the code layout for Ozone Manager
[ https://issues.apache.org/jira/browse/HDDS-1617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDDS-1617: --- Status: Open (was: Patch Available) Abandoning this patch, since it changes too much stuff so close to 0.4.1 > Restructure the code layout for Ozone Manager > - > > Key: HDDS-1617 > URL: https://issues.apache.org/jira/browse/HDDS-1617 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Manager >Reporter: Anu Engineer >Assignee: Anu Engineer >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > The Ozone Manager has a flat structure that deals with lot of specific > functions. This Jira proposes to refactor ozone managers code base and move > function specific packages. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1723) Create new OzoneManagerLock class
[ https://issues.apache.org/jira/browse/HDDS-1723?focusedWorklogId=266322=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266322 ] ASF GitHub Bot logged work on HDDS-1723: Author: ASF GitHub Bot Created on: 25/Jun/19 04:44 Start Date: 25/Jun/19 04:44 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #1006: HDDS-1723. Create new OzoneManagerLock class. URL: https://github.com/apache/hadoop/pull/1006#issuecomment-505280177 Thank You @anuengineer for the review. I have fixed the review comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266322) Time Spent: 5h 50m (was: 5h 40m) > Create new OzoneManagerLock class > - > > Key: HDDS-1723 > URL: https://issues.apache.org/jira/browse/HDDS-1723 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 5h 50m > Remaining Estimate: 0h > > This Jira is to use bit manipulation, instead of hashmap in OzoneManager lock > logic. And also this Jira follows the locking order based on the document > attached to HDDS-1672 jira. > This Jira is created based on [~anu] comment during review of HDDS-1672. > Not a suggestion for this patch. But more of a question, should we just > maintain a bitset here, and just flip that bit up and down to see if the lock > is held. Or we can just maintain 32 bit integer, and we can easily find if a > lock is held by Xoring with the correct mask. I feel that might be super > efficient. [@nandakumar131|https://github.com/nandakumar131] . But as I said > let us not do that in this patch. > > This Jira will add new class, integration of this new class into code will be > done in a new jira. > Clean up of old code also will be done in new jira. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1723) Create new OzoneManagerLock class
[
https://issues.apache.org/jira/browse/HDDS-1723?focusedWorklogId=266321=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266321
]
ASF GitHub Bot logged work on HDDS-1723:
Author: ASF GitHub Bot
Created on: 25/Jun/19 04:43
Start Date: 25/Jun/19 04:43
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #1006:
HDDS-1723. Create new OzoneManagerLock class.
URL: https://github.com/apache/hadoop/pull/1006#discussion_r297003720
##
File path:
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/OzoneManagerLock.java
##
@@ -0,0 +1,336 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.lock;
+
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ozone.lock.LockManager;
+
+/**
+ * Provides different locks to handle concurrency in OzoneMaster.
+ * We also maintain lock hierarchy, based on the weight.
+ *
+ *
+ *
+ *
+ * WEIGHT LOCK
+ *
+ *
+ * 0 S3 Bucket Lock
+ *
+ *
+ * 1 Volume Lock
+ *
+ *
+ * 2 Bucket Lock
+ *
+ *
+ * 3 User Lock
+ *
+ *
+ * 4 S3 Secret Lock
+ *
+ *
+ * 5 Prefix Lock
+ *
+ *
+ *
+ * One cannot obtain a lower weight lock while holding a lock with higher
+ * weight. The other way around is possible.
+ *
+ *
+ * For example:
+ *
+ * {@literal ->} acquire volume lock (will work)
+ * {@literal +->} acquire bucket lock (will work)
+ * {@literal +-->} acquire s3 bucket lock (will throw Exception)
+ *
+ *
+ */
+
+public class OzoneManagerLock {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerLock.class);
+
+ private final LockManager manager;
+ private final ThreadLocal lockSet = ThreadLocal.withInitial(
+ () -> Short.valueOf((short)0));
+
+
+ /**
+ * Creates new OzoneManagerLock instance.
+ * @param conf Configuration object
+ */
+ public OzoneManagerLock(Configuration conf) {
+manager = new LockManager<>(conf);
+ }
+
+ /**
+ * Acquire lock on resource.
+ *
+ * For S3_Bucket, VOLUME, BUCKET type resource, same thread acquiring lock
+ * again is allowed.
+ *
+ * For USER, PREFIX, S3_SECRET type resource, same thread acquiring lock
+ * again is not allowed.
+ *
+ * Special Note for UserLock: Single thread can acquire single user lock/
+ * multi user lock. But not both at the same time.
+ * @param resourceName - Resource name on which user want to acquire lock.
+ * @param resource - Type of the resource.
+ */
+ public void acquireLock(String resourceName, Resource resource) {
+if (!resource.canLock(lockSet.get())) {
+ String errorMessage = getErrorMessage(resource);
+ LOG.error(errorMessage);
+ throw new RuntimeException(errorMessage);
+} else {
+ manager.lock(resourceName);
+ lockSet.set(resource.setLock(lockSet.get()));
+}
+ }
+
+ private String getErrorMessage(Resource resource) {
+return "Thread '" + Thread.currentThread().getName() + "' cannot " +
+"acquire " + resource.name + " lock while holding " +
+getCurrentLocks().toString() + " lock(s).";
+
+ }
+
+ private List getCurrentLocks() {
+List currentLocks = new ArrayList<>();
+int i=0;
+short lockSetVal = lockSet.get();
+for (Resource value : Resource.values()) {
+ if (value.isLevelLocked(lockSetVal)) {
+currentLocks.add(value.getName());
+ }
+}
+return currentLocks;
+ }
+
+ /**
+ * Acquire lock on multiple users.
+ * @param oldUserResource
+ * @param newUserResource
+ */
+ public void acquireMultiUserLock(String oldUserResource,
+ String newUserResource) {
+Resource resource = Resource.USER;
+if (!resource.canLock(lockSet.get())) {
+ String errorMessage = getErrorMessage(resource);
+ LOG.error(errorMessage);
+ throw new RuntimeException(errorMessage);
+} else {
+ // When
[jira] [Work logged] (HDDS-1723) Create new OzoneManagerLock class
[
https://issues.apache.org/jira/browse/HDDS-1723?focusedWorklogId=266320=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266320
]
ASF GitHub Bot logged work on HDDS-1723:
Author: ASF GitHub Bot
Created on: 25/Jun/19 04:42
Start Date: 25/Jun/19 04:42
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #1006:
HDDS-1723. Create new OzoneManagerLock class.
URL: https://github.com/apache/hadoop/pull/1006#discussion_r297003664
##
File path:
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/OzoneManagerLock.java
##
@@ -0,0 +1,336 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.lock;
+
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ozone.lock.LockManager;
+
+/**
+ * Provides different locks to handle concurrency in OzoneMaster.
+ * We also maintain lock hierarchy, based on the weight.
+ *
+ *
+ *
+ *
+ * WEIGHT LOCK
+ *
+ *
+ * 0 S3 Bucket Lock
+ *
+ *
+ * 1 Volume Lock
+ *
+ *
+ * 2 Bucket Lock
+ *
+ *
+ * 3 User Lock
+ *
+ *
+ * 4 S3 Secret Lock
+ *
+ *
+ * 5 Prefix Lock
+ *
+ *
+ *
+ * One cannot obtain a lower weight lock while holding a lock with higher
+ * weight. The other way around is possible.
+ *
+ *
+ * For example:
+ *
+ * {@literal ->} acquire volume lock (will work)
+ * {@literal +->} acquire bucket lock (will work)
+ * {@literal +-->} acquire s3 bucket lock (will throw Exception)
+ *
+ *
+ */
+
+public class OzoneManagerLock {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerLock.class);
+
+ private final LockManager manager;
+ private final ThreadLocal lockSet = ThreadLocal.withInitial(
+ () -> Short.valueOf((short)0));
+
+
+ /**
+ * Creates new OzoneManagerLock instance.
+ * @param conf Configuration object
+ */
+ public OzoneManagerLock(Configuration conf) {
+manager = new LockManager<>(conf);
+ }
+
+ /**
+ * Acquire lock on resource.
+ *
+ * For S3_Bucket, VOLUME, BUCKET type resource, same thread acquiring lock
+ * again is allowed.
+ *
+ * For USER, PREFIX, S3_SECRET type resource, same thread acquiring lock
+ * again is not allowed.
+ *
+ * Special Note for UserLock: Single thread can acquire single user lock/
+ * multi user lock. But not both at the same time.
+ * @param resourceName - Resource name on which user want to acquire lock.
+ * @param resource - Type of the resource.
+ */
+ public void acquireLock(String resourceName, Resource resource) {
+if (!resource.canLock(lockSet.get())) {
+ String errorMessage = getErrorMessage(resource);
+ LOG.error(errorMessage);
+ throw new RuntimeException(errorMessage);
+} else {
+ manager.lock(resourceName);
+ lockSet.set(resource.setLock(lockSet.get()));
+}
+ }
+
+ private String getErrorMessage(Resource resource) {
+return "Thread '" + Thread.currentThread().getName() + "' cannot " +
+"acquire " + resource.name + " lock while holding " +
+getCurrentLocks().toString() + " lock(s).";
+
+ }
+
+ private List getCurrentLocks() {
+List currentLocks = new ArrayList<>();
+int i=0;
+short lockSetVal = lockSet.get();
+for (Resource value : Resource.values()) {
+ if (value.isLevelLocked(lockSetVal)) {
+currentLocks.add(value.getName());
+ }
+}
+return currentLocks;
+ }
+
+ /**
+ * Acquire lock on multiple users.
+ * @param oldUserResource
+ * @param newUserResource
+ */
+ public void acquireMultiUserLock(String oldUserResource,
+ String newUserResource) {
+Resource resource = Resource.USER;
+if (!resource.canLock(lockSet.get())) {
+ String errorMessage = getErrorMessage(resource);
+ LOG.error(errorMessage);
+ throw new RuntimeException(errorMessage);
+} else {
+ // When
[jira] [Updated] (HDDS-1617) Restructure the code layout for Ozone Manager
[ https://issues.apache.org/jira/browse/HDDS-1617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDDS-1617: --- Status: Patch Available (was: Open) > Restructure the code layout for Ozone Manager > - > > Key: HDDS-1617 > URL: https://issues.apache.org/jira/browse/HDDS-1617 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Manager >Reporter: Anu Engineer >Assignee: Anu Engineer >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > The Ozone Manager has a flat structure that deals with lot of specific > functions. This Jira proposes to refactor ozone managers code base and move > function specific packages. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14598) Findbugs warning caused by HDFS-12487
[
https://issues.apache.org/jira/browse/HDFS-14598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871986#comment-16871986
]
Wei-Chiu Chuang commented on HDFS-14598:
Cherrypick the commits into branch-3.2 branch-3.1. Thanks [~anu]!
> Findbugs warning caused by HDFS-12487
> -
>
> Key: HDFS-14598
> URL: https://issues.apache.org/jira/browse/HDFS-14598
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: diskbalancer
>Reporter: Wei-Chiu Chuang
>Assignee: He Xiaoqiao
>Priority: Minor
> Labels: newbie
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14598.001.patch
>
>
> https://builds.apache.org/job/PreCommit-HDFS-Build/27038/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
> {noformat}
> Redundant nullcheck of block, which is known to be non-null in
> org.apache.hadoop.hdfs.server.datanode.DiskBalancer$DiskBalancerMover.getBlockToCopy(FsVolumeSpi$BlockIterator,
> DiskBalancerWorkItem)
> Bug type RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE (click for details)
> In class org.apache.hadoop.hdfs.server.datanode.DiskBalancer$DiskBalancerMover
> In method
> org.apache.hadoop.hdfs.server.datanode.DiskBalancer$DiskBalancerMover.getBlockToCopy(FsVolumeSpi$BlockIterator,
> DiskBalancerWorkItem)
> Value loaded from block
> Return value of
> org.apache.hadoop.hdfs.server.datanode.fsdataset.FsVolumeSpi$BlockIterator.nextBlock()
> of type org.apache.hadoop.hdfs.protocol.ExtendedBlock
> Redundant null check at DiskBalancer.java:[line 912]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14598) Findbugs warning caused by HDFS-12487
[
https://issues.apache.org/jira/browse/HDFS-14598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-14598:
---
Fix Version/s: 3.1.3
3.2.1
> Findbugs warning caused by HDFS-12487
> -
>
> Key: HDFS-14598
> URL: https://issues.apache.org/jira/browse/HDFS-14598
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: diskbalancer
>Reporter: Wei-Chiu Chuang
>Assignee: He Xiaoqiao
>Priority: Minor
> Labels: newbie
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14598.001.patch
>
>
> https://builds.apache.org/job/PreCommit-HDFS-Build/27038/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
> {noformat}
> Redundant nullcheck of block, which is known to be non-null in
> org.apache.hadoop.hdfs.server.datanode.DiskBalancer$DiskBalancerMover.getBlockToCopy(FsVolumeSpi$BlockIterator,
> DiskBalancerWorkItem)
> Bug type RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE (click for details)
> In class org.apache.hadoop.hdfs.server.datanode.DiskBalancer$DiskBalancerMover
> In method
> org.apache.hadoop.hdfs.server.datanode.DiskBalancer$DiskBalancerMover.getBlockToCopy(FsVolumeSpi$BlockIterator,
> DiskBalancerWorkItem)
> Value loaded from block
> Return value of
> org.apache.hadoop.hdfs.server.datanode.fsdataset.FsVolumeSpi$BlockIterator.nextBlock()
> of type org.apache.hadoop.hdfs.protocol.ExtendedBlock
> Redundant null check at DiskBalancer.java:[line 912]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1723) Create new OzoneManagerLock class
[
https://issues.apache.org/jira/browse/HDDS-1723?focusedWorklogId=266316=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266316
]
ASF GitHub Bot logged work on HDDS-1723:
Author: ASF GitHub Bot
Created on: 25/Jun/19 04:27
Start Date: 25/Jun/19 04:27
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #1006:
HDDS-1723. Create new OzoneManagerLock class.
URL: https://github.com/apache/hadoop/pull/1006#discussion_r297000862
##
File path:
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/OzoneManagerLock.java
##
@@ -0,0 +1,336 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.lock;
+
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ozone.lock.LockManager;
+
+/**
+ * Provides different locks to handle concurrency in OzoneMaster.
+ * We also maintain lock hierarchy, based on the weight.
+ *
+ *
+ *
+ *
+ * WEIGHT LOCK
+ *
+ *
+ * 0 S3 Bucket Lock
+ *
+ *
+ * 1 Volume Lock
+ *
+ *
+ * 2 Bucket Lock
+ *
+ *
+ * 3 User Lock
+ *
+ *
+ * 4 S3 Secret Lock
+ *
+ *
+ * 5 Prefix Lock
+ *
+ *
+ *
+ * One cannot obtain a lower weight lock while holding a lock with higher
+ * weight. The other way around is possible.
+ *
+ *
+ * For example:
+ *
+ * {@literal ->} acquire volume lock (will work)
+ * {@literal +->} acquire bucket lock (will work)
+ * {@literal +-->} acquire s3 bucket lock (will throw Exception)
+ *
+ *
+ */
+
+public class OzoneManagerLock {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerLock.class);
+
+ private final LockManager manager;
+ private final ThreadLocal lockSet = ThreadLocal.withInitial(
+ () -> Short.valueOf((short)0));
+
+
+ /**
+ * Creates new OzoneManagerLock instance.
+ * @param conf Configuration object
+ */
+ public OzoneManagerLock(Configuration conf) {
+manager = new LockManager<>(conf);
+ }
+
+ /**
+ * Acquire lock on resource.
+ *
+ * For S3_Bucket, VOLUME, BUCKET type resource, same thread acquiring lock
+ * again is allowed.
+ *
+ * For USER, PREFIX, S3_SECRET type resource, same thread acquiring lock
+ * again is not allowed.
+ *
+ * Special Note for UserLock: Single thread can acquire single user lock/
+ * multi user lock. But not both at the same time.
+ * @param resourceName - Resource name on which user want to acquire lock.
+ * @param resource - Type of the resource.
+ */
+ public void acquireLock(String resourceName, Resource resource) {
+if (!resource.canLock(lockSet.get())) {
+ String errorMessage = getErrorMessage(resource);
+ LOG.error(errorMessage);
+ throw new RuntimeException(errorMessage);
+} else {
+ manager.lock(resourceName);
+ lockSet.set(resource.setLock(lockSet.get()));
+}
+ }
+
+ private String getErrorMessage(Resource resource) {
+return "Thread '" + Thread.currentThread().getName() + "' cannot " +
+"acquire " + resource.name + " lock while holding " +
+getCurrentLocks().toString() + " lock(s).";
+
+ }
+
+ private List getCurrentLocks() {
+List currentLocks = new ArrayList<>();
+int i=0;
+short lockSetVal = lockSet.get();
+for (Resource value : Resource.values()) {
+ if (value.isLevelLocked(lockSetVal)) {
+currentLocks.add(value.getName());
+ }
+}
+return currentLocks;
+ }
+
+ /**
+ * Acquire lock on multiple users.
+ * @param oldUserResource
+ * @param newUserResource
+ */
+ public void acquireMultiUserLock(String oldUserResource,
+ String newUserResource) {
+Resource resource = Resource.USER;
+if (!resource.canLock(lockSet.get())) {
+ String errorMessage = getErrorMessage(resource);
+ LOG.error(errorMessage);
+ throw new RuntimeException(errorMessage);
+} else {
+ // When
[jira] [Work logged] (HDDS-1723) Create new OzoneManagerLock class
[
https://issues.apache.org/jira/browse/HDDS-1723?focusedWorklogId=266315=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266315
]
ASF GitHub Bot logged work on HDDS-1723:
Author: ASF GitHub Bot
Created on: 25/Jun/19 04:24
Start Date: 25/Jun/19 04:24
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #1006:
HDDS-1723. Create new OzoneManagerLock class.
URL: https://github.com/apache/hadoop/pull/1006#discussion_r297000862
##
File path:
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/OzoneManagerLock.java
##
@@ -0,0 +1,336 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.lock;
+
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ozone.lock.LockManager;
+
+/**
+ * Provides different locks to handle concurrency in OzoneMaster.
+ * We also maintain lock hierarchy, based on the weight.
+ *
+ *
+ *
+ *
+ * WEIGHT LOCK
+ *
+ *
+ * 0 S3 Bucket Lock
+ *
+ *
+ * 1 Volume Lock
+ *
+ *
+ * 2 Bucket Lock
+ *
+ *
+ * 3 User Lock
+ *
+ *
+ * 4 S3 Secret Lock
+ *
+ *
+ * 5 Prefix Lock
+ *
+ *
+ *
+ * One cannot obtain a lower weight lock while holding a lock with higher
+ * weight. The other way around is possible.
+ *
+ *
+ * For example:
+ *
+ * {@literal ->} acquire volume lock (will work)
+ * {@literal +->} acquire bucket lock (will work)
+ * {@literal +-->} acquire s3 bucket lock (will throw Exception)
+ *
+ *
+ */
+
+public class OzoneManagerLock {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerLock.class);
+
+ private final LockManager manager;
+ private final ThreadLocal lockSet = ThreadLocal.withInitial(
+ () -> Short.valueOf((short)0));
+
+
+ /**
+ * Creates new OzoneManagerLock instance.
+ * @param conf Configuration object
+ */
+ public OzoneManagerLock(Configuration conf) {
+manager = new LockManager<>(conf);
+ }
+
+ /**
+ * Acquire lock on resource.
+ *
+ * For S3_Bucket, VOLUME, BUCKET type resource, same thread acquiring lock
+ * again is allowed.
+ *
+ * For USER, PREFIX, S3_SECRET type resource, same thread acquiring lock
+ * again is not allowed.
+ *
+ * Special Note for UserLock: Single thread can acquire single user lock/
+ * multi user lock. But not both at the same time.
+ * @param resourceName - Resource name on which user want to acquire lock.
+ * @param resource - Type of the resource.
+ */
+ public void acquireLock(String resourceName, Resource resource) {
+if (!resource.canLock(lockSet.get())) {
+ String errorMessage = getErrorMessage(resource);
+ LOG.error(errorMessage);
+ throw new RuntimeException(errorMessage);
+} else {
+ manager.lock(resourceName);
+ lockSet.set(resource.setLock(lockSet.get()));
+}
+ }
+
+ private String getErrorMessage(Resource resource) {
+return "Thread '" + Thread.currentThread().getName() + "' cannot " +
+"acquire " + resource.name + " lock while holding " +
+getCurrentLocks().toString() + " lock(s).";
+
+ }
+
+ private List getCurrentLocks() {
+List currentLocks = new ArrayList<>();
+int i=0;
+short lockSetVal = lockSet.get();
+for (Resource value : Resource.values()) {
+ if (value.isLevelLocked(lockSetVal)) {
+currentLocks.add(value.getName());
+ }
+}
+return currentLocks;
+ }
+
+ /**
+ * Acquire lock on multiple users.
+ * @param oldUserResource
+ * @param newUserResource
+ */
+ public void acquireMultiUserLock(String oldUserResource,
+ String newUserResource) {
+Resource resource = Resource.USER;
+if (!resource.canLock(lockSet.get())) {
+ String errorMessage = getErrorMessage(resource);
+ LOG.error(errorMessage);
+ throw new RuntimeException(errorMessage);
+} else {
+ // When
[jira] [Work logged] (HDDS-1723) Create new OzoneManagerLock class
[
https://issues.apache.org/jira/browse/HDDS-1723?focusedWorklogId=266313=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266313
]
ASF GitHub Bot logged work on HDDS-1723:
Author: ASF GitHub Bot
Created on: 25/Jun/19 04:22
Start Date: 25/Jun/19 04:22
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #1006:
HDDS-1723. Create new OzoneManagerLock class.
URL: https://github.com/apache/hadoop/pull/1006#discussion_r297000650
##
File path:
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/OzoneManagerLock.java
##
@@ -0,0 +1,336 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.lock;
+
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ozone.lock.LockManager;
+
+/**
+ * Provides different locks to handle concurrency in OzoneMaster.
+ * We also maintain lock hierarchy, based on the weight.
+ *
+ *
+ *
+ *
+ * WEIGHT LOCK
+ *
+ *
+ * 0 S3 Bucket Lock
+ *
+ *
+ * 1 Volume Lock
+ *
+ *
+ * 2 Bucket Lock
+ *
+ *
+ * 3 User Lock
+ *
+ *
+ * 4 S3 Secret Lock
+ *
+ *
+ * 5 Prefix Lock
+ *
+ *
+ *
+ * One cannot obtain a lower weight lock while holding a lock with higher
+ * weight. The other way around is possible.
+ *
+ *
+ * For example:
+ *
+ * {@literal ->} acquire volume lock (will work)
+ * {@literal +->} acquire bucket lock (will work)
+ * {@literal +-->} acquire s3 bucket lock (will throw Exception)
+ *
+ *
+ */
+
+public class OzoneManagerLock {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerLock.class);
+
+ private final LockManager manager;
+ private final ThreadLocal lockSet = ThreadLocal.withInitial(
+ () -> Short.valueOf((short)0));
+
+
+ /**
+ * Creates new OzoneManagerLock instance.
+ * @param conf Configuration object
+ */
+ public OzoneManagerLock(Configuration conf) {
+manager = new LockManager<>(conf);
+ }
+
+ /**
+ * Acquire lock on resource.
+ *
+ * For S3_Bucket, VOLUME, BUCKET type resource, same thread acquiring lock
+ * again is allowed.
+ *
+ * For USER, PREFIX, S3_SECRET type resource, same thread acquiring lock
+ * again is not allowed.
+ *
+ * Special Note for UserLock: Single thread can acquire single user lock/
+ * multi user lock. But not both at the same time.
+ * @param resourceName - Resource name on which user want to acquire lock.
+ * @param resource - Type of the resource.
+ */
+ public void acquireLock(String resourceName, Resource resource) {
+if (!resource.canLock(lockSet.get())) {
+ String errorMessage = getErrorMessage(resource);
+ LOG.error(errorMessage);
+ throw new RuntimeException(errorMessage);
+} else {
+ manager.lock(resourceName);
+ lockSet.set(resource.setLock(lockSet.get()));
+}
+ }
+
+ private String getErrorMessage(Resource resource) {
+return "Thread '" + Thread.currentThread().getName() + "' cannot " +
+"acquire " + resource.name + " lock while holding " +
+getCurrentLocks().toString() + " lock(s).";
+
+ }
+
+ private List getCurrentLocks() {
+List currentLocks = new ArrayList<>();
+int i=0;
+short lockSetVal = lockSet.get();
+for (Resource value : Resource.values()) {
+ if (value.isLevelLocked(lockSetVal)) {
+currentLocks.add(value.getName());
+ }
+}
+return currentLocks;
+ }
+
+ /**
+ * Acquire lock on multiple users.
+ * @param oldUserResource
+ * @param newUserResource
+ */
+ public void acquireMultiUserLock(String oldUserResource,
+ String newUserResource) {
+Resource resource = Resource.USER;
+if (!resource.canLock(lockSet.get())) {
+ String errorMessage = getErrorMessage(resource);
+ LOG.error(errorMessage);
+ throw new RuntimeException(errorMessage);
+} else {
+ // When
[jira] [Updated] (HDFS-14247) Repeat adding node description into network topology
[ https://issues.apache.org/jira/browse/HDFS-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-14247: --- Fix Version/s: 3.1.3 2.9.3 3.2.1 2.8.6 3.0.4 2.10.0 > Repeat adding node description into network topology > > > Key: HDFS-14247 > URL: https://issues.apache.org/jira/browse/HDFS-14247 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.3.0 >Reporter: HuangTao >Assignee: HuangTao >Priority: Minor > Fix For: 2.10.0, 3.0.4, 3.3.0, 2.8.6, 3.2.1, 2.9.3, 3.1.3 > > Attachments: HDFS-14247.001.patch > > > I find there is a duplicate code to add nodeDescr to networktopology in the > DatanodeManager.java#registerDatanode. > It firstly call networktopology.add(nodeDescr), and then call > addDatanode(nodeDescr) to add nodeDescr again -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13694) Making md5 computing being in parallel with image loading
[
https://issues.apache.org/jira/browse/HDFS-13694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871981#comment-16871981
]
Hadoop QA commented on HDFS-13694:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 31s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
56s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
54s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant
findbugs warnings. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 38s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 28 unchanged - 0 fixed = 29 total (was 28) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 45s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
6s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 1
unchanged - 0 fixed = 2 total (was 1) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 86m 39s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
0s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}149m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Should
org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader$DigestThread
be a _static_ inner class? At FSImageFormatProtobuf.java:inner class? At
FSImageFormatProtobuf.java:[lines 176-203] |
| Failed junit tests | hadoop.hdfs.server.diskbalancer.TestDiskBalancer |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/1010 |
| JIRA Issue |
[jira] [Updated] (HDFS-12172) Reduce EZ lookup overhead
[ https://issues.apache.org/jira/browse/HDFS-12172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-12172: --- Attachment: HDFS-12172.004.patch > Reduce EZ lookup overhead > - > > Key: HDFS-12172 > URL: https://issues.apache.org/jira/browse/HDFS-12172 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Major > Attachments: HDFS-12172.002.patch, HDFS-12172.003.patch, > HDFS-12172.004.patch, HDFS-12172.01.patch, HDFS-12172.patch > > > A number of inefficiencies exist in EZ lookups. These are amplified by > frequent operations like list status. Once one encryption zone exists, all > operations take the performance penalty. > Ex. Operations should not perform redundant lookups. EZ path reconstruction > should be lazy since it's not required in the common case. Renames do not > need to reallocate new IIPs to check parent dirs for EZ. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13470) RBF: Add Browse the Filesystem button to the UI
[
https://issues.apache.org/jira/browse/HDFS-13470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871965#comment-16871965
]
Hadoop QA commented on HDFS-13470:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
26m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 35 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 13s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-13470 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12919518/HDFS-13470.000.patch |
| Optional Tests | dupname asflicense shadedclient |
| uname | Linux 0bb5e3819c2f 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 041e7a7 |
| maven | version: Apache Maven 3.3.9 |
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/27065/artifact/out/whitespace-tabs.txt
|
| Max. process+thread count | 449 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/27065/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF: Add Browse the Filesystem button to the UI
> ---
>
> Key: HDFS-13470
> URL: https://issues.apache.org/jira/browse/HDFS-13470
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: HDFS-13470.000.patch
>
>
> After HDFS-12512 added WebHDFS, we can add the support to browse the
> filesystem to the UI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13694) Making md5 computing being in parallel with image loading
[ https://issues.apache.org/jira/browse/HDFS-13694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-13694: --- Attachment: HDFS-13694-004.patch > Making md5 computing being in parallel with image loading > - > > Key: HDFS-13694 > URL: https://issues.apache.org/jira/browse/HDFS-13694 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: zhouyingchao >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-13694-001.patch, HDFS-13694-002.patch, > HDFS-13694-003.patch, HDFS-13694-004.patch > > > During namenode image loading, it firstly compute the md5 and then load the > image. Actually these two steps can be in parallel. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13495) RBF: Support Router Admin REST API
[ https://issues.apache.org/jira/browse/HDFS-13495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871953#comment-16871953 ] Mohammad Arshad commented on HDFS-13495: I moved to some other work. If any body interested can take this JIRA. > RBF: Support Router Admin REST API > -- > > Key: HDFS-13495 > URL: https://issues.apache.org/jira/browse/HDFS-13495 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Mohammad Arshad >Assignee: Mohammad Arshad >Priority: Major > Labels: RBF > > This JIRA intends to add REST API support for all admin commands. Router > Admin REST APIs can be useful in managing the Routers from a central > management layer tool. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-13495) RBF: Support Router Admin REST API
[ https://issues.apache.org/jira/browse/HDFS-13495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Arshad reassigned HDFS-13495: -- Assignee: (was: Mohammad Arshad) > RBF: Support Router Admin REST API > -- > > Key: HDFS-13495 > URL: https://issues.apache.org/jira/browse/HDFS-13495 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Mohammad Arshad >Priority: Major > Labels: RBF > > This JIRA intends to add REST API support for all admin commands. Router > Admin REST APIs can be useful in managing the Routers from a central > management layer tool. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13245) RBF: State store DBMS implementation
[
https://issues.apache.org/jira/browse/HDFS-13245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871946#comment-16871946
]
Hadoop QA commented on HDFS-13245:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 13s{color}
| {color:red} HDFS-13245 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-13245 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12923649/HDFS-13245.012.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/27067/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF: State store DBMS implementation
>
>
> Key: HDFS-13245
> URL: https://issues.apache.org/jira/browse/HDFS-13245
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: maobaolong
>Assignee: Yiran Wu
>Priority: Major
> Attachments: HDFS-13245.001.patch, HDFS-13245.002.patch,
> HDFS-13245.003.patch, HDFS-13245.004.patch, HDFS-13245.005.patch,
> HDFS-13245.006.patch, HDFS-13245.007.patch, HDFS-13245.008.patch,
> HDFS-13245.009.patch, HDFS-13245.010.patch, HDFS-13245.011.patch,
> HDFS-13245.012.patch
>
>
> Add a DBMS implementation for the State Store.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13507) RBF: Remove update functionality from routeradmin's add cmd
[
https://issues.apache.org/jira/browse/HDFS-13507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871947#comment-16871947
]
Hadoop QA commented on HDFS-13507:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HDFS-13507 does not apply to HDFS-13891. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-13507 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12954870/HDFS-13507-HDFS-13891.004.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/27066/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF: Remove update functionality from routeradmin's add cmd
> ---
>
> Key: HDFS-13507
> URL: https://issues.apache.org/jira/browse/HDFS-13507
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Gang Li
>Priority: Minor
> Labels: incompatible
> Attachments: HDFS-13507-HDFS-13891.003.patch,
> HDFS-13507-HDFS-13891.004.patch, HDFS-13507.000.patch, HDFS-13507.001.patch,
> HDFS-13507.002.patch
>
>
> Follow up the discussion in HDFS-13326. We should remove the "update"
> functionality from routeradmin's add cmd, to make it consistent with RPC
> calls.
> Note that: this is an incompatible change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14090) RBF: Improved isolation for downstream name nodes.
[
https://issues.apache.org/jira/browse/HDFS-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871940#comment-16871940
]
Yiqun Lin commented on HDFS-14090:
--
Hi [~crh], some further review comments for the latest patch:
* Following log is not correct described
{noformat}
+ private void releasePermit(String nsId, UserGroupInformation ugi, Method m) {
+if (fairnessManager != null) {
+ fairnessManager.releasePermission(nsId);
+ LOG.trace("Permission denied for ugi: {} for method: {}",
+ ugi, m.getName());
+}
+ }
{noformat}
* Can we rename the unit test method
{{TestRouterFairnessManager#testHandlerAllocation1, testHandlerAllocation2..}}
to the more clear names? Actually when I first looked the name of this, I
didn't know what's the error case of this :D.
* I don't think we have covered all the handler assignments cases. Can we add
the new unit test for dedicated namespace handler assignment and then do the
handler count verification?
> RBF: Improved isolation for downstream name nodes.
> --
>
> Key: HDFS-14090
> URL: https://issues.apache.org/jira/browse/HDFS-14090
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: CR Hota
>Priority: Major
> Attachments: HDFS-14090-HDFS-13891.001.patch,
> HDFS-14090-HDFS-13891.002.patch, HDFS-14090-HDFS-13891.003.patch,
> HDFS-14090-HDFS-13891.004.patch, HDFS-14090-HDFS-13891.005.patch, RBF_
> Isolation design.pdf
>
>
> Router is a gateway to underlying name nodes. Gateway architectures, should
> help minimize impact of clients connecting to healthy clusters vs unhealthy
> clusters.
> For example - If there are 2 name nodes downstream, and one of them is
> heavily loaded with calls spiking rpc queue times, due to back pressure the
> same with start reflecting on the router. As a result of this, clients
> connecting to healthy/faster name nodes will also slow down as same rpc queue
> is maintained for all calls at the router layer. Essentially the same IPC
> thread pool is used by router to connect to all name nodes.
> Currently router uses one single rpc queue for all calls. Lets discuss how we
> can change the architecture and add some throttling logic for
> unhealthy/slow/overloaded name nodes.
> One way could be to read from current call queue, immediately identify
> downstream name node and maintain a separate queue for each underlying name
> node. Another simpler way is to maintain some sort of rate limiter configured
> for each name node and let routers drop/reject/send error requests after
> certain threshold.
> This won’t be a simple change as router’s ‘Server’ layer would need redesign
> and implementation. Currently this layer is the same as name node.
> Opening this ticket to discuss, design and implement this feature.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1723) Create new OzoneManagerLock class
[
https://issues.apache.org/jira/browse/HDDS-1723?focusedWorklogId=266294=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266294
]
ASF GitHub Bot logged work on HDDS-1723:
Author: ASF GitHub Bot
Created on: 25/Jun/19 02:45
Start Date: 25/Jun/19 02:45
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #1006: HDDS-1723.
Create new OzoneManagerLock class.
URL: https://github.com/apache/hadoop/pull/1006#discussion_r296983712
##
File path:
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/OzoneManagerLock.java
##
@@ -0,0 +1,336 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.lock;
+
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ozone.lock.LockManager;
+
+/**
+ * Provides different locks to handle concurrency in OzoneMaster.
+ * We also maintain lock hierarchy, based on the weight.
+ *
+ *
+ *
+ *
+ * WEIGHT LOCK
+ *
+ *
+ * 0 S3 Bucket Lock
+ *
+ *
+ * 1 Volume Lock
+ *
+ *
+ * 2 Bucket Lock
+ *
+ *
+ * 3 User Lock
+ *
+ *
+ * 4 S3 Secret Lock
+ *
+ *
+ * 5 Prefix Lock
+ *
+ *
+ *
+ * One cannot obtain a lower weight lock while holding a lock with higher
+ * weight. The other way around is possible.
+ *
+ *
+ * For example:
+ *
+ * {@literal ->} acquire volume lock (will work)
+ * {@literal +->} acquire bucket lock (will work)
+ * {@literal +-->} acquire s3 bucket lock (will throw Exception)
+ *
+ *
+ */
+
+public class OzoneManagerLock {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerLock.class);
+
+ private final LockManager manager;
+ private final ThreadLocal lockSet = ThreadLocal.withInitial(
+ () -> Short.valueOf((short)0));
+
+
+ /**
+ * Creates new OzoneManagerLock instance.
+ * @param conf Configuration object
+ */
+ public OzoneManagerLock(Configuration conf) {
+manager = new LockManager<>(conf);
+ }
+
+ /**
+ * Acquire lock on resource.
+ *
+ * For S3_Bucket, VOLUME, BUCKET type resource, same thread acquiring lock
+ * again is allowed.
+ *
+ * For USER, PREFIX, S3_SECRET type resource, same thread acquiring lock
+ * again is not allowed.
+ *
+ * Special Note for UserLock: Single thread can acquire single user lock/
+ * multi user lock. But not both at the same time.
+ * @param resourceName - Resource name on which user want to acquire lock.
+ * @param resource - Type of the resource.
+ */
+ public void acquireLock(String resourceName, Resource resource) {
+if (!resource.canLock(lockSet.get())) {
+ String errorMessage = getErrorMessage(resource);
+ LOG.error(errorMessage);
+ throw new RuntimeException(errorMessage);
+} else {
+ manager.lock(resourceName);
+ lockSet.set(resource.setLock(lockSet.get()));
+}
+ }
+
+ private String getErrorMessage(Resource resource) {
+return "Thread '" + Thread.currentThread().getName() + "' cannot " +
+"acquire " + resource.name + " lock while holding " +
+getCurrentLocks().toString() + " lock(s).";
+
+ }
+
+ private List getCurrentLocks() {
+List currentLocks = new ArrayList<>();
+int i=0;
+short lockSetVal = lockSet.get();
+for (Resource value : Resource.values()) {
+ if (value.isLevelLocked(lockSetVal)) {
+currentLocks.add(value.getName());
+ }
+}
+return currentLocks;
+ }
+
+ /**
+ * Acquire lock on multiple users.
+ * @param oldUserResource
+ * @param newUserResource
+ */
+ public void acquireMultiUserLock(String oldUserResource,
+ String newUserResource) {
+Resource resource = Resource.USER;
+if (!resource.canLock(lockSet.get())) {
+ String errorMessage = getErrorMessage(resource);
+ LOG.error(errorMessage);
+ throw new RuntimeException(errorMessage);
+} else {
+ // When acquiring
[jira] [Work logged] (HDDS-1723) Create new OzoneManagerLock class
[
https://issues.apache.org/jira/browse/HDDS-1723?focusedWorklogId=266295=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266295
]
ASF GitHub Bot logged work on HDDS-1723:
Author: ASF GitHub Bot
Created on: 25/Jun/19 02:45
Start Date: 25/Jun/19 02:45
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #1006: HDDS-1723.
Create new OzoneManagerLock class.
URL: https://github.com/apache/hadoop/pull/1006#discussion_r296984019
##
File path:
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/OzoneManagerLock.java
##
@@ -0,0 +1,336 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.lock;
+
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ozone.lock.LockManager;
+
+/**
+ * Provides different locks to handle concurrency in OzoneMaster.
+ * We also maintain lock hierarchy, based on the weight.
+ *
+ *
+ *
+ *
+ * WEIGHT LOCK
+ *
+ *
+ * 0 S3 Bucket Lock
+ *
+ *
+ * 1 Volume Lock
+ *
+ *
+ * 2 Bucket Lock
+ *
+ *
+ * 3 User Lock
+ *
+ *
+ * 4 S3 Secret Lock
+ *
+ *
+ * 5 Prefix Lock
+ *
+ *
+ *
+ * One cannot obtain a lower weight lock while holding a lock with higher
+ * weight. The other way around is possible.
+ *
+ *
+ * For example:
+ *
+ * {@literal ->} acquire volume lock (will work)
+ * {@literal +->} acquire bucket lock (will work)
+ * {@literal +-->} acquire s3 bucket lock (will throw Exception)
+ *
+ *
+ */
+
+public class OzoneManagerLock {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerLock.class);
+
+ private final LockManager manager;
+ private final ThreadLocal lockSet = ThreadLocal.withInitial(
+ () -> Short.valueOf((short)0));
+
+
+ /**
+ * Creates new OzoneManagerLock instance.
+ * @param conf Configuration object
+ */
+ public OzoneManagerLock(Configuration conf) {
+manager = new LockManager<>(conf);
+ }
+
+ /**
+ * Acquire lock on resource.
+ *
+ * For S3_Bucket, VOLUME, BUCKET type resource, same thread acquiring lock
+ * again is allowed.
+ *
+ * For USER, PREFIX, S3_SECRET type resource, same thread acquiring lock
+ * again is not allowed.
+ *
+ * Special Note for UserLock: Single thread can acquire single user lock/
+ * multi user lock. But not both at the same time.
+ * @param resourceName - Resource name on which user want to acquire lock.
+ * @param resource - Type of the resource.
+ */
+ public void acquireLock(String resourceName, Resource resource) {
+if (!resource.canLock(lockSet.get())) {
+ String errorMessage = getErrorMessage(resource);
+ LOG.error(errorMessage);
+ throw new RuntimeException(errorMessage);
+} else {
+ manager.lock(resourceName);
+ lockSet.set(resource.setLock(lockSet.get()));
+}
+ }
+
+ private String getErrorMessage(Resource resource) {
+return "Thread '" + Thread.currentThread().getName() + "' cannot " +
+"acquire " + resource.name + " lock while holding " +
+getCurrentLocks().toString() + " lock(s).";
+
+ }
+
+ private List getCurrentLocks() {
+List currentLocks = new ArrayList<>();
+int i=0;
+short lockSetVal = lockSet.get();
+for (Resource value : Resource.values()) {
+ if (value.isLevelLocked(lockSetVal)) {
+currentLocks.add(value.getName());
+ }
+}
+return currentLocks;
+ }
+
+ /**
+ * Acquire lock on multiple users.
+ * @param oldUserResource
+ * @param newUserResource
+ */
+ public void acquireMultiUserLock(String oldUserResource,
+ String newUserResource) {
+Resource resource = Resource.USER;
+if (!resource.canLock(lockSet.get())) {
+ String errorMessage = getErrorMessage(resource);
+ LOG.error(errorMessage);
+ throw new RuntimeException(errorMessage);
+} else {
+ // When acquiring
[jira] [Work logged] (HDDS-1723) Create new OzoneManagerLock class
[
https://issues.apache.org/jira/browse/HDDS-1723?focusedWorklogId=266298=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266298
]
ASF GitHub Bot logged work on HDDS-1723:
Author: ASF GitHub Bot
Created on: 25/Jun/19 02:45
Start Date: 25/Jun/19 02:45
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #1006: HDDS-1723.
Create new OzoneManagerLock class.
URL: https://github.com/apache/hadoop/pull/1006#discussion_r296986201
##
File path:
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/OzoneManagerLock.java
##
@@ -0,0 +1,336 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.lock;
+
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ozone.lock.LockManager;
+
+/**
+ * Provides different locks to handle concurrency in OzoneMaster.
+ * We also maintain lock hierarchy, based on the weight.
+ *
+ *
+ *
+ *
+ * WEIGHT LOCK
+ *
+ *
+ * 0 S3 Bucket Lock
+ *
+ *
+ * 1 Volume Lock
+ *
+ *
+ * 2 Bucket Lock
+ *
+ *
+ * 3 User Lock
+ *
+ *
+ * 4 S3 Secret Lock
+ *
+ *
+ * 5 Prefix Lock
+ *
+ *
+ *
+ * One cannot obtain a lower weight lock while holding a lock with higher
+ * weight. The other way around is possible.
+ *
+ *
+ * For example:
+ *
+ * {@literal ->} acquire volume lock (will work)
+ * {@literal +->} acquire bucket lock (will work)
+ * {@literal +-->} acquire s3 bucket lock (will throw Exception)
+ *
+ *
+ */
+
+public class OzoneManagerLock {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerLock.class);
+
+ private final LockManager manager;
+ private final ThreadLocal lockSet = ThreadLocal.withInitial(
+ () -> Short.valueOf((short)0));
+
+
+ /**
+ * Creates new OzoneManagerLock instance.
+ * @param conf Configuration object
+ */
+ public OzoneManagerLock(Configuration conf) {
+manager = new LockManager<>(conf);
+ }
+
+ /**
+ * Acquire lock on resource.
+ *
+ * For S3_Bucket, VOLUME, BUCKET type resource, same thread acquiring lock
+ * again is allowed.
+ *
+ * For USER, PREFIX, S3_SECRET type resource, same thread acquiring lock
+ * again is not allowed.
+ *
+ * Special Note for UserLock: Single thread can acquire single user lock/
+ * multi user lock. But not both at the same time.
+ * @param resourceName - Resource name on which user want to acquire lock.
+ * @param resource - Type of the resource.
+ */
+ public void acquireLock(String resourceName, Resource resource) {
+if (!resource.canLock(lockSet.get())) {
+ String errorMessage = getErrorMessage(resource);
+ LOG.error(errorMessage);
+ throw new RuntimeException(errorMessage);
+} else {
+ manager.lock(resourceName);
+ lockSet.set(resource.setLock(lockSet.get()));
+}
+ }
+
+ private String getErrorMessage(Resource resource) {
+return "Thread '" + Thread.currentThread().getName() + "' cannot " +
+"acquire " + resource.name + " lock while holding " +
+getCurrentLocks().toString() + " lock(s).";
+
+ }
+
+ private List getCurrentLocks() {
+List currentLocks = new ArrayList<>();
+int i=0;
+short lockSetVal = lockSet.get();
+for (Resource value : Resource.values()) {
+ if (value.isLevelLocked(lockSetVal)) {
+currentLocks.add(value.getName());
+ }
+}
+return currentLocks;
+ }
+
+ /**
+ * Acquire lock on multiple users.
+ * @param oldUserResource
+ * @param newUserResource
+ */
+ public void acquireMultiUserLock(String oldUserResource,
+ String newUserResource) {
+Resource resource = Resource.USER;
+if (!resource.canLock(lockSet.get())) {
+ String errorMessage = getErrorMessage(resource);
+ LOG.error(errorMessage);
+ throw new RuntimeException(errorMessage);
+} else {
+ // When acquiring
[jira] [Work logged] (HDDS-1723) Create new OzoneManagerLock class
[
https://issues.apache.org/jira/browse/HDDS-1723?focusedWorklogId=266297=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266297
]
ASF GitHub Bot logged work on HDDS-1723:
Author: ASF GitHub Bot
Created on: 25/Jun/19 02:45
Start Date: 25/Jun/19 02:45
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #1006: HDDS-1723.
Create new OzoneManagerLock class.
URL: https://github.com/apache/hadoop/pull/1006#discussion_r296984611
##
File path:
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/OzoneManagerLock.java
##
@@ -0,0 +1,336 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.lock;
+
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ozone.lock.LockManager;
+
+/**
+ * Provides different locks to handle concurrency in OzoneMaster.
+ * We also maintain lock hierarchy, based on the weight.
+ *
+ *
+ *
+ *
+ * WEIGHT LOCK
+ *
+ *
+ * 0 S3 Bucket Lock
+ *
+ *
+ * 1 Volume Lock
+ *
+ *
+ * 2 Bucket Lock
+ *
+ *
+ * 3 User Lock
+ *
+ *
+ * 4 S3 Secret Lock
+ *
+ *
+ * 5 Prefix Lock
+ *
+ *
+ *
+ * One cannot obtain a lower weight lock while holding a lock with higher
+ * weight. The other way around is possible.
+ *
+ *
+ * For example:
+ *
+ * {@literal ->} acquire volume lock (will work)
+ * {@literal +->} acquire bucket lock (will work)
+ * {@literal +-->} acquire s3 bucket lock (will throw Exception)
+ *
+ *
+ */
+
+public class OzoneManagerLock {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerLock.class);
+
+ private final LockManager manager;
+ private final ThreadLocal lockSet = ThreadLocal.withInitial(
+ () -> Short.valueOf((short)0));
+
+
+ /**
+ * Creates new OzoneManagerLock instance.
+ * @param conf Configuration object
+ */
+ public OzoneManagerLock(Configuration conf) {
+manager = new LockManager<>(conf);
+ }
+
+ /**
+ * Acquire lock on resource.
+ *
+ * For S3_Bucket, VOLUME, BUCKET type resource, same thread acquiring lock
+ * again is allowed.
+ *
+ * For USER, PREFIX, S3_SECRET type resource, same thread acquiring lock
+ * again is not allowed.
+ *
+ * Special Note for UserLock: Single thread can acquire single user lock/
+ * multi user lock. But not both at the same time.
+ * @param resourceName - Resource name on which user want to acquire lock.
+ * @param resource - Type of the resource.
+ */
+ public void acquireLock(String resourceName, Resource resource) {
+if (!resource.canLock(lockSet.get())) {
+ String errorMessage = getErrorMessage(resource);
+ LOG.error(errorMessage);
+ throw new RuntimeException(errorMessage);
+} else {
+ manager.lock(resourceName);
+ lockSet.set(resource.setLock(lockSet.get()));
+}
+ }
+
+ private String getErrorMessage(Resource resource) {
+return "Thread '" + Thread.currentThread().getName() + "' cannot " +
+"acquire " + resource.name + " lock while holding " +
+getCurrentLocks().toString() + " lock(s).";
+
+ }
+
+ private List getCurrentLocks() {
+List currentLocks = new ArrayList<>();
+int i=0;
+short lockSetVal = lockSet.get();
+for (Resource value : Resource.values()) {
+ if (value.isLevelLocked(lockSetVal)) {
+currentLocks.add(value.getName());
+ }
+}
+return currentLocks;
+ }
+
+ /**
+ * Acquire lock on multiple users.
+ * @param oldUserResource
+ * @param newUserResource
+ */
+ public void acquireMultiUserLock(String oldUserResource,
+ String newUserResource) {
+Resource resource = Resource.USER;
+if (!resource.canLock(lockSet.get())) {
+ String errorMessage = getErrorMessage(resource);
+ LOG.error(errorMessage);
+ throw new RuntimeException(errorMessage);
+} else {
+ // When acquiring
[jira] [Work logged] (HDDS-1723) Create new OzoneManagerLock class
[
https://issues.apache.org/jira/browse/HDDS-1723?focusedWorklogId=266296=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266296
]
ASF GitHub Bot logged work on HDDS-1723:
Author: ASF GitHub Bot
Created on: 25/Jun/19 02:45
Start Date: 25/Jun/19 02:45
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #1006: HDDS-1723.
Create new OzoneManagerLock class.
URL: https://github.com/apache/hadoop/pull/1006#discussion_r296984587
##
File path:
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/OzoneManagerLock.java
##
@@ -0,0 +1,336 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.lock;
+
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ozone.lock.LockManager;
+
+/**
+ * Provides different locks to handle concurrency in OzoneMaster.
+ * We also maintain lock hierarchy, based on the weight.
+ *
+ *
+ *
+ *
+ * WEIGHT LOCK
+ *
+ *
+ * 0 S3 Bucket Lock
+ *
+ *
+ * 1 Volume Lock
+ *
+ *
+ * 2 Bucket Lock
+ *
+ *
+ * 3 User Lock
+ *
+ *
+ * 4 S3 Secret Lock
+ *
+ *
+ * 5 Prefix Lock
+ *
+ *
+ *
+ * One cannot obtain a lower weight lock while holding a lock with higher
+ * weight. The other way around is possible.
+ *
+ *
+ * For example:
+ *
+ * {@literal ->} acquire volume lock (will work)
+ * {@literal +->} acquire bucket lock (will work)
+ * {@literal +-->} acquire s3 bucket lock (will throw Exception)
+ *
+ *
+ */
+
+public class OzoneManagerLock {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerLock.class);
+
+ private final LockManager manager;
+ private final ThreadLocal lockSet = ThreadLocal.withInitial(
+ () -> Short.valueOf((short)0));
+
+
+ /**
+ * Creates new OzoneManagerLock instance.
+ * @param conf Configuration object
+ */
+ public OzoneManagerLock(Configuration conf) {
+manager = new LockManager<>(conf);
+ }
+
+ /**
+ * Acquire lock on resource.
+ *
+ * For S3_Bucket, VOLUME, BUCKET type resource, same thread acquiring lock
+ * again is allowed.
+ *
+ * For USER, PREFIX, S3_SECRET type resource, same thread acquiring lock
+ * again is not allowed.
+ *
+ * Special Note for UserLock: Single thread can acquire single user lock/
+ * multi user lock. But not both at the same time.
+ * @param resourceName - Resource name on which user want to acquire lock.
+ * @param resource - Type of the resource.
+ */
+ public void acquireLock(String resourceName, Resource resource) {
+if (!resource.canLock(lockSet.get())) {
+ String errorMessage = getErrorMessage(resource);
+ LOG.error(errorMessage);
+ throw new RuntimeException(errorMessage);
+} else {
+ manager.lock(resourceName);
+ lockSet.set(resource.setLock(lockSet.get()));
+}
+ }
+
+ private String getErrorMessage(Resource resource) {
+return "Thread '" + Thread.currentThread().getName() + "' cannot " +
+"acquire " + resource.name + " lock while holding " +
+getCurrentLocks().toString() + " lock(s).";
+
+ }
+
+ private List getCurrentLocks() {
+List currentLocks = new ArrayList<>();
+int i=0;
+short lockSetVal = lockSet.get();
+for (Resource value : Resource.values()) {
+ if (value.isLevelLocked(lockSetVal)) {
+currentLocks.add(value.getName());
+ }
+}
+return currentLocks;
+ }
+
+ /**
+ * Acquire lock on multiple users.
+ * @param oldUserResource
+ * @param newUserResource
+ */
+ public void acquireMultiUserLock(String oldUserResource,
+ String newUserResource) {
+Resource resource = Resource.USER;
+if (!resource.canLock(lockSet.get())) {
+ String errorMessage = getErrorMessage(resource);
+ LOG.error(errorMessage);
+ throw new RuntimeException(errorMessage);
+} else {
+ // When acquiring
[jira] [Updated] (HDFS-14044) RBF: NamenodeHeartbeat test cases around jmx parameters
[ https://issues.apache.org/jira/browse/HDFS-14044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-14044: Parent Issue: HDFS-14603 (was: HDFS-13891) > RBF: NamenodeHeartbeat test cases around jmx parameters > --- > > Key: HDFS-14044 > URL: https://issues.apache.org/jira/browse/HDFS-14044 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: CR Hota >Assignee: CR Hota >Priority: Major > > NamenodeHeartbeat service get downstream name node report via jmx. This part > of the code should have better tests. Tests around unavailability of certain > params, backyard compatibility of downstream namenode version vs router > version etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13989) RBF: Add FSCK to the Router
[ https://issues.apache.org/jira/browse/HDFS-13989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13989: Parent Issue: HDFS-14603 (was: HDFS-13891) > RBF: Add FSCK to the Router > --- > > Key: HDFS-13989 > URL: https://issues.apache.org/jira/browse/HDFS-13989 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13989.001.patch > > > The namenode supports FSCK. > The Router should be able to forward FSCK to the right Namenode and aggregate > the results. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13912) RBF: Add methods to RouterAdmin to set order, read only, and chown
[ https://issues.apache.org/jira/browse/HDFS-13912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13912: Parent Issue: HDFS-14603 (was: HDFS-13891) > RBF: Add methods to RouterAdmin to set order, read only, and chown > -- > > Key: HDFS-13912 > URL: https://issues.apache.org/jira/browse/HDFS-13912 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-13912-01.patch, HDFS-13912-02.patch > > > Presently there is methods for only quotas for an existing mount entries. > Similarly it can be added for the other remaining parameters those even are > dynamic and requires changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13577) RBF: Failed mount point operations, returns wrong exit code.
[
https://issues.apache.org/jira/browse/HDFS-13577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-13577:
Parent Issue: HDFS-14603 (was: HDFS-13891)
> RBF: Failed mount point operations, returns wrong exit code.
>
>
> Key: HDFS-13577
> URL: https://issues.apache.org/jira/browse/HDFS-13577
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Dibyendu Karmakar
>Priority: Major
> Labels: RBF
>
> If client is performed add mount point with some special character, mount
> point add is failed.
> And prints the message like
> {noformat}
> 18/05/17 09:58:34 DEBUG ipc.ProtobufRpcEngine: Call: addMountTableEntry took
> 19ms Cannot add mount point /testSpecialCharMountPointCreation/test/
> {noformat}
> In the above case it should return the exist code is non zero value.
> {code:java|title=RouterAdmin.java|borderStyle=solid}
> Exception debugException = null;
> exitCode = 0;
> try {
> if ("-add".equals(cmd)) {
> if (addMount(argv, i)) {
> System.out.println("Successfully added mount point " + argv[i]);
> }
> {code}
> we should handle this kind of cases also.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13633) RBF: Synchronous way to create RPC client connections to NN
[ https://issues.apache.org/jira/browse/HDFS-13633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13633: Parent Issue: HDFS-14603 (was: HDFS-13891) > RBF: Synchronous way to create RPC client connections to NN > --- > > Key: HDFS-13633 > URL: https://issues.apache.org/jira/browse/HDFS-13633 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: federation >Reporter: CR Hota >Assignee: CR Hota >Priority: Major > > Currently the router code does the following. > # IPC handler thread gets a connection from the pool, even if the connection > is NOT usable. > # At the same time the IPC thread also submits a request to connection > creator thread for adding a new connection to the pool asynchronously. > # The new connection is NOT utilized by the IPC threads that get back an > unusable connection. > With this approach burst behaviors of clients, fill up the pool without > necessarily using the connections. Also the approach is indeterministic. > We propose a flag that can allow router admins to control the behavior of > getting connections by the IPC handler threads. The flag would allow to > toggle ON/OFF asynchronous vs synchronous way of connection creation. > In the new model, if a connection is unusable, IPC handler thread would go > ahead and create a connection and add to the pool and utilize it > subsequently. It would still utilize the unusable connection if the pool is > full. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13219) RBF: Cluster information on Router is not correct when the Federation shares datanodes
[
https://issues.apache.org/jira/browse/HDFS-13219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-13219:
Parent Issue: HDFS-14603 (was: HDFS-13891)
> RBF: Cluster information on Router is not correct when the Federation shares
> datanodes
> --
>
> Key: HDFS-13219
> URL: https://issues.apache.org/jira/browse/HDFS-13219
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.9.0
>Reporter: Tao Jie
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png,
> screenshot-4.png, screenshot-5.png
>
>
> Now summary information on Router website aggregates summary of each
> nameservice. However in a typical federation cluster deployment, datanodes
> are shared among nameservices. Consider we have 2 namespaces and 100
> datanodes in one cluster. 100 datanodes are available for each namespace, but
> we see 200 datanodes on the router website. So does other information such as
> {{Total capacity}}, {{Remaining capacity}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13254) RBF: Cannot mv/cp file cross namespace
[ https://issues.apache.org/jira/browse/HDFS-13254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13254: Parent Issue: HDFS-14603 (was: HDFS-13891) > RBF: Cannot mv/cp file cross namespace > -- > > Key: HDFS-13254 > URL: https://issues.apache.org/jira/browse/HDFS-13254 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wu Weiwei >Priority: Major > > When I try to mv a file from a namespace to another, the client return an > error. > > Do we have any plan to support cp/mv file cross namespace? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13469) RBF: Support InodeID in the Router
[ https://issues.apache.org/jira/browse/HDFS-13469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13469: Parent Issue: HDFS-14603 (was: HDFS-13891) > RBF: Support InodeID in the Router > -- > > Key: HDFS-13469 > URL: https://issues.apache.org/jira/browse/HDFS-13469 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Priority: Major > > The Namenode supports identifying files through inode identifiers. > Currently the Router does not handle this properly, we need to add this > functionality. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13270) RBF: Router audit logger
[ https://issues.apache.org/jira/browse/HDFS-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13270: Parent Issue: HDFS-14603 (was: HDFS-13891) > RBF: Router audit logger > > > Key: HDFS-13270 > URL: https://issues.apache.org/jira/browse/HDFS-13270 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Affects Versions: 3.2.0 >Reporter: maobaolong >Priority: Major > > We can use router auditlogger to log the client info and cmd, because the > FSNamesystem#Auditlogger's log think the client are all from router. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13248) RBF: Namenode need to choose block location for the client
[ https://issues.apache.org/jira/browse/HDFS-13248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13248: Parent Issue: HDFS-14603 (was: HDFS-13891) > RBF: Namenode need to choose block location for the client > -- > > Key: HDFS-13248 > URL: https://issues.apache.org/jira/browse/HDFS-13248 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wu Weiwei >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13248.000.patch, HDFS-13248.001.patch, > HDFS-13248.002.patch, HDFS-13248.003.patch, HDFS-13248.004.patch, > HDFS-13248.005.patch, HDFS-Router-Data-Locality.odt, RBF Data Locality > Design.pdf, clientMachine-call-path.jpeg, debug-info-1.jpeg, debug-info-2.jpeg > > > When execute a put operation via router, the NameNode will choose block > location for the router, not for the real client. This will affect the file's > locality. > I think on both NameNode and Router, we should add a new addBlock method, or > add a parameter for the current addBlock method, to pass the real client > information. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13274) RBF: Extend RouterRpcClient to use multiple sockets
[ https://issues.apache.org/jira/browse/HDFS-13274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13274: Parent Issue: HDFS-14603 (was: HDFS-13891) > RBF: Extend RouterRpcClient to use multiple sockets > --- > > Key: HDFS-13274 > URL: https://issues.apache.org/jira/browse/HDFS-13274 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > > HADOOP-13144 introduces the ability to create multiple connections for the > same user and use different sockets. The RouterRpcClient should use this > approach to get a better throughput. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13245) RBF: State store DBMS implementation
[ https://issues.apache.org/jira/browse/HDFS-13245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13245: Parent Issue: HDFS-14603 (was: HDFS-13891) > RBF: State store DBMS implementation > > > Key: HDFS-13245 > URL: https://issues.apache.org/jira/browse/HDFS-13245 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: maobaolong >Assignee: Yiran Wu >Priority: Major > Attachments: HDFS-13245.001.patch, HDFS-13245.002.patch, > HDFS-13245.003.patch, HDFS-13245.004.patch, HDFS-13245.005.patch, > HDFS-13245.006.patch, HDFS-13245.007.patch, HDFS-13245.008.patch, > HDFS-13245.009.patch, HDFS-13245.010.patch, HDFS-13245.011.patch, > HDFS-13245.012.patch > > > Add a DBMS implementation for the State Store. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14598) Findbugs warning caused by HDFS-12487
[
https://issues.apache.org/jira/browse/HDFS-14598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871933#comment-16871933
]
Hudson commented on HDFS-14598:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16818 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16818/])
HDFS-14598. Findbugs warning caused by HDFS-12487. Contributed by He
(aengineer: rev 041e7a7dee4a17714f31952dc6364c77a65b1b73)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DiskBalancer.java
> Findbugs warning caused by HDFS-12487
> -
>
> Key: HDFS-14598
> URL: https://issues.apache.org/jira/browse/HDFS-14598
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: diskbalancer
>Reporter: Wei-Chiu Chuang
>Assignee: He Xiaoqiao
>Priority: Minor
> Labels: newbie
> Fix For: 3.3.0
>
> Attachments: HDFS-14598.001.patch
>
>
> https://builds.apache.org/job/PreCommit-HDFS-Build/27038/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
> {noformat}
> Redundant nullcheck of block, which is known to be non-null in
> org.apache.hadoop.hdfs.server.datanode.DiskBalancer$DiskBalancerMover.getBlockToCopy(FsVolumeSpi$BlockIterator,
> DiskBalancerWorkItem)
> Bug type RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE (click for details)
> In class org.apache.hadoop.hdfs.server.datanode.DiskBalancer$DiskBalancerMover
> In method
> org.apache.hadoop.hdfs.server.datanode.DiskBalancer$DiskBalancerMover.getBlockToCopy(FsVolumeSpi$BlockIterator,
> DiskBalancerWorkItem)
> Value loaded from block
> Return value of
> org.apache.hadoop.hdfs.server.datanode.fsdataset.FsVolumeSpi$BlockIterator.nextBlock()
> of type org.apache.hadoop.hdfs.protocol.ExtendedBlock
> Redundant null check at DiskBalancer.java:[line 912]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13278) RBF: Correct the logic of mount validate to avoid the bad mountPoint
[ https://issues.apache.org/jira/browse/HDFS-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13278: Parent Issue: HDFS-14603 (was: HDFS-13891) > RBF: Correct the logic of mount validate to avoid the bad mountPoint > > > Key: HDFS-13278 > URL: https://issues.apache.org/jira/browse/HDFS-13278 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Affects Versions: 3.2.0 >Reporter: maobaolong >Priority: Major > Labels: RBF > > Correct the logic of mount validate to avoid the bad mountPoint. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12487) FsDatasetSpi.isValidBlock() lacks null pointer check inside and neither do the callers
[ https://issues.apache.org/jira/browse/HDFS-12487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871934#comment-16871934 ] Hudson commented on HDFS-12487: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16818 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16818/]) HDFS-14598. Findbugs warning caused by HDFS-12487. Contributed by He (aengineer: rev 041e7a7dee4a17714f31952dc6364c77a65b1b73) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DiskBalancer.java > FsDatasetSpi.isValidBlock() lacks null pointer check inside and neither do > the callers > -- > > Key: HDFS-12487 > URL: https://issues.apache.org/jira/browse/HDFS-12487 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer mover, diskbalancer >Affects Versions: 3.0.0 > Environment: CentOS 6.8 x64 > CPU:4 core > Memory:16GB > Hadoop: Release 3.0.0-alpha4 >Reporter: liumi >Assignee: liumi >Priority: Major > Fix For: 3.3.0, 3.2.1, 3.1.3 > > Attachments: HDFS-12487.002.patch, HDFS-12487.003.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > BlockIteratorImpl.nextBlock() will look for the blocks in the source volume, > if there are no blocks any more, it will return null up to > DiskBalancer.getBlockToCopy(). However, the DiskBalancer.getBlockToCopy() > will check whether it's a valid block. > When I look into the FsDatasetSpi.isValidBlock(), I find that it doesn't > check the null pointer! In fact, we firstly need to check whether it's null > or not, or exception will occur. > This bug is hard to find, because the DiskBalancer hardly copy all the data > of one volume to others. Even if some times we may copy all the data of one > volume to other volumes, when the bug occurs, the copy process has already > done. > However, when we try to copy all the data of two or more volumes to other > volumes in more than one step, the thread will be shut down, which is caused > by the bug above. > The bug can fixed by two ways: > 1)Before the call of FsDatasetSpi.isValidBlock(), we check the null pointer > 2)Check the null pointer inside the implementation of > FsDatasetSpi.isValidBlock() -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13470) RBF: Add Browse the Filesystem button to the UI
[ https://issues.apache.org/jira/browse/HDFS-13470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13470: Parent Issue: HDFS-14603 (was: HDFS-13891) > RBF: Add Browse the Filesystem button to the UI > --- > > Key: HDFS-13470 > URL: https://issues.apache.org/jira/browse/HDFS-13470 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13470.000.patch > > > After HDFS-12512 added WebHDFS, we can add the support to browse the > filesystem to the UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13495) RBF: Support Router Admin REST API
[ https://issues.apache.org/jira/browse/HDFS-13495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13495: Parent Issue: HDFS-14603 (was: HDFS-13891) > RBF: Support Router Admin REST API > -- > > Key: HDFS-13495 > URL: https://issues.apache.org/jira/browse/HDFS-13495 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Mohammad Arshad >Assignee: Mohammad Arshad >Priority: Major > Labels: RBF > > This JIRA intends to add REST API support for all admin commands. Router > Admin REST APIs can be useful in managing the Routers from a central > management layer tool. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13507) RBF: Remove update functionality from routeradmin's add cmd
[ https://issues.apache.org/jira/browse/HDFS-13507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13507: Parent Issue: HDFS-14603 (was: HDFS-13891) > RBF: Remove update functionality from routeradmin's add cmd > --- > > Key: HDFS-13507 > URL: https://issues.apache.org/jira/browse/HDFS-13507 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wei Yan >Assignee: Gang Li >Priority: Minor > Labels: incompatible > Attachments: HDFS-13507-HDFS-13891.003.patch, > HDFS-13507-HDFS-13891.004.patch, HDFS-13507.000.patch, HDFS-13507.001.patch, > HDFS-13507.002.patch > > > Follow up the discussion in HDFS-13326. We should remove the "update" > functionality from routeradmin's add cmd, to make it consistent with RPC > calls. > Note that: this is an incompatible change. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13576) RBF: Add destination path length validation for add/update mount entry
[
https://issues.apache.org/jira/browse/HDFS-13576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-13576:
Parent Issue: HDFS-14603 (was: HDFS-13891)
> RBF: Add destination path length validation for add/update mount entry
> --
>
> Key: HDFS-13576
> URL: https://issues.apache.org/jira/browse/HDFS-13576
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Dibyendu Karmakar
>Assignee: Ayush Saxena
>Priority: Minor
>
> Currently there is no validation to check destination path length while
> adding or updating mount entry. But while trying to create directory using
> this mount entry
> {noformat}
> RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$PathComponentTooLongException){noformat}
> is thrown with exception message as
> {noformat}
> "maximum path component name limit of ... directory / is
> exceeded: limit=255 length=1817"{noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13506) RBF: Create destination directory when adding mount entry using router admin cmds.
[ https://issues.apache.org/jira/browse/HDFS-13506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13506: Parent Issue: HDFS-14603 (was: HDFS-13891) > RBF: Create destination directory when adding mount entry using router admin > cmds. > -- > > Key: HDFS-13506 > URL: https://issues.apache.org/jira/browse/HDFS-13506 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Dibyendu Karmakar >Assignee: Dibyendu Karmakar >Priority: Major > > Currently there is no option to create destination when adding mount entry. > User has to create the destination separately. > In router admin -add command we can add an option -createDest to create the > destination for the mount entry. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12345) Scale testing HDFS NameNode with real metadata and workloads (Dynamometer)
[ https://issues.apache.org/jira/browse/HDFS-12345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871931#comment-16871931 ] Yiqun Lin commented on HDFS-12345: -- Thanks for cleaning up the patches, [~xkrogen]! I am also +1 for this. Looks nice to see this in the hadoop tool. I didn't see the doc of the Dynamometer, we need to document this to let more users know more of this tool. > Scale testing HDFS NameNode with real metadata and workloads (Dynamometer) > -- > > Key: HDFS-12345 > URL: https://issues.apache.org/jira/browse/HDFS-12345 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode, test >Reporter: Zhe Zhang >Assignee: Erik Krogen >Priority: Major > Attachments: HDFS-12345.000.patch, HDFS-12345.001.patch, > HDFS-12345.002.patch, HDFS-12345.003.patch, HDFS-12345.004.patch, > HDFS-12345.005.patch, HDFS-12345.006.patch, HDFS-12345.007.patch, > HDFS-12345.008.patch, HDFS-12345.009.patch > > > Dynamometer has now been open sourced on our [GitHub > page|https://github.com/linkedin/dynamometer]. Read more at our [recent blog > post|https://engineering.linkedin.com/blog/2018/02/dynamometer--scale-testing-hdfs-on-minimal-hardware-with-maximum]. > To encourage getting the tool into the open for others to use as quickly as > possible, we went through our standard open sourcing process of releasing on > GitHub. However we are interested in the possibility of donating this to > Apache as part of Hadoop itself and would appreciate feedback on whether or > not this is something that would be supported by the community. > Also of note, previous [discussions on the dev mail > lists|http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201707.mbox/%3c98fceffa-faff-4cf1-a14d-4faab6567...@gmail.com%3e] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1672) Improve locking in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871930#comment-16871930 ] Bharat Viswanadham commented on HDDS-1672: -- Canceled the patch. As we are coming up with new OzoneManagerLock.java. Once HDDS-1723 gets checked in, will work on this, using new API's implemented in HDDS-1723. > Improve locking in OzoneManager > --- > > Key: HDDS-1672 > URL: https://issues.apache.org/jira/browse/HDDS-1672 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Manager >Affects Versions: 0.4.0 >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Attachments: Ozone Locks in OM.pdf > > Time Spent: 7h 10m > Remaining Estimate: 0h > > In this Jira, we shall follow the new lock ordering. In this way, in volume > requests we can solve the issue of acquire/release/reacquire problem. And few > bugs in the current implementation of S3Bucket/Volume operations. > > Currently after acquiring volume lock, we cannot acquire user lock. > This is causing an issue in Volume request implementation, > acquire/release/reacquire volume lock. > > Case of Delete Volume Request: > # Acquire volume lock. > # Get Volume Info from DB > # Release Volume lock. (We are releasing the lock, because while acquiring > volume lock, we cannot acquire user lock0 > # Get owner from volume Info read from DB > # Acquire owner lock > # Acquire volume lock > # Do delete logic > # release volume lock > # release user lock > > We can avoid this acquire/release/reacquire lock issue by making volume lock > as low weight. > > In this way, the above deleteVolume request will change as below > # Acquire volume lock > # Get Volume Info from DB > # Get owner from volume Info read from DB > # Acquire owner lock > # Do delete logic > # release owner lock > # release volume lock. > Same issue is seen with SetOwner for Volume request also. > During HDDS-1620 [~arp] brought up this issue. > I am proposing the above solution to solve this issue. Any other > idea/suggestions are welcome. > This also resolves a bug in setOwner for Volume request. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1672) Improve locking in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1672: - Status: Open (was: Patch Available) > Improve locking in OzoneManager > --- > > Key: HDDS-1672 > URL: https://issues.apache.org/jira/browse/HDDS-1672 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Manager >Affects Versions: 0.4.0 >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Attachments: Ozone Locks in OM.pdf > > Time Spent: 7h 10m > Remaining Estimate: 0h > > In this Jira, we shall follow the new lock ordering. In this way, in volume > requests we can solve the issue of acquire/release/reacquire problem. And few > bugs in the current implementation of S3Bucket/Volume operations. > > Currently after acquiring volume lock, we cannot acquire user lock. > This is causing an issue in Volume request implementation, > acquire/release/reacquire volume lock. > > Case of Delete Volume Request: > # Acquire volume lock. > # Get Volume Info from DB > # Release Volume lock. (We are releasing the lock, because while acquiring > volume lock, we cannot acquire user lock0 > # Get owner from volume Info read from DB > # Acquire owner lock > # Acquire volume lock > # Do delete logic > # release volume lock > # release user lock > > We can avoid this acquire/release/reacquire lock issue by making volume lock > as low weight. > > In this way, the above deleteVolume request will change as below > # Acquire volume lock > # Get Volume Info from DB > # Get owner from volume Info read from DB > # Acquire owner lock > # Do delete logic > # release owner lock > # release volume lock. > Same issue is seen with SetOwner for Volume request also. > During HDDS-1620 [~arp] brought up this issue. > I am proposing the above solution to solve this issue. Any other > idea/suggestions are welcome. > This also resolves a bug in setOwner for Volume request. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1723) Create new OzoneManagerLock class
[ https://issues.apache.org/jira/browse/HDDS-1723?focusedWorklogId=266282=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266282 ] ASF GitHub Bot logged work on HDDS-1723: Author: ASF GitHub Bot Created on: 25/Jun/19 02:14 Start Date: 25/Jun/19 02:14 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #1006: HDDS-1723. Create new OzoneManagerLock class. URL: https://github.com/apache/hadoop/pull/1006#issuecomment-505250084 Test failures are not related to this patch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266282) Time Spent: 4h (was: 3h 50m) > Create new OzoneManagerLock class > - > > Key: HDDS-1723 > URL: https://issues.apache.org/jira/browse/HDDS-1723 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 4h > Remaining Estimate: 0h > > This Jira is to use bit manipulation, instead of hashmap in OzoneManager lock > logic. And also this Jira follows the locking order based on the document > attached to HDDS-1672 jira. > This Jira is created based on [~anu] comment during review of HDDS-1672. > Not a suggestion for this patch. But more of a question, should we just > maintain a bitset here, and just flip that bit up and down to see if the lock > is held. Or we can just maintain 32 bit integer, and we can easily find if a > lock is held by Xoring with the correct mask. I feel that might be super > efficient. [@nandakumar131|https://github.com/nandakumar131] . But as I said > let us not do that in this patch. > > This Jira will add new class, integration of this new class into code will be > done in a new jira. > Clean up of old code also will be done in new jira. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14598) Findbugs warning caused by HDFS-12487
[
https://issues.apache.org/jira/browse/HDFS-14598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871927#comment-16871927
]
Anu Engineer commented on HDFS-14598:
-
[~jojochuang] Please let me know if you would like this to be backported.
I do see the other patch went into other branches.
{quote}Pushed to trunk, branch-3.2 and branch-3.1
Thanks [~liumihust] for the contribution and [~anu] for review!
{quote}
> Findbugs warning caused by HDFS-12487
> -
>
> Key: HDFS-14598
> URL: https://issues.apache.org/jira/browse/HDFS-14598
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: diskbalancer
>Reporter: Wei-Chiu Chuang
>Assignee: He Xiaoqiao
>Priority: Minor
> Labels: newbie
> Fix For: 3.3.0
>
> Attachments: HDFS-14598.001.patch
>
>
> https://builds.apache.org/job/PreCommit-HDFS-Build/27038/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
> {noformat}
> Redundant nullcheck of block, which is known to be non-null in
> org.apache.hadoop.hdfs.server.datanode.DiskBalancer$DiskBalancerMover.getBlockToCopy(FsVolumeSpi$BlockIterator,
> DiskBalancerWorkItem)
> Bug type RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE (click for details)
> In class org.apache.hadoop.hdfs.server.datanode.DiskBalancer$DiskBalancerMover
> In method
> org.apache.hadoop.hdfs.server.datanode.DiskBalancer$DiskBalancerMover.getBlockToCopy(FsVolumeSpi$BlockIterator,
> DiskBalancerWorkItem)
> Value loaded from block
> Return value of
> org.apache.hadoop.hdfs.server.datanode.fsdataset.FsVolumeSpi$BlockIterator.nextBlock()
> of type org.apache.hadoop.hdfs.protocol.ExtendedBlock
> Redundant null check at DiskBalancer.java:[line 912]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14598) Findbugs warning caused by HDFS-12487
[
https://issues.apache.org/jira/browse/HDFS-14598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDFS-14598:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.3.0
Status: Resolved (was: Patch Available)
[~jojochuang] Thanks for filing the issue. [~hexiaoqiao] Thank you for the
contribution. I have committed this patch to the trunk.
> Findbugs warning caused by HDFS-12487
> -
>
> Key: HDFS-14598
> URL: https://issues.apache.org/jira/browse/HDFS-14598
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: diskbalancer
>Reporter: Wei-Chiu Chuang
>Assignee: He Xiaoqiao
>Priority: Minor
> Labels: newbie
> Fix For: 3.3.0
>
> Attachments: HDFS-14598.001.patch
>
>
> https://builds.apache.org/job/PreCommit-HDFS-Build/27038/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
> {noformat}
> Redundant nullcheck of block, which is known to be non-null in
> org.apache.hadoop.hdfs.server.datanode.DiskBalancer$DiskBalancerMover.getBlockToCopy(FsVolumeSpi$BlockIterator,
> DiskBalancerWorkItem)
> Bug type RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE (click for details)
> In class org.apache.hadoop.hdfs.server.datanode.DiskBalancer$DiskBalancerMover
> In method
> org.apache.hadoop.hdfs.server.datanode.DiskBalancer$DiskBalancerMover.getBlockToCopy(FsVolumeSpi$BlockIterator,
> DiskBalancerWorkItem)
> Value loaded from block
> Return value of
> org.apache.hadoop.hdfs.server.datanode.fsdataset.FsVolumeSpi$BlockIterator.nextBlock()
> of type org.apache.hadoop.hdfs.protocol.ExtendedBlock
> Redundant null check at DiskBalancer.java:[line 912]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14541) When evictableMmapped or evictable size is zero, do not throw NoSuchElementException
[
https://issues.apache.org/jira/browse/HDFS-14541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871922#comment-16871922
]
Íñigo Goiri commented on HDFS-14541:
Cherry-picked to branch-3.2, branch-3.1, branch-3.0, branch-2, and branch-2.9.
> When evictableMmapped or evictable size is zero, do not throw
> NoSuchElementException
> -
>
> Key: HDFS-14541
> URL: https://issues.apache.org/jira/browse/HDFS-14541
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, performance
>Reporter: Zheng Hu
>Assignee: Lisheng Sun
>Priority: Major
> Fix For: 2.10.0, 3.0.4, 3.3.0, 3.2.1, 2.9.3, 3.1.3
>
> Attachments: HDFS-14541.000.patch, HDFS-14541.001.patch,
> HDFS-14541.002.patch, after-QPS.png, after-cpu-flame-graph.svg,
> after-heap-flame-graph.svg, async-prof-pid-94152-alloc-2.svg,
> async-prof-pid-94152-cpu-1.svg, before-QPS.png, before-cpu-flame-graph.svg,
> before-heap-flame-graph.svg
>
>
> Our XiaoMi HBase team are evaluating the performence improvement of
> HBASE-21879, and we have few CPU flame graph & heap flame graph by using
> async-profiler, and find that there're some performence issues in DFSClient
> .
> See the attached two flame graphs, we can conclude that the try catch block
> in ShortCircuitCache#trimEvictionMaps has some serious perf problem , we
> should remove the try catch from DFSClient.
> {code}
> /**
>* Trim the eviction lists.
>*/
> private void trimEvictionMaps() {
> long now = Time.monotonicNow();
> demoteOldEvictableMmaped(now);
> while (true) {
> long evictableSize = evictable.size();
> long evictableMmappedSize = evictableMmapped.size();
> if (evictableSize + evictableMmappedSize <= maxTotalSize) {
> return;
> }
> ShortCircuitReplica replica;
> try {
> if (evictableSize == 0) {
> replica = (ShortCircuitReplica)evictableMmapped.get(evictableMmapped
> .firstKey());
> } else {
> replica = (ShortCircuitReplica)evictable.get(evictable.firstKey());
> }
> } catch (NoSuchElementException e) {
> break;
> }
> if (LOG.isTraceEnabled()) {
> LOG.trace(this + ": trimEvictionMaps is purging " + replica +
> StringUtils.getStackTrace(Thread.currentThread()));
> }
> purge(replica);
> }
> }
> {code}
> Our Xiaomi HDFS Team member [~leosun08] will prepare patch for this issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14541) When evictableMmapped or evictable size is zero, do not throw NoSuchElementException
[
https://issues.apache.org/jira/browse/HDFS-14541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-14541:
---
Fix Version/s: 3.1.3
2.9.3
3.2.1
3.0.4
2.10.0
> When evictableMmapped or evictable size is zero, do not throw
> NoSuchElementException
> -
>
> Key: HDFS-14541
> URL: https://issues.apache.org/jira/browse/HDFS-14541
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, performance
>Reporter: Zheng Hu
>Assignee: Lisheng Sun
>Priority: Major
> Fix For: 2.10.0, 3.0.4, 3.3.0, 3.2.1, 2.9.3, 3.1.3
>
> Attachments: HDFS-14541.000.patch, HDFS-14541.001.patch,
> HDFS-14541.002.patch, after-QPS.png, after-cpu-flame-graph.svg,
> after-heap-flame-graph.svg, async-prof-pid-94152-alloc-2.svg,
> async-prof-pid-94152-cpu-1.svg, before-QPS.png, before-cpu-flame-graph.svg,
> before-heap-flame-graph.svg
>
>
> Our XiaoMi HBase team are evaluating the performence improvement of
> HBASE-21879, and we have few CPU flame graph & heap flame graph by using
> async-profiler, and find that there're some performence issues in DFSClient
> .
> See the attached two flame graphs, we can conclude that the try catch block
> in ShortCircuitCache#trimEvictionMaps has some serious perf problem , we
> should remove the try catch from DFSClient.
> {code}
> /**
>* Trim the eviction lists.
>*/
> private void trimEvictionMaps() {
> long now = Time.monotonicNow();
> demoteOldEvictableMmaped(now);
> while (true) {
> long evictableSize = evictable.size();
> long evictableMmappedSize = evictableMmapped.size();
> if (evictableSize + evictableMmappedSize <= maxTotalSize) {
> return;
> }
> ShortCircuitReplica replica;
> try {
> if (evictableSize == 0) {
> replica = (ShortCircuitReplica)evictableMmapped.get(evictableMmapped
> .firstKey());
> } else {
> replica = (ShortCircuitReplica)evictable.get(evictable.firstKey());
> }
> } catch (NoSuchElementException e) {
> break;
> }
> if (LOG.isTraceEnabled()) {
> LOG.trace(this + ": trimEvictionMaps is purging " + replica +
> StringUtils.getStackTrace(Thread.currentThread()));
> }
> purge(replica);
> }
> }
> {code}
> Our Xiaomi HDFS Team member [~leosun08] will prepare patch for this issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14586) Trash missing delete the folder which near timeout checkpoint
[ https://issues.apache.org/jira/browse/HDFS-14586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871919#comment-16871919 ] hu yongfa edited comment on HDFS-14586 at 6/25/19 2:00 AM: --- [~hexiaoqiao] thanks for your good idea, the new patch is available, it keep the Date(now) aligned with deletionInterval before loop. and the new patch is based on branch trunk. was (Author: huyongfa): [~hexiaoqiao] thanks for your good idea, the new patch is available, it keep the Date(now) aligned with deletionInterval before loop. and the new patch is based on branch trunk. [^HDFS-14586.001.patch] > Trash missing delete the folder which near timeout checkpoint > - > > Key: HDFS-14586 > URL: https://issues.apache.org/jira/browse/HDFS-14586 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hu yongfa >Assignee: hu yongfa >Priority: Major > Attachments: HDFS-14586.001.patch > > > when trash timeout checkpoint coming, trash will delete the old folder first, > then create a new checkpoint folder. > as the delete action may spend a long time, such as 2 minutes, so the new > checkpoint folder created late. > at the next trash timeout checkpoint, trash will skip delete the new > checkpoint folder, because the new checkpoint folder is > less than a checkpoint interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14586) Trash missing delete the folder which near timeout checkpoint
[ https://issues.apache.org/jira/browse/HDFS-14586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hu yongfa updated HDFS-14586: - Attachment: (was: HDFS-14586.001.patch) > Trash missing delete the folder which near timeout checkpoint > - > > Key: HDFS-14586 > URL: https://issues.apache.org/jira/browse/HDFS-14586 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hu yongfa >Assignee: hu yongfa >Priority: Major > Attachments: HDFS-14586.001.patch > > > when trash timeout checkpoint coming, trash will delete the old folder first, > then create a new checkpoint folder. > as the delete action may spend a long time, such as 2 minutes, so the new > checkpoint folder created late. > at the next trash timeout checkpoint, trash will skip delete the new > checkpoint folder, because the new checkpoint folder is > less than a checkpoint interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14586) Trash missing delete the folder which near timeout checkpoint
[ https://issues.apache.org/jira/browse/HDFS-14586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hu yongfa updated HDFS-14586: - Attachment: HDFS-14586.001.patch > Trash missing delete the folder which near timeout checkpoint > - > > Key: HDFS-14586 > URL: https://issues.apache.org/jira/browse/HDFS-14586 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hu yongfa >Assignee: hu yongfa >Priority: Major > Attachments: HDFS-14586.001.patch > > > when trash timeout checkpoint coming, trash will delete the old folder first, > then create a new checkpoint folder. > as the delete action may spend a long time, such as 2 minutes, so the new > checkpoint folder created late. > at the next trash timeout checkpoint, trash will skip delete the new > checkpoint folder, because the new checkpoint folder is > less than a checkpoint interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14586) Trash missing delete the folder which near timeout checkpoint
[ https://issues.apache.org/jira/browse/HDFS-14586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hu yongfa updated HDFS-14586: - Attachment: (was: HDFS-14586.001.patch) > Trash missing delete the folder which near timeout checkpoint > - > > Key: HDFS-14586 > URL: https://issues.apache.org/jira/browse/HDFS-14586 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hu yongfa >Assignee: hu yongfa >Priority: Major > Attachments: HDFS-14586.001.patch > > > when trash timeout checkpoint coming, trash will delete the old folder first, > then create a new checkpoint folder. > as the delete action may spend a long time, such as 2 minutes, so the new > checkpoint folder created late. > at the next trash timeout checkpoint, trash will skip delete the new > checkpoint folder, because the new checkpoint folder is > less than a checkpoint interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14586) Trash missing delete the folder which near timeout checkpoint
[ https://issues.apache.org/jira/browse/HDFS-14586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871919#comment-16871919 ] hu yongfa commented on HDFS-14586: -- [~hexiaoqiao] thanks for your good idea, the new patch is available, it keep the Date(now) aligned with deletionInterval before loop. and the new patch is based on branch trunk. [^HDFS-14586.001.patch] > Trash missing delete the folder which near timeout checkpoint > - > > Key: HDFS-14586 > URL: https://issues.apache.org/jira/browse/HDFS-14586 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hu yongfa >Assignee: hu yongfa >Priority: Major > Attachments: HDFS-14586.001.patch, HDFS-14586.001.patch > > > when trash timeout checkpoint coming, trash will delete the old folder first, > then create a new checkpoint folder. > as the delete action may spend a long time, such as 2 minutes, so the new > checkpoint folder created late. > at the next trash timeout checkpoint, trash will skip delete the new > checkpoint folder, because the new checkpoint folder is > less than a checkpoint interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14586) Trash missing delete the folder which near timeout checkpoint
[ https://issues.apache.org/jira/browse/HDFS-14586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hu yongfa updated HDFS-14586: - Attachment: HDFS-14586.001.patch > Trash missing delete the folder which near timeout checkpoint > - > > Key: HDFS-14586 > URL: https://issues.apache.org/jira/browse/HDFS-14586 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hu yongfa >Assignee: hu yongfa >Priority: Major > Attachments: HDFS-14586.001.patch, HDFS-14586.001.patch > > > when trash timeout checkpoint coming, trash will delete the old folder first, > then create a new checkpoint folder. > as the delete action may spend a long time, such as 2 minutes, so the new > checkpoint folder created late. > at the next trash timeout checkpoint, trash will skip delete the new > checkpoint folder, because the new checkpoint folder is > less than a checkpoint interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14541) When evictableMmapped or evictable size is zero, do not throw NoSuchElementException
[
https://issues.apache.org/jira/browse/HDFS-14541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871916#comment-16871916
]
Duo Zhang commented on HDFS-14541:
--
The default option is what you selected last time, so after you click the
'squash and merge' once it will be the default for you in the future :)
> When evictableMmapped or evictable size is zero, do not throw
> NoSuchElementException
> -
>
> Key: HDFS-14541
> URL: https://issues.apache.org/jira/browse/HDFS-14541
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, performance
>Reporter: Zheng Hu
>Assignee: Lisheng Sun
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-14541.000.patch, HDFS-14541.001.patch,
> HDFS-14541.002.patch, after-QPS.png, after-cpu-flame-graph.svg,
> after-heap-flame-graph.svg, async-prof-pid-94152-alloc-2.svg,
> async-prof-pid-94152-cpu-1.svg, before-QPS.png, before-cpu-flame-graph.svg,
> before-heap-flame-graph.svg
>
>
> Our XiaoMi HBase team are evaluating the performence improvement of
> HBASE-21879, and we have few CPU flame graph & heap flame graph by using
> async-profiler, and find that there're some performence issues in DFSClient
> .
> See the attached two flame graphs, we can conclude that the try catch block
> in ShortCircuitCache#trimEvictionMaps has some serious perf problem , we
> should remove the try catch from DFSClient.
> {code}
> /**
>* Trim the eviction lists.
>*/
> private void trimEvictionMaps() {
> long now = Time.monotonicNow();
> demoteOldEvictableMmaped(now);
> while (true) {
> long evictableSize = evictable.size();
> long evictableMmappedSize = evictableMmapped.size();
> if (evictableSize + evictableMmappedSize <= maxTotalSize) {
> return;
> }
> ShortCircuitReplica replica;
> try {
> if (evictableSize == 0) {
> replica = (ShortCircuitReplica)evictableMmapped.get(evictableMmapped
> .firstKey());
> } else {
> replica = (ShortCircuitReplica)evictable.get(evictable.firstKey());
> }
> } catch (NoSuchElementException e) {
> break;
> }
> if (LOG.isTraceEnabled()) {
> LOG.trace(this + ": trimEvictionMaps is purging " + replica +
> StringUtils.getStackTrace(Thread.currentThread()));
> }
> purge(replica);
> }
> }
> {code}
> Our Xiaomi HDFS Team member [~leosun08] will prepare patch for this issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14541) When evictableMmapped or evictable size is zero, do not throw NoSuchElementException
[
https://issues.apache.org/jira/browse/HDFS-14541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871914#comment-16871914
]
Íñigo Goiri commented on HDFS-14541:
I just saw that the default was 'create a merge commit'.
I'll commit by hand to the other branches.
Let's make 'squash and merge the default.
> When evictableMmapped or evictable size is zero, do not throw
> NoSuchElementException
> -
>
> Key: HDFS-14541
> URL: https://issues.apache.org/jira/browse/HDFS-14541
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, performance
>Reporter: Zheng Hu
>Assignee: Lisheng Sun
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-14541.000.patch, HDFS-14541.001.patch,
> HDFS-14541.002.patch, after-QPS.png, after-cpu-flame-graph.svg,
> after-heap-flame-graph.svg, async-prof-pid-94152-alloc-2.svg,
> async-prof-pid-94152-cpu-1.svg, before-QPS.png, before-cpu-flame-graph.svg,
> before-heap-flame-graph.svg
>
>
> Our XiaoMi HBase team are evaluating the performence improvement of
> HBASE-21879, and we have few CPU flame graph & heap flame graph by using
> async-profiler, and find that there're some performence issues in DFSClient
> .
> See the attached two flame graphs, we can conclude that the try catch block
> in ShortCircuitCache#trimEvictionMaps has some serious perf problem , we
> should remove the try catch from DFSClient.
> {code}
> /**
>* Trim the eviction lists.
>*/
> private void trimEvictionMaps() {
> long now = Time.monotonicNow();
> demoteOldEvictableMmaped(now);
> while (true) {
> long evictableSize = evictable.size();
> long evictableMmappedSize = evictableMmapped.size();
> if (evictableSize + evictableMmappedSize <= maxTotalSize) {
> return;
> }
> ShortCircuitReplica replica;
> try {
> if (evictableSize == 0) {
> replica = (ShortCircuitReplica)evictableMmapped.get(evictableMmapped
> .firstKey());
> } else {
> replica = (ShortCircuitReplica)evictable.get(evictable.firstKey());
> }
> } catch (NoSuchElementException e) {
> break;
> }
> if (LOG.isTraceEnabled()) {
> LOG.trace(this + ": trimEvictionMaps is purging " + replica +
> StringUtils.getStackTrace(Thread.currentThread()));
> }
> purge(replica);
> }
> }
> {code}
> Our Xiaomi HDFS Team member [~leosun08] will prepare patch for this issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13694) Making md5 computing being in parallel with image loading
[ https://issues.apache.org/jira/browse/HDFS-13694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-13694: --- Attachment: HDFS-13694-003.patch > Making md5 computing being in parallel with image loading > - > > Key: HDFS-13694 > URL: https://issues.apache.org/jira/browse/HDFS-13694 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: zhouyingchao >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-13694-001.patch, HDFS-13694-002.patch, > HDFS-13694-003.patch > > > During namenode image loading, it firstly compute the md5 and then load the > image. Actually these two steps can be in parallel. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14541) When evictableMmapped or evictable size is zero, do not throw NoSuchElementException
[
https://issues.apache.org/jira/browse/HDFS-14541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HDFS-14541:
-
Component/s: performance
hdfs-client
> When evictableMmapped or evictable size is zero, do not throw
> NoSuchElementException
> -
>
> Key: HDFS-14541
> URL: https://issues.apache.org/jira/browse/HDFS-14541
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, performance
>Reporter: Zheng Hu
>Assignee: Lisheng Sun
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-14541.000.patch, HDFS-14541.001.patch,
> HDFS-14541.002.patch, after-QPS.png, after-cpu-flame-graph.svg,
> after-heap-flame-graph.svg, async-prof-pid-94152-alloc-2.svg,
> async-prof-pid-94152-cpu-1.svg, before-QPS.png, before-cpu-flame-graph.svg,
> before-heap-flame-graph.svg
>
>
> Our XiaoMi HBase team are evaluating the performence improvement of
> HBASE-21879, and we have few CPU flame graph & heap flame graph by using
> async-profiler, and find that there're some performence issues in DFSClient
> .
> See the attached two flame graphs, we can conclude that the try catch block
> in ShortCircuitCache#trimEvictionMaps has some serious perf problem , we
> should remove the try catch from DFSClient.
> {code}
> /**
>* Trim the eviction lists.
>*/
> private void trimEvictionMaps() {
> long now = Time.monotonicNow();
> demoteOldEvictableMmaped(now);
> while (true) {
> long evictableSize = evictable.size();
> long evictableMmappedSize = evictableMmapped.size();
> if (evictableSize + evictableMmappedSize <= maxTotalSize) {
> return;
> }
> ShortCircuitReplica replica;
> try {
> if (evictableSize == 0) {
> replica = (ShortCircuitReplica)evictableMmapped.get(evictableMmapped
> .firstKey());
> } else {
> replica = (ShortCircuitReplica)evictable.get(evictable.firstKey());
> }
> } catch (NoSuchElementException e) {
> break;
> }
> if (LOG.isTraceEnabled()) {
> LOG.trace(this + ": trimEvictionMaps is purging " + replica +
> StringUtils.getStackTrace(Thread.currentThread()));
> }
> purge(replica);
> }
> }
> {code}
> Our Xiaomi HDFS Team member [~leosun08] will prepare patch for this issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14541) When evictableMmapped or evictable size is zero, do not throw NoSuchElementException
[
https://issues.apache.org/jira/browse/HDFS-14541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871911#comment-16871911
]
Duo Zhang commented on HDFS-14541:
--
You should use 'squash and merge' or 'rebase and merge', instead of 'create a
merge commit'. And I think we should file an issue at INFRA to disable the
'create a merge commit' button...
And I think should go into all branches effected? Not only trunk?
Thanks [~jojochuang] [~elgoiri].
> When evictableMmapped or evictable size is zero, do not throw
> NoSuchElementException
> -
>
> Key: HDFS-14541
> URL: https://issues.apache.org/jira/browse/HDFS-14541
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Zheng Hu
>Assignee: Lisheng Sun
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-14541.000.patch, HDFS-14541.001.patch,
> HDFS-14541.002.patch, after-QPS.png, after-cpu-flame-graph.svg,
> after-heap-flame-graph.svg, async-prof-pid-94152-alloc-2.svg,
> async-prof-pid-94152-cpu-1.svg, before-QPS.png, before-cpu-flame-graph.svg,
> before-heap-flame-graph.svg
>
>
> Our XiaoMi HBase team are evaluating the performence improvement of
> HBASE-21879, and we have few CPU flame graph & heap flame graph by using
> async-profiler, and find that there're some performence issues in DFSClient
> .
> See the attached two flame graphs, we can conclude that the try catch block
> in ShortCircuitCache#trimEvictionMaps has some serious perf problem , we
> should remove the try catch from DFSClient.
> {code}
> /**
>* Trim the eviction lists.
>*/
> private void trimEvictionMaps() {
> long now = Time.monotonicNow();
> demoteOldEvictableMmaped(now);
> while (true) {
> long evictableSize = evictable.size();
> long evictableMmappedSize = evictableMmapped.size();
> if (evictableSize + evictableMmappedSize <= maxTotalSize) {
> return;
> }
> ShortCircuitReplica replica;
> try {
> if (evictableSize == 0) {
> replica = (ShortCircuitReplica)evictableMmapped.get(evictableMmapped
> .firstKey());
> } else {
> replica = (ShortCircuitReplica)evictable.get(evictable.firstKey());
> }
> } catch (NoSuchElementException e) {
> break;
> }
> if (LOG.isTraceEnabled()) {
> LOG.trace(this + ": trimEvictionMaps is purging " + replica +
> StringUtils.getStackTrace(Thread.currentThread()));
> }
> purge(replica);
> }
> }
> {code}
> Our Xiaomi HDFS Team member [~leosun08] will prepare patch for this issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1685) Recon: Add support for "start" query param to containers and containers/{id} endpoints
[
https://issues.apache.org/jira/browse/HDDS-1685?focusedWorklogId=266258=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266258
]
ASF GitHub Bot logged work on HDDS-1685:
Author: ASF GitHub Bot
Created on: 25/Jun/19 01:21
Start Date: 25/Jun/19 01:21
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on issue #987: HDDS-1685. Recon:
Add support for 'start' query param to containers…
URL: https://github.com/apache/hadoop/pull/987#issuecomment-505239525
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|::|--:|:|:|
| 0 | reexec | 500 | Docker mode activated. |
||| _ Prechecks _ |
| +1 | dupname | 0 | No case conflicting files found. |
| +1 | @author | 0 | The patch does not contain any @author tags. |
| +1 | test4tests | 0 | The patch appears to include 2 new or modified test
files. |
||| _ trunk Compile Tests _ |
| +1 | mvninstall | 496 | trunk passed |
| +1 | compile | 246 | trunk passed |
| +1 | checkstyle | 60 | trunk passed |
| +1 | mvnsite | 0 | trunk passed |
| +1 | shadedclient | 830 | branch has no errors when building and testing
our client artifacts. |
| +1 | javadoc | 148 | trunk passed |
| 0 | spotbugs | 317 | Used deprecated FindBugs config; considering
switching to SpotBugs. |
| +1 | findbugs | 512 | trunk passed |
||| _ Patch Compile Tests _ |
| +1 | mvninstall | 464 | the patch passed |
| -1 | compile | 88 | hadoop-hdds in the patch failed. |
| -1 | javac | 88 | hadoop-hdds in the patch failed. |
| -0 | checkstyle | 34 | hadoop-ozone: The patch generated 3 new + 0
unchanged - 0 fixed = 3 total (was 0) |
| +1 | mvnsite | 0 | the patch passed |
| +1 | whitespace | 0 | The patch has no whitespace issues. |
| +1 | shadedclient | 671 | patch has no errors when building and testing
our client artifacts. |
| +1 | javadoc | 152 | the patch passed |
| -1 | findbugs | 112 | hadoop-hdds in the patch failed. |
| -1 | findbugs | 27 | hadoop-ozone in the patch failed. |
||| _ Other Tests _ |
| -1 | unit | 29 | hadoop-hdds in the patch failed. |
| -1 | unit | 27 | hadoop-ozone in the patch failed. |
| 0 | asflicense | 28 | ASF License check generated no output? |
| | | 4756 | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | Client=17.05.0-ce Server=17.05.0-ce base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-987/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/987 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux d652ac3b24b7 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 129576f |
| Default Java | 1.8.0_212 |
| compile |
https://builds.apache.org/job/hadoop-multibranch/job/PR-987/4/artifact/out/patch-compile-hadoop-hdds.txt
|
| javac |
https://builds.apache.org/job/hadoop-multibranch/job/PR-987/4/artifact/out/patch-compile-hadoop-hdds.txt
|
| checkstyle |
https://builds.apache.org/job/hadoop-multibranch/job/PR-987/4/artifact/out/diff-checkstyle-hadoop-ozone.txt
|
| findbugs |
https://builds.apache.org/job/hadoop-multibranch/job/PR-987/4/artifact/out/patch-findbugs-hadoop-hdds.txt
|
| findbugs |
https://builds.apache.org/job/hadoop-multibranch/job/PR-987/4/artifact/out/patch-findbugs-hadoop-ozone.txt
|
| unit |
https://builds.apache.org/job/hadoop-multibranch/job/PR-987/4/artifact/out/patch-unit-hadoop-hdds.txt
|
| unit |
https://builds.apache.org/job/hadoop-multibranch/job/PR-987/4/artifact/out/patch-unit-hadoop-ozone.txt
|
| Test Results |
https://builds.apache.org/job/hadoop-multibranch/job/PR-987/4/testReport/ |
| Max. process+thread count | 412 (vs. ulimit of 5500) |
| modules | C: hadoop-ozone/ozone-recon U: hadoop-ozone/ozone-recon |
| Console output |
https://builds.apache.org/job/hadoop-multibranch/job/PR-987/4/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.10.0 http://yetus.apache.org |
This message was automatically generated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 266258)
Time Spent: 2h (was: 1h 50m)
> Recon: Add support for "start" query param to containers and containers/{id}
>
[jira] [Commented] (HDFS-13371) NPE for FsServerDefaults.getKeyProviderUri() for clientProtocol communication between 2.7 and 3.X
[ https://issues.apache.org/jira/browse/HDFS-13371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871905#comment-16871905 ] Hudson commented on HDFS-13371: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16817 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16817/]) HDFS-13371. NPE for FsServerDefaults.getKeyProviderUri() for (inigoiri: rev b76b843c8bd3906aaa5ad633d8a939aebc671907) * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java > NPE for FsServerDefaults.getKeyProviderUri() for clientProtocol communication > between 2.7 and 3.X > - > > Key: HDFS-13371 > URL: https://issues.apache.org/jira/browse/HDFS-13371 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.1.0, 3.2.0 >Reporter: Sherwood Zheng >Assignee: Sherwood Zheng >Priority: Minor > Labels: RBF > Fix For: 3.3.0 > > Attachments: HDFS-13371.000.patch > > > KeyProviderUri is not available in 2.7 so when 2.7 clients contact with 3.2 > services, it cannot find the key provider URI and triggers a > NullPointerException. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13371) NPE for FsServerDefaults.getKeyProviderUri() for clientProtocol communication between 2.7 and 3.X
[ https://issues.apache.org/jira/browse/HDFS-13371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13371: --- Summary: NPE for FsServerDefaults.getKeyProviderUri() for clientProtocol communication between 2.7 and 3.X (was: NPE for FsServerDefaults.getKeyProviderUri() for clientProtocol communication between 2.7 and 3.2) > NPE for FsServerDefaults.getKeyProviderUri() for clientProtocol communication > between 2.7 and 3.X > - > > Key: HDFS-13371 > URL: https://issues.apache.org/jira/browse/HDFS-13371 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.1.0, 3.2.0 >Reporter: Sherwood Zheng >Assignee: Sherwood Zheng >Priority: Minor > Labels: RBF > Attachments: HDFS-13371.000.patch > > > KeyProviderUri is not available in 2.7 so when 2.7 clients contact with 3.2 > services, it cannot find the key provider URI and triggers a > NullPointerException. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13371) NPE for FsServerDefaults.getKeyProviderUri() for clientProtocol communication between 2.7 and 3.X
[ https://issues.apache.org/jira/browse/HDFS-13371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13371: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.3.0 Status: Resolved (was: Patch Available) > NPE for FsServerDefaults.getKeyProviderUri() for clientProtocol communication > between 2.7 and 3.X > - > > Key: HDFS-13371 > URL: https://issues.apache.org/jira/browse/HDFS-13371 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.1.0, 3.2.0 >Reporter: Sherwood Zheng >Assignee: Sherwood Zheng >Priority: Minor > Labels: RBF > Fix For: 3.3.0 > > Attachments: HDFS-13371.000.patch > > > KeyProviderUri is not available in 2.7 so when 2.7 clients contact with 3.2 > services, it cannot find the key provider URI and triggers a > NullPointerException. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-14541) When evictableMmapped or evictable size is zero, do not throw NoSuchElementException
[
https://issues.apache.org/jira/browse/HDFS-14541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri resolved HDFS-14541.
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.3.0
> When evictableMmapped or evictable size is zero, do not throw
> NoSuchElementException
> -
>
> Key: HDFS-14541
> URL: https://issues.apache.org/jira/browse/HDFS-14541
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Zheng Hu
>Assignee: Lisheng Sun
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-14541.000.patch, HDFS-14541.001.patch,
> HDFS-14541.002.patch, after-QPS.png, after-cpu-flame-graph.svg,
> after-heap-flame-graph.svg, async-prof-pid-94152-alloc-2.svg,
> async-prof-pid-94152-cpu-1.svg, before-QPS.png, before-cpu-flame-graph.svg,
> before-heap-flame-graph.svg
>
>
> Our XiaoMi HBase team are evaluating the performence improvement of
> HBASE-21879, and we have few CPU flame graph & heap flame graph by using
> async-profiler, and find that there're some performence issues in DFSClient
> .
> See the attached two flame graphs, we can conclude that the try catch block
> in ShortCircuitCache#trimEvictionMaps has some serious perf problem , we
> should remove the try catch from DFSClient.
> {code}
> /**
>* Trim the eviction lists.
>*/
> private void trimEvictionMaps() {
> long now = Time.monotonicNow();
> demoteOldEvictableMmaped(now);
> while (true) {
> long evictableSize = evictable.size();
> long evictableMmappedSize = evictableMmapped.size();
> if (evictableSize + evictableMmappedSize <= maxTotalSize) {
> return;
> }
> ShortCircuitReplica replica;
> try {
> if (evictableSize == 0) {
> replica = (ShortCircuitReplica)evictableMmapped.get(evictableMmapped
> .firstKey());
> } else {
> replica = (ShortCircuitReplica)evictable.get(evictable.firstKey());
> }
> } catch (NoSuchElementException e) {
> break;
> }
> if (LOG.isTraceEnabled()) {
> LOG.trace(this + ": trimEvictionMaps is purging " + replica +
> StringUtils.getStackTrace(Thread.currentThread()));
> }
> purge(replica);
> }
> }
> {code}
> Our Xiaomi HDFS Team member [~leosun08] will prepare patch for this issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14541) When evictableMmapped or evictable size is zero, do not throw NoSuchElementException
[
https://issues.apache.org/jira/browse/HDFS-14541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871899#comment-16871899
]
Íñigo Goiri commented on HDFS-14541:
I did the whole process and merge to trunk.
Unfortunately it added "Merge pull request #977 from leosunli/trunk" to the
commit message.
It would be nice to have the full guide for merging PRs.
> When evictableMmapped or evictable size is zero, do not throw
> NoSuchElementException
> -
>
> Key: HDFS-14541
> URL: https://issues.apache.org/jira/browse/HDFS-14541
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Zheng Hu
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-14541.000.patch, HDFS-14541.001.patch,
> HDFS-14541.002.patch, after-QPS.png, after-cpu-flame-graph.svg,
> after-heap-flame-graph.svg, async-prof-pid-94152-alloc-2.svg,
> async-prof-pid-94152-cpu-1.svg, before-QPS.png, before-cpu-flame-graph.svg,
> before-heap-flame-graph.svg
>
>
> Our XiaoMi HBase team are evaluating the performence improvement of
> HBASE-21879, and we have few CPU flame graph & heap flame graph by using
> async-profiler, and find that there're some performence issues in DFSClient
> .
> See the attached two flame graphs, we can conclude that the try catch block
> in ShortCircuitCache#trimEvictionMaps has some serious perf problem , we
> should remove the try catch from DFSClient.
> {code}
> /**
>* Trim the eviction lists.
>*/
> private void trimEvictionMaps() {
> long now = Time.monotonicNow();
> demoteOldEvictableMmaped(now);
> while (true) {
> long evictableSize = evictable.size();
> long evictableMmappedSize = evictableMmapped.size();
> if (evictableSize + evictableMmappedSize <= maxTotalSize) {
> return;
> }
> ShortCircuitReplica replica;
> try {
> if (evictableSize == 0) {
> replica = (ShortCircuitReplica)evictableMmapped.get(evictableMmapped
> .firstKey());
> } else {
> replica = (ShortCircuitReplica)evictable.get(evictable.firstKey());
> }
> } catch (NoSuchElementException e) {
> break;
> }
> if (LOG.isTraceEnabled()) {
> LOG.trace(this + ": trimEvictionMaps is purging " + replica +
> StringUtils.getStackTrace(Thread.currentThread()));
> }
> purge(replica);
> }
> }
> {code}
> Our Xiaomi HDFS Team member [~leosun08] will prepare patch for this issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14541) When evictableMmapped or evictable size is zero, do not throw NoSuchElementException
[
https://issues.apache.org/jira/browse/HDFS-14541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-14541:
---
Summary: When evictableMmapped or evictable size is zero, do not throw
NoSuchElementException (was: ShortCircuitReplica#unref cost about 6% cpu and
6% heap allocation because of the frequent thrown NoSuchElementException in
our HBase benchmark)
> When evictableMmapped or evictable size is zero, do not throw
> NoSuchElementException
> -
>
> Key: HDFS-14541
> URL: https://issues.apache.org/jira/browse/HDFS-14541
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Zheng Hu
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-14541.000.patch, HDFS-14541.001.patch,
> HDFS-14541.002.patch, after-QPS.png, after-cpu-flame-graph.svg,
> after-heap-flame-graph.svg, async-prof-pid-94152-alloc-2.svg,
> async-prof-pid-94152-cpu-1.svg, before-QPS.png, before-cpu-flame-graph.svg,
> before-heap-flame-graph.svg
>
>
> Our XiaoMi HBase team are evaluating the performence improvement of
> HBASE-21879, and we have few CPU flame graph & heap flame graph by using
> async-profiler, and find that there're some performence issues in DFSClient
> .
> See the attached two flame graphs, we can conclude that the try catch block
> in ShortCircuitCache#trimEvictionMaps has some serious perf problem , we
> should remove the try catch from DFSClient.
> {code}
> /**
>* Trim the eviction lists.
>*/
> private void trimEvictionMaps() {
> long now = Time.monotonicNow();
> demoteOldEvictableMmaped(now);
> while (true) {
> long evictableSize = evictable.size();
> long evictableMmappedSize = evictableMmapped.size();
> if (evictableSize + evictableMmappedSize <= maxTotalSize) {
> return;
> }
> ShortCircuitReplica replica;
> try {
> if (evictableSize == 0) {
> replica = (ShortCircuitReplica)evictableMmapped.get(evictableMmapped
> .firstKey());
> } else {
> replica = (ShortCircuitReplica)evictable.get(evictable.firstKey());
> }
> } catch (NoSuchElementException e) {
> break;
> }
> if (LOG.isTraceEnabled()) {
> LOG.trace(this + ": trimEvictionMaps is purging " + replica +
> StringUtils.getStackTrace(Thread.currentThread()));
> }
> purge(replica);
> }
> }
> {code}
> Our Xiaomi HDFS Team member [~leosun08] will prepare patch for this issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1651) Create a http.policy config for Ozone
[ https://issues.apache.org/jira/browse/HDDS-1651?focusedWorklogId=266234=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266234 ] ASF GitHub Bot logged work on HDDS-1651: Author: ASF GitHub Bot Created on: 25/Jun/19 00:28 Start Date: 25/Jun/19 00:28 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #930: HDDS-1651. Create a http.policy config for Ozone URL: https://github.com/apache/hadoop/pull/930#discussion_r295128201 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsUtils.java ## @@ -31,6 +31,7 @@ import java.util.Optional; import java.util.TimeZone; +import org.apache.hadoop.HadoopIllegalArgumentException; Review comment: Minor: Unused import. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266234) Time Spent: 3h (was: 2h 50m) > Create a http.policy config for Ozone > - > > Key: HDDS-1651 > URL: https://issues.apache.org/jira/browse/HDDS-1651 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Hanisha Koneru >Assignee: Shweta >Priority: Major > Labels: newbie, pull-request-available > Time Spent: 3h > Remaining Estimate: 0h > > Ozone currently uses dfs.http.policy for HTTP policy. Ozone should have its > own ozone.http.policy configuration and if undefined, then fallback to > dfs.http.policy. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1651) Create a http.policy config for Ozone
[ https://issues.apache.org/jira/browse/HDDS-1651?focusedWorklogId=266232=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266232 ] ASF GitHub Bot logged work on HDDS-1651: Author: ASF GitHub Bot Created on: 25/Jun/19 00:28 Start Date: 25/Jun/19 00:28 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #930: HDDS-1651. Create a http.policy config for Ozone URL: https://github.com/apache/hadoop/pull/930#issuecomment-505229785 Hi, I have a question: What is the reason to create a new http policy for ozone? Because when https is enabled, some of the additional config like key store location we still use dfs.https.server.keystore.resource. I feel ozone can also re-use the hdfs config. In this way, we don't miss the code changes where HTTP policy is being used, like OzoneManagerSnapShotProvider which uses still DFSUtils.getHttpPolicy. And also we use to create HttpServer2.Builder builder with below code. That again uses dfs.http.policy. So, this is the reason we need to set ozone.http.policy to dfs.http.policy to make it work. So, I feel to avoid all these, we can use dfs.http.policy as before. Let me know your thoughts on this. ``` builder = DFSUtil.httpServerTemplateForNNAndJN(conf, this.httpAddress, this.httpsAddress, name, getSpnegoPrincipal(), getKeytabFile()); ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266232) Time Spent: 2h 50m (was: 2h 40m) > Create a http.policy config for Ozone > - > > Key: HDDS-1651 > URL: https://issues.apache.org/jira/browse/HDDS-1651 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Hanisha Koneru >Assignee: Shweta >Priority: Major > Labels: newbie, pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > Ozone currently uses dfs.http.policy for HTTP policy. Ozone should have its > own ozone.http.policy configuration and if undefined, then fallback to > dfs.http.policy. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1651) Create a http.policy config for Ozone
[ https://issues.apache.org/jira/browse/HDDS-1651?focusedWorklogId=266231=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266231 ] ASF GitHub Bot logged work on HDDS-1651: Author: ASF GitHub Bot Created on: 25/Jun/19 00:26 Start Date: 25/Jun/19 00:26 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #930: HDDS-1651. Create a http.policy config for Ozone URL: https://github.com/apache/hadoop/pull/930#issuecomment-505229785 Hi, I have a question: is there a reason to create a new http policy for config. Because when https is enabled, some of the additional config like key store location we use dfs.https.server.keystore.resource. I feel ozone can also re-use the hdfs config. In this way, we don't miss the code changes where HTTP policy is being used, like OzoneManagerSnapShotProvider which uses still DFSUtils.getHttpPolicy. And also we use to create HttpServer2.Builder builder with below code. That again uses dfs.http.policy. So, this is the reason we need to set ozone.http.policy to dfs.http.policy to make it work. So, I feel to avoid all these, we can use dfs.http.policy as before. Let me know your thoughts on this. ``` builder = DFSUtil.httpServerTemplateForNNAndJN(conf, this.httpAddress, this.httpsAddress, name, getSpnegoPrincipal(), getKeytabFile()); ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266231) Time Spent: 2h 40m (was: 2.5h) > Create a http.policy config for Ozone > - > > Key: HDDS-1651 > URL: https://issues.apache.org/jira/browse/HDDS-1651 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Hanisha Koneru >Assignee: Shweta >Priority: Major > Labels: newbie, pull-request-available > Time Spent: 2h 40m > Remaining Estimate: 0h > > Ozone currently uses dfs.http.policy for HTTP policy. Ozone should have its > own ozone.http.policy configuration and if undefined, then fallback to > dfs.http.policy. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1723) Create new OzoneManagerLock class
[ https://issues.apache.org/jira/browse/HDDS-1723?focusedWorklogId=266228=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266228 ] ASF GitHub Bot logged work on HDDS-1723: Author: ASF GitHub Bot Created on: 25/Jun/19 00:20 Start Date: 25/Jun/19 00:20 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1006: HDDS-1723. Create new OzoneManagerLock class. URL: https://github.com/apache/hadoop/pull/1006#issuecomment-505228694 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 893 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 64 | Maven dependency ordering for branch | | +1 | mvninstall | 475 | trunk passed | | +1 | compile | 243 | trunk passed | | +1 | checkstyle | 67 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 892 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 154 | trunk passed | | 0 | spotbugs | 306 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 491 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 29 | Maven dependency ordering for patch | | +1 | mvninstall | 421 | the patch passed | | +1 | compile | 247 | the patch passed | | +1 | javac | 247 | the patch passed | | +1 | checkstyle | 72 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 1 | The patch has no whitespace issues. | | +1 | shadedclient | 686 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 149 | the patch passed | | +1 | findbugs | 505 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 262 | hadoop-hdds in the patch passed. | | -1 | unit | 1710 | hadoop-ozone in the patch failed. | | +1 | asflicense | 40 | The patch does not generate ASF License warnings. | | | | 7548 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestCommitWatcher | | | hadoop.ozone.om.TestOzoneManager | | | hadoop.ozone.client.rpc.TestBCSID | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.ozone.om.TestOzoneManagerHA | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=18.09.5 Server=18.09.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1006/10/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1006 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 95409dcc34c1 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 129576f | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1006/10/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1006/10/testReport/ | | Max. process+thread count | 5389 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-ozone/common U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1006/10/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266228) Time Spent: 3h 50m (was: 3h 40m) > Create new OzoneManagerLock class > - > > Key: HDDS-1723 > URL: https://issues.apache.org/jira/browse/HDDS-1723 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat
[jira] [Work logged] (HDDS-1617) Restructure the code layout for Ozone Manager
[ https://issues.apache.org/jira/browse/HDDS-1617?focusedWorklogId=266224=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266224 ] ASF GitHub Bot logged work on HDDS-1617: Author: ASF GitHub Bot Created on: 25/Jun/19 00:07 Start Date: 25/Jun/19 00:07 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #880: HDDS-1617. Restructure the code layout for Ozone Manager URL: https://github.com/apache/hadoop/pull/880#issuecomment-505226258 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 30 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 7 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 42 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 65 | Maven dependency ordering for branch | | +1 | mvninstall | 469 | trunk passed | | +1 | compile | 243 | trunk passed | | +1 | checkstyle | 75 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 858 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 161 | trunk passed | | 0 | spotbugs | 309 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 500 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 29 | Maven dependency ordering for patch | | +1 | mvninstall | 429 | the patch passed | | +1 | compile | 263 | the patch passed | | +1 | cc | 263 | the patch passed | | +1 | javac | 263 | the patch passed | | -0 | checkstyle | 42 | hadoop-ozone: The patch generated 11 new + 0 unchanged - 0 fixed = 11 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 1 | The patch has no whitespace issues. | | +1 | shadedclient | 667 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 160 | the patch passed | | +1 | findbugs | 518 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 242 | hadoop-hdds in the patch passed. | | -1 | unit | 1095 | hadoop-ozone in the patch failed. | | +1 | asflicense | 43 | The patch does not generate ASF License warnings. | | | | 6121 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.TestMiniOzoneCluster | | | hadoop.ozone.om.TestMultipleContainerReadWrite | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-880/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/880 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux 5cc20d5333f3 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 129576f | | Default Java | 1.8.0_212 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-880/2/artifact/out/diff-checkstyle-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-880/2/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-880/2/testReport/ | | Max. process+thread count | 5068 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-ozone/common hadoop-ozone/integration-test hadoop-ozone/ozone-manager hadoop-ozone/ozone-recon hadoop-ozone/ozonefs hadoop-ozone/tools U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-880/2/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266224) Time Spent: 2h 20m (was: 2h 10m) > Restructure the code layout for Ozone Manager > - > >
[jira] [Commented] (HDFS-13984) getFileInfo of libhdfs call NameNode#getFileStatus twice
[
https://issues.apache.org/jira/browse/HDFS-13984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871888#comment-16871888
]
Sahil Takiar commented on HDFS-13984:
-
Yeah, this makes sense to me.
[~yangjiandan]do you have plans to continue working on this?
A few comments:
* Use {{javaObjectIsOfClass}} in {{jni_helper.h}}
* The {{FileNotFoundException}} needs to be freed before returning null
> getFileInfo of libhdfs call NameNode#getFileStatus twice
>
>
> Key: HDFS-13984
> URL: https://issues.apache.org/jira/browse/HDFS-13984
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: libhdfs
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: HDFS-13984.001.patch, HDFS-13984.002.patch,
> HDFS-13984.003.patch
>
>
> getFileInfo in hdfs.c calls *FileSystem#exists* first, then calls
> *FileSystem#getFileStatus*.
> *FileSystem#exists* also call *FileSystem#getFileStatus*, just as follows:
> {code:java}
> public boolean exists(Path f) throws IOException {
> try {
> return getFileStatus(f) != null;
> } catch (FileNotFoundException e) {
> return false;
> }
> }
> {code}
> and finally this leads to call NameNodeRpcServer#getFileInfo twice.
> Actually we can implement by calling once.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1709) TestScmSafeNode is flaky
[
https://issues.apache.org/jira/browse/HDDS-1709?focusedWorklogId=266220=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266220
]
ASF GitHub Bot logged work on HDDS-1709:
Author: ASF GitHub Bot
Created on: 24/Jun/19 23:51
Start Date: 24/Jun/19 23:51
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on issue #993: HDDS-1709.
TestScmSafeNode is flaky
URL: https://github.com/apache/hadoop/pull/993#issuecomment-505223323
Thank You @elek for reporting and fixing the issue.
+1 LGTM. Can you take care of checkstyle issue during the commit?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 266220)
Time Spent: 0.5h (was: 20m)
> TestScmSafeNode is flaky
>
>
> Key: HDDS-1709
> URL: https://issues.apache.org/jira/browse/HDDS-1709
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM, test
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> org.apache.hadoop.ozone.om.TestScmSafeMode.testSCMSafeMode is failed at last
> night with the following error:
> {code:java}
> java.lang.AssertionError at org.junit.Assert.fail(Assert.java:86) at
> org.junit.Assert.assertTrue(Assert.java:41) at
> org.junit.Assert.assertTrue(Assert.java:52) at
> org.apache.hadoop.ozone.om.TestScmSafeMode.testSCMSafeMode(TestScmSafeMode.java:285)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74){code}
> Locally it can be tested but it's very easy to reproduce by adding an
> additional sleep DataNodeSafeModeRule:
> {code:java}
> +++
> b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/DataNodeSafeModeRule.java
> @@ -63,7 +63,11 @@ protected boolean validate() {
>
> @Override
> protected void process(NodeRegistrationContainerReport reportsProto) {
> -
> + try {
> + Thread.sleep(3000);
> + } catch (InterruptedException e) {
> + e.printStackTrace();
> + }{code}
> This is a clear race condition:
> DatanodeSafeModeRule and ContainerSafeModeRule are processing the same events
> but it can be possible (in case of an accidental sleep) that the container
> safe mode rule is done, but DatanodeSafeModeRule didn't process the new event
> (yet).
> As a result the test execution will continue:
> {code:java}
> GenericTestUtils
> .waitFor(() -> scm.getCurrentContainerThreshold() == 1.0, 100, 2);
> {code}
> (This line is waiting ONLY for the ContainerSafeModeRule).
> The fix is easy, let's wait for the processing of all the async events:
> {code:java}
> EventQueue eventQueue =
> (EventQueue) cluster.getStorageContainerManager().getEventQueue();
> eventQueue.processAll(5000L);{code}
> As we are sure that the events are already sent to the EventQueue (because we
> have the previous waitFor), it should be enough.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1666) Improve logic in openKey when allocating block
[ https://issues.apache.org/jira/browse/HDDS-1666?focusedWorklogId=266219=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266219 ] ASF GitHub Bot logged work on HDDS-1666: Author: ASF GitHub Bot Created on: 24/Jun/19 23:50 Start Date: 24/Jun/19 23:50 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #943: HDDS-1666. Issue in openKey when allocating block. URL: https://github.com/apache/hadoop/pull/943#issuecomment-505223055 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 30 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 471 | trunk passed | | +1 | compile | 241 | trunk passed | | +1 | checkstyle | 61 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 787 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 149 | trunk passed | | 0 | spotbugs | 345 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 558 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 464 | the patch passed | | +1 | compile | 256 | the patch passed | | +1 | javac | 256 | the patch passed | | +1 | checkstyle | 65 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 658 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 159 | the patch passed | | +1 | findbugs | 586 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 248 | hadoop-hdds in the patch passed. | | -1 | unit | 1689 | hadoop-ozone in the patch failed. | | +1 | asflicense | 36 | The patch does not generate ASF License warnings. | | | | 6585 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.web.client.TestKeys | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.om.TestKeyManagerImpl | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-943/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/943 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux cd17417896f4 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 129576f | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-943/2/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-943/2/testReport/ | | Max. process+thread count | 4169 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/ozone-manager U: hadoop-ozone/ozone-manager | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-943/2/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266219) Time Spent: 0.5h (was: 20m) > Improve logic in openKey when allocating block > -- > > Key: HDDS-1666 > URL: https://issues.apache.org/jira/browse/HDDS-1666 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent:
[jira] [Updated] (HDDS-1696) RocksDB use separate Write-ahead-log location for RocksDB.
[ https://issues.apache.org/jira/browse/HDDS-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-1696: - Labels: pull-request-available (was: ) > RocksDB use separate Write-ahead-log location for RocksDB. > -- > > Key: HDDS-1696 > URL: https://issues.apache.org/jira/browse/HDDS-1696 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > This will help on production systems where WAL logs and db actual data files > will be in a different location. During compaction, it will not affect actual > writes. > > Suggested by [~msingh] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1696) RocksDB use separate Write-ahead-log location for RocksDB.
[ https://issues.apache.org/jira/browse/HDDS-1696?focusedWorklogId=266214=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266214 ] ASF GitHub Bot logged work on HDDS-1696: Author: ASF GitHub Bot Created on: 24/Jun/19 23:31 Start Date: 24/Jun/19 23:31 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #981: HDDS-1696. RocksDB use separate Write-ahead-log location for OM RocksDB. URL: https://github.com/apache/hadoop/pull/981 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266214) Time Spent: 2h (was: 1h 50m) > RocksDB use separate Write-ahead-log location for RocksDB. > -- > > Key: HDDS-1696 > URL: https://issues.apache.org/jira/browse/HDDS-1696 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > This will help on production systems where WAL logs and db actual data files > will be in a different location. During compaction, it will not affect actual > writes. > > Suggested by [~msingh] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1723) Create new OzoneManagerLock class
[ https://issues.apache.org/jira/browse/HDDS-1723?focusedWorklogId=266212=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266212 ] ASF GitHub Bot logged work on HDDS-1723: Author: ASF GitHub Bot Created on: 24/Jun/19 23:31 Start Date: 24/Jun/19 23:31 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1006: HDDS-1723. Create new OzoneManagerLock class. URL: https://github.com/apache/hadoop/pull/1006#issuecomment-505219271 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 29 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 20 | Maven dependency ordering for branch | | +1 | mvninstall | 449 | trunk passed | | +1 | compile | 241 | trunk passed | | +1 | checkstyle | 63 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 784 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 141 | trunk passed | | 0 | spotbugs | 312 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 497 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 28 | Maven dependency ordering for patch | | +1 | mvninstall | 412 | the patch passed | | +1 | compile | 238 | the patch passed | | +1 | javac | 238 | the patch passed | | -0 | checkstyle | 36 | hadoop-ozone: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 611 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 159 | the patch passed | | +1 | findbugs | 531 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 233 | hadoop-hdds in the patch passed. | | -1 | unit | 1058 | hadoop-ozone in the patch failed. | | +1 | asflicense | 49 | The patch does not generate ASF License warnings. | | | | 5743 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1006/9/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1006 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux b89d15c3ec24 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 129576f | | Default Java | 1.8.0_212 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1006/9/artifact/out/diff-checkstyle-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1006/9/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1006/9/testReport/ | | Max. process+thread count | 4563 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-ozone/common U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1006/9/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266212) Time Spent: 3h 40m (was: 3.5h) > Create new OzoneManagerLock class > - > > Key: HDDS-1723 > URL: https://issues.apache.org/jira/browse/HDDS-1723 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Manager >
[jira] [Work logged] (HDDS-1696) RocksDB use separate Write-ahead-log location for RocksDB.
[ https://issues.apache.org/jira/browse/HDDS-1696?focusedWorklogId=266213=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266213 ] ASF GitHub Bot logged work on HDDS-1696: Author: ASF GitHub Bot Created on: 24/Jun/19 23:31 Start Date: 24/Jun/19 23:31 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #981: HDDS-1696. RocksDB use separate Write-ahead-log location for OM RocksDB. URL: https://github.com/apache/hadoop/pull/981#issuecomment-505219413 Closing this, as there is a way to pass a .ini file and read the values and set the required rocksdb. Thank You @anuengineer and @arp7 for the info. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266213) Time Spent: 1h 50m (was: 1h 40m) > RocksDB use separate Write-ahead-log location for RocksDB. > -- > > Key: HDDS-1696 > URL: https://issues.apache.org/jira/browse/HDDS-1696 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > This will help on production systems where WAL logs and db actual data files > will be in a different location. During compaction, it will not affect actual > writes. > > Suggested by [~msingh] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
