[jira] [Updated] (HDFS-12696) BlockPoolManager#startAll is called twice during DataNode startup
[
https://issues.apache.org/jira/browse/HDFS-12696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandakumar updated HDFS-12696:
--
Resolution: Duplicate
Status: Resolved (was: Patch Available)
> BlockPoolManager#startAll is called twice during DataNode startup
> -
>
> Key: HDFS-12696
> URL: https://issues.apache.org/jira/browse/HDFS-12696
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Nandakumar
>Assignee: Nandakumar
>Priority: Minor
> Attachments: HDFS-12696.000.patch
>
>
> As part of Datanode startup, {{BlockPoolManager#startAll}} which starts all
> {{BPServiceActor}} threads is called twice.
> First in {{Datanode}} constructor, {{Datanode#startDataNode}} is called which
> does {{BlockPoolManager#refreshNamenodes}} inside which we do {{startAll}}
> And as part of {{Datanode#runDatanodeDaemon}} we again call
> {{BlockPoolManager#startAll}}.
> Since {{BPServiceActor}} checks if {{bpThread}} is already running, before
> starting them again, the second call is ignored.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-12213) Ozone: Corona: Support for online mode
[ https://issues.apache.org/jira/browse/HDFS-12213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandakumar resolved HDFS-12213. --- Resolution: Fixed > Ozone: Corona: Support for online mode > -- > > Key: HDFS-12213 > URL: https://issues.apache.org/jira/browse/HDFS-12213 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Nandakumar >Assignee: Nandakumar >Priority: Major > Labels: OzonePostMerge, tool > > This jira brings support for online mode in corona. > In online mode, common crawl data from AWS will be used to populate ozone > with data. Default source is [CC-MAIN-2017-17/warc.paths.gz | > https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2017-17/warc.paths.gz] > (it contains the path to actual data segment), user can override this using > -source. > The following values are derived from URL of Common Crawl data > * Domain will be used as Volume > * URL will be used as Bucket > * FileName will be used as Key -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12696) BlockPoolManager#startAll is called twice during DataNode startup
[
https://issues.apache.org/jira/browse/HDFS-12696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandakumar updated HDFS-12696:
--
Status: Patch Available (was: Open)
> BlockPoolManager#startAll is called twice during DataNode startup
> -
>
> Key: HDFS-12696
> URL: https://issues.apache.org/jira/browse/HDFS-12696
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Nandakumar
>Assignee: Nandakumar
>Priority: Minor
> Attachments: HDFS-12696.000.patch
>
>
> As part of Datanode startup, {{BlockPoolManager#startAll}} which starts all
> {{BPServiceActor}} threads is called twice.
> First in {{Datanode}} constructor, {{Datanode#startDataNode}} is called which
> does {{BlockPoolManager#refreshNamenodes}} inside which we do {{startAll}}
> And as part of {{Datanode#runDatanodeDaemon}} we again call
> {{BlockPoolManager#startAll}}.
> Since {{BPServiceActor}} checks if {{bpThread}} is already running, before
> starting them again, the second call is ignored.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12647) DN commands processing should be async
[ https://issues.apache.org/jira/browse/HDFS-12647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandakumar reassigned HDFS-12647: - Assignee: Nandakumar > DN commands processing should be async > -- > > Key: HDFS-12647 > URL: https://issues.apache.org/jira/browse/HDFS-12647 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Affects Versions: 2.8.0 >Reporter: Daryn Sharp >Assignee: Nandakumar > > Due to dataset lock contention, service actors may encounter significant > latency while processing DN commands. Even the queuing of async deletions > require multiple lock acquisitions. A slow disk will cause a backlog of > xceivers instantiating block sender/receivers which starves the actor and > leads to the NN falsely declaring the node dead. > Async processing of all commands will free the actor to perform its primary > purpose of heartbeating and block reporting. Note that FBRs will be > dependent on queued block invalidations not being included in the report. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12519) Ozone: Lease Manager framework
[ https://issues.apache.org/jira/browse/HDFS-12519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandakumar updated HDFS-12519: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Ozone: Lease Manager framework > -- > > Key: HDFS-12519 > URL: https://issues.apache.org/jira/browse/HDFS-12519 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Assignee: Nandakumar > Labels: ozoneMerge > Attachments: HDFS-12519-HDFS-7240.000.patch, > HDFS-12519-HDFS-7240.001.patch, HDFS-12519-HDFS-7240.002.patch, > HDFS-12519-HDFS-7240.003.patch, HDFS-12519-HDFS-7240.003.patch > > > Many objects, including Containers and pipelines can time out during creating > process. We need a way to track these timeouts. This lease Manager allows SCM > to hold a lease on these objects and helps SCM timeout waiting for creating > of these objects. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12519) Ozone: Lease Manager framework
[ https://issues.apache.org/jira/browse/HDFS-12519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201953#comment-16201953 ] Nandakumar commented on HDFS-12519: --- Committed this to HDFS-7240. Thanks [~linyiqun], [~vagarychen] & [~anu] for the review. > Ozone: Lease Manager framework > -- > > Key: HDFS-12519 > URL: https://issues.apache.org/jira/browse/HDFS-12519 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Assignee: Nandakumar > Labels: ozoneMerge > Attachments: HDFS-12519-HDFS-7240.000.patch, > HDFS-12519-HDFS-7240.001.patch, HDFS-12519-HDFS-7240.002.patch, > HDFS-12519-HDFS-7240.003.patch, HDFS-12519-HDFS-7240.003.patch > > > Many objects, including Containers and pipelines can time out during creating > process. We need a way to track these timeouts. This lease Manager allows SCM > to hold a lease on these objects and helps SCM timeout waiting for creating > of these objects. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12519) Ozone: Lease Manager framework
[ https://issues.apache.org/jira/browse/HDFS-12519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandakumar updated HDFS-12519: -- Summary: Ozone: Lease Manager framework (was: Ozone: Add a Lease Manager to SCM) > Ozone: Lease Manager framework > -- > > Key: HDFS-12519 > URL: https://issues.apache.org/jira/browse/HDFS-12519 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Assignee: Nandakumar > Labels: ozoneMerge > Attachments: HDFS-12519-HDFS-7240.000.patch, > HDFS-12519-HDFS-7240.001.patch, HDFS-12519-HDFS-7240.002.patch, > HDFS-12519-HDFS-7240.003.patch, HDFS-12519-HDFS-7240.003.patch > > > Many objects, including Containers and pipelines can time out during creating > process. We need a way to track these timeouts. This lease Manager allows SCM > to hold a lease on these objects and helps SCM timeout waiting for creating > of these objects. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12519) Ozone: Add a Lease Manager to SCM
[ https://issues.apache.org/jira/browse/HDFS-12519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201930#comment-16201930 ] Nandakumar commented on HDFS-12519: --- Test failures are not related, will commit it shortly. > Ozone: Add a Lease Manager to SCM > - > > Key: HDFS-12519 > URL: https://issues.apache.org/jira/browse/HDFS-12519 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Assignee: Nandakumar > Labels: ozoneMerge > Attachments: HDFS-12519-HDFS-7240.000.patch, > HDFS-12519-HDFS-7240.001.patch, HDFS-12519-HDFS-7240.002.patch, > HDFS-12519-HDFS-7240.003.patch, HDFS-12519-HDFS-7240.003.patch > > > Many objects, including Containers and pipelines can time out during creating > process. We need a way to track these timeouts. This lease Manager allows SCM > to hold a lease on these objects and helps SCM timeout waiting for creating > of these objects. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12490) Ozone: OzoneClient: Add creation/modification time information in OzoneVolume/OzoneBucket/OzoneKey
[ https://issues.apache.org/jira/browse/HDFS-12490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201721#comment-16201721 ] Nandakumar commented on HDFS-12490: --- Thanks [~msingh] for updating the patch. +1 on v003, pending Jenkins. > Ozone: OzoneClient: Add creation/modification time information in > OzoneVolume/OzoneBucket/OzoneKey > -- > > Key: HDFS-12490 > URL: https://issues.apache.org/jira/browse/HDFS-12490 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Fix For: HDFS-7240 > > Attachments: HDFS-12490-HDFS-7240.001.patch, > HDFS-12490-HDFS-7240.002.patch, HDFS-12490-HDFS-7240.003.patch > > > OzoneBucket should have information about the bucket creation time. > OzoneFileSystem needs creation time to display the file status information > for the root of the filesystem. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12519) Ozone: Add a Lease Manager to SCM
[ https://issues.apache.org/jira/browse/HDFS-12519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandakumar updated HDFS-12519: -- Attachment: HDFS-12519-HDFS-7240.003.patch Uploading v003 again for jenkins re-trigger. > Ozone: Add a Lease Manager to SCM > - > > Key: HDFS-12519 > URL: https://issues.apache.org/jira/browse/HDFS-12519 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Assignee: Nandakumar > Labels: ozoneMerge > Attachments: HDFS-12519-HDFS-7240.000.patch, > HDFS-12519-HDFS-7240.001.patch, HDFS-12519-HDFS-7240.002.patch, > HDFS-12519-HDFS-7240.003.patch, HDFS-12519-HDFS-7240.003.patch > > > Many objects, including Containers and pipelines can time out during creating > process. We need a way to track these timeouts. This lease Manager allows SCM > to hold a lease on these objects and helps SCM timeout waiting for creating > of these objects. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12490) Ozone: OzoneClient: OzoneBucket/OzoneKey should have information about the bucket/key creation/modification time
[
https://issues.apache.org/jira/browse/HDFS-12490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201510#comment-16201510
]
Nandakumar commented on HDFS-12490:
---
I think we should update the {{modificationTime}} in
{{KeyManagerImpl#commitKey}}. This is when the key is written completely into
ozone and visible for the users to read.
> Ozone: OzoneClient: OzoneBucket/OzoneKey should have information about the
> bucket/key creation/modification time
>
>
> Key: HDFS-12490
> URL: https://issues.apache.org/jira/browse/HDFS-12490
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Fix For: HDFS-7240
>
> Attachments: HDFS-12490-HDFS-7240.001.patch,
> HDFS-12490-HDFS-7240.002.patch
>
>
> OzoneBucket should have information about the bucket creation time.
> OzoneFileSystem needs creation time to display the file status information
> for the root of the filesystem.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12549) Ozone: OzoneClient: Support for REST protocol
[ https://issues.apache.org/jira/browse/HDFS-12549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200918#comment-16200918 ] Nandakumar commented on HDFS-12549: --- Uploaded patch on top of latest changes. > Ozone: OzoneClient: Support for REST protocol > - > > Key: HDFS-12549 > URL: https://issues.apache.org/jira/browse/HDFS-12549 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Nandakumar >Assignee: Nandakumar > Attachments: HDFS-12549-HDFS-7240.000.patch, > HDFS-12549-HDFS-7240.001.patch > > > Support for REST protocol in OzoneClient. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12549) Ozone: OzoneClient: Support for REST protocol
[ https://issues.apache.org/jira/browse/HDFS-12549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandakumar updated HDFS-12549: -- Attachment: HDFS-12549-HDFS-7240.001.patch > Ozone: OzoneClient: Support for REST protocol > - > > Key: HDFS-12549 > URL: https://issues.apache.org/jira/browse/HDFS-12549 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Nandakumar >Assignee: Nandakumar > Attachments: HDFS-12549-HDFS-7240.000.patch, > HDFS-12549-HDFS-7240.001.patch > > > Support for REST protocol in OzoneClient. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12519) Ozone: Add a Lease Manager to SCM
[ https://issues.apache.org/jira/browse/HDFS-12519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200875#comment-16200875 ] Nandakumar commented on HDFS-12519: --- Thanks [~vagarychen] & [~anu] for the review. Comments are addressed in patch v003. > Ozone: Add a Lease Manager to SCM > - > > Key: HDFS-12519 > URL: https://issues.apache.org/jira/browse/HDFS-12519 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Assignee: Nandakumar > Labels: ozoneMerge > Attachments: HDFS-12519-HDFS-7240.000.patch, > HDFS-12519-HDFS-7240.001.patch, HDFS-12519-HDFS-7240.002.patch, > HDFS-12519-HDFS-7240.003.patch > > > Many objects, including Containers and pipelines can time out during creating > process. We need a way to track these timeouts. This lease Manager allows SCM > to hold a lease on these objects and helps SCM timeout waiting for creating > of these objects. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12519) Ozone: Add a Lease Manager to SCM
[ https://issues.apache.org/jira/browse/HDFS-12519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandakumar updated HDFS-12519: -- Attachment: HDFS-12519-HDFS-7240.003.patch > Ozone: Add a Lease Manager to SCM > - > > Key: HDFS-12519 > URL: https://issues.apache.org/jira/browse/HDFS-12519 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Assignee: Nandakumar > Labels: ozoneMerge > Attachments: HDFS-12519-HDFS-7240.000.patch, > HDFS-12519-HDFS-7240.001.patch, HDFS-12519-HDFS-7240.002.patch, > HDFS-12519-HDFS-7240.003.patch > > > Many objects, including Containers and pipelines can time out during creating > process. We need a way to track these timeouts. This lease Manager allows SCM > to hold a lease on these objects and helps SCM timeout waiting for creating > of these objects. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12631) Ozone: ContainerStorageLocation#scmUsage should count only SCM usage
[
https://issues.apache.org/jira/browse/HDFS-12631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandakumar updated HDFS-12631:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Ozone: ContainerStorageLocation#scmUsage should count only SCM usage
> -
>
> Key: HDFS-12631
> URL: https://issues.apache.org/jira/browse/HDFS-12631
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: scm
>Affects Versions: HDFS-7240
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Labels: ozoneMerge
> Attachments: HDFS-12631-HDFS-7240.001.patch,
> HDFS-12631-HDFS-7240.002.patch
>
>
> Currently the ContainerStorageLocation#ContainerStorageLocation does not
> resolve to the actual container prefix before passing it to
> CachingGetSpaceUsed for calculation. As a result of that, if SCM datanode is
> running along with HDFS datanode, the HDFS datanode usage will be included in
> scmUsage too, which is incorrect. This ticket is opened to fix it.
> {code}
> this.scmUsage = new CachingGetSpaceUsed.Builder().setPath(dataDir)
> .setConf(conf)
> .setInitialUsed(loadScmUsed())
> .build();
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12631) Ozone: ContainerStorageLocation#scmUsage should count only SCM usage
[
https://issues.apache.org/jira/browse/HDFS-12631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200156#comment-16200156
]
Nandakumar commented on HDFS-12631:
---
I have committed this to the feature branch, thanks [~xyao] for the
contribution and [~anu] for the review.
> Ozone: ContainerStorageLocation#scmUsage should count only SCM usage
> -
>
> Key: HDFS-12631
> URL: https://issues.apache.org/jira/browse/HDFS-12631
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: scm
>Affects Versions: HDFS-7240
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Labels: ozoneMerge
> Attachments: HDFS-12631-HDFS-7240.001.patch,
> HDFS-12631-HDFS-7240.002.patch
>
>
> Currently the ContainerStorageLocation#ContainerStorageLocation does not
> resolve to the actual container prefix before passing it to
> CachingGetSpaceUsed for calculation. As a result of that, if SCM datanode is
> running along with HDFS datanode, the HDFS datanode usage will be included in
> scmUsage too, which is incorrect. This ticket is opened to fix it.
> {code}
> this.scmUsage = new CachingGetSpaceUsed.Builder().setPath(dataDir)
> .setConf(conf)
> .setInitialUsed(loadScmUsed())
> .build();
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12631) Ozone: ContainerStorageLocation#scmUsage should count only SCM usage
[
https://issues.apache.org/jira/browse/HDFS-12631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200146#comment-16200146
]
Nandakumar commented on HDFS-12631:
---
+1, test case failures are not related. Will commit this shortly.
> Ozone: ContainerStorageLocation#scmUsage should count only SCM usage
> -
>
> Key: HDFS-12631
> URL: https://issues.apache.org/jira/browse/HDFS-12631
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: scm
>Affects Versions: HDFS-7240
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Labels: ozoneMerge
> Attachments: HDFS-12631-HDFS-7240.001.patch,
> HDFS-12631-HDFS-7240.002.patch
>
>
> Currently the ContainerStorageLocation#ContainerStorageLocation does not
> resolve to the actual container prefix before passing it to
> CachingGetSpaceUsed for calculation. As a result of that, if SCM datanode is
> running along with HDFS datanode, the HDFS datanode usage will be included in
> scmUsage too, which is incorrect. This ticket is opened to fix it.
> {code}
> this.scmUsage = new CachingGetSpaceUsed.Builder().setPath(dataDir)
> .setConf(conf)
> .setInitialUsed(loadScmUsed())
> .build();
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12519) Ozone: Add a Lease Manager to SCM
[
https://issues.apache.org/jira/browse/HDFS-12519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199700#comment-16199700
]
Nandakumar commented on HDFS-12519:
---
Thanks [~vagarychen] for the through review.
bq. {{LeaseManager}}, how about adding a heap in addition to activeLeases?
The problem with using {{PriorityQueue}} is that re-ordering doesn't happen if
an element inside queue is modified. We have {{public void renew(long
timeout)}} in {{Lease}} which modifies the timeout value based on which the
ordering will be done. I agree on the performance gain we will get by adding
{{PriorityQueue}}, in that case we might have to remove the support for lease
renewal.
bq. {{LeaseManager#LeaseMonitor#run}} About the interrupt in acquire, will the
following case happen?
Good point.
>>will the lease monitor thread got stopped?
When interrupt is called on a running thread the interrupt flag is set, the
thread won't be stopped. We can check the interrupt flag and perform required
operations.
>>it seems it is possible that whenever leaseMonitor is checking timeout, an
>>acquire call may come in and interrupt leaseMonitor...
This can be handled, will update the patch to handle this situation.
> Ozone: Add a Lease Manager to SCM
> -
>
> Key: HDFS-12519
> URL: https://issues.apache.org/jira/browse/HDFS-12519
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Anu Engineer
>Assignee: Nandakumar
> Labels: ozoneMerge
> Attachments: HDFS-12519-HDFS-7240.000.patch,
> HDFS-12519-HDFS-7240.001.patch, HDFS-12519-HDFS-7240.002.patch
>
>
> Many objects, including Containers and pipelines can time out during creating
> process. We need a way to track these timeouts. This lease Manager allows SCM
> to hold a lease on these objects and helps SCM timeout waiting for creating
> of these objects.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12519) Ozone: Add a Lease Manager to SCM
[
https://issues.apache.org/jira/browse/HDFS-12519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198319#comment-16198319
]
Nandakumar commented on HDFS-12519:
---
[~linyiqun], The reason for having {{leaseMonitorThread.interrupt()}} is
because we support custom timeout in {{acquire(T resource, long timeout)}},
custom timeout value can be set less than default timeout. In this case we need
to re-calculate the sleep time in {{LeaseMonitor}}, because new lease might
have less {{remainingTime}} than the one we calculate previously.
> Ozone: Add a Lease Manager to SCM
> -
>
> Key: HDFS-12519
> URL: https://issues.apache.org/jira/browse/HDFS-12519
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Anu Engineer
>Assignee: Nandakumar
> Labels: OzonePostMerge
> Attachments: HDFS-12519-HDFS-7240.000.patch,
> HDFS-12519-HDFS-7240.001.patch, HDFS-12519-HDFS-7240.002.patch
>
>
> Many objects, including Containers and pipelines can time out during creating
> process. We need a way to track these timeouts. This lease Manager allows SCM
> to hold a lease on these objects and helps SCM timeout waiting for creating
> of these objects.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12609) Ozone: SCM: Refactoring of chill mode logic in NodeManager
[
https://issues.apache.org/jira/browse/HDFS-12609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16196904#comment-16196904
]
Nandakumar commented on HDFS-12609:
---
Thanks [~vagarychen] for the reivew and thanks [~yuanbo] for the review and
commit
> Ozone: SCM: Refactoring of chill mode logic in NodeManager
> --
>
> Key: HDFS-12609
> URL: https://issues.apache.org/jira/browse/HDFS-12609
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nandakumar
>Assignee: Nandakumar
> Fix For: HDFS-7240
>
> Attachments: HDFS-12609-HDFS-7240.000.patch,
> HDFS-12609-HDFS-7240.001.patch
>
>
> As per the discussion in HDFS-12467, this jira is for refactoring chill mode
> logic in SCM's {{NodeManager}}.
> The following methods will be supported to manipulate chill mode status of SCM
> * {{forceExitChillMode}} - will bring SCM out of chill mode (both startup and
> manual)
> * {{enterChillMode}} - will set manual chill mode flag to true
> * {{exitChillMode}} - will set manual chill mode flag to false
> * {{getChillModeStatus}} - will return chill mode status string
> * {{isOutOfChillMode}} - will return true if SCM is out of chill mode, else
> false
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12519) Ozone: Add a Lease Manager to SCM
[ https://issues.apache.org/jira/browse/HDFS-12519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandakumar updated HDFS-12519: -- Attachment: HDFS-12519-HDFS-7240.002.patch > Ozone: Add a Lease Manager to SCM > - > > Key: HDFS-12519 > URL: https://issues.apache.org/jira/browse/HDFS-12519 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Assignee: Nandakumar > Labels: OzonePostMerge > Attachments: HDFS-12519-HDFS-7240.000.patch, > HDFS-12519-HDFS-7240.001.patch, HDFS-12519-HDFS-7240.002.patch > > > Many objects, including Containers and pipelines can time out during creating > process. We need a way to track these timeouts. This lease Manager allows SCM > to hold a lease on these objects and helps SCM timeout waiting for creating > of these objects. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12519) Ozone: Add a Lease Manager to SCM
[
https://issues.apache.org/jira/browse/HDFS-12519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16196104#comment-16196104
]
Nandakumar commented on HDFS-12519:
---
Thanks [~linyiqun] for the feedback.
bq.This comment seems not very clear. LOG.isDebug() is not always needed when
calling LOG.debug. If there are some parameters passed, the check LOG.isDebug()
is not needed.
The reason for having {{LOG.isDebugEnabled()}} is to avoid too many String
operations when debug is not enabled and debug message has parameters.
Consider the following cases:
Case 1:
{{LOG.debug("Starting LeaseManager#LeaseMonitor Thread")}}
Here we actually don't need {{LOG.isDebugEnabled()}}, it will add additional
overhead.
Case 2:
{{LOG.debug("string" + obj + "string" + obj + "string" + obj)}}
In this case there will be {{toString}} calls on the objects used in message
and string concatenation also happens, which might affect the performance. And
these string operations are done even if log level is not set to debug. Here it
makes sense to do {{LOG.isDebugEnabled()}} check.
Performing {{LOG.isDebugEnabled()}} check always will not affect the
performance that much, and in future if someone does change the log message and
make is something like the one in case 2, it will help us.
Anyways, I will remove {{LOG.isDebugEnabled()}} check wherever we are doing
constant string logging. Is this ok?
bq. I am thinking if the method public synchronized Lease acquire(T
resource, long timeout) is needed. Different timeout value resource can just
use separated lease manager to deal with. Multiple timeout required resources
make single lease manager inconveniently to deal with.
I agree on this point, but even if we remove the timeout option while acquiring
lease we will be stuck with the same problem.
Lets assume we are running lease monitor in fixed interval of time
{noformat}
t1 t2 t3t4 t5 t6
t7
|--|--|--|--|--|--|
|| | | ||
A L1 R L1 A L2A L3 R L2 R L3
{noformat}
A L - Acquire Lease
R L - Release Lease
We need to run monitor thread once the lease times out which will not always
fall on our fixed time interval. In the above scenario we will be releasing
lease L1 during t3, but the lease has expired long before, and will be
releasing L2 and L3 at t5 which is not correct.
Sorry if I have confused you more, please let me know if it's not clear.
We can remove the finally part in {{LeaseMonitor#run}} as it's already running
inside while loop.
{code}
finally {
run();
}
{code}
I will update the patch.
> Ozone: Add a Lease Manager to SCM
> -
>
> Key: HDFS-12519
> URL: https://issues.apache.org/jira/browse/HDFS-12519
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Anu Engineer
>Assignee: Nandakumar
> Labels: OzonePostMerge
> Attachments: HDFS-12519-HDFS-7240.000.patch,
> HDFS-12519-HDFS-7240.001.patch
>
>
> Many objects, including Containers and pipelines can time out during creating
> process. We need a way to track these timeouts. This lease Manager allows SCM
> to hold a lease on these objects and helps SCM timeout waiting for creating
> of these objects.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12616) Ozone: SCM: Open containers are not reused for block allocation after restart
[
https://issues.apache.org/jira/browse/HDFS-12616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195791#comment-16195791
]
Nandakumar commented on HDFS-12616:
---
My understanding was that HDFS-12521 is for improving the performance by
caching container info in {{ContainerMapping}} during startup, my bad. I'm ok
with closing either one of them.
> Ozone: SCM: Open containers are not reused for block allocation after restart
> -
>
> Key: HDFS-12616
> URL: https://issues.apache.org/jira/browse/HDFS-12616
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nandakumar
>Assignee: Nandakumar
>
> When SCM is restarted, previously opened containers are not loaded by
> {{ContainerStateManager}}. This causes creation of new container for
> {{BlockManangerImpl#allocateBlock}} call.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12519) Ozone: Add a Lease Manager to SCM
[
https://issues.apache.org/jira/browse/HDFS-12519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195786#comment-16195786
]
Nandakumar commented on HDFS-12519:
---
Thanks for the review [~linyiqun].
bq. line64 and line107: Can use {{Time.now()}} to replace of
{{System.currentTimeMillis()}}?
Used {{Time.monotonicNow()}} instead of {{Time.now()}}, according to
Times.now() java doc
{code}
* Current system time. Do not use this to calculate a duration or interval
* to sleep, because it will be broken by settimeofday. Instead, use
* monotonicNow.
{code}
bq. line32: Log instance seems get incorrect.
This was done intentionally, so that the logging will be done as {{Lease}}.
Please let me know if it's misleading, I'm ok with changing it too.
bq. line65, line70 line154, line193: should add the check {{LOG.isDebug()}}
before invoking {{LOG.debug("")}};
done.
bq. line152: the running flag {{isRunning}} doesn't set to false when shutting
down lease manager.
Thanks for the catch, fixed.
bq. Why not use ScheduledExecutorService and make this as a periodical task?
Periodic execution will not help us here.
Consider the following cases:
1. Service 'A' wants to use LeaseManager with timeout of 60 seconds
2. Service 'B' wants to use LeaseManager with timeout of 10 minutes
We can't decide a proper interval which can suit for all the cases. We can add
a parameter in LeaseManager constructor to get lease check interval, but this
will make the API complex for someone who is using it. Using
ScheduledExecutorService will also bring in the complexity of two Threads
executing {{LeaseMonitor#run}}, one is executed whenever a lease is expired and
the other will be the periodic task. Also with ScheduledExecutorService we will
be unnecessarily executing the monitor periodically even if LeaseManager is
started and never used at all i.e. no lease is obtained.
> Ozone: Add a Lease Manager to SCM
> -
>
> Key: HDFS-12519
> URL: https://issues.apache.org/jira/browse/HDFS-12519
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Anu Engineer
>Assignee: Nandakumar
> Labels: OzonePostMerge
> Attachments: HDFS-12519-HDFS-7240.000.patch,
> HDFS-12519-HDFS-7240.001.patch
>
>
> Many objects, including Containers and pipelines can time out during creating
> process. We need a way to track these timeouts. This lease Manager allows SCM
> to hold a lease on these objects and helps SCM timeout waiting for creating
> of these objects.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12519) Ozone: Add a Lease Manager to SCM
[ https://issues.apache.org/jira/browse/HDFS-12519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandakumar updated HDFS-12519: -- Attachment: HDFS-12519-HDFS-7240.001.patch > Ozone: Add a Lease Manager to SCM > - > > Key: HDFS-12519 > URL: https://issues.apache.org/jira/browse/HDFS-12519 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Assignee: Nandakumar > Labels: OzonePostMerge > Attachments: HDFS-12519-HDFS-7240.000.patch, > HDFS-12519-HDFS-7240.001.patch > > > Many objects, including Containers and pipelines can time out during creating > process. We need a way to track these timeouts. This lease Manager allows SCM > to hold a lease on these objects and helps SCM timeout waiting for creating > of these objects. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12616) Ozone: SCM: Open containers are not reused for block allocation after restart
Nandakumar created HDFS-12616:
-
Summary: Ozone: SCM: Open containers are not reused for block
allocation after restart
Key: HDFS-12616
URL: https://issues.apache.org/jira/browse/HDFS-12616
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone
Reporter: Nandakumar
Assignee: Nandakumar
When SCM is restarted, previously opened containers are not loaded by
{{ContainerStateManager}}. This causes creation of new container for
{{BlockManangerImpl#allocateBlock}} call.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12537) Ozone: Reduce key creation overhead in Corona
[ https://issues.apache.org/jira/browse/HDFS-12537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandakumar updated HDFS-12537: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Ozone: Reduce key creation overhead in Corona > - > > Key: HDFS-12537 > URL: https://issues.apache.org/jira/browse/HDFS-12537 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Lokesh Jain >Assignee: Lokesh Jain > Attachments: HDFS-12537-HDFS-7240.001.patch, > HDFS-12537-HDFS-7240.002.patch, HDFS-12537-HDFS-7240.003.patch, > HDFS-12537-HDFS-7240.004.patch, HDFS-12537-HDFS-7240.005.patch, > HDFS-12537-HDFS-7240.006.patch > > > Currently Corona creates random key values for each key. This creates a lot > of overhead. An option should be provided to use a single key value. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12537) Ozone: Reduce key creation overhead in Corona
[ https://issues.apache.org/jira/browse/HDFS-12537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195627#comment-16195627 ] Nandakumar commented on HDFS-12537: --- Thanks [~ljain] for the contribution and [~anu] for the review. I have committed the patch to feature branch. > Ozone: Reduce key creation overhead in Corona > - > > Key: HDFS-12537 > URL: https://issues.apache.org/jira/browse/HDFS-12537 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Lokesh Jain >Assignee: Lokesh Jain > Attachments: HDFS-12537-HDFS-7240.001.patch, > HDFS-12537-HDFS-7240.002.patch, HDFS-12537-HDFS-7240.003.patch, > HDFS-12537-HDFS-7240.004.patch, HDFS-12537-HDFS-7240.005.patch, > HDFS-12537-HDFS-7240.006.patch > > > Currently Corona creates random key values for each key. This creates a lot > of overhead. An option should be provided to use a single key value. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12537) Ozone: Reduce key creation overhead in Corona
[ https://issues.apache.org/jira/browse/HDFS-12537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195625#comment-16195625 ] Nandakumar commented on HDFS-12537: --- +1 on the latest patch, I will commit this shortly > Ozone: Reduce key creation overhead in Corona > - > > Key: HDFS-12537 > URL: https://issues.apache.org/jira/browse/HDFS-12537 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Lokesh Jain >Assignee: Lokesh Jain > Attachments: HDFS-12537-HDFS-7240.001.patch, > HDFS-12537-HDFS-7240.002.patch, HDFS-12537-HDFS-7240.003.patch, > HDFS-12537-HDFS-7240.004.patch, HDFS-12537-HDFS-7240.005.patch, > HDFS-12537-HDFS-7240.006.patch > > > Currently Corona creates random key values for each key. This creates a lot > of overhead. An option should be provided to use a single key value. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12610) Ozone: OzoneClient: RpcClient list calls throw NPE when iterating over empty list
[ https://issues.apache.org/jira/browse/HDFS-12610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandakumar updated HDFS-12610: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Ozone: OzoneClient: RpcClient list calls throw NPE when iterating over empty > list > - > > Key: HDFS-12610 > URL: https://issues.apache.org/jira/browse/HDFS-12610 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Elek, Marton >Assignee: Nandakumar > Labels: ozoneMerge > Attachments: HDFS-12610-HDFS-7240.000.patch > > > While doing a list on empty volume/bucket and iterating over the result > iterator throws NullPointerException. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12610) Ozone: OzoneClient: RpcClient list calls throw NPE when iterating over empty list
[ https://issues.apache.org/jira/browse/HDFS-12610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195578#comment-16195578 ] Nandakumar commented on HDFS-12610: --- Patch committed to HDFS-7240 branch. > Ozone: OzoneClient: RpcClient list calls throw NPE when iterating over empty > list > - > > Key: HDFS-12610 > URL: https://issues.apache.org/jira/browse/HDFS-12610 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Elek, Marton >Assignee: Nandakumar > Labels: ozoneMerge > Attachments: HDFS-12610-HDFS-7240.000.patch > > > While doing a list on empty volume/bucket and iterating over the result > iterator throws NullPointerException. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12610) Ozone: OzoneClient: RpcClient list calls throw NPE when iterating over empty list
[ https://issues.apache.org/jira/browse/HDFS-12610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195576#comment-16195576 ] Nandakumar commented on HDFS-12610: --- Thanks for the review [~anu]. Test failures are not related, I will commit this shortly. > Ozone: OzoneClient: RpcClient list calls throw NPE when iterating over empty > list > - > > Key: HDFS-12610 > URL: https://issues.apache.org/jira/browse/HDFS-12610 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Elek, Marton >Assignee: Nandakumar > Labels: ozoneMerge > Attachments: HDFS-12610-HDFS-7240.000.patch > > > While doing a list on empty volume/bucket and iterating over the result > iterator throws NullPointerException. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12609) Ozone: SCM: Refactoring of chill mode logic in NodeManager
[
https://issues.apache.org/jira/browse/HDFS-12609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195424#comment-16195424
]
Nandakumar commented on HDFS-12609:
---
Thanks [~vagarychen] for the review.
Updated patch v001 addressing the review comments.
> Ozone: SCM: Refactoring of chill mode logic in NodeManager
> --
>
> Key: HDFS-12609
> URL: https://issues.apache.org/jira/browse/HDFS-12609
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nandakumar
>Assignee: Nandakumar
> Attachments: HDFS-12609-HDFS-7240.000.patch,
> HDFS-12609-HDFS-7240.001.patch
>
>
> As per the discussion in HDFS-12467, this jira is for refactoring chill mode
> logic in SCM's {{NodeManager}}.
> The following methods will be supported to manipulate chill mode status of SCM
> * {{forceExitChillMode}} - will bring SCM out of chill mode (both startup and
> manual)
> * {{enterChillMode}} - will set manual chill mode flag to true
> * {{exitChillMode}} - will set manual chill mode flag to false
> * {{getChillModeStatus}} - will return chill mode status string
> * {{isOutOfChillMode}} - will return true if SCM is out of chill mode, else
> false
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12609) Ozone: SCM: Refactoring of chill mode logic in NodeManager
[
https://issues.apache.org/jira/browse/HDFS-12609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandakumar updated HDFS-12609:
--
Attachment: HDFS-12609-HDFS-7240.001.patch
> Ozone: SCM: Refactoring of chill mode logic in NodeManager
> --
>
> Key: HDFS-12609
> URL: https://issues.apache.org/jira/browse/HDFS-12609
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nandakumar
>Assignee: Nandakumar
> Attachments: HDFS-12609-HDFS-7240.000.patch,
> HDFS-12609-HDFS-7240.001.patch
>
>
> As per the discussion in HDFS-12467, this jira is for refactoring chill mode
> logic in SCM's {{NodeManager}}.
> The following methods will be supported to manipulate chill mode status of SCM
> * {{forceExitChillMode}} - will bring SCM out of chill mode (both startup and
> manual)
> * {{enterChillMode}} - will set manual chill mode flag to true
> * {{exitChillMode}} - will set manual chill mode flag to false
> * {{getChillModeStatus}} - will return chill mode status string
> * {{isOutOfChillMode}} - will return true if SCM is out of chill mode, else
> false
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12610) Ozone: OzoneClient: RpcClient list calls throw NPE when iterating over empty list
[ https://issues.apache.org/jira/browse/HDFS-12610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandakumar updated HDFS-12610: -- Description: While doing a list on empty volume/bucket and iterating over the result iterator throws NullPointerException. (was: While doing a list on empty volume/bucket and iterating over the result iterator throws NullPointerException. Reported by [~elek]) > Ozone: OzoneClient: RpcClient list calls throw NPE when iterating over empty > list > - > > Key: HDFS-12610 > URL: https://issues.apache.org/jira/browse/HDFS-12610 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Elek, Marton >Assignee: Nandakumar > Attachments: HDFS-12610-HDFS-7240.000.patch > > > While doing a list on empty volume/bucket and iterating over the result > iterator throws NullPointerException. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12610) Ozone: OzoneClient: RpcClient list calls throw NPE when iterating over empty list
[ https://issues.apache.org/jira/browse/HDFS-12610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandakumar updated HDFS-12610: -- Status: Patch Available (was: Open) > Ozone: OzoneClient: RpcClient list calls throw NPE when iterating over empty > list > - > > Key: HDFS-12610 > URL: https://issues.apache.org/jira/browse/HDFS-12610 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Nandakumar >Assignee: Nandakumar > Attachments: HDFS-12610-HDFS-7240.000.patch > > > While doing a list on empty volume/bucket and iterating over the result > iterator throws NullPointerException. > Reported by [~elek] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12610) Ozone: OzoneClient: RpcClient list calls throw NPE when iterating over empty list
[ https://issues.apache.org/jira/browse/HDFS-12610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandakumar updated HDFS-12610: -- Attachment: HDFS-12610-HDFS-7240.000.patch > Ozone: OzoneClient: RpcClient list calls throw NPE when iterating over empty > list > - > > Key: HDFS-12610 > URL: https://issues.apache.org/jira/browse/HDFS-12610 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Nandakumar >Assignee: Nandakumar > Attachments: HDFS-12610-HDFS-7240.000.patch > > > While doing a list on empty volume/bucket and iterating over the result > iterator throws NullPointerException. > Reported by [~elek] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12610) Ozone: OzoneClient: RpcClient list calls throw NPE when iterating over empty list
Nandakumar created HDFS-12610: - Summary: Ozone: OzoneClient: RpcClient list calls throw NPE when iterating over empty list Key: HDFS-12610 URL: https://issues.apache.org/jira/browse/HDFS-12610 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Reporter: Nandakumar Assignee: Nandakumar While doing a list on empty volume/bucket and iterating over the result iterator throws NullPointerException. Reported by [~elek] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12609) Ozone: SCM: Refactoring of chill mode logic in NodeManager
[
https://issues.apache.org/jira/browse/HDFS-12609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandakumar updated HDFS-12609:
--
Attachment: HDFS-12609-HDFS-7240.000.patch
> Ozone: SCM: Refactoring of chill mode logic in NodeManager
> --
>
> Key: HDFS-12609
> URL: https://issues.apache.org/jira/browse/HDFS-12609
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nandakumar
>Assignee: Nandakumar
> Attachments: HDFS-12609-HDFS-7240.000.patch
>
>
> As per the discussion in HDFS-12467, this jira is for refactoring chill mode
> logic in SCM's {{NodeManager}}.
> The following methods will be supported to manipulate chill mode status of SCM
> * {{forceExitChillMode}} - will bring SCM out of chill mode (both startup and
> manual)
> * {{enterChillMode}} - will set manual chill mode flag to true
> * {{exitChillMode}} - will set manual chill mode flag to false
> * {{getChillModeStatus}} - will return chill mode status string
> * {{isOutOfChillMode}} - will return true if SCM is out of chill mode, else
> false
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12609) Ozone: SCM: Refactoring of chill mode logic in NodeManager
[
https://issues.apache.org/jira/browse/HDFS-12609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandakumar updated HDFS-12609:
--
Status: Patch Available (was: Open)
> Ozone: SCM: Refactoring of chill mode logic in NodeManager
> --
>
> Key: HDFS-12609
> URL: https://issues.apache.org/jira/browse/HDFS-12609
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nandakumar
>Assignee: Nandakumar
> Attachments: HDFS-12609-HDFS-7240.000.patch
>
>
> As per the discussion in HDFS-12467, this jira is for refactoring chill mode
> logic in SCM's {{NodeManager}}.
> The following methods will be supported to manipulate chill mode status of SCM
> * {{forceExitChillMode}} - will bring SCM out of chill mode (both startup and
> manual)
> * {{enterChillMode}} - will set manual chill mode flag to true
> * {{exitChillMode}} - will set manual chill mode flag to false
> * {{getChillModeStatus}} - will return chill mode status string
> * {{isOutOfChillMode}} - will return true if SCM is out of chill mode, else
> false
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12467) Ozone: SCM: NodeManager should log when it comes out of chill mode
[
https://issues.apache.org/jira/browse/HDFS-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195173#comment-16195173
]
Nandakumar commented on HDFS-12467:
---
HDFS-12609 is created for refactoring chill mode logic.
> Ozone: SCM: NodeManager should log when it comes out of chill mode
> --
>
> Key: HDFS-12467
> URL: https://issues.apache.org/jira/browse/HDFS-12467
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nandakumar
>Assignee: Nandakumar
>Priority: Minor
> Fix For: HDFS-7240
>
> Attachments: HDFS-12467-HDFS-7240.000.patch,
> HDFS-12467-HDFS-7240.001.patch, HDFS-12467-HDFS-7240.002.patch,
> HDFS-12467-HDFS-7240.003.patch, HDFS-12467-HDFS-7240.004.patch
>
>
> {{NodeManager}} should add a log message when it comes out of chill mode.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12609) Ozone: SCM: Refactoring of chill mode logic in NodeManager
Nandakumar created HDFS-12609:
-
Summary: Ozone: SCM: Refactoring of chill mode logic in NodeManager
Key: HDFS-12609
URL: https://issues.apache.org/jira/browse/HDFS-12609
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone
Reporter: Nandakumar
Assignee: Nandakumar
As per the discussion in HDFS-12467, this jira is for refactoring chill mode
logic in SCM's {{NodeManager}}.
The following methods will be supported to manipulate chill mode status of SCM
* {{forceExitChillMode}} - will bring SCM out of chill mode (both startup and
manual)
* {{enterChillMode}} - will set manual chill mode flag to true
* {{exitChillMode}} - will set manual chill mode flag to false
* {{getChillModeStatus}} - will return chill mode status string
* {{isOutOfChillMode}} - will return true if SCM is out of chill mode, else
false
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12588) Use GenericOptionsParser for scm and ksm daemon
[
https://issues.apache.org/jira/browse/HDFS-12588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194832#comment-16194832
]
Nandakumar commented on HDFS-12588:
---
Thanks [~elek] for updating.
The patch looks good to me, some minor NITs.
*KeySpaceManager.java*
Line 21 & 24: Unused imports, please remove them.
Line 201: {{System.err.println("USAGE: hdfs ksm \[genericOptions\]\n");}} can
be changed to {{System.err.println("USAGE: " + USAGE + "\n");}}
*StorageContainerManager.java*
Line 38: Unused import, can be removed
Line 328: {{System.err.println(USAGE + "\n");}} can be change to
{{System.err.println("USAGE: " + USAGE + "\n");}}
> Use GenericOptionsParser for scm and ksm daemon
> ---
>
> Key: HDFS-12588
> URL: https://issues.apache.org/jira/browse/HDFS-12588
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
> Attachments: HDFS-12588-HDFS-7240.001.patch,
> HDFS-12588-HDFS-7240.002.patch
>
>
> Most of the hadoop commands use the GenericOptionsParser to use some common
> CLI arguments (such as -conf or -D or -libjars to define configuration/modify
> configuration/modify classpath).
> I suggest to use the same common options to scm and ksm daemons as well, as:
> 1. It allows to use the existing cluster management tools/scripts as the
> daemons could be configured in the same way as namenode and datanode
> 2. It follows the convention from the hadoop common.
> 3. It's easier to develop from the IDE (I start the ksm/scm/datanode/namenode
> from intellij but I need to add the configuration to the classpath. With
> -conf I would able to use external configration.)
> I found one problem during the implementation. Until now we used `hdfs scm`
> command both for the daemon and the scm command line client. If there were no
> parameters the daemon is started, with parameters the cli is started. The
> help listed only the damon.
> The -conf (GenericOptionParser) could be used only if we separate the scm and
> scmcli commands. But any way, it's a more clean and visible if we have
> separated `hdfs scm` and `hdfs scmcli`.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12519) Ozone: Add a Lease Manager to SCM
[ https://issues.apache.org/jira/browse/HDFS-12519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandakumar updated HDFS-12519: -- Status: Patch Available (was: Open) > Ozone: Add a Lease Manager to SCM > - > > Key: HDFS-12519 > URL: https://issues.apache.org/jira/browse/HDFS-12519 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Assignee: Nandakumar > Labels: OzonePostMerge > Attachments: HDFS-12519-HDFS-7240.000.patch > > > Many objects, including Containers and pipelines can time out during creating > process. We need a way to track these timeouts. This lease Manager allows SCM > to hold a lease on these objects and helps SCM timeout waiting for creating > of these objects. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12519) Ozone: Add a Lease Manager to SCM
[ https://issues.apache.org/jira/browse/HDFS-12519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandakumar updated HDFS-12519: -- Attachment: HDFS-12519-HDFS-7240.000.patch > Ozone: Add a Lease Manager to SCM > - > > Key: HDFS-12519 > URL: https://issues.apache.org/jira/browse/HDFS-12519 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Assignee: Nandakumar > Labels: OzonePostMerge > Attachments: HDFS-12519-HDFS-7240.000.patch > > > Many objects, including Containers and pipelines can time out during creating > process. We need a way to track these timeouts. This lease Manager allows SCM > to hold a lease on these objects and helps SCM timeout waiting for creating > of these objects. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12543) Ozone : allow create key without specifying size
[ https://issues.apache.org/jira/browse/HDFS-12543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194218#comment-16194218 ] Nandakumar commented on HDFS-12543: --- Patch committed to HDFS-7240 branch. Thanks [~vagarychen] for the contribution and [~xyao] for the review. > Ozone : allow create key without specifying size > > > Key: HDFS-12543 > URL: https://issues.apache.org/jira/browse/HDFS-12543 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang > Labels: ozoneMerge > Attachments: HDFS-12543-HDFS-7240.001.patch, > HDFS-12543-HDFS-7240.002.patch, HDFS-12543-HDFS-7240.003.patch, > HDFS-12543-HDFS-7240.004.patch, HDFS-12543-HDFS-7240.005.patch, > HDFS-12543-HDFS-7240.006.patch, HDFS-12543-HDFS-7240.007.patch, > HDFS-12543-HDFS-7240.008.patch, HDFS-12543-HDFS-7240.009.patch, > HDFS-12543-HDFS-7240.010.patch, HDFS-12543-HDFS-7240.011.patch, > HDFS-12543-HDFS-7240.012.patch > > > Currently when creating a key, it is required to specify the total size of > the key. This makes it inconvenient for the case where a key is created and > data keeps coming and being appended. This JIRA is remove the requirement of > specifying the size on key creation, and allows appending to the key > indefinitely. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12543) Ozone : allow create key without specifying size
[ https://issues.apache.org/jira/browse/HDFS-12543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194206#comment-16194206 ] Nandakumar edited comment on HDFS-12543 at 10/6/17 6:31 AM: +1 for latest patch, thanks [~vagarychen] for updating it. Test failures are not related, I will commit this shortly. was (Author: nandakumar131): +1 for latest patch, thanks [~vagarychen] for updating it. I will commit it shortly. > Ozone : allow create key without specifying size > > > Key: HDFS-12543 > URL: https://issues.apache.org/jira/browse/HDFS-12543 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang > Labels: ozoneMerge > Attachments: HDFS-12543-HDFS-7240.001.patch, > HDFS-12543-HDFS-7240.002.patch, HDFS-12543-HDFS-7240.003.patch, > HDFS-12543-HDFS-7240.004.patch, HDFS-12543-HDFS-7240.005.patch, > HDFS-12543-HDFS-7240.006.patch, HDFS-12543-HDFS-7240.007.patch, > HDFS-12543-HDFS-7240.008.patch, HDFS-12543-HDFS-7240.009.patch, > HDFS-12543-HDFS-7240.010.patch, HDFS-12543-HDFS-7240.011.patch, > HDFS-12543-HDFS-7240.012.patch > > > Currently when creating a key, it is required to specify the total size of > the key. This makes it inconvenient for the case where a key is created and > data keeps coming and being appended. This JIRA is remove the requirement of > specifying the size on key creation, and allows appending to the key > indefinitely. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12543) Ozone : allow create key without specifying size
[ https://issues.apache.org/jira/browse/HDFS-12543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194206#comment-16194206 ] Nandakumar commented on HDFS-12543: --- +1 for latest patch, thanks [~vagarychen] for updating it. I will commit it shortly. > Ozone : allow create key without specifying size > > > Key: HDFS-12543 > URL: https://issues.apache.org/jira/browse/HDFS-12543 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang > Labels: ozoneMerge > Attachments: HDFS-12543-HDFS-7240.001.patch, > HDFS-12543-HDFS-7240.002.patch, HDFS-12543-HDFS-7240.003.patch, > HDFS-12543-HDFS-7240.004.patch, HDFS-12543-HDFS-7240.005.patch, > HDFS-12543-HDFS-7240.006.patch, HDFS-12543-HDFS-7240.007.patch, > HDFS-12543-HDFS-7240.008.patch, HDFS-12543-HDFS-7240.009.patch, > HDFS-12543-HDFS-7240.010.patch, HDFS-12543-HDFS-7240.011.patch, > HDFS-12543-HDFS-7240.012.patch > > > Currently when creating a key, it is required to specify the total size of > the key. This makes it inconvenient for the case where a key is created and > data keeps coming and being appended. This JIRA is remove the requirement of > specifying the size on key creation, and allows appending to the key > indefinitely. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12543) Ozone : allow create key without specifying size
[ https://issues.apache.org/jira/browse/HDFS-12543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193501#comment-16193501 ] Nandakumar commented on HDFS-12543: --- Thanks [~vagarychen] for updating the patch. v011 looks good to me. +1 pending Jenkins. > Ozone : allow create key without specifying size > > > Key: HDFS-12543 > URL: https://issues.apache.org/jira/browse/HDFS-12543 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang > Labels: ozoneMerge > Attachments: HDFS-12543-HDFS-7240.001.patch, > HDFS-12543-HDFS-7240.002.patch, HDFS-12543-HDFS-7240.003.patch, > HDFS-12543-HDFS-7240.004.patch, HDFS-12543-HDFS-7240.005.patch, > HDFS-12543-HDFS-7240.006.patch, HDFS-12543-HDFS-7240.007.patch, > HDFS-12543-HDFS-7240.008.patch, HDFS-12543-HDFS-7240.009.patch, > HDFS-12543-HDFS-7240.010.patch, HDFS-12543-HDFS-7240.011.patch > > > Currently when creating a key, it is required to specify the total size of > the key. This makes it inconvenient for the case where a key is created and > data keeps coming and being appended. This JIRA is remove the requirement of > specifying the size on key creation, and allows appending to the key > indefinitely. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12537) Ozone: Reduce key creation overhead in Corona
[
https://issues.apache.org/jira/browse/HDFS-12537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16191549#comment-16191549
]
Nandakumar commented on HDFS-12537:
---
Thanks [~ljain] for updating the patch.
In *Corona.java*
If {{threadThroughput}} is just for holding the thread's throughput, you can
use a simple list and make it thread safe with
{{Collections.synchronizedList()}} instead of using {{BlockingQueue}}
Line: 325
This
{code}
jsonDir = cmdLine.hasOption(JSON_WRITE_DIRECTORY) ?
cmdLine.getOptionValue(JSON_WRITE_DIRECTORY) : null;
{code}
can be replaced with
{code}
jsonDir = cmdLine.getOptionValue(JSON_WRITE_DIRECTORY);
{code}
{{CommandLine#getOptionValue}} returns the Value of argument if option is set,
and has an argument, otherwise null.
Line 412: There is no need to assign {{keyValue}} to the local variable
{{value}}, keyValue can be directly used in {{os.write(keyValue)}}
Line 428: Incomplete value is added for validation, this will cause the
validation of writes to fail. Since the size of value can be huge we can use
checksum to optimize data validation, this can be done in follow up jira. For
now you can add the complete value for validation.
> Ozone: Reduce key creation overhead in Corona
> -
>
> Key: HDFS-12537
> URL: https://issues.apache.org/jira/browse/HDFS-12537
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Lokesh Jain
>Assignee: Lokesh Jain
> Attachments: HDFS-12537-HDFS-7240.001.patch,
> HDFS-12537-HDFS-7240.002.patch, HDFS-12537-HDFS-7240.003.patch
>
>
> Currently Corona creates random key values for each key. This creates a lot
> of overhead. An option should be provided to use a single key value.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12588) Use GenericOptionsParser for scm and ksm daemon
[ https://issues.apache.org/jira/browse/HDFS-12588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16191400#comment-16191400 ] Nandakumar commented on HDFS-12588: --- Thanks [~elek] for reporting this and working on it. The patch is not applying against latest HDFS-7240 branch. Can you please rebase and update the patch. > Use GenericOptionsParser for scm and ksm daemon > --- > > Key: HDFS-12588 > URL: https://issues.apache.org/jira/browse/HDFS-12588 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton > Attachments: HDFS-12588-HDFS-7240.001.patch > > > Most of the hadoop commands use the GenericOptionsParser to use some common > CLI arguments (such as -conf or -D or -libjars to define configuration/modify > configuration/modify classpath). > I suggest to use the same common options to scm and ksm daemons as well, as: > 1. It allows to use the existing cluster management tools/scripts as the > daemons could be configured in the same way as namenode and datanode > 2. It follows the convention from the hadoop common. > 3. It's easier to develop from the IDE (I start the ksm/scm/datanode/namenode > from intellij but I need to add the configuration to the classpath. With > -conf I would able to use external configration.) > I found one problem during the implementation. Until now we used `hdfs scm` > command both for the daemon and the scm command line client. If there were no > parameters the daemon is started, with parameters the cli is started. The > help listed only the damon. > The -conf (GenericOptionParser) could be used only if we separate the scm and > scmcli commands. But any way, it's a more clean and visible if we have > separated `hdfs scm` and `hdfs scmcli`. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12543) Ozone : allow create key without specifying size
[
https://issues.apache.org/jira/browse/HDFS-12543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16191348#comment-16191348
]
Nandakumar commented on HDFS-12543:
---
Thanks [~vagarychen] for working on this, the patch looks pretty good to me.
Few comments:
bq. It might be better to remove size completely from {{KsmKeyArgs}} I would
like to leave it as a separate JIRA after this is done.
If the plan is to remove size from {{KsmKeyArgs}}, can we also remove the
dependency of size from {{KeyManagerImpl#openKey}}. I understand the
optimization done with key size, but if we are going to remove it later why
depend on it now? I don't have any cleaner approach to suggest for optimizing
block allocation for big keys.
*KeyManagerImpl.java*
{{commitKey}}: we have to update actual size of the key during commit.
Line 201: Even for a 0 length key we are initially updating the size as
{{scmBlockSize}} which is not necessary, we can set the size to 0.
Line 137: The log message can be changed, we are not actually committing the
key here.
*KeySpaceManager.java*
Line 497: {{metrics.incNumBlockAllocates()}} this doesn't give us the actual
number of blocks allocated, since we are also allocating blocks as part of
{{openKey}}. Can we change this metrics name from {{numBlockAllocate}} to
{{numAllocateBlockCalls}}?
> Ozone : allow create key without specifying size
>
>
> Key: HDFS-12543
> URL: https://issues.apache.org/jira/browse/HDFS-12543
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Labels: ozoneMerge
> Attachments: HDFS-12543-HDFS-7240.001.patch,
> HDFS-12543-HDFS-7240.002.patch, HDFS-12543-HDFS-7240.003.patch,
> HDFS-12543-HDFS-7240.004.patch, HDFS-12543-HDFS-7240.005.patch,
> HDFS-12543-HDFS-7240.006.patch, HDFS-12543-HDFS-7240.007.patch,
> HDFS-12543-HDFS-7240.008.patch, HDFS-12543-HDFS-7240.009.patch
>
>
> Currently when creating a key, it is required to specify the total size of
> the key. This makes it inconvenient for the case where a key is created and
> data keeps coming and being appended. This JIRA is remove the requirement of
> specifying the size on key creation, and allows appending to the key
> indefinitely.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12038) Ozone: Non-admin user is unable to run InfoVolume to the volume owned by itself
[
https://issues.apache.org/jira/browse/HDFS-12038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190992#comment-16190992
]
Nandakumar commented on HDFS-12038:
---
Thanks [~anu] for the ping, and thanks [~ljain] for taking this up and working
on it.
As of KSM's current state we don't have any authorization mechanism in place,
i.e we don't do authorization on any client calls. Authorization of
createVolume calls are done in OzoneHandler's {{VolumeHandler}} (datanode REST
server), this is not an ideal place to do it as RPC clients will bypass this.
We have to authorize all the calls made to KSM in {{KeySpaceManager}}, which
can be done in another jira.
For this issue we should properly set {{client.setUserAuth(userName)}} which is
not happening in first place; If {{-root}} is not specified we are setting
UserAuth as null and HTTP header {{Authorization}} is not set in the HttpGet
request which is causing the issue.
As pointed out by [~cheersyang], we have to remove line 89
{code}
client.setUserAuth(rootName);
{code}
Additionally we can add logic in
{{VolumeProcessTemplate#getVolumeInfoResponse}} to check if the user is admin
or owner of the volume, with this we can make sure that unauthorized user
doesn't have access to InfoVolume calls. Still with RPC client anyone can make
any calls.
> Ozone: Non-admin user is unable to run InfoVolume to the volume owned by
> itself
> ---
>
> Key: HDFS-12038
> URL: https://issues.apache.org/jira/browse/HDFS-12038
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Lokesh Jain
> Labels: OzonePostMerge
> Attachments: HDFS-12038-HDFS-7240.001.patch
>
>
> Reproduce steps
> 1. Create a volume with a non-admin user
> {code}
> hdfs oz -createVolume http://ozone1.fyre.ibm.com:9864/volume-wwei-0 -user
> wwei -root -quota 2TB
> {code}
> 2. Run infoVolume command to get this volume info
> {noformat}
> hdfs oz -infoVolume http://ozone1.fyre.ibm.com:9864/volume-wwei-0 -user wwei
> Command Failed :
> {"httpCode":400,"shortMessage":"badAuthorization","resource":null,"message":"Missing
> authorization or authorization has to be
> unique.","requestID":"221efb47-72b9-498d-ac19-907257428573","hostName":"ozone1.fyre.ibm.com"}
> {noformat}
> add {{-root}} to run as admin user could bypass this issue
> {noformat}
> hdfs oz -infoVolume http://ozone1.fyre.ibm.com:9864/volume-wwei-0 -user wwei
> -root
> {
> "owner" : {
> "name" : "wwei"
> },
> "quota" : {
> "unit" : "TB",
> "size" : 2
> },
> "volumeName" : "volume-wwei-0",
> "createdOn" : null,
> "createdBy" : "hdfs"
> }
> {noformat}
> expecting: both volume owner and admin should be able to run infoVolume
> command.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12519) Ozone: Add a Lease Manager to SCM
[ https://issues.apache.org/jira/browse/HDFS-12519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandakumar reassigned HDFS-12519: - Assignee: Nandakumar (was: Anu Engineer) > Ozone: Add a Lease Manager to SCM > - > > Key: HDFS-12519 > URL: https://issues.apache.org/jira/browse/HDFS-12519 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Assignee: Nandakumar > Labels: OzonePostMerge > > Many objects, including Containers and pipelines can time out during creating > process. We need a way to track these timeouts. This lease Manager allows SCM > to hold a lease on these objects and helps SCM timeout waiting for creating > of these objects. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12467) Ozone: SCM: NodeManager should log when it comes out of chill mode
[
https://issues.apache.org/jira/browse/HDFS-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16187456#comment-16187456
]
Nandakumar commented on HDFS-12467:
---
Thanks [~vagarychen] for the detailed description. I agree with whatever you
have said.
Previously there was no way to bring SCM out of startup chill mode with any API
calls, it happened only through datanode registration. I used
{{forceExitChillMode}} for this, now with forceExitChillMode() the system will
come out of both the chill modes.
I would like to propose the following changes for chillMode (refactor the whole
chill mode logic)
The following methods will be supported to manipulate chill mode status of SCM
* {{forceExitChillMode}} - will bring SCM out of chill mode (both startup and
manual)
* {{enterChillMode}} - will set manual chill mode flag to true
* {{exitChillMode}} - will set manual chill mode flag to false
* {{getChillModeStatus}} - will return chill mode status string
* {{inChillMode}} - will return true if SCM is in chill mode, else false
The following methods will be removed
* {{clearChillModeFlag}}
* {{forceEnterChillMode}}
* {{isInManualChillMode}}
* {{isOutOfNodeChillMode}}
The getter and setter for MinimumChillModeNodes will remain as it is
* {{setMinimumChillModeNodes}}
* {{getMinimumChillModeNodes}}
This will make it easy to understand and use.
> Ozone: SCM: NodeManager should log when it comes out of chill mode
> --
>
> Key: HDFS-12467
> URL: https://issues.apache.org/jira/browse/HDFS-12467
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nandakumar
>Assignee: Nandakumar
>Priority: Minor
> Attachments: HDFS-12467-HDFS-7240.000.patch,
> HDFS-12467-HDFS-7240.001.patch, HDFS-12467-HDFS-7240.002.patch,
> HDFS-12467-HDFS-7240.003.patch, HDFS-12467-HDFS-7240.004.patch
>
>
> {{NodeManager}} should add a log message when it comes out of chill mode.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12467) Ozone: SCM: NodeManager should log when it comes out of chill mode
[
https://issues.apache.org/jira/browse/HDFS-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16186153#comment-16186153
]
Nandakumar commented on HDFS-12467:
---
Thanks [~vagarychen] for the update, fixed the test case issue in patch v004.
The following cases are covered in the test case
* Still in chill mode, waiting on nodes to report in. (Startup chill mode)
* Out of chill mode.
* Out of startup chill mode, but in manual chill mode. (Manual chill mode)
* Out of chill mode.
> Ozone: SCM: NodeManager should log when it comes out of chill mode
> --
>
> Key: HDFS-12467
> URL: https://issues.apache.org/jira/browse/HDFS-12467
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nandakumar
>Assignee: Nandakumar
>Priority: Minor
> Attachments: HDFS-12467-HDFS-7240.000.patch,
> HDFS-12467-HDFS-7240.001.patch, HDFS-12467-HDFS-7240.002.patch,
> HDFS-12467-HDFS-7240.003.patch, HDFS-12467-HDFS-7240.004.patch
>
>
> {{NodeManager}} should add a log message when it comes out of chill mode.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12467) Ozone: SCM: NodeManager should log when it comes out of chill mode
[
https://issues.apache.org/jira/browse/HDFS-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandakumar updated HDFS-12467:
--
Attachment: HDFS-12467-HDFS-7240.004.patch
> Ozone: SCM: NodeManager should log when it comes out of chill mode
> --
>
> Key: HDFS-12467
> URL: https://issues.apache.org/jira/browse/HDFS-12467
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nandakumar
>Assignee: Nandakumar
>Priority: Minor
> Attachments: HDFS-12467-HDFS-7240.000.patch,
> HDFS-12467-HDFS-7240.001.patch, HDFS-12467-HDFS-7240.002.patch,
> HDFS-12467-HDFS-7240.003.patch, HDFS-12467-HDFS-7240.004.patch
>
>
> {{NodeManager}} should add a log message when it comes out of chill mode.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12537) Ozone: Reduce key creation overhead in Corona
[
https://issues.apache.org/jira/browse/HDFS-12537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185746#comment-16185746
]
Nandakumar commented on HDFS-12537:
---
Thanks [~ljain] for working on this. Some minor comments.
1. Do we really need a flag for {{optimizedWrites}}? we can have this as
default behavior. Since we are adding {{UUID.randomUUID()}} to the data it
becomes unique. This randomness should be enough for testing. We can remove
that option and have it as default behavior, it will make life of the user more
simple.
2. +1 for [~anu]'s suggestion on having date and time of execution in json
output.
3. Current json file name format is something like {{2017-09-29
17:25:15,717+0530}}, can we change it to {{MMDDHHMMSS}}?
3. "HH:mm:ss,SSS" can be defined as static final variable.
4: line 734: null check has to be done on {{os}} before {{os.close()}}
> Ozone: Reduce key creation overhead in Corona
> -
>
> Key: HDFS-12537
> URL: https://issues.apache.org/jira/browse/HDFS-12537
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Lokesh Jain
>Assignee: Lokesh Jain
> Attachments: HDFS-12537-HDFS-7240.001.patch,
> HDFS-12537-HDFS-7240.002.patch
>
>
> Currently Corona creates random key values for each key. This creates a lot
> of overhead. An option should be provided to use a single key value.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12060) Ozone: OzoneClient: Add list calls in RpcClient
[
https://issues.apache.org/jira/browse/HDFS-12060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185697#comment-16185697
]
Nandakumar edited comment on HDFS-12060 at 9/29/17 11:14 AM:
-
Thanks [~xyao] for the review. updated patch v003 adressing review comments.
bq. Do we need to update the KSM CLI to use these new list APIs for better
performance?
Once HDFS-12549 is done, I will create a new jira to make all the {{oz}}
commands to use new API. we have HDFS-12550 to track the removal of old Rest
client.
bq. At interface level, I would like to have a maxListResult as a parameter to
the API instead of assuming ozone.key.cache.size. This applies to the other
list APIs as well.
APIs have been changed.
bq. prevKey should be prevBucket
Done
bq. Do we limit the maximum number bucket returned per listBuckets call with
ozone.key.cache.size?
Yeah, we do. Updated the javadoc
bq. we should use the UGI#getShortUserName()
Thanks for the catch. fixed it in new patch.
bq. If we could make the change in the API with the additional parameter, we
will use the ozone.key.cache.size only if the maxListResult is not specified.
Now maxListResult is a mandatory parameter, we use {{ozone.client.list.cache}}
in {{ObjectStore}}, {{OzoneVolume}} and {{OzoneBucket}} to calculate
maxListResult.
was (Author: nandakumar131):
Thanks [~xyao] for the review. updated patch v003 adressing review comments.
bq. Do we need to update the KSM CLI to use these new list APIs for better
performance?
Once HDFS-12549 is done, I will create a new jira to make all the {{oz}}
commands to use new API. we have HDFS-12550 to track the removal of old Rest
client.
bq. At interface level, I would like to have a maxListResult as a parameter to
the API instead of assuming ozone.key.cache.size. This applies to the other
list APIs as well.
APIs have been changed.
bq. prevKey should be prevBucket
Done
bq. Do we limit the maximum number bucket returned per listBuckets call with
ozone.key.cache.size?
Yeah, we do. Updated the javadoc
bq. we should use the UGI#getShortUserName()
Thanks for the catch. fixed it in new patch.
bq. If we could make the change in the API with the additional parameter, we
will use the ozone.key.cache.size only if the maxListResult is not specified.
Now maxListResult is a mandatory parameter, we use {{ozone.key.cache.size}} in
{{ObjectStore}}, {{OzoneVolume}} and {{OzoneBucket}} to calculate maxListResult.
> Ozone: OzoneClient: Add list calls in RpcClient
> ---
>
> Key: HDFS-12060
> URL: https://issues.apache.org/jira/browse/HDFS-12060
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Nandakumar
>Assignee: Nandakumar
> Labels: ozoneMerge
> Attachments: HDFS-12060-HDFS-7240.000.patch,
> HDFS-12060-HDFS-7240.001.patch, HDFS-12060-HDFS-7240.002.patch,
> HDFS-12060-HDFS-7240.003.patch
>
>
> Support for {{listVolumes}}, {{listBuckets}}, {{listKeys}} in {{RpcClient}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12060) Ozone: OzoneClient: Add list calls in RpcClient
[
https://issues.apache.org/jira/browse/HDFS-12060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185697#comment-16185697
]
Nandakumar commented on HDFS-12060:
---
Thanks [~xyao] for the review. updated patch v003 adressing review comments.
bq. Do we need to update the KSM CLI to use these new list APIs for better
performance?
Once HDFS-12549 is done, I will create a new jira to make all the {{oz}}
commands to use new API. we have HDFS-12550 to track the removal of old Rest
client.
bq. At interface level, I would like to have a maxListResult as a parameter to
the API instead of assuming ozone.key.cache.size. This applies to the other
list APIs as well.
APIs have been changed.
bq. prevKey should be prevBucket
Done
bq. Do we limit the maximum number bucket returned per listBuckets call with
ozone.key.cache.size?
Yeah, we do. Updated the javadoc
bq. we should use the UGI#getShortUserName()
Thanks for the catch. fixed it in new patch.
bq. If we could make the change in the API with the additional parameter, we
will use the ozone.key.cache.size only if the maxListResult is not specified.
Now maxListResult is a mandatory parameter, we use {{ozone.key.cache.size}} in
{{ObjectStore}}, {{OzoneVolume}} and {{OzoneBucket}} to calculate maxListResult.
> Ozone: OzoneClient: Add list calls in RpcClient
> ---
>
> Key: HDFS-12060
> URL: https://issues.apache.org/jira/browse/HDFS-12060
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Nandakumar
>Assignee: Nandakumar
> Labels: ozoneMerge
> Attachments: HDFS-12060-HDFS-7240.000.patch,
> HDFS-12060-HDFS-7240.001.patch, HDFS-12060-HDFS-7240.002.patch,
> HDFS-12060-HDFS-7240.003.patch
>
>
> Support for {{listVolumes}}, {{listBuckets}}, {{listKeys}} in {{RpcClient}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12060) Ozone: OzoneClient: Add list calls in RpcClient
[
https://issues.apache.org/jira/browse/HDFS-12060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandakumar updated HDFS-12060:
--
Attachment: HDFS-12060-HDFS-7240.003.patch
> Ozone: OzoneClient: Add list calls in RpcClient
> ---
>
> Key: HDFS-12060
> URL: https://issues.apache.org/jira/browse/HDFS-12060
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Nandakumar
>Assignee: Nandakumar
> Labels: ozoneMerge
> Attachments: HDFS-12060-HDFS-7240.000.patch,
> HDFS-12060-HDFS-7240.001.patch, HDFS-12060-HDFS-7240.002.patch,
> HDFS-12060-HDFS-7240.003.patch
>
>
> Support for {{listVolumes}}, {{listBuckets}}, {{listKeys}} in {{RpcClient}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12467) Ozone: SCM: NodeManager should log when it comes out of chill mode
[
https://issues.apache.org/jira/browse/HDFS-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185429#comment-16185429
]
Nandakumar commented on HDFS-12467:
---
Thanks [~vagarychen] & [~anu] for the review and suggestion. I have updated
patch v003 addressing the review comment.
> Ozone: SCM: NodeManager should log when it comes out of chill mode
> --
>
> Key: HDFS-12467
> URL: https://issues.apache.org/jira/browse/HDFS-12467
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nandakumar
>Assignee: Nandakumar
>Priority: Minor
> Attachments: HDFS-12467-HDFS-7240.000.patch,
> HDFS-12467-HDFS-7240.001.patch, HDFS-12467-HDFS-7240.002.patch,
> HDFS-12467-HDFS-7240.003.patch
>
>
> {{NodeManager}} should add a log message when it comes out of chill mode.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12467) Ozone: SCM: NodeManager should log when it comes out of chill mode
[
https://issues.apache.org/jira/browse/HDFS-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandakumar updated HDFS-12467:
--
Attachment: HDFS-12467-HDFS-7240.003.patch
> Ozone: SCM: NodeManager should log when it comes out of chill mode
> --
>
> Key: HDFS-12467
> URL: https://issues.apache.org/jira/browse/HDFS-12467
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nandakumar
>Assignee: Nandakumar
>Priority: Minor
> Attachments: HDFS-12467-HDFS-7240.000.patch,
> HDFS-12467-HDFS-7240.001.patch, HDFS-12467-HDFS-7240.002.patch,
> HDFS-12467-HDFS-7240.003.patch
>
>
> {{NodeManager}} should add a log message when it comes out of chill mode.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12549) Ozone: OzoneClient: Support for REST protocol
[ https://issues.apache.org/jira/browse/HDFS-12549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandakumar updated HDFS-12549: -- Status: Patch Available (was: Open) > Ozone: OzoneClient: Support for REST protocol > - > > Key: HDFS-12549 > URL: https://issues.apache.org/jira/browse/HDFS-12549 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Nandakumar >Assignee: Nandakumar > Attachments: HDFS-12549-HDFS-7240.000.patch > > > Support for REST protocol in OzoneClient. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12549) Ozone: OzoneClient: Support for REST protocol
[ https://issues.apache.org/jira/browse/HDFS-12549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandakumar updated HDFS-12549: -- Attachment: HDFS-12549-HDFS-7240.000.patch Uploading initial version of patch. > Ozone: OzoneClient: Support for REST protocol > - > > Key: HDFS-12549 > URL: https://issues.apache.org/jira/browse/HDFS-12549 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Nandakumar >Assignee: Nandakumar > Attachments: HDFS-12549-HDFS-7240.000.patch > > > Support for REST protocol in OzoneClient. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12060) Ozone: OzoneClient: Add list calls in RpcClient
[
https://issues.apache.org/jira/browse/HDFS-12060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandakumar updated HDFS-12060:
--
Attachment: HDFS-12060-HDFS-7240.002.patch
Uploading patch v002 after rebase.
> Ozone: OzoneClient: Add list calls in RpcClient
> ---
>
> Key: HDFS-12060
> URL: https://issues.apache.org/jira/browse/HDFS-12060
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Nandakumar
>Assignee: Nandakumar
> Labels: ozoneMerge
> Attachments: HDFS-12060-HDFS-7240.000.patch,
> HDFS-12060-HDFS-7240.001.patch, HDFS-12060-HDFS-7240.002.patch
>
>
> Support for {{listVolumes}}, {{listBuckets}}, {{listKeys}} in {{RpcClient}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12525) Ozone: OzoneClient: Verify bucket/volume name in create calls
[
https://issues.apache.org/jira/browse/HDFS-12525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181084#comment-16181084
]
Nandakumar commented on HDFS-12525:
---
Thanks [~linyiqun] for the review.
I have updated the patch based on your comment and also created HDFS-12550 to
track the removal of old {{OzoneRestClient}}
> Ozone: OzoneClient: Verify bucket/volume name in create calls
> -
>
> Key: HDFS-12525
> URL: https://issues.apache.org/jira/browse/HDFS-12525
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nandakumar
>Assignee: Nandakumar
> Labels: ozoneMerge
> Attachments: HDFS-12525-HDFS-7240.000.patch,
> HDFS-12525-HDFS-7240.000.patch, HDFS-12525-HDFS-7240.001.patch
>
>
> The new OzoneClient API has to verify bucket/volume name during creation
> call. Volume/Bucket name shouldn't support any special characters other {{.}}
> and {{-}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12550) Ozone: OzoneClient: Removal of old OzoneRestClient
Nandakumar created HDFS-12550: - Summary: Ozone: OzoneClient: Removal of old OzoneRestClient Key: HDFS-12550 URL: https://issues.apache.org/jira/browse/HDFS-12550 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Reporter: Nandakumar Assignee: Nandakumar Once new REST based OzoneClient is read, the old OzoneRestClient can be removed. This jira is to track it. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12549) Ozone: OzoneClient: Support for REST protocol
Nandakumar created HDFS-12549: - Summary: Ozone: OzoneClient: Support for REST protocol Key: HDFS-12549 URL: https://issues.apache.org/jira/browse/HDFS-12549 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Reporter: Nandakumar Assignee: Nandakumar Support for REST protocol in OzoneClient. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12525) Ozone: OzoneClient: Verify bucket/volume name in create calls
[
https://issues.apache.org/jira/browse/HDFS-12525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandakumar updated HDFS-12525:
--
Attachment: HDFS-12525-HDFS-7240.001.patch
> Ozone: OzoneClient: Verify bucket/volume name in create calls
> -
>
> Key: HDFS-12525
> URL: https://issues.apache.org/jira/browse/HDFS-12525
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nandakumar
>Assignee: Nandakumar
> Labels: ozoneMerge
> Attachments: HDFS-12525-HDFS-7240.000.patch,
> HDFS-12525-HDFS-7240.000.patch, HDFS-12525-HDFS-7240.001.patch
>
>
> The new OzoneClient API has to verify bucket/volume name during creation
> call. Volume/Bucket name shouldn't support any special characters other {{.}}
> and {{-}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11613) Ozone: Cleanup findbugs issues
[
https://issues.apache.org/jira/browse/HDFS-11613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16180335#comment-16180335
]
Nandakumar commented on HDFS-11613:
---
[~msingh], the findbug issue in {{RpcClient}} is handled/fixed in HDFS-12017.
> Ozone: Cleanup findbugs issues
> --
>
> Key: HDFS-11613
> URL: https://issues.apache.org/jira/browse/HDFS-11613
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Mukul Kumar Singh
>Priority: Blocker
> Labels: ozoneMerge
> Attachments: HDFS-11613-HDFS-7240.001.patch
>
>
> Some of the ozone checkins happened before Findbugs started running on test
> files. This will cause issues when we attempt to merge with trunk. This jira
> tracks cleaning up all Findbugs issue under ozone.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12525) Ozone: OzoneClient: Verify bucket/volume name in create calls
[
https://issues.apache.org/jira/browse/HDFS-12525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178557#comment-16178557
]
Nandakumar commented on HDFS-12525:
---
Thanks [~linyiqun] and [~anu] for the review.
I agree that we have to have one function to verify the names. The one in
hadoop-hdfs {{OzoneUtils}} is used by the old {{OzoneRestClient}}, which we are
planning to remove once new REST based {{OzoneClient}} is ready (based on
HDFS-12385). The plan is to remove the {{OzoneUtils#verifyResourceName}} while
removing the old {{OzoneRestClient}}. Removing it now will result in a bigger
patch for this jira with changes in all files inside old rest client.
> Ozone: OzoneClient: Verify bucket/volume name in create calls
> -
>
> Key: HDFS-12525
> URL: https://issues.apache.org/jira/browse/HDFS-12525
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nandakumar
>Assignee: Nandakumar
> Labels: ozoneMerge
> Attachments: HDFS-12525-HDFS-7240.000.patch,
> HDFS-12525-HDFS-7240.000.patch
>
>
> The new OzoneClient API has to verify bucket/volume name during creation
> call. Volume/Bucket name shouldn't support any special characters other {{.}}
> and {{-}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12525) Ozone: OzoneClient: Verify bucket/volume name in create calls
[
https://issues.apache.org/jira/browse/HDFS-12525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandakumar updated HDFS-12525:
--
Attachment: HDFS-12525-HDFS-7240.000.patch
Uploading patch to re-trigger build.
> Ozone: OzoneClient: Verify bucket/volume name in create calls
> -
>
> Key: HDFS-12525
> URL: https://issues.apache.org/jira/browse/HDFS-12525
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nandakumar
>Assignee: Nandakumar
> Labels: ozoneMerge
> Attachments: HDFS-12525-HDFS-7240.000.patch,
> HDFS-12525-HDFS-7240.000.patch
>
>
> The new OzoneClient API has to verify bucket/volume name during creation
> call. Volume/Bucket name shouldn't support any special characters other {{.}}
> and {{-}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11897) Ozone: KSM: Changing log level for client calls in KSM
[ https://issues.apache.org/jira/browse/HDFS-11897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandakumar updated HDFS-11897: -- Resolution: Fixed Fix Version/s: HDFS-7240 Target Version/s: HDFS-7240 Status: Resolved (was: Patch Available) > Ozone: KSM: Changing log level for client calls in KSM > -- > > Key: HDFS-11897 > URL: https://issues.apache.org/jira/browse/HDFS-11897 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Nandakumar >Assignee: Shashikant Banerjee > Labels: ozoneMerge > Fix For: HDFS-7240 > > Attachments: HDFS-11897-HDFS-7240.001.patch, > HDFS-11897-HDFS-7240.002.patch, HDFS-11897-HDFS-7240.003.patch > > > Whenever there is no Volume/Bucker/Key found in MetadataDB for a client call, > KSM logs ERROR which is not necessary. The level of these log messages can be > changed to DEBUG, which will be helpful in debugging. > Changes are to be made in the following classes > * VolumeManagerImpl > * BucketManagerImpl > * KeyManagerImpl -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11897) Ozone: KSM: Changing log level for client calls in KSM
[ https://issues.apache.org/jira/browse/HDFS-11897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176564#comment-16176564 ] Nandakumar commented on HDFS-11897: --- Thanks [~shashikant] for the contribution. I've commit the patch to feature branch. > Ozone: KSM: Changing log level for client calls in KSM > -- > > Key: HDFS-11897 > URL: https://issues.apache.org/jira/browse/HDFS-11897 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Nandakumar >Assignee: Shashikant Banerjee > Labels: ozoneMerge > Fix For: HDFS-7240 > > Attachments: HDFS-11897-HDFS-7240.001.patch, > HDFS-11897-HDFS-7240.002.patch, HDFS-11897-HDFS-7240.003.patch > > > Whenever there is no Volume/Bucker/Key found in MetadataDB for a client call, > KSM logs ERROR which is not necessary. The level of these log messages can be > changed to DEBUG, which will be helpful in debugging. > Changes are to be made in the following classes > * VolumeManagerImpl > * BucketManagerImpl > * KeyManagerImpl -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11897) Ozone: KSM: Changing log level for client calls in KSM
[
https://issues.apache.org/jira/browse/HDFS-11897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176426#comment-16176426
]
Nandakumar commented on HDFS-11897:
---
Thanks [~shashikant] for working on this. +1, the patch looks good to me. I
will commit this shortly.
I will take care of the minor typo in VolumeManagerImpl:145 while committing.
Based on the offline discussion, the log level is retained as {{error}} since
the logging will happen only if there is any problem while reading metadata
from db.
> Ozone: KSM: Changing log level for client calls in KSM
> --
>
> Key: HDFS-11897
> URL: https://issues.apache.org/jira/browse/HDFS-11897
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Nandakumar
>Assignee: Shashikant Banerjee
> Labels: ozoneMerge
> Attachments: HDFS-11897-HDFS-7240.001.patch,
> HDFS-11897-HDFS-7240.002.patch, HDFS-11897-HDFS-7240.003.patch
>
>
> Whenever there is no Volume/Bucker/Key found in MetadataDB for a client call,
> KSM logs ERROR which is not necessary. The level of these log messages can be
> changed to DEBUG, which will be helpful in debugging.
> Changes are to be made in the following classes
> * VolumeManagerImpl
> * BucketManagerImpl
> * KeyManagerImpl
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12525) Ozone: OzoneClient: Verify bucket/volume name in create calls
[
https://issues.apache.org/jira/browse/HDFS-12525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandakumar updated HDFS-12525:
--
Status: Patch Available (was: Open)
> Ozone: OzoneClient: Verify bucket/volume name in create calls
> -
>
> Key: HDFS-12525
> URL: https://issues.apache.org/jira/browse/HDFS-12525
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nandakumar
>Assignee: Nandakumar
> Labels: ozoneMerge
> Attachments: HDFS-12525-HDFS-7240.000.patch
>
>
> The new OzoneClient API has to verify bucket/volume name during creation
> call. Volume/Bucket name shouldn't support any special characters other {{.}}
> and {{-}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12525) Ozone: OzoneClient: Verify bucket/volume name in create calls
[
https://issues.apache.org/jira/browse/HDFS-12525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandakumar updated HDFS-12525:
--
Attachment: HDFS-12525-HDFS-7240.000.patch
> Ozone: OzoneClient: Verify bucket/volume name in create calls
> -
>
> Key: HDFS-12525
> URL: https://issues.apache.org/jira/browse/HDFS-12525
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nandakumar
>Assignee: Nandakumar
> Labels: ozoneMerge
> Attachments: HDFS-12525-HDFS-7240.000.patch
>
>
> The new OzoneClient API has to verify bucket/volume name during creation
> call. Volume/Bucket name shouldn't support any special characters other {{.}}
> and {{-}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11897) Ozone: KSM: Changing log level for client calls in KSM
[
https://issues.apache.org/jira/browse/HDFS-11897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174677#comment-16174677
]
Nandakumar commented on HDFS-11897:
---
Thanks [~shashikant] for working on this, please find my comments below.
The log level in catch block can be changed from {{error}} to {{warn}}
*BucketManagerImpl*
217 & 218: Incorrect indentation
*VolumeManagerImpl*
220: Typo in the log message
260: debug log needs to be added
268: Incorrect indentation
268: handling of {{ex instanceof KSMException}} has to be done
290: debug log to be added
294: debug log to be added
331: debug log to be added
339: handling of {{ex instanceof KSMException}} has to be done
> Ozone: KSM: Changing log level for client calls in KSM
> --
>
> Key: HDFS-11897
> URL: https://issues.apache.org/jira/browse/HDFS-11897
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Nandakumar
>Assignee: Shashikant Banerjee
> Labels: ozoneMerge
> Attachments: HDFS-11897-HDFS-7240.001.patch
>
>
> Whenever there is no Volume/Bucker/Key found in MetadataDB for a client call,
> KSM logs ERROR which is not necessary. The level of these log messages can be
> changed to DEBUG, which will be helpful in debugging.
> Changes are to be made in the following classes
> * VolumeManagerImpl
> * BucketManagerImpl
> * KeyManagerImpl
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12525) Ozone: OzoneClient: Verify bucket/volume name in create calls
[
https://issues.apache.org/jira/browse/HDFS-12525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandakumar updated HDFS-12525:
--
Labels: ozoneMerge (was: )
> Ozone: OzoneClient: Verify bucket/volume name in create calls
> -
>
> Key: HDFS-12525
> URL: https://issues.apache.org/jira/browse/HDFS-12525
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nandakumar
>Assignee: Nandakumar
> Labels: ozoneMerge
>
> The new OzoneClient API has to verify bucket/volume name during creation
> call. Volume/Bucket name shouldn't support any special characters other {{.}}
> and {{-}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12506) Ozone: ListBucket is too slow
[
https://issues.apache.org/jira/browse/HDFS-12506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174503#comment-16174503
]
Nandakumar commented on HDFS-12506:
---
Created HDFS-12525 to verify volume/bucket name in OzoneClient.
> Ozone: ListBucket is too slow
> -
>
> Key: HDFS-12506
> URL: https://issues.apache.org/jira/browse/HDFS-12506
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Blocker
> Labels: ozoneMerge, performance
> Attachments: HDFS-12506-HDFS-7240.001.patch
>
>
> Generated 3 million keys in ozone, and run {{listBucket}} command to get a
> list of buckets under a volume,
> {code}
> bin/hdfs oz -listBucket http://15oz1.fyre.ibm.com:9864/vol-0-15143 -user wwei
> {code}
> this call spent over *15 seconds* to finish. The problem was caused by the
> inflexible structure of KSM DB. Right now {{ksm.db}} stores keys like
> following
> {code}
> /v1/b1
> /v1/b1/k1
> /v1/b1/k2
> /v1/b1/k3
> /v1/b2
> /v1/b2/k1
> /v1/b2/k2
> /v1/b2/k3
> /v1/b3
> /v1/b4
> {code}
> keys are sorted in nature order so when we do list buckets under a volume e.g
> /v1, we need to seek to /v1 point and start to iterate and filter keys, this
> ends up with scanning all keys under volume /v1. The problem with this design
> is we don't have an efficient approach to locate all buckets without scanning
> the keys.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12525) Ozone: OzoneClient: Verify bucket/volume name in create calls
Nandakumar created HDFS-12525:
-
Summary: Ozone: OzoneClient: Verify bucket/volume name in create
calls
Key: HDFS-12525
URL: https://issues.apache.org/jira/browse/HDFS-12525
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone
Reporter: Nandakumar
Assignee: Nandakumar
The new OzoneClient API has to verify bucket/volume name during creation call.
Volume/Bucket name shouldn't support any special characters other {{.}} and
{{-}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12506) Ozone: ListBucket is too slow
[
https://issues.apache.org/jira/browse/HDFS-12506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174501#comment-16174501
]
Nandakumar commented on HDFS-12506:
---
[~cheersyang], [~linyiqun]
bq. With such prefix definition, we will need to avoid user from adding volumes
with name like "#volumeName", bucket with name like "#bucketName".
We don't support any special character other than "." and "-" in bucket/volume
name, it is handled in {{OzoneUtils#verifyResourceName}}.
This is not the case with new OzoneClient implementation (HDFS-12385), I will
file a jira to add this behavior in it. Thanks for bringing it up.
> Ozone: ListBucket is too slow
> -
>
> Key: HDFS-12506
> URL: https://issues.apache.org/jira/browse/HDFS-12506
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Blocker
> Labels: ozoneMerge, performance
> Attachments: HDFS-12506-HDFS-7240.001.patch
>
>
> Generated 3 million keys in ozone, and run {{listBucket}} command to get a
> list of buckets under a volume,
> {code}
> bin/hdfs oz -listBucket http://15oz1.fyre.ibm.com:9864/vol-0-15143 -user wwei
> {code}
> this call spent over *15 seconds* to finish. The problem was caused by the
> inflexible structure of KSM DB. Right now {{ksm.db}} stores keys like
> following
> {code}
> /v1/b1
> /v1/b1/k1
> /v1/b1/k2
> /v1/b1/k3
> /v1/b2
> /v1/b2/k1
> /v1/b2/k2
> /v1/b2/k3
> /v1/b3
> /v1/b4
> {code}
> keys are sorted in nature order so when we do list buckets under a volume e.g
> /v1, we need to seek to /v1 point and start to iterate and filter keys, this
> ends up with scanning all keys under volume /v1. The problem with this design
> is we don't have an efficient approach to locate all buckets without scanning
> the keys.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12490) Ozone: OzoneClient: OzoneBucket should have information about the bucket creation time
[ https://issues.apache.org/jira/browse/HDFS-12490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173624#comment-16173624 ] Nandakumar commented on HDFS-12490: --- Thanks [~msingh] for working on this. +1, The patch looks good to me. > Ozone: OzoneClient: OzoneBucket should have information about the bucket > creation time > -- > > Key: HDFS-12490 > URL: https://issues.apache.org/jira/browse/HDFS-12490 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Fix For: HDFS-7240 > > Attachments: HDFS-12490-HDFS-7240.001.patch > > > OzoneBucket should have information about the bucket creation time. > OzoneFileSystem needs creation time to display the file status information > for the root of the filesystem. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12506) Ozone: ListBucket is too slow
[
https://issues.apache.org/jira/browse/HDFS-12506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173430#comment-16173430
]
Nandakumar edited comment on HDFS-12506 at 9/20/17 4:19 PM:
+1 for [~xyao]'s idea, I was also thinking of the same.
One small change though
For Volume
/#v1
For Bucket
/v1/#b1
Keys can be stored as they are stored now
With this we can iterate and get list of volumes without iterating over
buckets, and get list of buckets without iterating over keys.
Something like
{code}
/#v1
/#v2
/#v3
/v1/#b1
/v1/#b2
/v2/#b1
/v3/#b1
/v1/b1/k1
/v2/b2/k2
{code}
was (Author: nandakumar131):
+1 for [~xyao]'s idea, I was also thinking of the same.
One small change though
For Volume
/#v1
For Bucket
/v1/#b1
Keys can be stored as they are stored now
With this we can iterate and get list of volumes without iterating over
buckets, and get list of buckets without iterating over keys.
Something lime
{code}
/#v1
/#v2
/#v3
/v1/#b1
/v1/#b2
/v2/#b1
/v3/#b1
/v1/b1/k1
/v2/b2/k2
{code}
> Ozone: ListBucket is too slow
> -
>
> Key: HDFS-12506
> URL: https://issues.apache.org/jira/browse/HDFS-12506
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Priority: Blocker
> Labels: ozoneMerge
>
> Generated 3 million keys in ozone, and run {{listBucket}} command to get a
> list of buckets under a volume,
> {code}
> bin/hdfs oz -listBucket http://15oz1.fyre.ibm.com:9864/vol-0-15143 -user wwei
> {code}
> this call spent over *15 seconds* to finish. The problem was caused by the
> inflexible structure of KSM DB. Right now {{ksm.db}} stores keys like
> following
> {code}
> /v1/b1
> /v1/b1/k1
> /v1/b1/k2
> /v1/b1/k3
> /v1/b2
> /v1/b2/k1
> /v1/b2/k2
> /v1/b2/k3
> /v1/b3
> /v1/b4
> {code}
> keys are sorted in nature order so when we do list buckets under a volume e.g
> /v1, we need to seek to /v1 point and start to iterate and filter keys, this
> ends up with scanning all keys under volume /v1. The problem with this design
> is we don't have an efficient approach to locate all buckets without scanning
> the keys.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12506) Ozone: ListBucket is too slow
[
https://issues.apache.org/jira/browse/HDFS-12506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173430#comment-16173430
]
Nandakumar commented on HDFS-12506:
---
+1 for [~xyao]'s idea, I was also thinking of the same.
One small change though
For Volume
/#v1
For Bucket
/v1/#b1
Keys can be stored as they are stored now
With this we can iterate and get list of volumes without iterating over
buckets, and get list of buckets without iterating over keys.
Something lime
{code}
/#v1
/#v2
/#v3
/v1/#b1
/v1/#b2
/v2/#b1
/v3/#b1
/v1/b1/k1
/v2/b2/k2
{code}
> Ozone: ListBucket is too slow
> -
>
> Key: HDFS-12506
> URL: https://issues.apache.org/jira/browse/HDFS-12506
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Priority: Blocker
> Labels: ozoneMerge
>
> Generated 3 million keys in ozone, and run {{listBucket}} command to get a
> list of buckets under a volume,
> {code}
> bin/hdfs oz -listBucket http://15oz1.fyre.ibm.com:9864/vol-0-15143 -user wwei
> {code}
> this call spent over *15 seconds* to finish. The problem was caused by the
> inflexible structure of KSM DB. Right now {{ksm.db}} stores keys like
> following
> {code}
> /v1/b1
> /v1/b1/k1
> /v1/b1/k2
> /v1/b1/k3
> /v1/b2
> /v1/b2/k1
> /v1/b2/k2
> /v1/b2/k3
> /v1/b3
> /v1/b4
> {code}
> keys are sorted in nature order so when we do list buckets under a volume e.g
> /v1, we need to seek to /v1 point and start to iterate and filter keys, this
> ends up with scanning all keys under volume /v1. The problem with this design
> is we don't have an efficient approach to locate all buckets without scanning
> the keys.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12466) Ozone: KSM: Make "ozone.ksm.address" as mandatory property for client
[
https://issues.apache.org/jira/browse/HDFS-12466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172705#comment-16172705
]
Nandakumar commented on HDFS-12466:
---
[~cheersyang], the movement of create {{keySpaceManagerClient}} was intentional.
There is an idea of bringing service discovery API in KSM, from which we can
get details about other services in the cluster. In that case we need to
connect to KSM first. This change will make it easier to do that later,
whenever we develop service discovery API.
> Ozone: KSM: Make "ozone.ksm.address" as mandatory property for client
> -
>
> Key: HDFS-12466
> URL: https://issues.apache.org/jira/browse/HDFS-12466
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nandakumar
>Assignee: Nandakumar
>Priority: Minor
> Attachments: HDFS-12466-HDFS-7240.000.patch,
> HDFS-12466-HDFS-7240.001.patch
>
>
> This jira makes {{ozone.ksm.address}} as a mandatory property at client side.
> User has to explicitly configure it now. We should not use default
> {{0.0.0.0}} if the value is not present, throw exception instead.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12017) Ozone: Container: Move IPC port to 98xx range
[ https://issues.apache.org/jira/browse/HDFS-12017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172226#comment-16172226 ] Nandakumar edited comment on HDFS-12017 at 9/19/17 9:43 PM: Thanks for the update [~xyao], fixed it in patch v002. was (Author: nandakumar131): Thanks for the update [~xyao], fix it in patch v002. > Ozone: Container: Move IPC port to 98xx range > - > > Key: HDFS-12017 > URL: https://issues.apache.org/jira/browse/HDFS-12017 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Nandakumar > Labels: ozoneMerge > Attachments: HDFS-12017-HDFS-7240.000.patch, > HDFS-12017-HDFS-7240.001.patch, HDFS-12017-HDFS-7240.002.patch > > > We use port 50011 port -- This is an old format of choosing port. Hadoop 3 > has already moved to port ranges under 98xx. In fact all of KSM/SCM and > CBlock is using 98xx range. We should move 50011 to a port under 98xx. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12060) Ozone: OzoneClient: Add list calls in RpcClient
[
https://issues.apache.org/jira/browse/HDFS-12060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172367#comment-16172367
]
Nandakumar commented on HDFS-12060:
---
Patch updated.
> Ozone: OzoneClient: Add list calls in RpcClient
> ---
>
> Key: HDFS-12060
> URL: https://issues.apache.org/jira/browse/HDFS-12060
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Nandakumar
>Assignee: Nandakumar
> Labels: ozoneMerge
> Attachments: HDFS-12060-HDFS-7240.000.patch,
> HDFS-12060-HDFS-7240.001.patch
>
>
> Support for {{listVolumes}}, {{listBuckets}}, {{listKeys}} in {{RpcClient}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12060) Ozone: OzoneClient: Add list calls in RpcClient
[
https://issues.apache.org/jira/browse/HDFS-12060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandakumar updated HDFS-12060:
--
Status: Patch Available (was: Open)
> Ozone: OzoneClient: Add list calls in RpcClient
> ---
>
> Key: HDFS-12060
> URL: https://issues.apache.org/jira/browse/HDFS-12060
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Nandakumar
>Assignee: Nandakumar
> Labels: ozoneMerge
> Attachments: HDFS-12060-HDFS-7240.000.patch,
> HDFS-12060-HDFS-7240.001.patch
>
>
> Support for {{listVolumes}}, {{listBuckets}}, {{listKeys}} in {{RpcClient}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12060) Ozone: OzoneClient: Add list calls in RpcClient
[
https://issues.apache.org/jira/browse/HDFS-12060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandakumar updated HDFS-12060:
--
Attachment: HDFS-12060-HDFS-7240.001.patch
> Ozone: OzoneClient: Add list calls in RpcClient
> ---
>
> Key: HDFS-12060
> URL: https://issues.apache.org/jira/browse/HDFS-12060
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Nandakumar
>Assignee: Nandakumar
> Labels: ozoneMerge
> Attachments: HDFS-12060-HDFS-7240.000.patch,
> HDFS-12060-HDFS-7240.001.patch
>
>
> Support for {{listVolumes}}, {{listBuckets}}, {{listKeys}} in {{RpcClient}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12466) Ozone: KSM: Make "ozone.ksm.address" as mandatory property for client
[
https://issues.apache.org/jira/browse/HDFS-12466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172228#comment-16172228
]
Nandakumar commented on HDFS-12466:
---
findbug issue is fixed as part of HDFS-12017.
> Ozone: KSM: Make "ozone.ksm.address" as mandatory property for client
> -
>
> Key: HDFS-12466
> URL: https://issues.apache.org/jira/browse/HDFS-12466
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nandakumar
>Assignee: Nandakumar
>Priority: Minor
> Attachments: HDFS-12466-HDFS-7240.000.patch,
> HDFS-12466-HDFS-7240.001.patch
>
>
> This jira makes {{ozone.ksm.address}} as a mandatory property at client side.
> User has to explicitly configure it now. We should not use default
> {{0.0.0.0}} if the value is not present, throw exception instead.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12017) Ozone: Container: Move IPC port to 98xx range
[ https://issues.apache.org/jira/browse/HDFS-12017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandakumar updated HDFS-12017: -- Attachment: HDFS-12017-HDFS-7240.002.patch > Ozone: Container: Move IPC port to 98xx range > - > > Key: HDFS-12017 > URL: https://issues.apache.org/jira/browse/HDFS-12017 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Nandakumar > Labels: ozoneMerge > Attachments: HDFS-12017-HDFS-7240.000.patch, > HDFS-12017-HDFS-7240.001.patch, HDFS-12017-HDFS-7240.002.patch > > > We use port 50011 port -- This is an old format of choosing port. Hadoop 3 > has already moved to port ranges under 98xx. In fact all of KSM/SCM and > CBlock is using 98xx range. We should move 50011 to a port under 98xx. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12017) Ozone: Container: Move IPC port to 98xx range
[ https://issues.apache.org/jira/browse/HDFS-12017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172226#comment-16172226 ] Nandakumar commented on HDFS-12017: --- Thanks for the update [~xyao], fix it in patch v002. > Ozone: Container: Move IPC port to 98xx range > - > > Key: HDFS-12017 > URL: https://issues.apache.org/jira/browse/HDFS-12017 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Nandakumar > Labels: ozoneMerge > Attachments: HDFS-12017-HDFS-7240.000.patch, > HDFS-12017-HDFS-7240.001.patch, HDFS-12017-HDFS-7240.002.patch > > > We use port 50011 port -- This is an old format of choosing port. Hadoop 3 > has already moved to port ranges under 98xx. In fact all of KSM/SCM and > CBlock is using 98xx range. We should move 50011 to a port under 98xx. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12060) Ozone: OzoneClient: Add list calls in RpcClient
[
https://issues.apache.org/jira/browse/HDFS-12060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandakumar updated HDFS-12060:
--
Description: Support for {{listVolumes}}, {{listBuckets}}, {{listKeys}} in
{{RpcClient}} (was: Support for {{listVolumes}}, {{listBuckets}}, {{listKeys}}
in {{OzoneClient}})
> Ozone: OzoneClient: Add list calls in RpcClient
> ---
>
> Key: HDFS-12060
> URL: https://issues.apache.org/jira/browse/HDFS-12060
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Nandakumar
>Assignee: Nandakumar
> Labels: ozoneMerge
> Attachments: HDFS-12060-HDFS-7240.000.patch
>
>
> Support for {{listVolumes}}, {{listBuckets}}, {{listKeys}} in {{RpcClient}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12060) Ozone: OzoneClient: Add list calls in RpcClient
[
https://issues.apache.org/jira/browse/HDFS-12060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandakumar updated HDFS-12060:
--
Summary: Ozone: OzoneClient: Add list calls in RpcClient (was: Ozone:
OzoneClient: Add list calls)
> Ozone: OzoneClient: Add list calls in RpcClient
> ---
>
> Key: HDFS-12060
> URL: https://issues.apache.org/jira/browse/HDFS-12060
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Nandakumar
>Assignee: Nandakumar
> Labels: ozoneMerge
> Attachments: HDFS-12060-HDFS-7240.000.patch
>
>
> Support for {{listVolumes}}, {{listBuckets}}, {{listKeys}} in {{OzoneClient}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12017) Ozone: Container: Move IPC port to 98xx range
[
https://issues.apache.org/jira/browse/HDFS-12017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171887#comment-16171887
]
Nandakumar commented on HDFS-12017:
---
Thanks for the review [~xyao]. I have addressed all the review comments in
patch v001.
Now {{dfs.container.ratis.ipc}} will use 9858.
> Ozone: Container: Move IPC port to 98xx range
> -
>
> Key: HDFS-12017
> URL: https://issues.apache.org/jira/browse/HDFS-12017
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Nandakumar
> Labels: ozoneMerge
> Attachments: HDFS-12017-HDFS-7240.000.patch,
> HDFS-12017-HDFS-7240.001.patch
>
>
> We use port 50011 port -- This is an old format of choosing port. Hadoop 3
> has already moved to port ranges under 98xx. In fact all of KSM/SCM and
> CBlock is using 98xx range. We should move 50011 to a port under 98xx.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12017) Ozone: Container: Move IPC port to 98xx range
[ https://issues.apache.org/jira/browse/HDFS-12017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandakumar updated HDFS-12017: -- Attachment: HDFS-12017-HDFS-7240.001.patch > Ozone: Container: Move IPC port to 98xx range > - > > Key: HDFS-12017 > URL: https://issues.apache.org/jira/browse/HDFS-12017 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Nandakumar > Labels: ozoneMerge > Attachments: HDFS-12017-HDFS-7240.000.patch, > HDFS-12017-HDFS-7240.001.patch > > > We use port 50011 port -- This is an old format of choosing port. Hadoop 3 > has already moved to port ranges under 98xx. In fact all of KSM/SCM and > CBlock is using 98xx range. We should move 50011 to a port under 98xx. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12466) Ozone: KSM: Make "ozone.ksm.address" as mandatory property for client
[
https://issues.apache.org/jira/browse/HDFS-12466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171855#comment-16171855
]
Nandakumar commented on HDFS-12466:
---
[~cheersyang], thanks for the quick review. The value {{0.0.0.0}} was there in
ozone-default.xml which was causing the issue which you have described (never
reach to {{IllegalArgumentException}} ), this patch removes the value from
ozone-default.xml.
> Ozone: KSM: Make "ozone.ksm.address" as mandatory property for client
> -
>
> Key: HDFS-12466
> URL: https://issues.apache.org/jira/browse/HDFS-12466
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nandakumar
>Assignee: Nandakumar
>Priority: Minor
> Attachments: HDFS-12466-HDFS-7240.000.patch,
> HDFS-12466-HDFS-7240.001.patch
>
>
> This jira makes {{ozone.ksm.address}} as a mandatory property at client side.
> User has to explicitly configure it now. We should not use default
> {{0.0.0.0}} if the value is not present, throw exception instead.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
