[jira] [Commented] (HDDS-4308) Fix issue with quota update
[ https://issues.apache.org/jira/browse/HDDS-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212631#comment-17212631 ] Rui Wang commented on HDDS-4308: [~micahzhao] thanks for this reminder. Will follow up on discussions above to understand necessary steps. > Fix issue with quota update > --- > > Key: HDDS-4308 > URL: https://issues.apache.org/jira/browse/HDDS-4308 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Priority: Blocker > > Currently volumeArgs using getCacheValue and put the same object in > doubleBuffer, this might cause issue. > Let's take the below scenario: > InitialVolumeArgs quotaBytes -> 1 > 1. T1 -> Update VolumeArgs, and subtracting 1000 and put this updated > volumeArgs to DoubleBuffer. > 2. T2-> Update VolumeArgs, and subtracting 2000 and has not still updated to > double buffer. > *Now at the end of flushing these transactions, our DB should have 7000 as > bytes used.* > Now T1 is picked by double Buffer and when it commits, and as it uses cached > Object put into doubleBuffer, it flushes to DB with the updated value from > T2(As it is a cache object) and update DB with bytesUsed as 7000. > And now OM has restarted, and only DB has transactions till T1. (We get this > info from TransactionInfo > Table(https://issues.apache.org/jira/browse/HDDS-3685) > Now T2 is again replayed, as it is not committed to DB, now DB will be again > subtracted with 2000, and now DB will have 5000. > But after T2, the value should be 7000, so we have DB in an incorrect state. > Issue here: > 1. As we use a cached object and put the same cached object into double > buffer this can cause this kind of issue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3728) Bucket space: check quotaUsageInBytes when write key
[ https://issues.apache.org/jira/browse/HDDS-3728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211239#comment-17211239 ] Rui Wang commented on HDDS-3728: [~micahzhao] thank you! I send other space quota related PR soon after rebased against this one. > Bucket space: check quotaUsageInBytes when write key > > > Key: HDDS-3728 > URL: https://issues.apache.org/jira/browse/HDDS-3728 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Simon Su >Assignee: mingchao zhao >Priority: Major > Labels: pull-request-available > Fix For: 1.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDDS-3728) Bucket space: check quotaUsageInBytes when write key
[ https://issues.apache.org/jira/browse/HDDS-3728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated HDDS-3728: --- Comment: was deleted (was: [~micahzhao] thank you! I send other space quota related PR soon after rebased against this one.) > Bucket space: check quotaUsageInBytes when write key > > > Key: HDDS-3728 > URL: https://issues.apache.org/jira/browse/HDDS-3728 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Simon Su >Assignee: mingchao zhao >Priority: Major > Labels: pull-request-available > Fix For: 1.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4317) Copy UserVolumeInfo to PersistedUserVolumeInfo in interface-storage
Rui Wang created HDDS-4317: -- Summary: Copy UserVolumeInfo to PersistedUserVolumeInfo in interface-storage Key: HDDS-4317 URL: https://issues.apache.org/jira/browse/HDDS-4317 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Rui Wang Assignee: Rui Wang -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4237) Testing Infrastructure Random Failures
[ https://issues.apache.org/jira/browse/HDDS-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205210#comment-17205210 ] Rui Wang commented on HDDS-4237: Hi Nicholas, that's great! This Jira is supposed to find more bugs and there will also work to to during/after Jira. All will need people to work on :) > Testing Infrastructure Random Failures > -- > > Key: HDDS-4237 > URL: https://issues.apache.org/jira/browse/HDDS-4237 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Rui Wang >Priority: Major > > Network partitioning can cause brian-split case where there are two leaders > exist. We need some sort of testing Infrastructure/framework to simulate such > case and verify whether our SCM HA implementation can achieve strong > consistency under partitioned network. > There might be two ways suggested by Mukul Kumar Singh: > a) Blockade tests, blockade is a docker based framework where the > network for one DN can be isolated from the other > b) MiniOzoneChaosCluster - This is a unit test based test, where a > random datanode was killed and this helped in finding out issues with > the consistency. > We might need similar solution for SCM: block SCM leader network and also > increase timeout to make old leader do not turn into candidate. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-4237) Testing Infrastructure Random Failures
[ https://issues.apache.org/jira/browse/HDDS-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205210#comment-17205210 ] Rui Wang edited comment on HDDS-4237 at 10/1/20, 2:22 AM: -- Hi Nicholas, that's great! This Jira is supposed to find more bugs and there will also work to to during/after this Jira. All will need people to work on :) was (Author: amaliujia): Hi Nicholas, that's great! This Jira is supposed to find more bugs and there will also work to to during/after Jira. All will need people to work on :) > Testing Infrastructure Random Failures > -- > > Key: HDDS-4237 > URL: https://issues.apache.org/jira/browse/HDDS-4237 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Rui Wang >Priority: Major > > Network partitioning can cause brian-split case where there are two leaders > exist. We need some sort of testing Infrastructure/framework to simulate such > case and verify whether our SCM HA implementation can achieve strong > consistency under partitioned network. > There might be two ways suggested by Mukul Kumar Singh: > a) Blockade tests, blockade is a docker based framework where the > network for one DN can be isolated from the other > b) MiniOzoneChaosCluster - This is a unit test based test, where a > random datanode was killed and this helped in finding out issues with > the consistency. > We might need similar solution for SCM: block SCM leader network and also > increase timeout to make old leader do not turn into candidate. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4273) Volume Namespace:`usedNamespace` works by `ozone sh vol info`
[ https://issues.apache.org/jira/browse/HDDS-4273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated HDDS-4273: --- Summary: Volume Namespace:`usedNamespace` works by `ozone sh vol info` (was: `usedNamespace` works by `ozone sh vol info`) > Volume Namespace:`usedNamespace` works by `ozone sh vol info` > - > > Key: HDDS-4273 > URL: https://issues.apache.org/jira/browse/HDDS-4273 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3730) Support Namespace Level quota for bucket
[ https://issues.apache.org/jira/browse/HDDS-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang resolved HDDS-3730. Resolution: Duplicate Has broken down this Jira to multiple smaller Jira > Support Namespace Level quota for bucket > > > Key: HDDS-3730 > URL: https://issues.apache.org/jira/browse/HDDS-3730 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Simon Su >Assignee: Rui Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4279) Bucket Namespace: support clear namespace quota
Rui Wang created HDDS-4279: -- Summary: Bucket Namespace: support clear namespace quota Key: HDDS-4279 URL: https://issues.apache.org/jira/browse/HDDS-4279 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Rui Wang Assignee: Rui Wang -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4278) Bucket Namespace: check usedNamespace when create key in bucket
Rui Wang created HDDS-4278: -- Summary: Bucket Namespace: check usedNamespace when create key in bucket Key: HDDS-4278 URL: https://issues.apache.org/jira/browse/HDDS-4278 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Rui Wang Assignee: Rui Wang -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4277) Bucket Namespace: add namespaceQuotaUsage and update it when create and delete key in a bucket
Rui Wang created HDDS-4277: -- Summary: Bucket Namespace: add namespaceQuotaUsage and update it when create and delete key in a bucket Key: HDDS-4277 URL: https://issues.apache.org/jira/browse/HDDS-4277 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Rui Wang Assignee: Rui Wang -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3729) Support Namespace Level quota for Volume
[ https://issues.apache.org/jira/browse/HDDS-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang resolved HDDS-3729. Resolution: Duplicate This is Jira is broken down to more smaller Jira. > Support Namespace Level quota for Volume > > > Key: HDDS-3729 > URL: https://issues.apache.org/jira/browse/HDDS-3729 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Simon Su >Assignee: Rui Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4276) Volume Namespace: support clear namespace quota
Rui Wang created HDDS-4276: -- Summary: Volume Namespace: support clear namespace quota Key: HDDS-4276 URL: https://issues.apache.org/jira/browse/HDDS-4276 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Rui Wang Assignee: Rui Wang -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4275) Volume Namespace: check usedNamespace when create bucket
Rui Wang created HDDS-4275: -- Summary: Volume Namespace: check usedNamespace when create bucket Key: HDDS-4275 URL: https://issues.apache.org/jira/browse/HDDS-4275 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Rui Wang Assignee: Rui Wang -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4273) `usedNamespace` works by `ozone sh vol info`
Rui Wang created HDDS-4273: -- Summary: `usedNamespace` works by `ozone sh vol info` Key: HDDS-4273 URL: https://issues.apache.org/jira/browse/HDDS-4273 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Rui Wang Assignee: Rui Wang -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-4272) Volume namespace: add namespaceQuotaUsage and update it when create and delete bucket
[ https://issues.apache.org/jira/browse/HDDS-4272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang reassigned HDDS-4272: -- Assignee: Rui Wang > Volume namespace: add namespaceQuotaUsage and update it when create and > delete bucket > - > > Key: HDDS-4272 > URL: https://issues.apache.org/jira/browse/HDDS-4272 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4272) Volume namespace: add namespaceQuotaUsage and update it when create and delete bucket
Rui Wang created HDDS-4272: -- Summary: Volume namespace: add namespaceQuotaUsage and update it when create and delete bucket Key: HDDS-4272 URL: https://issues.apache.org/jira/browse/HDDS-4272 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Rui Wang -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4242) Copy PrefixInfo proto to new project hadoop-ozone/interface-storage
[ https://issues.apache.org/jira/browse/HDDS-4242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated HDDS-4242: --- Summary: Copy PrefixInfo proto to new project hadoop-ozone/interface-storage (was: Copy storage proto to new project hadoop-ozone/interface-storage) > Copy PrefixInfo proto to new project hadoop-ozone/interface-storage > --- > > Key: HDDS-4242 > URL: https://issues.apache.org/jira/browse/HDDS-4242 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-541) Ozone Quota support.
[
https://issues.apache.org/jira/browse/HDDS-541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200538#comment-17200538
]
Rui Wang commented on HDDS-541:
---
Thanks [~micahzhao]!
I will start from the NameSpace jira.
> Ozone Quota support.
>
>
> Key: HDDS-541
> URL: https://issues.apache.org/jira/browse/HDDS-541
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Namit Maheshwari
>Assignee: mingchao zhao
>Priority: Major
> Labels: Triaged
> Time Spent: 96h
> Remaining Estimate: 120h
>
> Create a volume with just 1 MB as quota
> {code:java}
> [root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone sh volume create
> --quota=1MB --user=root /hive
> 2018-09-23 02:10:11,283 [main] INFO - Creating Volume: hive, with root as
> owner and quota set to 1048576 bytes.
> {code}
> Now create a bucket and put a big key greater than 1MB in the bucket
> {code:java}
> [root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone sh bucket create
> /hive/bucket1
> 2018-09-23 02:10:38,003 [main] INFO - Creating Bucket: hive/bucket1, with
> Versioning false and Storage Type set to DISK
> [root@ctr-e138-1518143905142-481027-01-02 bin]# ls -l
> ../../ozone-0.3.0-SNAPSHOT.tar.gz
> -rw-r--r-- 1 root root 165903437 Sep 21 13:16
> ../../ozone-0.3.0-SNAPSHOT.tar.gz
> [root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone sh key put
> /hive/ozone-0.3.0-SNAPSHOT.tar.gz ../../ozone-0.3.0-SNAPSHOT.tar.gz
> volume/bucket/key name required in putKey
> [root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone sh key put
> /hive/bucket1/ozone-0.3.0-SNAPSHOT.tar.gz ../../ozone-0.3.0-SNAPSHOT.tar.gz
> [root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone sh key info
> /hive/bucket1/ozone-0.3.0-SNAPSHOT.tar.gz
> {
> "version" : 0,
> "md5hash" : null,
> "createdOn" : "Sun, 23 Sep 2018 02:13:02 GMT",
> "modifiedOn" : "Sun, 23 Sep 2018 02:13:08 GMT",
> "size" : 165903437,
> "keyName" : "ozone-0.3.0-SNAPSHOT.tar.gz",
> "keyLocations" : [ {
> "containerID" : 2,
> "localID" : 100772661343420416,
> "length" : 134217728,
> "offset" : 0
> }, {
> "containerID" : 3,
> "localID" : 100772661661007873,
> "length" : 31685709,
> "offset" : 0
> } ]
> }{code}
> It was able to put a 165 MB file on a volume with just 1MB quota.
>
> Currently Ozone haven't support Quota, So I think this should be a new
> feature .
> The design document can be referred to the attachment. ([design google
> docs|https://docs.google.com/document/d/1ohbGn5N7FN6OD15xMShHH2SrtZRYx0-zUf9vjatn_OM/edit])
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-792) Use md5 hash as ETag for Ozone S3 objects
[ https://issues.apache.org/jira/browse/HDDS-792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang reassigned HDDS-792: - Assignee: Rui Wang > Use md5 hash as ETag for Ozone S3 objects > - > > Key: HDDS-792 > URL: https://issues.apache.org/jira/browse/HDDS-792 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: S3 >Reporter: Marton Elek >Assignee: Rui Wang >Priority: Major > > AWS S3 uses md5 hash of the files as ETag. > Not a strict requirement, but s3 tests (https://github.com/gaul/s3-tests/) > can not been executed without that. > It requires to support custom key/value annotations on key objects. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-792) Use md5 hash as ETag for Ozone S3 objects
[ https://issues.apache.org/jira/browse/HDDS-792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200509#comment-17200509 ] Rui Wang commented on HDDS-792: --- I can try to look at this one. > Use md5 hash as ETag for Ozone S3 objects > - > > Key: HDDS-792 > URL: https://issues.apache.org/jira/browse/HDDS-792 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: S3 >Reporter: Marton Elek >Priority: Major > > AWS S3 uses md5 hash of the files as ETag. > Not a strict requirement, but s3 tests (https://github.com/gaul/s3-tests/) > can not been executed without that. > It requires to support custom key/value annotations on key objects. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3208) Implement Ratis Snapshots on SCM
[ https://issues.apache.org/jira/browse/HDDS-3208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200503#comment-17200503 ] Rui Wang commented on HDDS-3208: OzoneManager.installSnapshotFromLeader Download part: OzoneManager.getDBCheckpointFromLeader > Implement Ratis Snapshots on SCM > > > Key: HDDS-3208 > URL: https://issues.apache.org/jira/browse/HDDS-3208 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Li Cheng >Assignee: Rui Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-3730) Support Namespace Level quota for bucket
[ https://issues.apache.org/jira/browse/HDDS-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang reassigned HDDS-3730: -- Assignee: Rui Wang > Support Namespace Level quota for bucket > > > Key: HDDS-3730 > URL: https://issues.apache.org/jira/browse/HDDS-3730 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Simon Su >Assignee: Rui Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3728) Support Storage space Level Quota for bucket
[ https://issues.apache.org/jira/browse/HDDS-3728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200212#comment-17200212 ] Rui Wang commented on HDDS-3728: [~simonss] [~micahzhao] do you mind me working on this JIRA? > Support Storage space Level Quota for bucket > > > Key: HDDS-3728 > URL: https://issues.apache.org/jira/browse/HDDS-3728 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Simon Su >Assignee: Rui Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3730) Support Namespace Level quota for bucket
[ https://issues.apache.org/jira/browse/HDDS-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200213#comment-17200213 ] Rui Wang commented on HDDS-3730: [~simonss] [~micahzhao] do you mind me working on this JIRA? This one seems related to HDDS-3728 > Support Namespace Level quota for bucket > > > Key: HDDS-3730 > URL: https://issues.apache.org/jira/browse/HDDS-3730 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Simon Su >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-3728) Support Storage space Level Quota for bucket
[ https://issues.apache.org/jira/browse/HDDS-3728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang reassigned HDDS-3728: -- Assignee: Rui Wang > Support Storage space Level Quota for bucket > > > Key: HDDS-3728 > URL: https://issues.apache.org/jira/browse/HDDS-3728 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Simon Su >Assignee: Rui Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-541) Ozone Quota support.
[
https://issues.apache.org/jira/browse/HDDS-541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199638#comment-17199638
]
Rui Wang commented on HDDS-541:
---
[~micahzhao]
Do you mind me helping on this JIRA? I am seeing lots of open work here.
> Ozone Quota support.
>
>
> Key: HDDS-541
> URL: https://issues.apache.org/jira/browse/HDDS-541
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Namit Maheshwari
>Assignee: mingchao zhao
>Priority: Major
> Labels: Triaged
> Attachments: Ozone Quota Design.pdf
>
> Time Spent: 96h
> Remaining Estimate: 120h
>
> Create a volume with just 1 MB as quota
> {code:java}
> [root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone sh volume create
> --quota=1MB --user=root /hive
> 2018-09-23 02:10:11,283 [main] INFO - Creating Volume: hive, with root as
> owner and quota set to 1048576 bytes.
> {code}
> Now create a bucket and put a big key greater than 1MB in the bucket
> {code:java}
> [root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone sh bucket create
> /hive/bucket1
> 2018-09-23 02:10:38,003 [main] INFO - Creating Bucket: hive/bucket1, with
> Versioning false and Storage Type set to DISK
> [root@ctr-e138-1518143905142-481027-01-02 bin]# ls -l
> ../../ozone-0.3.0-SNAPSHOT.tar.gz
> -rw-r--r-- 1 root root 165903437 Sep 21 13:16
> ../../ozone-0.3.0-SNAPSHOT.tar.gz
> [root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone sh key put
> /hive/ozone-0.3.0-SNAPSHOT.tar.gz ../../ozone-0.3.0-SNAPSHOT.tar.gz
> volume/bucket/key name required in putKey
> [root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone sh key put
> /hive/bucket1/ozone-0.3.0-SNAPSHOT.tar.gz ../../ozone-0.3.0-SNAPSHOT.tar.gz
> [root@ctr-e138-1518143905142-481027-01-02 bin]# ./ozone sh key info
> /hive/bucket1/ozone-0.3.0-SNAPSHOT.tar.gz
> {
> "version" : 0,
> "md5hash" : null,
> "createdOn" : "Sun, 23 Sep 2018 02:13:02 GMT",
> "modifiedOn" : "Sun, 23 Sep 2018 02:13:08 GMT",
> "size" : 165903437,
> "keyName" : "ozone-0.3.0-SNAPSHOT.tar.gz",
> "keyLocations" : [ {
> "containerID" : 2,
> "localID" : 100772661343420416,
> "length" : 134217728,
> "offset" : 0
> }, {
> "containerID" : 3,
> "localID" : 100772661661007873,
> "length" : 31685709,
> "offset" : 0
> } ]
> }{code}
> It was able to put a 165 MB file on a volume with just 1MB quota.
>
> Currently Ozone haven't support Quota, So I think this should be a new
> feature .
> The design document can be referred to the attachment.
> ([link|https://docs.google.com/document/d/1ohbGn5N7FN6OD15xMShHH2SrtZRYx0-zUf9vjatn_OM/edit?usp=sharing]
> to google docs)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3091) Assign Object ID from OMKeyInfo in PrefixManagerImpl.addAcl and setAcl
[ https://issues.apache.org/jira/browse/HDDS-3091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199624#comment-17199624 ] Rui Wang commented on HDDS-3091: [~sdeka] do you have any sub-tasks under HDDS-2939 that you think I can help with? > Assign Object ID from OMKeyInfo in PrefixManagerImpl.addAcl and setAcl > --- > > Key: HDDS-3091 > URL: https://issues.apache.org/jira/browse/HDDS-3091 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Major > > HDDS-2940 creates a separate prefix entry in the key table for every > directory in the pathname. > Each prefix directory is assigned a unique object id during the File create > request OR Mkdir(Directory create request). > PrefixManagerImpl needs to retrieve the object id from the Key Info of the > prefix/directory when doing a addAcl() or a setAcl(). This ensures > consistency between the key table and the ACL prefix table. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-903) Compile OzoneManager Ratis protobuf files with proto3 compiler
[ https://issues.apache.org/jira/browse/HDDS-903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1719#comment-1719 ] Rui Wang commented on HDDS-903: --- [~hanishakoneru] Can I help on this JIRA? > Compile OzoneManager Ratis protobuf files with proto3 compiler > -- > > Key: HDDS-903 > URL: https://issues.apache.org/jira/browse/HDDS-903 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: OM HA, Ozone Manager >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > > Ratis requires the payload to be proto3 ByteString. If the OM ratis protos > are compiled using proto2, we would have to make the conversion to proto3 for > every request to be submitted to Ratis. Instead, we can compile OM's Ratis > protocol files using proto3 compiler. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-4212) TestSCMStateMachine
[ https://issues.apache.org/jira/browse/HDDS-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang reassigned HDDS-4212: -- Assignee: (was: Rui Wang) > TestSCMStateMachine > --- > > Key: HDDS-4212 > URL: https://issues.apache.org/jira/browse/HDDS-4212 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Rui Wang >Priority: Major > > I cannot find a TestSCMStateMachine in HDDS-2323 feature branch. So in this > JIRA I propose add tests for SCMStateMachine -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4237) Testing Infrastructure Random Failures
[ https://issues.apache.org/jira/browse/HDDS-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated HDDS-4237: --- Summary: Testing Infrastructure Random Failures (was: Testing Infrastructure for network partitioning) > Testing Infrastructure Random Failures > -- > > Key: HDDS-4237 > URL: https://issues.apache.org/jira/browse/HDDS-4237 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Rui Wang >Priority: Major > > Network partitioning can cause brian-split case where there are two leaders > exist. We need some sort of testing Infrastructure/framework to simulate such > case and verify whether our SCM HA implementation can achieve strong > consistency under partitioned network. > There might be two ways suggested by Mukul Kumar Singh: > a) Blockade tests, blockade is a docker based framework where the > network for one DN can be isolated from the other > b) MiniOzoneChaosCluster - This is a unit test based test, where a > random datanode was killed and this helped in finding out issues with > the consistency. > We might need similar solution for SCM: block SCM leader network and also > increase timeout to make old leader do not turn into candidate. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4237) Testing Infrastructure for network partitioning
[ https://issues.apache.org/jira/browse/HDDS-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17196672#comment-17196672 ] Rui Wang commented on HDDS-4237: https://github.com/apache/hadoop-ozone/blob/676610ef40b0f8701d950b347fb916d28268f1da/hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/failure/Failures.java#L55 https://github.com/apache/hadoop-ozone/blob/676610ef40b0f8701d950b347fb916d28268f1da/hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/TestMiniChaosOzoneCluster.java#L113 > Testing Infrastructure for network partitioning > --- > > Key: HDDS-4237 > URL: https://issues.apache.org/jira/browse/HDDS-4237 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Rui Wang >Priority: Major > > Network partitioning can cause brian-split case where there are two leaders > exist. We need some sort of testing Infrastructure/framework to simulate such > case and verify whether our SCM HA implementation can achieve strong > consistency under partitioned network. > There might be two ways suggested by Mukul Kumar Singh: > a) Blockade tests, blockade is a docker based framework where the > network for one DN can be isolated from the other > b) MiniOzoneChaosCluster - This is a unit test based test, where a > random datanode was killed and this helped in finding out issues with > the consistency. > We might need similar solution for SCM: block SCM leader network and also > increase timeout to make old leader do not turn into candidate. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4237) Testing Infrastructure for network partitioning
[ https://issues.apache.org/jira/browse/HDDS-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17196670#comment-17196670 ] Rui Wang commented on HDDS-4237: might start from MiniOzoneChaosCluster > Testing Infrastructure for network partitioning > --- > > Key: HDDS-4237 > URL: https://issues.apache.org/jira/browse/HDDS-4237 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Rui Wang >Priority: Major > > Network partitioning can cause brian-split case where there are two leaders > exist. We need some sort of testing Infrastructure/framework to simulate such > case and verify whether our SCM HA implementation can achieve strong > consistency under partitioned network. > There might be two ways suggested by Mukul Kumar Singh: > a) Blockade tests, blockade is a docker based framework where the > network for one DN can be isolated from the other > b) MiniOzoneChaosCluster - This is a unit test based test, where a > random datanode was killed and this helped in finding out issues with > the consistency. > We might need similar solution for SCM: block SCM leader network and also > increase timeout to make old leader do not turn into candidate. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4242) Copy storage proto to new project hadoop-ozone/interface-storage
[ https://issues.apache.org/jira/browse/HDDS-4242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated HDDS-4242: --- Summary: Copy storage proto to new project hadoop-ozone/interface-storage (was: Move storage proto to new project hadoop-ozone/interface-storage) > Copy storage proto to new project hadoop-ozone/interface-storage > > > Key: HDDS-4242 > URL: https://issues.apache.org/jira/browse/HDDS-4242 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4242) Move storage proto to new project hadoop-ozone/interface-storage
Rui Wang created HDDS-4242: -- Summary: Move storage proto to new project hadoop-ozone/interface-storage Key: HDDS-4242 URL: https://issues.apache.org/jira/browse/HDDS-4242 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Rui Wang Assignee: Rui Wang -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-4237) Testing Infrastructure for network partitioning
[ https://issues.apache.org/jira/browse/HDDS-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195579#comment-17195579 ] Rui Wang edited comment on HDDS-4237 at 9/14/20, 4:35 PM: -- Other potential ideas: blockade: https://github.com/apache/hadoop-ozone/tree/master/hadoop-ozone/fault-injection-test/network-tests/src/test https://chaos-mesh.org/ https://jepsen.io/ was (Author: amaliujia): Other potential ideas: https://github.com/apache/hadoop-ozone/tree/master/hadoop-ozone/fault-injection-test/network-tests/src/test https://chaos-mesh.org/ https://jepsen.io/ > Testing Infrastructure for network partitioning > --- > > Key: HDDS-4237 > URL: https://issues.apache.org/jira/browse/HDDS-4237 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Rui Wang >Priority: Major > > Network partitioning can cause brian-split case where there are two leaders > exist. We need some sort of testing Infrastructure/framework to simulate such > case and verify whether our SCM HA implementation can achieve strong > consistency under partitioned network. > There might be two ways suggested by Mukul Kumar Singh: > a) Blockade tests, blockade is a docker based framework where the > network for one DN can be isolated from the other > b) MiniOzoneChaosCluster - This is a unit test based test, where a > random datanode was killed and this helped in finding out issues with > the consistency. > We might need similar solution for SCM: block SCM leader network and also > increase timeout to make old leader do not turn into candidate. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4237) Testing Infrastructure for network partitioning
[ https://issues.apache.org/jira/browse/HDDS-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195579#comment-17195579 ] Rui Wang commented on HDDS-4237: Other potential ideas: https://github.com/apache/hadoop-ozone/tree/master/hadoop-ozone/fault-injection-test/network-tests/src/test https://chaos-mesh.org/ https://jepsen.io/ > Testing Infrastructure for network partitioning > --- > > Key: HDDS-4237 > URL: https://issues.apache.org/jira/browse/HDDS-4237 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Rui Wang >Priority: Major > > Network partitioning can cause brian-split case where there are two leaders > exist. We need some sort of testing Infrastructure/framework to simulate such > case and verify whether our SCM HA implementation can achieve strong > consistency under partitioned network. > There might be two ways suggested by Mukul Kumar Singh: > a) Blockade tests, blockade is a docker based framework where the > network for one DN can be isolated from the other > b) MiniOzoneChaosCluster - This is a unit test based test, where a > random datanode was killed and this helped in finding out issues with > the consistency. > We might need similar solution for SCM: block SCM leader network and also > increase timeout to make old leader do not turn into candidate. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4237) Testing Infrastructure for network partitioning
[ https://issues.apache.org/jira/browse/HDDS-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated HDDS-4237: --- Description: Network partitioning can cause brian-split case where there are two leaders exist. We need some sort of testing Infrastructure/framework to simulate such case and verify whether our SCM HA implementation can achieve strong consistency under partitioned network. There might be two ways suggested by Mukul Kumar Singh: a) Blockade tests, blockade is a docker based framework where the network for one DN can be isolated from the other b) MiniOzoneChaosCluster - This is a unit test based test, where a random datanode was killed and this helped in finding out issues with the consistency. We might need similar solution for SCM: block SCM leader network and also increase timeout to make old leader do not turn into candidate. was: Network partitioning can cause Brian-split case where there are two leaders exist. We need some sort of testing Infrastructure/framework to simulate such case and verify whether our SCM HA implementation can achieve strong consistency. There might be two ways suggested by Mukul Kumar Singh: a) Blockade tests, blockade is a docker based framework where the network for one DN can be isolated from the other b) MiniOzoneChaosCluster - This is a unit test based test, where a random datanode was killed and this helped in finding out issues with the consistency. We might need similar solution for SCM: block SCM leader network and also increase timeout to make old leader do not turn into candidate. > Testing Infrastructure for network partitioning > --- > > Key: HDDS-4237 > URL: https://issues.apache.org/jira/browse/HDDS-4237 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Rui Wang >Priority: Major > > Network partitioning can cause brian-split case where there are two leaders > exist. We need some sort of testing Infrastructure/framework to simulate such > case and verify whether our SCM HA implementation can achieve strong > consistency under partitioned network. > There might be two ways suggested by Mukul Kumar Singh: > a) Blockade tests, blockade is a docker based framework where the > network for one DN can be isolated from the other > b) MiniOzoneChaosCluster - This is a unit test based test, where a > random datanode was killed and this helped in finding out issues with > the consistency. > We might need similar solution for SCM: block SCM leader network and also > increase timeout to make old leader do not turn into candidate. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4237) Testing Infrastructure for network partitioning
[ https://issues.apache.org/jira/browse/HDDS-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated HDDS-4237: --- Description: Network partitioning can cause Brian-split case where there are two leaders exist. We need some sort of testing Infrastructure/framework to simulate such case and verify whether our SCM HA implementation can achieve strong consistency. There might be two ways suggested by Mukul Kumar Singh: a) Blockade tests, blockade is a docker based framework where the network for one DN can be isolated from the other b) MiniOzoneChaosCluster - This is a unit test based test, where a random datanode was killed and this helped in finding out issues with the consistency. We might need similar solution for SCM: block SCM leader network and also increase timeout to make old leader do not turn into candidate. was: Network partitioning can cause Brian-split case where there are two leaders exist. We need some sort of testing Infrastructure/framework to simulate such case and verify whether our SCM HA implementation can achieve strong consistency. There might be two ways suggested by Mukul Kumar Singh: a) Blockade tests, blockade is a docker based framework where the network for one DN can be isolated from the other b) MiniOzoneChaosCluster - This is a unit test based test, where a random datanode was killed and this helped in finding out issues with the consistency. > Testing Infrastructure for network partitioning > --- > > Key: HDDS-4237 > URL: https://issues.apache.org/jira/browse/HDDS-4237 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Rui Wang >Priority: Major > > Network partitioning can cause Brian-split case where there are two leaders > exist. We need some sort of testing Infrastructure/framework to simulate such > case and verify whether our SCM HA implementation can achieve strong > consistency. > There might be two ways suggested by Mukul Kumar Singh: > a) Blockade tests, blockade is a docker based framework where the > network for one DN can be isolated from the other > b) MiniOzoneChaosCluster - This is a unit test based test, where a > random datanode was killed and this helped in finding out issues with > the consistency. > We might need similar solution for SCM: block SCM leader network and also > increase timeout to make old leader do not turn into candidate. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4237) Testing Infrastructure for network partitioning
Rui Wang created HDDS-4237: -- Summary: Testing Infrastructure for network partitioning Key: HDDS-4237 URL: https://issues.apache.org/jira/browse/HDDS-4237 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Rui Wang Network partitioning can cause Brian-split case where there are two leaders exist. We need some sort of testing Infrastructure/framework to simulate such case and verify whether our SCM HA implementation can achieve strong consistency. There might be two ways suggested by Mukul Kumar Singh: a) Blockade tests, blockade is a docker based framework where the network for one DN can be isolated from the other b) MiniOzoneChaosCluster - This is a unit test based test, where a random datanode was killed and this helped in finding out issues with the consistency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-3890) Stop BackgroundPipelineCreator when PipelineManager is closed
[
https://issues.apache.org/jira/browse/HDDS-3890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Wang reassigned HDDS-3890:
--
Assignee: Rui Wang
> Stop BackgroundPipelineCreator when PipelineManager is closed
> -
>
> Key: HDDS-3890
> URL: https://issues.apache.org/jira/browse/HDDS-3890
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: SCM HA
>Reporter: Nanda kumar
>Assignee: Rui Wang
>Priority: Major
>
> {{BackgroundPipelineCreator}} is started by {{PipelineManager}} but never
> stopped. We should stop the {{BackgroundPipelineCreator}} when
> {{PipelineManager}} is closed
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4236) Move "Om*Codec.java" to new project hadoop-ozone/interface-storage
Rui Wang created HDDS-4236: -- Summary: Move "Om*Codec.java" to new project hadoop-ozone/interface-storage Key: HDDS-4236 URL: https://issues.apache.org/jira/browse/HDDS-4236 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Rui Wang Assignee: Rui Wang This is the first step to separate storage and RPC proto files. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4115) CLI command to show current SCM leader and follower status
[ https://issues.apache.org/jira/browse/HDDS-4115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated HDDS-4115: --- Status: Patch Available (was: In Progress) > CLI command to show current SCM leader and follower status > -- > > Key: HDDS-4115 > URL: https://issues.apache.org/jira/browse/HDDS-4115 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Li Cheng >Assignee: Rui Wang >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3684) Add tests for replication annotation
[ https://issues.apache.org/jira/browse/HDDS-3684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated HDDS-3684: --- Status: Patch Available (was: In Progress) > Add tests for replication annotation > > > Key: HDDS-3684 > URL: https://issues.apache.org/jira/browse/HDDS-3684 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Li Cheng >Assignee: Rui Wang >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-3102) ozone getconf command should use the GenericCli parent class
[ https://issues.apache.org/jira/browse/HDDS-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang reassigned HDDS-3102: -- Assignee: Rui Wang > ozone getconf command should use the GenericCli parent class > > > Key: HDDS-3102 > URL: https://issues.apache.org/jira/browse/HDDS-3102 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone CLI >Reporter: Marton Elek >Assignee: Rui Wang >Priority: Major > Labels: newbie, pull-request-available > > org.apache.hadoop.ozone.freon.OzoneGetCOnf implements a tool to print out > current configuration values > With all the other CLI tools we already started to use picocli and the > GenericCli parent class. > To provide better user experience we should migrate the tool to use > GenericCli (+move it to the tools project + remove freon from the package > name) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4216) Separate storage / RPC proto files
[ https://issues.apache.org/jira/browse/HDDS-4216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191833#comment-17191833 ] Rui Wang commented on HDDS-4216: [~elek] Can you share some code pointers to the proto structures in the write path? > Separate storage / RPC proto files > -- > > Key: HDDS-4216 > URL: https://issues.apache.org/jira/browse/HDDS-4216 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Marton Elek >Assignee: Rui Wang >Priority: Major > > The change of HDDS-3792 separated client/admin/tools proto files and > introduced a new way to check the proto files backward compatiblity. > To make it easier to check the compatibility of persistent data (proto > structures persisted in RocksDB) we need to separated RPC and storage proto > files: > 1. create a hadoop-ozone/inteface-storage project > 2. Clone the required proto files and generate java with different package > name > 3. Copy all the Codec implementation together with the code -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-4216) Separate storage / RPC proto files
[ https://issues.apache.org/jira/browse/HDDS-4216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang reassigned HDDS-4216: -- Assignee: Rui Wang (was: Marton Elek) > Separate storage / RPC proto files > -- > > Key: HDDS-4216 > URL: https://issues.apache.org/jira/browse/HDDS-4216 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Marton Elek >Assignee: Rui Wang >Priority: Major > > The change of HDDS-3792 separated client/admin/tools proto files and > introduced a new way to check the proto files backward compatiblity. > To make it easier to check the compatibility of persistent data (proto > structures persisted in RocksDB) we need to separated RPC and storage proto > files: > 1. create a hadoop-ozone/inteface-storage project > 2. Clone the required proto files and generate java with different package > name > 3. Copy all the Codec implementation together with the code -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4212) TestSCMStateMachine
[ https://issues.apache.org/jira/browse/HDDS-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191358#comment-17191358 ] Rui Wang commented on HDDS-4212: To clarify, this JIRA should work on: org.apache.hadoop.hdds.scm.ha.SCMStateMachine > TestSCMStateMachine > --- > > Key: HDDS-4212 > URL: https://issues.apache.org/jira/browse/HDDS-4212 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > > I cannot find a TestSCMStateMachine in HDDS-2323 feature branch. So in this > JIRA I propose add tests for SCMStateMachine -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4212) TestSCMStateMachine
Rui Wang created HDDS-4212: -- Summary: TestSCMStateMachine Key: HDDS-4212 URL: https://issues.apache.org/jira/browse/HDDS-4212 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Rui Wang Assignee: Rui Wang I cannot find a TestSCMStateMachine in HDDS-2323 feature branch. So in this JIRA I propose add tests for SCMStateMachine -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3661) Add Snapshot into new SCMRatisServer and SCMStateMachine
[ https://issues.apache.org/jira/browse/HDDS-3661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191129#comment-17191129 ] Rui Wang commented on HDDS-3661: +1 > Add Snapshot into new SCMRatisServer and SCMStateMachine > - > > Key: HDDS-3661 > URL: https://issues.apache.org/jira/browse/HDDS-3661 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Li Cheng >Assignee: Li Cheng >Priority: Major > > Now we have prototype SCMRatisServer and SCMStateMachine under Ratis and HA > path. Implement Snapshot support into new SCMRatisServer and SCMStateMachine > as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-3212) Purge SCM Ratis log
[ https://issues.apache.org/jira/browse/HDDS-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang reassigned HDDS-3212: -- Assignee: Rui Wang > Purge SCM Ratis log > --- > > Key: HDDS-3212 > URL: https://issues.apache.org/jira/browse/HDDS-3212 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Li Cheng >Assignee: Rui Wang >Priority: Major > > Based on Ratis snapshot. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-3209) Load Snapshot info upon SCM Ratis starts
[ https://issues.apache.org/jira/browse/HDDS-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang reassigned HDDS-3209: -- Assignee: Rui Wang > Load Snapshot info upon SCM Ratis starts > > > Key: HDDS-3209 > URL: https://issues.apache.org/jira/browse/HDDS-3209 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Li Cheng >Assignee: Rui Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3656) Update javadoc in SCMHA related classes
[ https://issues.apache.org/jira/browse/HDDS-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190994#comment-17190994 ] Rui Wang commented on HDDS-3656: Big + 1! > Update javadoc in SCMHA related classes > --- > > Key: HDDS-3656 > URL: https://issues.apache.org/jira/browse/HDDS-3656 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: documentation >Reporter: Nanda kumar >Assignee: Nanda kumar >Priority: Major > > Javadoc in SCM HA related classes should be updated for better > understandability. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3208) Implement Ratis Snapshots on SCM
[ https://issues.apache.org/jira/browse/HDDS-3208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190992#comment-17190992 ] Rui Wang commented on HDDS-3208: Will try to take a look at this one. > Implement Ratis Snapshots on SCM > > > Key: HDDS-3208 > URL: https://issues.apache.org/jira/browse/HDDS-3208 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Li Cheng >Assignee: Rui Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-3208) Implement Ratis Snapshots on SCM
[ https://issues.apache.org/jira/browse/HDDS-3208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang reassigned HDDS-3208: -- Assignee: Rui Wang > Implement Ratis Snapshots on SCM > > > Key: HDDS-3208 > URL: https://issues.apache.org/jira/browse/HDDS-3208 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Li Cheng >Assignee: Rui Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-3684) Add tests for replication annotation
[ https://issues.apache.org/jira/browse/HDDS-3684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang reassigned HDDS-3684: -- Assignee: Rui Wang > Add tests for replication annotation > > > Key: HDDS-3684 > URL: https://issues.apache.org/jira/browse/HDDS-3684 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Li Cheng >Assignee: Rui Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4189) Use a unified Cli syntax for both getting OM and SCM status
[ https://issues.apache.org/jira/browse/HDDS-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated HDDS-4189: --- Description: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r481380452 suggests a unification for OM and SCM for getting status by Cli. https://github.com/apache/hadoop-ozone/pull/1346 updated for SCM case. This JIRA proposes to change ozone admin om getserviceroles to ozone admin om roles was: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r481380452 suggests a unification for OM and SCM for getting status by Cli. https://github.com/apache/hadoop-ozone/pull/1346 updated for SCM case. This JIRA proposes to change ozone admin om getserviceroles to ozone admin om status > Use a unified Cli syntax for both getting OM and SCM status > --- > > Key: HDDS-4189 > URL: https://issues.apache.org/jira/browse/HDDS-4189 > Project: Hadoop Distributed Data Store > Issue Type: Task >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Labels: pull-request-available > > https://github.com/apache/hadoop-ozone/pull/1346#discussion_r481380452 > suggests a unification for OM and SCM for getting status by Cli. > https://github.com/apache/hadoop-ozone/pull/1346 updated for SCM case. > This JIRA proposes to change > ozone admin om getserviceroles > to > ozone admin om roles -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3103) Have multi-raft pipeline calculator to recommend best pipeline number per datanode
[ https://issues.apache.org/jira/browse/HDDS-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188980#comment-17188980 ] Rui Wang commented on HDDS-3103: [~timmylicheng] Do you have a plan to work on this JIRA in near term? Could I take this one? > Have multi-raft pipeline calculator to recommend best pipeline number per > datanode > -- > > Key: HDDS-3103 > URL: https://issues.apache.org/jira/browse/HDDS-3103 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Affects Versions: 0.5.0 >Reporter: Li Cheng >Assignee: Li Cheng >Priority: Critical > > PipelinePlacementPolicy should have a calculator method to recommend better > number for pipeline number per node. The number used to come from > ozone.datanode.pipeline.limit in config. SCM should be able to consider how > many ratis dir and the ratis retry timeout to recommend the best pipeline > number for every node. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4189) Use a unified Cli syntax for both getting OM and SCM status
Rui Wang created HDDS-4189: -- Summary: Use a unified Cli syntax for both getting OM and SCM status Key: HDDS-4189 URL: https://issues.apache.org/jira/browse/HDDS-4189 Project: Hadoop Distributed Data Store Issue Type: Task Reporter: Rui Wang Assignee: Rui Wang https://github.com/apache/hadoop-ozone/pull/1346#discussion_r481380452 suggests a unification for OM and SCM for getting status by Cli. https://github.com/apache/hadoop-ozone/pull/1346 updated for SCM case. This JIRA proposes to change ozone admin om getserviceroles to ozone admin om status -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3188) Enable Multiple SCMs
[ https://issues.apache.org/jira/browse/HDDS-3188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188114#comment-17188114 ] Rui Wang commented on HDDS-3188: Update JIRA description to say support "2N + 1" SCMs. (The original text was "2N"). > Enable Multiple SCMs > > > Key: HDDS-3188 > URL: https://issues.apache.org/jira/browse/HDDS-3188 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Li Cheng >Assignee: Li Cheng >Priority: Major > Labels: pull-request-available > > Need to supports 2N + 1 SCMs. Add configs and logic to support multiple SCMs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3188) Enable Multiple SCMs
[ https://issues.apache.org/jira/browse/HDDS-3188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated HDDS-3188: --- Description: Need to supports 2N + 1 SCMs. Add configs and logic to support multiple SCMs. (was: Need to supports 2N SCMs. Add configs and logic to support multiple SCMs.) > Enable Multiple SCMs > > > Key: HDDS-3188 > URL: https://issues.apache.org/jira/browse/HDDS-3188 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Li Cheng >Assignee: Li Cheng >Priority: Major > Labels: pull-request-available > > Need to supports 2N + 1 SCMs. Add configs and logic to support multiple SCMs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-4115) CLI command to show current SCM leader and follower status
[ https://issues.apache.org/jira/browse/HDDS-4115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang reassigned HDDS-4115: -- Assignee: Rui Wang > CLI command to show current SCM leader and follower status > -- > > Key: HDDS-4115 > URL: https://issues.apache.org/jira/browse/HDDS-4115 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Li Cheng >Assignee: Rui Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-3409) Update download links
[ https://issues.apache.org/jira/browse/HDDS-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175079#comment-17175079 ] Rui Wang edited comment on HDDS-3409 at 8/10/20, 10:09 PM: --- [~arp] I can help on this JIRA. Do you know where is the source of the website? was (Author: amaliujia): [~arp] I can help on this JIRA. Do you know where is the source of webstie? > Update download links > - > > Key: HDDS-3409 > URL: https://issues.apache.org/jira/browse/HDDS-3409 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: website >Reporter: Arpit Agarwal >Priority: Major > Labels: newbie > > The download links for signatures/checksums/KEYS should be updated from > dist.apache.org to https://downloads.apache.org/hadoop/ozone/. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3409) Update download links
[ https://issues.apache.org/jira/browse/HDDS-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175079#comment-17175079 ] Rui Wang commented on HDDS-3409: [~arp] I can help on this JIRA. Do you know where is the source of webstie? > Update download links > - > > Key: HDDS-3409 > URL: https://issues.apache.org/jira/browse/HDDS-3409 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: website >Reporter: Arpit Agarwal >Priority: Major > Labels: newbie > > The download links for signatures/checksums/KEYS should be updated from > dist.apache.org to https://downloads.apache.org/hadoop/ozone/. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3190) SCM needs to replay RaftLog for recovery
[ https://issues.apache.org/jira/browse/HDDS-3190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175024#comment-17175024 ] Rui Wang commented on HDDS-3190: [~timmylicheng] Can I work on this JIRA? I am willing to contribute to SCM HA project. Do you have any recommendation that what JIRA you think i can help to contribute to? > SCM needs to replay RaftLog for recovery > > > Key: HDDS-3190 > URL: https://issues.apache.org/jira/browse/HDDS-3190 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Li Cheng >Assignee: Rui Wang >Priority: Major > > Need to add a big Proto file for all types of requests to store in RaftLog. > SCM needs to replay RaftLog for recovery. > Note that Proto may have further changes. Until all data structures are > finished, need to leave some room for compatibility. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-3190) SCM needs to replay RaftLog for recovery
[ https://issues.apache.org/jira/browse/HDDS-3190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang reassigned HDDS-3190: -- Assignee: Rui Wang > SCM needs to replay RaftLog for recovery > > > Key: HDDS-3190 > URL: https://issues.apache.org/jira/browse/HDDS-3190 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Li Cheng >Assignee: Rui Wang >Priority: Major > > Need to add a big Proto file for all types of requests to store in RaftLog. > SCM needs to replay RaftLog for recovery. > Note that Proto may have further changes. Until all data structures are > finished, need to leave some room for compatibility. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3684) Add tests for replication annotation
[ https://issues.apache.org/jira/browse/HDDS-3684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17173579#comment-17173579 ] Rui Wang commented on HDDS-3684: [~timmylicheng] I am interested in what you are implementing and seeking for a way to contribute. It might be good for me to start from improving the testing coverage. Can you share more context on this JIRA so I can pick it up? > Add tests for replication annotation > > > Key: HDDS-3684 > URL: https://issues.apache.org/jira/browse/HDDS-3684 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Li Cheng >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1745) Add integration test for createDirectory for OM HA
[ https://issues.apache.org/jira/browse/HDDS-1745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang reassigned HDDS-1745: -- Assignee: Rui Wang > Add integration test for createDirectory for OM HA > -- > > Key: HDDS-1745 > URL: https://issues.apache.org/jira/browse/HDDS-1745 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: OM HA, Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Rui Wang >Priority: Major > Labels: newbie > > Add an integration test for createDirectory which is implemented as part of > HDDS-1730 for OM HA. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4067) Implement toString for OMTransactionInfo
[
https://issues.apache.org/jira/browse/HDDS-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17172933#comment-17172933
]
Rui Wang commented on HDDS-4067:
Thank you! I have created a PR. Any hint for how to test this PR?
> Implement toString for OMTransactionInfo
>
>
> Key: HDDS-4067
> URL: https://issues.apache.org/jira/browse/HDDS-4067
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Rui Wang
>Priority: Major
> Labels: newbie, pull-request-available
>
> During debug
> We see the logs
> {code:java}
> 23:30:36,175 INFO org.apache.hadoop.ozone.om.OzoneManager: Installing
> checkpoint with OMTransactionInfo
> org.apache.hadoop.ozone.om.ratis.OMTransactionInfo@e19e
> {code}
> It would be helpful to print actual transaction info. For this toString need
> to be implemented for OMTransactionInfo.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4067) Implement toString for OMTransactionInfo
[
https://issues.apache.org/jira/browse/HDDS-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171842#comment-17171842
]
Rui Wang commented on HDDS-4067:
Thanks Bharat, I will take this one.
(I will send an email to dev@ later to ask for JIRA permissions).
> Implement toString for OMTransactionInfo
>
>
> Key: HDDS-4067
> URL: https://issues.apache.org/jira/browse/HDDS-4067
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Priority: Major
> Labels: newbie
>
> During debug
> We see the logs
> {code:java}
> 23:30:36,175 INFO org.apache.hadoop.ozone.om.OzoneManager: Installing
> checkpoint with OMTransactionInfo
> org.apache.hadoop.ozone.om.ratis.OMTransactionInfo@e19e
> {code}
> It would be helpful to print actual transaction info. For this toString need
> to be implemented for OMTransactionInfo.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1745) Add integration test for createDirectory for OM HA
[ https://issues.apache.org/jira/browse/HDDS-1745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171631#comment-17171631 ] Rui Wang commented on HDDS-1745: [~hanishakoneru] Appreciate for your response. I was looking for `newbie` label to pick up some easier tasks to work on. All your help will really be useful for me at this moment. > Add integration test for createDirectory for OM HA > -- > > Key: HDDS-1745 > URL: https://issues.apache.org/jira/browse/HDDS-1745 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: OM HA, Ozone Manager >Reporter: Bharat Viswanadham >Priority: Major > Labels: newbie > > Add an integration test for createDirectory which is implemented as part of > HDDS-1730 for OM HA. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1745) Add integration test for createDirectory for OM HA
[ https://issues.apache.org/jira/browse/HDDS-1745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171031#comment-17171031 ] Rui Wang commented on HDDS-1745: In fact, for 1 I find the model in https://issues.apache.org/jira/browse/HDDS-505 also cc [~hanishakoneru] in case you know the answer to 2. > Add integration test for createDirectory for OM HA > -- > > Key: HDDS-1745 > URL: https://issues.apache.org/jira/browse/HDDS-1745 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: OM HA, Ozone Manager >Reporter: Bharat Viswanadham >Priority: Major > Labels: newbie > > Add an integration test for createDirectory which is implemented as part of > HDDS-1730 for OM HA. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-1745) Add integration test for createDirectory for OM HA
[ https://issues.apache.org/jira/browse/HDDS-1745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17169706#comment-17169706 ] Rui Wang edited comment on HDDS-1745 at 8/3/20, 4:20 AM: - [~bharat] Can I work on this JIRA? Can you share some context? Right now I want to know: 1. For OM HA model, which JIRA/paper describes it? (thus I can understand the correct behavior for createDirectory in HA mode) 2. what kind of integration test infrastructure/tool existing so I can utilize to create the test? Any existing integration test that I can use as a reference? was (Author: amaliujia): [~bharat] Can I work on this JIRA? Can you share some context? Right now I want to know: 1. For OM HA model, which JIRA/paper describes it? (thus I can understand the correct behavior for createDirectory in HA mode) 2. what kind of integration test infrastructure/tool existing so I can utilize to create the test? > Add integration test for createDirectory for OM HA > -- > > Key: HDDS-1745 > URL: https://issues.apache.org/jira/browse/HDDS-1745 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: OM HA, Ozone Manager >Reporter: Bharat Viswanadham >Priority: Major > Labels: newbie > > Add an integration test for createDirectory which is implemented as part of > HDDS-1730 for OM HA. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1745) Add integration test for createDirectory for OM HA
[ https://issues.apache.org/jira/browse/HDDS-1745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17169706#comment-17169706 ] Rui Wang commented on HDDS-1745: [~bharat] Can I work on this JIRA? Can you share some context? Right now I want to know: 1. For OM HA model, which JIRA/paper describes it? (thus I can understand the correct behavior for createDirectory in HA mode) 2. what kind of integration test infrastructure/tool existing so I can utilize to create the test? > Add integration test for createDirectory for OM HA > -- > > Key: HDDS-1745 > URL: https://issues.apache.org/jira/browse/HDDS-1745 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: OM HA, Ozone Manager >Reporter: Bharat Viswanadham >Priority: Major > Labels: newbie > > Add an integration test for createDirectory which is implemented as part of > HDDS-1730 for OM HA. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
