[jira] [Commented] (HDFS-11601) Ozone: Compact DB should be called on Open Containers.
[ https://issues.apache.org/jira/browse/HDFS-11601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172499#comment-16172499 ] Weiwei Yang commented on HDFS-11601: Hi [~anu] I think the problem is a bit more complex, with 3 million keys, I was able to see a significant performance issue on list buckets (haven't fully tested, but there might be more places like listbuckets). This happens when accessing the KSM DB with getRangeKVs call, it seems when iterating entries from a point with a filter is pretty slow (filter entries is slow), I am trying to narrow down the issue right now, not really get into the compact part yet. Will keep you posted. Thanks for checking. > Ozone: Compact DB should be called on Open Containers. > -- > > Key: HDFS-11601 > URL: https://issues.apache.org/jira/browse/HDFS-11601 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Weiwei Yang > Labels: ozoneMerge, tocheck > > The discussion in HDFS-11594 pointed to a potential issue that we might run > into. That is too many delete key operations can take place and make a DB > slow. Running compactDB in those cases are useful. Currently we run compactDB > when we close a container. This JIRA tracks a potential improvement of > running compactDB even on open containers. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11601) Ozone: Compact DB should be called on Open Containers.
[ https://issues.apache.org/jira/browse/HDFS-11601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172133#comment-16172133 ] Anu Engineer commented on HDFS-11601: - [~cheersyang] With the new scale that we have seen, more than 3 million keys, do you know if this is an issue? or do we much larger numbers? say 10 million ozone keys? > Ozone: Compact DB should be called on Open Containers. > -- > > Key: HDFS-11601 > URL: https://issues.apache.org/jira/browse/HDFS-11601 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Weiwei Yang > Labels: ozoneMerge, tocheck > > The discussion in HDFS-11594 pointed to a potential issue that we might run > into. That is too many delete key operations can take place and make a DB > slow. Running compactDB in those cases are useful. Currently we run compactDB > when we close a container. This JIRA tracks a potential improvement of > running compactDB even on open containers. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11601) Ozone: Compact DB should be called on Open Containers.
[ https://issues.apache.org/jira/browse/HDFS-11601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156643#comment-16156643 ] Weiwei Yang commented on HDFS-11601: Hi [~anu], I agree. I am trying to test in scale too. Will see if I can find out the problem we deal with. Thanks > Ozone: Compact DB should be called on Open Containers. > -- > > Key: HDFS-11601 > URL: https://issues.apache.org/jira/browse/HDFS-11601 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Weiwei Yang > Labels: ozoneMerge, tocheck > > The discussion in HDFS-11594 pointed to a potential issue that we might run > into. That is too many delete key operations can take place and make a DB > slow. Running compactDB in those cases are useful. Currently we run compactDB > when we close a container. This JIRA tracks a potential improvement of > running compactDB even on open containers. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11601) Ozone: Compact DB should be called on Open Containers.
[ https://issues.apache.org/jira/browse/HDFS-11601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156142#comment-16156142 ] Anu Engineer commented on HDFS-11601: - [~cheersyang] Since we have RocksDB now, I think we should wait until we get more testing with scale and see if we really need to do anything here. I have marked it as ozoneMerge so that we will be forced to look at this before merge. > Ozone: Compact DB should be called on Open Containers. > -- > > Key: HDFS-11601 > URL: https://issues.apache.org/jira/browse/HDFS-11601 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Weiwei Yang > Labels: ozoneMerge, tocheck > > The discussion in HDFS-11594 pointed to a potential issue that we might run > into. That is too many delete key operations can take place and make a DB > slow. Running compactDB in those cases are useful. Currently we run compactDB > when we close a container. This JIRA tracks a potential improvement of > running compactDB even on open containers. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11601) Ozone: Compact DB should be called on Open Containers.
[ https://issues.apache.org/jira/browse/HDFS-11601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16003980#comment-16003980 ] Weiwei Yang commented on HDFS-11601: Thanks [~anu] for the heads up, I will keep an eye on the statues of rocksDB. I may not start to work on this until we get to there. Let me decrease this one's priority for now on my list. Thanks. > Ozone: Compact DB should be called on Open Containers. > -- > > Key: HDFS-11601 > URL: https://issues.apache.org/jira/browse/HDFS-11601 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Weiwei Yang > > The discussion in HDFS-11594 pointed to a potential issue that we might run > into. That is too many delete key operations can take place and make a DB > slow. Running compactDB in those cases are useful. Currently we run compactDB > when we close a container. This JIRA tracks a potential improvement of > running compactDB even on open containers. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11601) Ozone: Compact DB should be called on Open Containers.
[ https://issues.apache.org/jira/browse/HDFS-11601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16003053#comment-16003053 ] Anu Engineer commented on HDFS-11601: - [~cheersyang] I am toying with the idea of bring in rocksDB and one of the features that rocks have is better compaction. Just a heads up. > Ozone: Compact DB should be called on Open Containers. > -- > > Key: HDFS-11601 > URL: https://issues.apache.org/jira/browse/HDFS-11601 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Weiwei Yang > > The discussion in HDFS-11594 pointed to a potential issue that we might run > into. That is too many delete key operations can take place and make a DB > slow. Running compactDB in those cases are useful. Currently we run compactDB > when we close a container. This JIRA tracks a potential improvement of > running compactDB even on open containers. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11601) Ozone: Compact DB should be called on Open Containers.
[ https://issues.apache.org/jira/browse/HDFS-11601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15950164#comment-15950164 ] Anu Engineer commented on HDFS-11601: - Thanks for the offer. I have assigned this JIRA to you. > Ozone: Compact DB should be called on Open Containers. > -- > > Key: HDFS-11601 > URL: https://issues.apache.org/jira/browse/HDFS-11601 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Weiwei Yang > > The discussion in HDFS-11594 pointed to a potential issue that we might run > into. That is too many delete key operations can take place and make a DB > slow. Running compactDB in those cases are useful. Currently we run compactDB > when we close a container. This JIRA tracks a potential improvement of > running compactDB even on open containers. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11601) Ozone: Compact DB should be called on Open Containers.
[ https://issues.apache.org/jira/browse/HDFS-11601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15950161#comment-15950161 ] Weiwei Yang commented on HDFS-11601: Hi [~anu] I think we need to do some performance evaluation to get a better understanding the problem we deal with, then decide if we need or how to do compact properly at the backend. If you don't have plan to work on this anytime soon, you can assign it to me so I can do some work when other higher priority tasks are done. Thanks. > Ozone: Compact DB should be called on Open Containers. > -- > > Key: HDFS-11601 > URL: https://issues.apache.org/jira/browse/HDFS-11601 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer > > The discussion in HDFS-11594 pointed to a potential issue that we might run > into. That is too many delete key operations can take place and make a DB > slow. Running compactDB in those cases are useful. Currently we run compactDB > when we close a container. This JIRA tracks a potential improvement of > running compactDB even on open containers. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
