Re: [DISCUSS] BookKeeper RocksDB configuration changed default blockCacheSize since 4.15.0
For the pr https://github.com/apache/bookkeeper/pull/3523 In BookKeeper and Pulsar's conf folder, it already has default RocksDB configuration files. In this Pr, the loading order is RocksDB Configuration file > conf/bk_server.conf, which means it will still load the RocksDB configuration file by default and will cause a compatibility issue in the BookKeeper version upgrade. Thanks, Hang Nicolò Boschi 于2022年10月17日周一 16:08写道： > > The pull has been merged and cherry-picked to branch-4.15 > > Thanks, > Nicolò Boschi > > > Il giorno gio 13 ott 2022 alle ore 12:58 Shiji Lu ha > scritto: > > > I think we can start a new discussion about the modification. The > > modification method can be done better. > > we automatically calculate the blockCacheSize through the configuration > > started by bin/bookkeeper, and then update > > conf/entry_location_rocksdb.conf, or I also mention a pr to make a scheme > > comparison ? Or make this function a switch to choose the implementation > > > > > > On 2022/10/09 15:03:01 Hang Chen wrote: > > > I found the PR https://github.com/apache/bookkeeper/pull/3056 has > > > changed the rocksDB default cache size from 10% of direct memory to > > > 206150041(196MB), which will lead to entry read performance decrease > > > when there are huge number of entries stored in the ledger directory. > > > It will have a huge impact on the read performance when the BookKeeper > > > cluster upgrades from 4.14.x to 4.15.x. > > > > > > The motivation of PR 3056 is to change RocksDB configuration to a > > > single configuration file, and it introduces the following drawbacks. > > > 1. Most users, especially those unfamiliar with RocksDB, will be > > > confused about tuning RocksDB performance due to lack of guidance on > > > some important key parameters. > > > 2. The RocksDB blockCacheSize configuration can only be set to a fixed > > > value, not a percentage of direct memory. > > > > > > In order to simplify the RocksDB configuration and make it easy to > > > tune the performance of the RocksDB, I prefer to separate the RocksDB > > > configuration into two parts. > > > 1. The most important configurations, which is usually changed to tune > > > RocksDB performance, will be located in conf/bk_server.conf > > > 2. Other advanced configuration will be in a separate RocksDB > > configuration file > > > > > > For the default RocksDB blockCacheSize change, Do you need to trigger > > > a new release to change it back? Current Pulsar master branch uses > > > BookKeeper 4.14.x, and we can change the default value back to 10% of > > > direct memory to avoid the entry read performance degradation caused > > > by upgrading Pulsar's BookKeeper dependency to 4.15.x. > > > > > > Do you guys have any suggestions? > > > > > > Thanks, > > > Hang > > > > >
Re: [DISCUSS] BookKeeper RocksDB configuration changed default blockCacheSize since 4.15.0
Thank you, Nicolò Boschi! I have updated the website and release note to mention this. https://github.com/apache/bookkeeper/pull/3540 Yong On Mon, 17 Oct 2022 at 16:08, Nicolò Boschi wrote: > The pull has been merged and cherry-picked to branch-4.15 > > Thanks, > Nicolò Boschi > > > Il giorno gio 13 ott 2022 alle ore 12:58 Shiji Lu ha > scritto: > > > I think we can start a new discussion about the modification. The > > modification method can be done better. > > we automatically calculate the blockCacheSize through the configuration > > started by bin/bookkeeper, and then update > > conf/entry_location_rocksdb.conf, or I also mention a pr to make a scheme > > comparison ? Or make this function a switch to choose the implementation > > > > > > On 2022/10/09 15:03:01 Hang Chen wrote: > > > I found the PR https://github.com/apache/bookkeeper/pull/3056 has > > > changed the rocksDB default cache size from 10% of direct memory to > > > 206150041(196MB), which will lead to entry read performance decrease > > > when there are huge number of entries stored in the ledger directory. > > > It will have a huge impact on the read performance when the BookKeeper > > > cluster upgrades from 4.14.x to 4.15.x. > > > > > > The motivation of PR 3056 is to change RocksDB configuration to a > > > single configuration file, and it introduces the following drawbacks. > > > 1. Most users, especially those unfamiliar with RocksDB, will be > > > confused about tuning RocksDB performance due to lack of guidance on > > > some important key parameters. > > > 2. The RocksDB blockCacheSize configuration can only be set to a fixed > > > value, not a percentage of direct memory. > > > > > > In order to simplify the RocksDB configuration and make it easy to > > > tune the performance of the RocksDB, I prefer to separate the RocksDB > > > configuration into two parts. > > > 1. The most important configurations, which is usually changed to tune > > > RocksDB performance, will be located in conf/bk_server.conf > > > 2. Other advanced configuration will be in a separate RocksDB > > configuration file > > > > > > For the default RocksDB blockCacheSize change, Do you need to trigger > > > a new release to change it back? Current Pulsar master branch uses > > > BookKeeper 4.14.x, and we can change the default value back to 10% of > > > direct memory to avoid the entry read performance degradation caused > > > by upgrading Pulsar's BookKeeper dependency to 4.15.x. > > > > > > Do you guys have any suggestions? > > > > > > Thanks, > > > Hang > > > > > >
Re: [DISCUSS] BookKeeper RocksDB configuration changed default blockCacheSize since 4.15.0
The pull has been merged and cherry-picked to branch-4.15 Thanks, Nicolò Boschi Il giorno gio 13 ott 2022 alle ore 12:58 Shiji Lu ha scritto: > I think we can start a new discussion about the modification. The > modification method can be done better. > we automatically calculate the blockCacheSize through the configuration > started by bin/bookkeeper, and then update > conf/entry_location_rocksdb.conf, or I also mention a pr to make a scheme > comparison ? Or make this function a switch to choose the implementation > > > On 2022/10/09 15:03:01 Hang Chen wrote: > > I found the PR https://github.com/apache/bookkeeper/pull/3056 has > > changed the rocksDB default cache size from 10% of direct memory to > > 206150041(196MB), which will lead to entry read performance decrease > > when there are huge number of entries stored in the ledger directory. > > It will have a huge impact on the read performance when the BookKeeper > > cluster upgrades from 4.14.x to 4.15.x. > > > > The motivation of PR 3056 is to change RocksDB configuration to a > > single configuration file, and it introduces the following drawbacks. > > 1. Most users, especially those unfamiliar with RocksDB, will be > > confused about tuning RocksDB performance due to lack of guidance on > > some important key parameters. > > 2. The RocksDB blockCacheSize configuration can only be set to a fixed > > value, not a percentage of direct memory. > > > > In order to simplify the RocksDB configuration and make it easy to > > tune the performance of the RocksDB, I prefer to separate the RocksDB > > configuration into two parts. > > 1. The most important configurations, which is usually changed to tune > > RocksDB performance, will be located in conf/bk_server.conf > > 2. Other advanced configuration will be in a separate RocksDB > configuration file > > > > For the default RocksDB blockCacheSize change, Do you need to trigger > > a new release to change it back? Current Pulsar master branch uses > > BookKeeper 4.14.x, and we can change the default value back to 10% of > > direct memory to avoid the entry read performance degradation caused > > by upgrading Pulsar's BookKeeper dependency to 4.15.x. > > > > Do you guys have any suggestions? > > > > Thanks, > > Hang > > >
Re: [DISCUSS] BookKeeper RocksDB configuration changed default blockCacheSize since 4.15.0
I think we can start a new discussion about the modification. The modification method can be done better. we automatically calculate the blockCacheSize through the configuration started by bin/bookkeeper, and then update conf/entry_location_rocksdb.conf, or I also mention a pr to make a scheme comparison ? Or make this function a switch to choose the implementation On 2022/10/09 15:03:01 Hang Chen wrote: > I found the PR https://github.com/apache/bookkeeper/pull/3056 has > changed the rocksDB default cache size from 10% of direct memory to > 206150041(196MB), which will lead to entry read performance decrease > when there are huge number of entries stored in the ledger directory. > It will have a huge impact on the read performance when the BookKeeper > cluster upgrades from 4.14.x to 4.15.x. > > The motivation of PR 3056 is to change RocksDB configuration to a > single configuration file, and it introduces the following drawbacks. > 1. Most users, especially those unfamiliar with RocksDB, will be > confused about tuning RocksDB performance due to lack of guidance on > some important key parameters. > 2. The RocksDB blockCacheSize configuration can only be set to a fixed > value, not a percentage of direct memory. > > In order to simplify the RocksDB configuration and make it easy to > tune the performance of the RocksDB, I prefer to separate the RocksDB > configuration into two parts. > 1. The most important configurations, which is usually changed to tune > RocksDB performance, will be located in conf/bk_server.conf > 2. Other advanced configuration will be in a separate RocksDB configuration > file > > For the default RocksDB blockCacheSize change, Do you need to trigger > a new release to change it back? Current Pulsar master branch uses > BookKeeper 4.14.x, and we can change the default value back to 10% of > direct memory to avoid the entry read performance degradation caused > by upgrading Pulsar's BookKeeper dependency to 4.15.x. > > Do you guys have any suggestions? > > Thanks, > Hang >
Re: [DISCUSS] BookKeeper RocksDB configuration changed default blockCacheSize since 4.15.0
Il giorno gio 13 ott 2022 alle ore 04:35 Yong Zhang ha scritto: > > Hi, > > I have pushed a PR for fixing this issue. > https://github.com/apache/bookkeeper/pull/3523 > > As we discussed, we ignore the configuration file if they are not exists. > Then using the configurations in the bk server configuration file. > > The rocksDB related configuration file already include in our release. > I want to rename those files with `.default` suffix then every thing would > keep the compatibility with previous versions. Then document it at website > (https://bookkeeper.apache.org/docs/deployment/manual) > to tell people how to use the independent configuration file to configure > the rocksDB. > > But we already have two releases including this, renaming the file also may > impact some people. > Another way is let users to decide which way they want to use. We > will document this in the release note 4.15.0, 4.15.1, and 4.15.2. Tell > people > how to avoid this change impactting their cluster. > > WDYT? As far as I know the main user or the RocksDB based storage is Pulsar I don't think that other users use that backend. The RocksDB backend is usually needed by applications, like Pulsar, that perform catch up reads frequently but most of the BK users (to my knowledge, that is limited) use BK as a Write-Ahead-Log. I believe that we can do the way you suggest and add it to the release notes We also missed to advertise this big breaking change in the 4.15.0 release notes and AFAIK nobody moved to BK 4.15 due to a few problems we discovered and that are still under evaluation Enrico > > Yong > > > On Mon, 10 Oct 2022 at 21:56, Hang Chen wrote: > > > I agree with Yong, we should back-support the existing configuration, > > otherwise, the pulsar helm chart also need to be changed when upgrade > > BookKeeper 4.14.x to 4.15.x > > > > Thanks, > > Hang > > > > Enrico Olivelli 于2022年10月10日周一 16:07写道： > > > > > > Il Lun 10 Ott 2022, 07:44 Yong Zhang ha > > > scritto: > > > > > > > We shouldn't remove any existing configuration or feature in a so short > > > > time. I think at least we need to back-support the existing > > configuration. > > > > If we must do the break, we should make the existing things in a > > > > deprecation > > > > state for a while. > > > > > > > > Breaking existing configurations would make a huge impact. > > > > Maybe we can add back the previous configurations in the next minor > > > > release for 4.15. So user can upgrade their cluster to 4.15.3 without > > > > changing any configuration. > > > > > > > > WDYT? > > > > > > > > > > Great idea > > > > > > Enrico > > > > > > > > > > Yong > > > > > > > > > > > > > > > > > > > > On Sun, 9 Oct 2022 at 23:20, Enrico Olivelli > > wrote: > > > > > > > > > Il Dom 9 Ott 2022, 17:03 Hang Chen ha scritto: > > > > > > > > > > > I found the PR https://github.com/apache/bookkeeper/pull/3056 has > > > > > > changed the rocksDB default cache size from 10% of direct memory to > > > > > > 206150041(196MB), which will lead to entry read performance > > decrease > > > > > > when there are huge number of entries stored in the ledger > > directory. > > > > > > It will have a huge impact on the read performance when the > > BookKeeper > > > > > > cluster upgrades from 4.14.x to 4.15.x. > > > > > > > > > > > > > > > > > > > > > I think that that patch shoud had passed from the BP process. > > > > > It is a huge breaking change. > > > > > Personally I missed it. > > > > > > > > > > BTW now it is there, we should have talked about it in the release > > notes, > > > > > and we now must add some guidance to users upgrading to this version > > of > > > > BK. > > > > > > > > > > It is a pity that we cannot keep the automatic value. > > > > > BTW in all the Pulsar cluster I know I think that they are > > overriding > > > > that > > > > > value because the default value is not big enough. > > > > > > > > > > I think that at this point we only have to document on the website > > the > > > > > breaking change and add a page about BK and RocksDB > > > > > > > > > > Enrico > > > > > > > > > > > > > > > > The motivation of PR 3056 is to change RocksDB configuration to a > > > > > > single configuration file, and it introduces the following > > drawbacks. > > > > > > 1. Most users, especially those unfamiliar with RocksDB, will be > > > > > > confused about tuning RocksDB performance due to lack of guidance > > on > > > > > > some important key parameters. > > > > > > 2. The RocksDB blockCacheSize configuration can only be set to a > > fixed > > > > > > value, not a percentage of direct memory. > > > > > > > > > > > > In order to simplify the RocksDB configuration and make it easy to > > > > > > tune the performance of the RocksDB, I prefer to separate the > > RocksDB > > > > > > configuration into two parts. > > > > > > 1. The most important configurations, which is usually changed to > > tune > > > > > > RocksDB performance, will be located in conf/bk_server.conf > > > > > > 2. Oth
Re: [DISCUSS] BookKeeper RocksDB configuration changed default blockCacheSize since 4.15.0
Hi, I have pushed a PR for fixing this issue. https://github.com/apache/bookkeeper/pull/3523 As we discussed, we ignore the configuration file if they are not exists. Then using the configurations in the bk server configuration file. The rocksDB related configuration file already include in our release. I want to rename those files with `.default` suffix then every thing would keep the compatibility with previous versions. Then document it at website (https://bookkeeper.apache.org/docs/deployment/manual) to tell people how to use the independent configuration file to configure the rocksDB. But we already have two releases including this, renaming the file also may impact some people. Another way is let users to decide which way they want to use. We will document this in the release note 4.15.0, 4.15.1, and 4.15.2. Tell people how to avoid this change impactting their cluster. WDYT? Yong On Mon, 10 Oct 2022 at 21:56, Hang Chen wrote: > I agree with Yong, we should back-support the existing configuration, > otherwise, the pulsar helm chart also need to be changed when upgrade > BookKeeper 4.14.x to 4.15.x > > Thanks, > Hang > > Enrico Olivelli 于2022年10月10日周一 16:07写道： > > > > Il Lun 10 Ott 2022, 07:44 Yong Zhang ha > > scritto: > > > > > We shouldn't remove any existing configuration or feature in a so short > > > time. I think at least we need to back-support the existing > configuration. > > > If we must do the break, we should make the existing things in a > > > deprecation > > > state for a while. > > > > > > Breaking existing configurations would make a huge impact. > > > Maybe we can add back the previous configurations in the next minor > > > release for 4.15. So user can upgrade their cluster to 4.15.3 without > > > changing any configuration. > > > > > > WDYT? > > > > > > > Great idea > > > > Enrico > > > > > > > Yong > > > > > > > > > > > > > > > On Sun, 9 Oct 2022 at 23:20, Enrico Olivelli > wrote: > > > > > > > Il Dom 9 Ott 2022, 17:03 Hang Chen ha scritto: > > > > > > > > > I found the PR https://github.com/apache/bookkeeper/pull/3056 has > > > > > changed the rocksDB default cache size from 10% of direct memory to > > > > > 206150041(196MB), which will lead to entry read performance > decrease > > > > > when there are huge number of entries stored in the ledger > directory. > > > > > It will have a huge impact on the read performance when the > BookKeeper > > > > > cluster upgrades from 4.14.x to 4.15.x. > > > > > > > > > > > > > > > > > I think that that patch shoud had passed from the BP process. > > > > It is a huge breaking change. > > > > Personally I missed it. > > > > > > > > BTW now it is there, we should have talked about it in the release > notes, > > > > and we now must add some guidance to users upgrading to this version > of > > > BK. > > > > > > > > It is a pity that we cannot keep the automatic value. > > > > BTW in all the Pulsar cluster I know I think that they are > overriding > > > that > > > > value because the default value is not big enough. > > > > > > > > I think that at this point we only have to document on the website > the > > > > breaking change and add a page about BK and RocksDB > > > > > > > > Enrico > > > > > > > > > > > > > The motivation of PR 3056 is to change RocksDB configuration to a > > > > > single configuration file, and it introduces the following > drawbacks. > > > > > 1. Most users, especially those unfamiliar with RocksDB, will be > > > > > confused about tuning RocksDB performance due to lack of guidance > on > > > > > some important key parameters. > > > > > 2. The RocksDB blockCacheSize configuration can only be set to a > fixed > > > > > value, not a percentage of direct memory. > > > > > > > > > > In order to simplify the RocksDB configuration and make it easy to > > > > > tune the performance of the RocksDB, I prefer to separate the > RocksDB > > > > > configuration into two parts. > > > > > 1. The most important configurations, which is usually changed to > tune > > > > > RocksDB performance, will be located in conf/bk_server.conf > > > > > 2. Other advanced configuration will be in a separate RocksDB > > > > > configuration file > > > > > > > > > > For the default RocksDB blockCacheSize change, Do you need to > trigger > > > > > a new release to change it back? Current Pulsar master branch uses > > > > > BookKeeper 4.14.x, and we can change the default value back to 10% > of > > > > > direct memory to avoid the entry read performance degradation > caused > > > > > by upgrading Pulsar's BookKeeper dependency to 4.15.x. > > > > > > > > > > Do you guys have any suggestions? > > > > > > > > > > Thanks, > > > > > Hang > > > > > > > > > > > > >
Re: [DISCUSS] BookKeeper RocksDB configuration changed default blockCacheSize since 4.15.0
I agree with Yong, we should back-support the existing configuration, otherwise, the pulsar helm chart also need to be changed when upgrade BookKeeper 4.14.x to 4.15.x Thanks, Hang Enrico Olivelli 于2022年10月10日周一 16:07写道： > > Il Lun 10 Ott 2022, 07:44 Yong Zhang ha > scritto: > > > We shouldn't remove any existing configuration or feature in a so short > > time. I think at least we need to back-support the existing configuration. > > If we must do the break, we should make the existing things in a > > deprecation > > state for a while. > > > > Breaking existing configurations would make a huge impact. > > Maybe we can add back the previous configurations in the next minor > > release for 4.15. So user can upgrade their cluster to 4.15.3 without > > changing any configuration. > > > > WDYT? > > > > Great idea > > Enrico > > > > Yong > > > > > > > > > > On Sun, 9 Oct 2022 at 23:20, Enrico Olivelli wrote: > > > > > Il Dom 9 Ott 2022, 17:03 Hang Chen ha scritto: > > > > > > > I found the PR https://github.com/apache/bookkeeper/pull/3056 has > > > > changed the rocksDB default cache size from 10% of direct memory to > > > > 206150041(196MB), which will lead to entry read performance decrease > > > > when there are huge number of entries stored in the ledger directory. > > > > It will have a huge impact on the read performance when the BookKeeper > > > > cluster upgrades from 4.14.x to 4.15.x. > > > > > > > > > > > > > I think that that patch shoud had passed from the BP process. > > > It is a huge breaking change. > > > Personally I missed it. > > > > > > BTW now it is there, we should have talked about it in the release notes, > > > and we now must add some guidance to users upgrading to this version of > > BK. > > > > > > It is a pity that we cannot keep the automatic value. > > > BTW in all the Pulsar cluster I know I think that they are overriding > > that > > > value because the default value is not big enough. > > > > > > I think that at this point we only have to document on the website the > > > breaking change and add a page about BK and RocksDB > > > > > > Enrico > > > > > > > > > > The motivation of PR 3056 is to change RocksDB configuration to a > > > > single configuration file, and it introduces the following drawbacks. > > > > 1. Most users, especially those unfamiliar with RocksDB, will be > > > > confused about tuning RocksDB performance due to lack of guidance on > > > > some important key parameters. > > > > 2. The RocksDB blockCacheSize configuration can only be set to a fixed > > > > value, not a percentage of direct memory. > > > > > > > > In order to simplify the RocksDB configuration and make it easy to > > > > tune the performance of the RocksDB, I prefer to separate the RocksDB > > > > configuration into two parts. > > > > 1. The most important configurations, which is usually changed to tune > > > > RocksDB performance, will be located in conf/bk_server.conf > > > > 2. Other advanced configuration will be in a separate RocksDB > > > > configuration file > > > > > > > > For the default RocksDB blockCacheSize change, Do you need to trigger > > > > a new release to change it back? Current Pulsar master branch uses > > > > BookKeeper 4.14.x, and we can change the default value back to 10% of > > > > direct memory to avoid the entry read performance degradation caused > > > > by upgrading Pulsar's BookKeeper dependency to 4.15.x. > > > > > > > > Do you guys have any suggestions? > > > > > > > > Thanks, > > > > Hang > > > > > > > > >
Re: [DISCUSS] BookKeeper RocksDB configuration changed default blockCacheSize since 4.15.0
Il Lun 10 Ott 2022, 07:44 Yong Zhang ha scritto: > We shouldn't remove any existing configuration or feature in a so short > time. I think at least we need to back-support the existing configuration. > If we must do the break, we should make the existing things in a > deprecation > state for a while. > > Breaking existing configurations would make a huge impact. > Maybe we can add back the previous configurations in the next minor > release for 4.15. So user can upgrade their cluster to 4.15.3 without > changing any configuration. > > WDYT? > Great idea Enrico > Yong > > > > > On Sun, 9 Oct 2022 at 23:20, Enrico Olivelli wrote: > > > Il Dom 9 Ott 2022, 17:03 Hang Chen ha scritto: > > > > > I found the PR https://github.com/apache/bookkeeper/pull/3056 has > > > changed the rocksDB default cache size from 10% of direct memory to > > > 206150041(196MB), which will lead to entry read performance decrease > > > when there are huge number of entries stored in the ledger directory. > > > It will have a huge impact on the read performance when the BookKeeper > > > cluster upgrades from 4.14.x to 4.15.x. > > > > > > > > > I think that that patch shoud had passed from the BP process. > > It is a huge breaking change. > > Personally I missed it. > > > > BTW now it is there, we should have talked about it in the release notes, > > and we now must add some guidance to users upgrading to this version of > BK. > > > > It is a pity that we cannot keep the automatic value. > > BTW in all the Pulsar cluster I know I think that they are overriding > that > > value because the default value is not big enough. > > > > I think that at this point we only have to document on the website the > > breaking change and add a page about BK and RocksDB > > > > Enrico > > > > > > > The motivation of PR 3056 is to change RocksDB configuration to a > > > single configuration file, and it introduces the following drawbacks. > > > 1. Most users, especially those unfamiliar with RocksDB, will be > > > confused about tuning RocksDB performance due to lack of guidance on > > > some important key parameters. > > > 2. The RocksDB blockCacheSize configuration can only be set to a fixed > > > value, not a percentage of direct memory. > > > > > > In order to simplify the RocksDB configuration and make it easy to > > > tune the performance of the RocksDB, I prefer to separate the RocksDB > > > configuration into two parts. > > > 1. The most important configurations, which is usually changed to tune > > > RocksDB performance, will be located in conf/bk_server.conf > > > 2. Other advanced configuration will be in a separate RocksDB > > > configuration file > > > > > > For the default RocksDB blockCacheSize change, Do you need to trigger > > > a new release to change it back? Current Pulsar master branch uses > > > BookKeeper 4.14.x, and we can change the default value back to 10% of > > > direct memory to avoid the entry read performance degradation caused > > > by upgrading Pulsar's BookKeeper dependency to 4.15.x. > > > > > > Do you guys have any suggestions? > > > > > > Thanks, > > > Hang > > > > > >
Re: [DISCUSS] BookKeeper RocksDB configuration changed default blockCacheSize since 4.15.0
We shouldn't remove any existing configuration or feature in a so short time. I think at least we need to back-support the existing configuration. If we must do the break, we should make the existing things in a deprecation state for a while. Breaking existing configurations would make a huge impact. Maybe we can add back the previous configurations in the next minor release for 4.15. So user can upgrade their cluster to 4.15.3 without changing any configuration. WDYT? Yong On Sun, 9 Oct 2022 at 23:20, Enrico Olivelli wrote: > Il Dom 9 Ott 2022, 17:03 Hang Chen ha scritto: > > > I found the PR https://github.com/apache/bookkeeper/pull/3056 has > > changed the rocksDB default cache size from 10% of direct memory to > > 206150041(196MB), which will lead to entry read performance decrease > > when there are huge number of entries stored in the ledger directory. > > It will have a huge impact on the read performance when the BookKeeper > > cluster upgrades from 4.14.x to 4.15.x. > > > > > I think that that patch shoud had passed from the BP process. > It is a huge breaking change. > Personally I missed it. > > BTW now it is there, we should have talked about it in the release notes, > and we now must add some guidance to users upgrading to this version of BK. > > It is a pity that we cannot keep the automatic value. > BTW in all the Pulsar cluster I know I think that they are overriding that > value because the default value is not big enough. > > I think that at this point we only have to document on the website the > breaking change and add a page about BK and RocksDB > > Enrico > > > > The motivation of PR 3056 is to change RocksDB configuration to a > > single configuration file, and it introduces the following drawbacks. > > 1. Most users, especially those unfamiliar with RocksDB, will be > > confused about tuning RocksDB performance due to lack of guidance on > > some important key parameters. > > 2. The RocksDB blockCacheSize configuration can only be set to a fixed > > value, not a percentage of direct memory. > > > > In order to simplify the RocksDB configuration and make it easy to > > tune the performance of the RocksDB, I prefer to separate the RocksDB > > configuration into two parts. > > 1. The most important configurations, which is usually changed to tune > > RocksDB performance, will be located in conf/bk_server.conf > > 2. Other advanced configuration will be in a separate RocksDB > > configuration file > > > > For the default RocksDB blockCacheSize change, Do you need to trigger > > a new release to change it back? Current Pulsar master branch uses > > BookKeeper 4.14.x, and we can change the default value back to 10% of > > direct memory to avoid the entry read performance degradation caused > > by upgrading Pulsar's BookKeeper dependency to 4.15.x. > > > > Do you guys have any suggestions? > > > > Thanks, > > Hang > > >
Re: [DISCUSS] BookKeeper RocksDB configuration changed default blockCacheSize since 4.15.0
Il Dom 9 Ott 2022, 17:03 Hang Chen ha scritto: > I found the PR https://github.com/apache/bookkeeper/pull/3056 has > changed the rocksDB default cache size from 10% of direct memory to > 206150041(196MB), which will lead to entry read performance decrease > when there are huge number of entries stored in the ledger directory. > It will have a huge impact on the read performance when the BookKeeper > cluster upgrades from 4.14.x to 4.15.x. > I think that that patch shoud had passed from the BP process. It is a huge breaking change. Personally I missed it. BTW now it is there, we should have talked about it in the release notes, and we now must add some guidance to users upgrading to this version of BK. It is a pity that we cannot keep the automatic value. BTW in all the Pulsar cluster I know I think that they are overriding that value because the default value is not big enough. I think that at this point we only have to document on the website the breaking change and add a page about BK and RocksDB Enrico > The motivation of PR 3056 is to change RocksDB configuration to a > single configuration file, and it introduces the following drawbacks. > 1. Most users, especially those unfamiliar with RocksDB, will be > confused about tuning RocksDB performance due to lack of guidance on > some important key parameters. > 2. The RocksDB blockCacheSize configuration can only be set to a fixed > value, not a percentage of direct memory. > > In order to simplify the RocksDB configuration and make it easy to > tune the performance of the RocksDB, I prefer to separate the RocksDB > configuration into two parts. > 1. The most important configurations, which is usually changed to tune > RocksDB performance, will be located in conf/bk_server.conf > 2. Other advanced configuration will be in a separate RocksDB > configuration file > > For the default RocksDB blockCacheSize change, Do you need to trigger > a new release to change it back? Current Pulsar master branch uses > BookKeeper 4.14.x, and we can change the default value back to 10% of > direct memory to avoid the entry read performance degradation caused > by upgrading Pulsar's BookKeeper dependency to 4.15.x. > > Do you guys have any suggestions? > > Thanks, > Hang >
[DISCUSS] BookKeeper RocksDB configuration changed default blockCacheSize since 4.15.0
I found the PR https://github.com/apache/bookkeeper/pull/3056 has changed the rocksDB default cache size from 10% of direct memory to 206150041(196MB), which will lead to entry read performance decrease when there are huge number of entries stored in the ledger directory. It will have a huge impact on the read performance when the BookKeeper cluster upgrades from 4.14.x to 4.15.x. The motivation of PR 3056 is to change RocksDB configuration to a single configuration file, and it introduces the following drawbacks. 1. Most users, especially those unfamiliar with RocksDB, will be confused about tuning RocksDB performance due to lack of guidance on some important key parameters. 2. The RocksDB blockCacheSize configuration can only be set to a fixed value, not a percentage of direct memory. In order to simplify the RocksDB configuration and make it easy to tune the performance of the RocksDB, I prefer to separate the RocksDB configuration into two parts. 1. The most important configurations, which is usually changed to tune RocksDB performance, will be located in conf/bk_server.conf 2. Other advanced configuration will be in a separate RocksDB configuration file For the default RocksDB blockCacheSize change, Do you need to trigger a new release to change it back? Current Pulsar master branch uses BookKeeper 4.14.x, and we can change the default value back to 10% of direct memory to avoid the entry read performance degradation caused by upgrading Pulsar's BookKeeper dependency to 4.15.x. Do you guys have any suggestions? Thanks, Hang
