[kudu-CR] [log block manager] Write lock for deletion
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20901 ) Change subject: [log_block_manager] Write lock for deletion .. Patch Set 16: Build Started http://jenkins.kudu.apache.org/job/pre_commit/193/ -- To view, visit http://gerrit.cloudera.org:8080/20901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73712a5fd8eccf23621f8abda3e99ebdf10a769f Gerrit-Change-Number: 20901 Gerrit-PatchSet: 16 Gerrit-Owner: Ádám Bakai Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Comment-Date: Tue, 04 Jun 2024 09:38:10 + Gerrit-HasComments: No
[kudu-CR] [log block manager] Write lock for deletion
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20901 ) Change subject: [log_block_manager] Write lock for deletion .. Patch Set 16: Verified-1 Build Failed http://jenkins.kudu.apache.org/job/pre_commit/193/ : FAILURE -- To view, visit http://gerrit.cloudera.org:8080/20901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73712a5fd8eccf23621f8abda3e99ebdf10a769f Gerrit-Change-Number: 20901 Gerrit-PatchSet: 16 Gerrit-Owner: Ádám Bakai Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Comment-Date: Tue, 04 Jun 2024 10:37:09 + Gerrit-HasComments: No
[kudu-CR] Add a benchmark for CBTree concurrent writes.
Zoltan Martonka has posted comments on this change. ( http://gerrit.cloudera.org:8080/21447 ) Change subject: Add a benchmark for CBTree concurrent writes. .. Patch Set 8: (2 comments) I think it is cleaner to add it to benchmark suit in a separate commit. I checked the Jenkins runs. Only TSAN fails with tests other than this file. @Alexey, is this ok to override Jenkins results, or should I keep updating until it passes? http://gerrit.cloudera.org:8080/#/c/21447/4/src/kudu/tablet/cbtree-test.cc File src/kudu/tablet/cbtree-test.cc: http://gerrit.cloudera.org:8080/#/c/21447/4/src/kudu/tablet/cbtree-test.cc@57 PS4, Line 57: N > You could use the same logic to output of hardware_concurrency() as you did I would like to keep it this way. http://gerrit.cloudera.org:8080/#/c/21447/8/src/kudu/tablet/cbtree-test.cc File src/kudu/tablet/cbtree-test.cc: http://gerrit.cloudera.org:8080/#/c/21447/8/src/kudu/tablet/cbtree-test.cc@924 PS8, Line 924: ConcurrentReadWriteBenchmark > Don't want to delay your commit. I would prefer to add it in a separate change. I will push the change to gerrit, when this is merged. -- To view, visit http://gerrit.cloudera.org:8080/21447 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1b0b16e269c70716962fc5ebb4ddca1e2cbe68a4 Gerrit-Change-Number: 21447 Gerrit-PatchSet: 8 Gerrit-Owner: Zoltan Martonka Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Ashwani Raina Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Zoltan Chovan Gerrit-Reviewer: Zoltan Martonka Gerrit-Comment-Date: Tue, 04 Jun 2024 11:04:25 + Gerrit-HasComments: Yes
[kudu-CR] Add a benchmark for CBTree concurrent writes.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/21447 ) Change subject: Add a benchmark for CBTree concurrent writes. .. Patch Set 8: > (2 comments) > > I think it is cleaner to add it to benchmark suit in a separate > commit. > I checked the Jenkins runs. Only TSAN fails with tests other than > this file. > @Alexey, is this ok to override Jenkins results, or should I keep > updating until it passes? If none of the relevant tests fails, then it's better to override the pre-commit test results: there are many flaky tests, and it's hard to rely on when they all pass (flaky tests stats and retries should help with that). -- To view, visit http://gerrit.cloudera.org:8080/21447 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1b0b16e269c70716962fc5ebb4ddca1e2cbe68a4 Gerrit-Change-Number: 21447 Gerrit-PatchSet: 8 Gerrit-Owner: Zoltan Martonka Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Ashwani Raina Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Zoltan Chovan Gerrit-Reviewer: Zoltan Martonka Gerrit-Comment-Date: Tue, 04 Jun 2024 14:04:28 + Gerrit-HasComments: No
[kudu-CR] Add a benchmark for CBTree concurrent writes.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/21447 ) Change subject: Add a benchmark for CBTree concurrent writes. .. Patch Set 8: Verified+1 unrelated test failures -- To view, visit http://gerrit.cloudera.org:8080/21447 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1b0b16e269c70716962fc5ebb4ddca1e2cbe68a4 Gerrit-Change-Number: 21447 Gerrit-PatchSet: 8 Gerrit-Owner: Zoltan Martonka Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Ashwani Raina Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Zoltan Chovan Gerrit-Reviewer: Zoltan Martonka Gerrit-Comment-Date: Tue, 04 Jun 2024 14:04:43 + Gerrit-HasComments: No
[kudu-CR] Add a benchmark for CBTree concurrent writes.
Alexey Serbin has removed a vote on this change. Change subject: Add a benchmark for CBTree concurrent writes. .. Removed Verified-1 by Kudu Jenkins (120) -- To view, visit http://gerrit.cloudera.org:8080/21447 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I1b0b16e269c70716962fc5ebb4ddca1e2cbe68a4 Gerrit-Change-Number: 21447 Gerrit-PatchSet: 8 Gerrit-Owner: Zoltan Martonka Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Ashwani Raina Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Zoltan Chovan Gerrit-Reviewer: Zoltan Martonka
[kudu-CR] Add a benchmark for CBTree concurrent writes.
Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21447 ) Change subject: Add a benchmark for CBTree concurrent writes. .. Add a benchmark for CBTree concurrent writes. Before updating CBTree for ARM (where it is misbehaving currently), we should have a proper test for two scenarios: + Writing on multiple threads. + Reading on multiple threads while there are also active writes. If read threads wait for values to be inserted, it defeats the purpose of benchmarking. Therefore, we should first populate a tree with values for the read threads. The read threads will then read values that are already in the tree, while the write threads continue to insert new values. Setting up the tree for the second scenario essentially involves performing the first scenario. This is why both scenarios are combined into a single test. The new test provides the following new features (compared to just running DoTestConcurrentInsert with higher parameters): + Different threads read the value that inserted it + Reader threads can't be assigned to a certain writer thread. + Keys are better distributed than the previous shuffle method. + Allows measuring read-heavy performance (with a flag). Reading threads start concurrently with writing threads, not at the end of each write thread (unlike DoTestConcurrentInsert). Note that running only concurrent reads should not differ from TestScanPerformance, since no locking takes place and they do not sabotage each other. So no new test is required for that scenario. Change-Id: I1b0b16e269c70716962fc5ebb4ddca1e2cbe68a4 Reviewed-on: http://gerrit.cloudera.org:8080/21447 Reviewed-by: Zoltan Chovan Reviewed-by: Ashwani Raina Reviewed-by: Alexey Serbin Tested-by: Alexey Serbin --- M src/kudu/tablet/cbtree-test.cc 1 file changed, 197 insertions(+), 0 deletions(-) Approvals: Zoltan Chovan: Looks good to me, but someone else must approve Ashwani Raina: Looks good to me, but someone else must approve Alexey Serbin: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/21447 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I1b0b16e269c70716962fc5ebb4ddca1e2cbe68a4 Gerrit-Change-Number: 21447 Gerrit-PatchSet: 9 Gerrit-Owner: Zoltan Martonka Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Ashwani Raina Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Zoltan Chovan Gerrit-Reviewer: Zoltan Martonka
[kudu-CR] [build] bootstrap-dev-env.sh fix for ubuntu 22.04+
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20559 ) Change subject: [build] bootstrap-dev-env.sh fix for ubuntu 22.04+ .. Patch Set 2: Verified+1 Build Successful http://jenkins.kudu.apache.org/job/pre_commit/194/ : SUCCESS -- To view, visit http://gerrit.cloudera.org:8080/20559 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I275fa06214fde6ede1c6dd8e85f786fdc950cf8c Gerrit-Change-Number: 20559 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Martonka Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber Gerrit-Comment-Date: Tue, 04 Jun 2024 15:37:13 + Gerrit-HasComments: No
[kudu-CR] [build] bootstrap-dev-env.sh fix for ubuntu 22.04+
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20559 ) Change subject: [build] bootstrap-dev-env.sh fix for ubuntu 22.04+ .. Patch Set 2: Build Started http://jenkins.kudu.apache.org/job/pre_commit/194/ -- To view, visit http://gerrit.cloudera.org:8080/20559 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I275fa06214fde6ede1c6dd8e85f786fdc950cf8c Gerrit-Change-Number: 20559 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Martonka Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber Gerrit-Comment-Date: Tue, 04 Jun 2024 15:36:51 + Gerrit-HasComments: No
[kudu-CR] [build] bootstrap-dev-env.sh fix for ubuntu 22.04+
Hello Marton Greber, Alexey Serbin, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20559 to look at the new patch set (#2). Change subject: [build] bootstrap-dev-env.sh fix for ubuntu 22.04+ .. [build] bootstrap-dev-env.sh fix for ubuntu 22.04+ On Ubuntu 22.04 python package is renamed to python2 On 23.10 and 24.04 python2 is no longer available. We should just use python3 on newer platforms. Tested on: 18.04, 20.04. 22.04, 23.10, 24.04 Change-Id: I275fa06214fde6ede1c6dd8e85f786fdc950cf8c --- M docker/bootstrap-dev-env.sh 1 file changed, 8 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/59/20559/2 -- To view, visit http://gerrit.cloudera.org:8080/20559 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I275fa06214fde6ede1c6dd8e85f786fdc950cf8c Gerrit-Change-Number: 20559 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Martonka Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber
[kudu-CR] [build] bootstrap-dev-env.sh fix for ubuntu 22.04+
Anonymous Coward (763) has posted comments on this change. ( http://gerrit.cloudera.org:8080/20559 ) Change subject: [build] bootstrap-dev-env.sh fix for ubuntu 22.04+ .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/20559/1/docker/bootstrap-dev-env.sh File docker/bootstrap-dev-env.sh: http://gerrit.cloudera.org:8080/#/c/20559/1/docker/bootstrap-dev-env.sh@116 PS1, Line 116: apt-get update -y : > I'm not sure this does what it's supposed to do. On ubuntu 20.04 if you run "apt install python" it will install python2. apt-cache search says there is no such package :D. So the script works. Not exatly how it should, but it does. There is this line during install: Note, selecting 'python-is-python2' instead of 'python' I updated with a different aproach. This should work on all current non-deprecated versions (+ 18.04). -- To view, visit http://gerrit.cloudera.org:8080/20559 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I275fa06214fde6ede1c6dd8e85f786fdc950cf8c Gerrit-Change-Number: 20559 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Martonka Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Anonymous Coward (763) Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber Gerrit-Comment-Date: Tue, 04 Jun 2024 15:54:07 + Gerrit-HasComments: Yes
[kudu-CR] KUDU-613: Integrate SLRU cache into block cache
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21390 ) Change subject: KUDU-613: Integrate SLRU cache into block cache .. Patch Set 4: Build Started http://jenkins.kudu.apache.org/job/pre_commit/196/ -- To view, visit http://gerrit.cloudera.org:8080/21390 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I04411ab2756045f15a272f3397d46d871b087b03 Gerrit-Change-Number: 21390 Gerrit-PatchSet: 4 Gerrit-Owner: Mahesh Reddy Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mahesh Reddy Gerrit-Comment-Date: Tue, 04 Jun 2024 20:54:38 + Gerrit-HasComments: No
[kudu-CR] KUDU-613: Introduce SLRU cache
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20607 ) Change subject: KUDU-613: Introduce SLRU cache .. Patch Set 20: Build Started http://jenkins.kudu.apache.org/job/pre_commit/195/ -- To view, visit http://gerrit.cloudera.org:8080/20607 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45531534a2049dd38c002f4dc7982df9fd46e0bb Gerrit-Change-Number: 20607 Gerrit-PatchSet: 20 Gerrit-Owner: Mahesh Reddy Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mahesh Reddy Gerrit-Reviewer: Marton Greber Gerrit-Reviewer: Tidy Bot (241) Gerrit-Comment-Date: Tue, 04 Jun 2024 20:54:37 + Gerrit-HasComments: No
[kudu-CR] KUDU-613: Add SLRU Cache metrics
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21389 ) Change subject: KUDU-613: Add SLRU Cache metrics .. Patch Set 6: Build Started http://jenkins.kudu.apache.org/job/pre_commit/197/ -- To view, visit http://gerrit.cloudera.org:8080/21389 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1c8181ec6bea301605aaef7db0003c3eaef3072d Gerrit-Change-Number: 21389 Gerrit-PatchSet: 6 Gerrit-Owner: Mahesh Reddy Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mahesh Reddy Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Comment-Date: Tue, 04 Jun 2024 20:54:38 + Gerrit-HasComments: No
[kudu-CR] KUDU-613: Introduce SLRU cache
Hello Tidy Bot, Marton Greber, Alexey Serbin, Attila Bukor, Kudu Jenkins, Abhishek Chennaka, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20607 to look at the new patch set (#20). Change subject: KUDU-613: Introduce SLRU cache .. KUDU-613: Introduce SLRU cache This patch introduces the SLRU cache that has two internal segments, the probationary and protected, to protect the cache from long/random reads. The SLRU cache has a parameter named 'lookups_threshold_' that determines the minimum amount of times an entry can be accessed before it's upgraded to the protected segment. Any random scans would then only evict entries from the probationary segment. Both the protected and probationary segment have their own configurable capacities. When the protected segment is at capacity, any entries evicted will be added to the MRU end of the probationary segment. Metrics will be added in a follow-up patch. Ran cache benchmarks for RELEASE build. Build ran locally on macOS, 6 cores and 2.6GHz. Here are some benchmark numbers for SLRU cache: Test case | Algorithm | Lookups/sec | Hit rate ZIPFIAN ratio=1.00x | LRU | 10.68M | 99.5% ZIPFIAN ratio=1.00x | SLRU | 10.68M | 99.9% ZIPFIAN ratio=3.00x | LRU | 10.68M | 95.9% ZIPFIAN ratio=3.00x | SLRU | 9.86M| 95.9% UNIFORM ratio=1.00x | LRU | 9.61M| 99.7% UNIFORM ratio=1.00x | SLRU | 7.14M| 99.7% UNIFORM ratio=3.00x | LRU | 6.43M| 33.3% UNIFORM ratio=3.00x | SLRU | 5.09M| 27.6% Change-Id: I45531534a2049dd38c002f4dc7982df9fd46e0bb --- M src/kudu/util/CMakeLists.txt M src/kudu/util/cache-bench.cc M src/kudu/util/cache.cc M src/kudu/util/cache.h A src/kudu/util/slru_cache-test.cc A src/kudu/util/slru_cache.cc A src/kudu/util/slru_cache.h 7 files changed, 1,460 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/07/20607/20 -- To view, visit http://gerrit.cloudera.org:8080/20607 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I45531534a2049dd38c002f4dc7982df9fd46e0bb Gerrit-Change-Number: 20607 Gerrit-PatchSet: 20 Gerrit-Owner: Mahesh Reddy Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mahesh Reddy Gerrit-Reviewer: Marton Greber Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-613: Add SLRU Cache metrics
Hello Alexey Serbin, Kudu Jenkins, Abhishek Chennaka, Wang Xixu, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21389 to look at the new patch set (#6). Change subject: KUDU-613: Add SLRU Cache metrics .. KUDU-613: Add SLRU Cache metrics This patch adds segment-specific metrics for both the probationary and the protected segment along with high-level metrics for the entire SLRU cache. It also adds these same metrics to the block cache so it can support a cache with SLRU eviction policy. Change-Id: I1c8181ec6bea301605aaef7db0003c3eaef3072d --- M src/kudu/util/block_cache_metrics.cc M src/kudu/util/cache_metrics.h M src/kudu/util/slru_cache-test.cc M src/kudu/util/slru_cache.cc M src/kudu/util/slru_cache.h 5 files changed, 314 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/89/21389/6 -- To view, visit http://gerrit.cloudera.org:8080/21389 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1c8181ec6bea301605aaef7db0003c3eaef3072d Gerrit-Change-Number: 21389 Gerrit-PatchSet: 6 Gerrit-Owner: Mahesh Reddy Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mahesh Reddy Gerrit-Reviewer: Wang Xixu <1450306...@qq.com>
[kudu-CR] KUDU-613: Integrate SLRU cache into block cache
Hello Alexey Serbin, Kudu Jenkins, Abhishek Chennaka, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21390 to look at the new patch set (#4). Change subject: KUDU-613: Integrate SLRU cache into block cache .. KUDU-613: Integrate SLRU cache into block cache This patch integrates the new SLRU cache into the block cache. It adds several flags to make the SLRU cache configurable for capacity and lookups as well as adding an eviction_policy flag to give users the option to use the new SLRU cache. The default value is still the old LRU cache. Change-Id: I04411ab2756045f15a272f3397d46d871b087b03 --- M src/kudu/cfile/block_cache.cc M src/kudu/cfile/block_cache.h M src/kudu/integration-tests/dense_node-itest.cc M src/kudu/integration-tests/full_stack-insert-scan-test.cc M src/kudu/integration-tests/heavy-update-compaction-itest.cc M src/kudu/tserver/tablet_server-stress-test.cc 6 files changed, 192 insertions(+), 17 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/90/21390/4 -- To view, visit http://gerrit.cloudera.org:8080/21390 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I04411ab2756045f15a272f3397d46d871b087b03 Gerrit-Change-Number: 21390 Gerrit-PatchSet: 4 Gerrit-Owner: Mahesh Reddy Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mahesh Reddy
[kudu-CR] KUDU-613: Introduce SLRU cache
Mahesh Reddy has posted comments on this change. (
http://gerrit.cloudera.org:8080/20607 )
Change subject: KUDU-613: Introduce SLRU cache
..
Patch Set 20:
(14 comments)
http://gerrit.cloudera.org:8080/#/c/20607/16//COMMIT_MSG
Commit Message:
PS16:
> Don't forget to re-base this patch on top of the HEAD in the main branch, o
Done
http://gerrit.cloudera.org:8080/#/c/20607/16//COMMIT_MSG@22
PS16, Line 22:
> I guess it's not a big deal to build in RELEASE configuration and run the t
Done
http://gerrit.cloudera.org:8080/#/c/20607/19//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/20607/19//COMMIT_MSG@23
PS19, Line 23: Ran cache benchmarks for RELEASE build.
: Build ran locally on macOS, 6 cores and 2.6GHz.
> We need stats for RELEASE build, not DEBUG. Please add those when you have
Done
http://gerrit.cloudera.org:8080/#/c/20607/19/src/kudu/util/cache-bench.cc
File src/kudu/util/cache-bench.cc:
http://gerrit.cloudera.org:8080/#/c/20607/19/src/kudu/util/cache-bench.cc@104
PS19, Line 104: / kEntrySize;
> I think this should have been (kProbationarySegmentCapacity + kProtectedSeg
Good catch. I was mistaken that max_key had to do with the entry size.
http://gerrit.cloudera.org:8080/#/c/20607/19/src/kudu/util/nvm_cache.cc
File src/kudu/util/nvm_cache.cc:
http://gerrit.cloudera.org:8080/#/c/20607/19/src/kudu/util/nvm_cache.cc@556
PS19, Line 556: UniquePendingHandle Allocate(Slice key, int val_len, int
charge) override {
> nit: why to change this at all?
Done
http://gerrit.cloudera.org:8080/#/c/20607/19/src/kudu/util/slru_cache.h
File src/kudu/util/slru_cache.h:
http://gerrit.cloudera.org:8080/#/c/20607/19/src/kudu/util/slru_cache.h@74
PS19, Line 74: // Removes entry from shard, frees the entry if no refs are
remaining.
: void Erase(const Slice& key, uint32_t hash);
: // Like Erase, but underlying entry is not freed.
: // Necessary when upgrading entry to protected segment.
: void SoftErase(const Slice& key, uint32_t hash);
: // Returns true if shard contains entry, false if not.
: bool Contains(const Slice& key, uint32_t
> It seems the code evolved and these methods are now used only in tests. If
Done
http://gerrit.cloudera.org:8080/#/c/20607/19/src/kudu/util/slru_cache.h@113
PS19, Line 113: cause the cache implementation is sharded, we do this tracking
in a
> The name of the method already assumes this cannot be anything but Cache::E
Removed it. Only template parameter now is the MemoryType but even that will
only be DRAM. What template parameters would you like to see here?
http://gerrit.cloudera.org:8080/#/c/20607/19/src/kudu/util/slru_cache.cc
File src/kudu/util/slru_cache.cc:
http://gerrit.cloudera.org:8080/#/c/20607/19/src/kudu/util/slru_cache.cc@63
PS19, Line 63: RLHan
> nit for here and elsewhere in this file: maybe, add corresponding 'using' d
Done
http://gerrit.cloudera.org:8080/#/c/20607/19/src/kudu/util/slru_cache.cc@66
PS19, Line 66: if (Unref(e)) {
> For this and other methods that are specific to protected or probational se
Added it as a TODO for now, this is a good idea, thanks for the feedback.
http://gerrit.cloudera.org:8080/#/c/20607/19/src/kudu/util/slru_cache.cc@231
PS19, Line 231: return reinterpret_cast(handle);
: }
> nit: could this be just
Done
http://gerrit.cloudera.org:8080/#/c/20607/19/src/kudu/util/slru_cache.cc@468
PS19, Line 468: int bits = PREDICT_FALSE(FLAGS_cache_force_single_shard) ?
: 0 : Bits::Log2Ceiling(base::NumCPUs());
: VLOG(1) << "Will use " << (1 << bits) << " shards for SLRU
cache.";
: return bits;
: }
:
> After consideration, inserting MRU first seems counter-productive. For sim
Good catch, changed it to insert the LRU entries first. Also changed this same
logic in SLRUCacheShardPair::Insert().
http://gerrit.cloudera.org:8080/#/c/20607/19/src/kudu/util/slru_cache.cc@516
PS19, Line 516:
:
:
:
:
> nit: it could be
Done
http://gerrit.cloudera.org:8080/#/c/20607/19/src/kudu/util/slru_cache.cc@526
PS19, Line 526:
:
> Instead of doing this, maybe just switch the order of the 'shards_' and t
Done
http://gerrit.cloudera.org:8080/#/c/20607/19/src/kudu/util/slru_cache.cc@539
PS19, Line 539:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
> This and other methods below look to be exactly identical to the code in Sh
Done
--
To view, visit
[kudu-CR] KUDU-613: Integrate SLRU cache into block cache
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21390 ) Change subject: KUDU-613: Integrate SLRU cache into block cache .. Patch Set 4: Verified-1 Build Failed http://jenkins.kudu.apache.org/job/pre_commit/196/ : FAILURE -- To view, visit http://gerrit.cloudera.org:8080/21390 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I04411ab2756045f15a272f3397d46d871b087b03 Gerrit-Change-Number: 21390 Gerrit-PatchSet: 4 Gerrit-Owner: Mahesh Reddy Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mahesh Reddy Gerrit-Comment-Date: Tue, 04 Jun 2024 21:07:50 + Gerrit-HasComments: No
[kudu-CR] KUDU-613: Introduce SLRU cache
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20607 ) Change subject: KUDU-613: Introduce SLRU cache .. Patch Set 20: Verified-1 Build Failed http://jenkins.kudu.apache.org/job/pre_commit/195/ : FAILURE -- To view, visit http://gerrit.cloudera.org:8080/20607 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45531534a2049dd38c002f4dc7982df9fd46e0bb Gerrit-Change-Number: 20607 Gerrit-PatchSet: 20 Gerrit-Owner: Mahesh Reddy Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mahesh Reddy Gerrit-Reviewer: Marton Greber Gerrit-Reviewer: Tidy Bot (241) Gerrit-Comment-Date: Tue, 04 Jun 2024 21:11:41 + Gerrit-HasComments: No
[kudu-CR] KUDU-613: Add SLRU Cache metrics
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21389 ) Change subject: KUDU-613: Add SLRU Cache metrics .. Patch Set 6: Verified-1 Build Failed http://jenkins.kudu.apache.org/job/pre_commit/197/ : FAILURE -- To view, visit http://gerrit.cloudera.org:8080/21389 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1c8181ec6bea301605aaef7db0003c3eaef3072d Gerrit-Change-Number: 21389 Gerrit-PatchSet: 6 Gerrit-Owner: Mahesh Reddy Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mahesh Reddy Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Comment-Date: Tue, 04 Jun 2024 21:13:46 + Gerrit-HasComments: No
[kudu-CR] [build] bootstrap-dev-env.sh fix for ubuntu 22.04+
Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20559 ) Change subject: [build] bootstrap-dev-env.sh fix for ubuntu 22.04+ .. [build] bootstrap-dev-env.sh fix for ubuntu 22.04+ On Ubuntu 22.04 python package is renamed to python2 On 23.10 and 24.04 python2 is no longer available. We should just use python3 on newer platforms. Tested on: 18.04, 20.04. 22.04, 23.10, 24.04 Change-Id: I275fa06214fde6ede1c6dd8e85f786fdc950cf8c Reviewed-on: http://gerrit.cloudera.org:8080/20559 Tested-by: Kudu Jenkins Reviewed-by: Alexey Serbin --- M docker/bootstrap-dev-env.sh 1 file changed, 8 insertions(+), 1 deletion(-) Approvals: Kudu Jenkins: Verified Alexey Serbin: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/20559 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I275fa06214fde6ede1c6dd8e85f786fdc950cf8c Gerrit-Change-Number: 20559 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Martonka Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Anonymous Coward (763) Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber
[kudu-CR] [metrics] Add metrics for tablet copy op time
Hello Alexey Serbin, Yingchun Lai, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21356 to look at the new patch set (#12). Change subject: [metrics] Add metrics for tablet copy op time .. [metrics] Add metrics for tablet copy op time Add server-level statistics to track the time consumption of copy tablet operations. This is effective both for the source tablet and destination tablet during the copy operation. The addition of monitoring items will aid in historical issue tracking and analysis, as well as facilitate the configuration of monitoring alarms. Change-Id: I088f6a9a8a07ad39ca95ae8b4995ce00d1a0d00c --- M src/kudu/integration-tests/tablet_copy-itest.cc M src/kudu/tserver/tablet_copy_client.cc M src/kudu/tserver/tablet_copy_client.h M src/kudu/tserver/tablet_copy_service.cc M src/kudu/tserver/tablet_copy_source_session.cc M src/kudu/tserver/tablet_copy_source_session.h 6 files changed, 83 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/56/21356/12 -- To view, visit http://gerrit.cloudera.org:8080/21356 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I088f6a9a8a07ad39ca95ae8b4995ce00d1a0d00c Gerrit-Change-Number: 21356 Gerrit-PatchSet: 12 Gerrit-Owner: KeDeng Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: KeDeng Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Yingchun Lai
[kudu-CR] [metrics] Add metrics for tablet copy op time
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21356 ) Change subject: [metrics] Add metrics for tablet copy op time .. Patch Set 12: Build Started http://jenkins.kudu.apache.org/job/pre_commit/198/ -- To view, visit http://gerrit.cloudera.org:8080/21356 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I088f6a9a8a07ad39ca95ae8b4995ce00d1a0d00c Gerrit-Change-Number: 21356 Gerrit-PatchSet: 12 Gerrit-Owner: KeDeng Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: KeDeng Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Yingchun Lai Gerrit-Comment-Date: Wed, 05 Jun 2024 06:32:54 + Gerrit-HasComments: No
[kudu-CR] [metrics] Add metrics for tablet copy op time
KeDeng has posted comments on this change. ( http://gerrit.cloudera.org:8080/21356 ) Change subject: [metrics] Add metrics for tablet copy op time .. Patch Set 12: (1 comment) http://gerrit.cloudera.org:8080/#/c/21356/11/src/kudu/tablet/tablet_metrics.cc File src/kudu/tablet/tablet_metrics.cc: http://gerrit.cloudera.org:8080/#/c/21356/11/src/kudu/tablet/tablet_metrics.cc@258 PS11, Line 258: o CLIENT_PROPAGATED.", > Well, after looking a bit more into the code it seems this is vice-versa: t Hi Alexey, As we discussed on Slack, I have re-implemented the server-level histogram monitoring item. To ensure there are no misunderstandings like before, I have opted to validate the new monitoring item through unit tests to confirm its effectiveness. Since the current version differs significantly from the one you reviewed earlier, I apologize for not addressing each comment from the previous review. I would greatly appreciate it if you could review my new implementation again. Thank you very much. -- To view, visit http://gerrit.cloudera.org:8080/21356 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I088f6a9a8a07ad39ca95ae8b4995ce00d1a0d00c Gerrit-Change-Number: 21356 Gerrit-PatchSet: 12 Gerrit-Owner: KeDeng Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: KeDeng Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Yingchun Lai Gerrit-Comment-Date: Wed, 05 Jun 2024 06:39:10 + Gerrit-HasComments: Yes
