[Impala-ASF-CR] IMPALA-5528: Bump total thread cache size when KRPC is enabled
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9058 ) Change subject: IMPALA-5528: Bump total thread cache size when KRPC is enabled .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/9058 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8407528942051fb19a0222491347c9090d4b4b8d Gerrit-Change-Number: 9058 Gerrit-PatchSet: 2 Gerrit-Owner: Michael HoGerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sun, 28 Jan 2018 04:49:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5528: Bump total thread cache size when KRPC is enabled
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/9058 ) Change subject: IMPALA-5528: Bump total thread cache size when KRPC is enabled .. IMPALA-5528: Bump total thread cache size when KRPC is enabled KRPC in general tends to put more pressure on the thread caches due to allocations of more small objects (i.e. <1MB). While some of them are being addressed in KUDU-1865, it's shown that the following TCMalloc workarounds will provide reasonable performance with KRPC: - TCMALLOC_TRANSFER_NUM_OBJ: - maximum number of object per classe type to transfer between thread and central caches. - the default value of 512 in 2.5.2 seems to cause the spin lock in the central cache to be held for too long with KRPC. 2.5.90 and latter reverts this value to 32 by default. - TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES - total amount of memory allocated to all thread caches in bytes - the default value is 32MB. We need to bump it to 1GB which is the internal cap in TCMalloc. This change bumps the thread cache sizes to 1GB when KRPC is enabled and FLAGS_TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES has the default value of 0. GPerfTools/TCMalloc needs to be upgraded to 2.5.90 or above to pick up the change of the default value of TCMALLOC_TRANSFER_NUM_OBJ. Previous attempt to upgrade GPerfTools to 2.6.3 failed on certain platforms (IMPALA-6414). Also fixes a couple of BE tests to initialize the test environment properly. Change-Id: I8407528942051fb19a0222491347c9090d4b4b8d Reviewed-on: http://gerrit.cloudera.org:8080/9058 Reviewed-by: Michael HoReviewed-by: Sailesh Mukil Tested-by: Impala Public Jenkins --- M be/src/runtime/bufferpool/buffer-allocator-test.cc M be/src/runtime/bufferpool/free-list-test.cc M be/src/runtime/bufferpool/suballocator-test.cc M be/src/runtime/exec-env.cc 4 files changed, 43 insertions(+), 12 deletions(-) Approvals: Michael Ho: Looks good to me, but someone else must approve Sailesh Mukil: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/9058 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8407528942051fb19a0222491347c9090d4b4b8d Gerrit-Change-Number: 9058 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-5528: Bump total thread cache size when KRPC is enabled
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9058 ) Change subject: IMPALA-5528: Bump total thread cache size when KRPC is enabled .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1812/ -- To view, visit http://gerrit.cloudera.org:8080/9058 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8407528942051fb19a0222491347c9090d4b4b8d Gerrit-Change-Number: 9058 Gerrit-PatchSet: 2 Gerrit-Owner: Michael HoGerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sun, 28 Jan 2018 01:00:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5528: Bump total thread cache size when KRPC is enabled
Sailesh Mukil has posted comments on this change. ( http://gerrit.cloudera.org:8080/9058 ) Change subject: IMPALA-5528: Bump total thread cache size when KRPC is enabled .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/9058 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8407528942051fb19a0222491347c9090d4b4b8d Gerrit-Change-Number: 9058 Gerrit-PatchSet: 2 Gerrit-Owner: Michael HoGerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 27 Jan 2018 03:17:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5528: Bump total thread cache size when KRPC is enabled
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/9058 ) Change subject: IMPALA-5528: Bump total thread cache size when KRPC is enabled .. Patch Set 1: (1 comment) The changes to the BE tests weren't strictly needed until we upgrade to 2.6.3 when the default value of aggressive-decommit changes. However, we are already setting that flag in the initialization code so we may as well fix those tests to pick up the correct value of that flag. http://gerrit.cloudera.org:8080/#/c/9058/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/9058/1//COMMIT_MSG@30 PS1, Line 30: TCMALLOC_TRANSFER_NUM_OBJ will be tuned in a separate change. Previous attempt > Nit: GPerfTools/TCMalloc needs to be upgraded to 2.6.3 to pick up the Done -- To view, visit http://gerrit.cloudera.org:8080/9058 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8407528942051fb19a0222491347c9090d4b4b8d Gerrit-Change-Number: 9058 Gerrit-PatchSet: 1 Gerrit-Owner: Michael HoGerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 26 Jan 2018 23:56:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5528: Bump total thread cache size when KRPC is enabled
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/9058 ) Change subject: IMPALA-5528: Bump total thread cache size when KRPC is enabled .. Patch Set 2: Code-Review+1 Carry +1 -- To view, visit http://gerrit.cloudera.org:8080/9058 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8407528942051fb19a0222491347c9090d4b4b8d Gerrit-Change-Number: 9058 Gerrit-PatchSet: 2 Gerrit-Owner: Michael HoGerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 26 Jan 2018 23:56:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5528: Bump total thread cache size when KRPC is enabled
Hello Sailesh Mukil, Tim Armstrong, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/9058 to look at the new patch set (#2). Change subject: IMPALA-5528: Bump total thread cache size when KRPC is enabled .. IMPALA-5528: Bump total thread cache size when KRPC is enabled KRPC in general tends to put more pressure on the thread caches due to allocations of more small objects (i.e. <1MB). While some of them are being addressed in KUDU-1865, it's shown that the following TCMalloc workarounds will provide reasonable performance with KRPC: - TCMALLOC_TRANSFER_NUM_OBJ: - maximum number of object per classe type to transfer between thread and central caches. - the default value of 512 in 2.5.2 seems to cause the spin lock in the central cache to be held for too long with KRPC. 2.5.90 and latter reverts this value to 32 by default. - TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES - total amount of memory allocated to all thread caches in bytes - the default value is 32MB. We need to bump it to 1GB which is the internal cap in TCMalloc. This change bumps the thread cache sizes to 1GB when KRPC is enabled and FLAGS_TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES has the default value of 0. GPerfTools/TCMalloc needs to be upgraded to 2.5.90 or above to pick up the change of the default value of TCMALLOC_TRANSFER_NUM_OBJ. Previous attempt to upgrade GPerfTools to 2.6.3 failed on certain platforms (IMPALA-6414). Also fixes a couple of BE tests to initialize the test environment properly. Change-Id: I8407528942051fb19a0222491347c9090d4b4b8d --- M be/src/runtime/bufferpool/buffer-allocator-test.cc M be/src/runtime/bufferpool/free-list-test.cc M be/src/runtime/bufferpool/suballocator-test.cc M be/src/runtime/exec-env.cc 4 files changed, 43 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/9058/2 -- To view, visit http://gerrit.cloudera.org:8080/9058 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8407528942051fb19a0222491347c9090d4b4b8d Gerrit-Change-Number: 9058 Gerrit-PatchSet: 2 Gerrit-Owner: Michael HoGerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-5528: Bump total thread cache size when KRPC is enabled
Sailesh Mukil has posted comments on this change. ( http://gerrit.cloudera.org:8080/9058 ) Change subject: IMPALA-5528: Bump total thread cache size when KRPC is enabled .. Patch Set 1: (1 comment) Are the corresponding test changes to buffer-allocator-test, free-list-test and suballocator-test not required? http://gerrit.cloudera.org:8080/#/c/9058/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/9058/1//COMMIT_MSG@30 PS1, Line 30: TCMALLOC_TRANSFER_NUM_OBJ will be tuned in a separate change. Previous attempt Nit: GPerfTools/TCMalloc needs to be upgraded to 2.6.3 to pick up the change of the default value of TCMALLOC_TRANSFER_NUM_OBJ. -- To view, visit http://gerrit.cloudera.org:8080/9058 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8407528942051fb19a0222491347c9090d4b4b8d Gerrit-Change-Number: 9058 Gerrit-PatchSet: 1 Gerrit-Owner: Michael HoGerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Thu, 18 Jan 2018 22:06:35 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5528: Bump total thread cache size when KRPC is enabled
Michael Ho has uploaded this change for review. ( http://gerrit.cloudera.org:8080/9058 Change subject: IMPALA-5528: Bump total thread cache size when KRPC is enabled .. IMPALA-5528: Bump total thread cache size when KRPC is enabled KRPC in general tends to put more pressure on the thread caches due to allocations of more small objects (i.e. <1MB). While some of them are being addressed in KUDU-1865, it's shown that the following TCMalloc workarounds will provide reasonable performance with KRPC: - TCMALLOC_TRANSFER_NUM_OBJ: - maximum number of object per classe type to transfer between thread and central caches. - the default value of 512 in 2.5.2 seems to cause the spin lock in the central cache to be held for too long with KRPC. 2.5.90 and latter reverts this value to 32 by default. - TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES - total amount of memory allocated to all thread caches in bytes - the default value is 32MB. We need to bump it to 1GB which is the internal cap in TCMalloc. This change bumps the thread cache sizes to 1GB when KRPC is enabled and FLAGS_TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES has the default value of 0. TCMALLOC_TRANSFER_NUM_OBJ will be tuned in a separate change. Previous attempt to upgrade GPerfTools to 2.6.3 failed on certain platforms (IMPALA-6414). Change-Id: I8407528942051fb19a0222491347c9090d4b4b8d --- M be/src/runtime/exec-env.cc 1 file changed, 12 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/9058/1 -- To view, visit http://gerrit.cloudera.org:8080/9058 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I8407528942051fb19a0222491347c9090d4b4b8d Gerrit-Change-Number: 9058 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho