[Impala-ASF-CR] IMPALA-12076: Use ZSTD interfaces with reusable context
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19773 ) Change subject: IMPALA-12076: Use ZSTD interfaces with reusable context .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19773 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5010a56bf8202ccb3f1710425002f81587fd412b Gerrit-Change-Number: 19773 Gerrit-PatchSet: 5 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Sat, 07 Oct 2023 00:28:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12076: Use ZSTD interfaces with reusable context
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19773 ) Change subject: IMPALA-12076: Use ZSTD interfaces with reusable context .. IMPALA-12076: Use ZSTD interfaces with reusable context For repeated compression/decompression, ZSTD recommends constructing a context once via ZSTD_createCCtx()/ZSTD_createDCtx() and using the set of interfaces that passes in the context explicitly to avoid constructing the context on each call. This follows the recommendation and allocates the ZSTD context once for each compressor / decompressor and reuses it for the lifetime of the compressor / decompressor. This gets a minor speedup for small-scale ZSTD TPC-H: +--++-++++ | Workload | File Format| Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) | +--++-++++ | TPCH(42) | parquet / zstd / block | 3.55| -1.40% | 2.52 | -1.63% | +--++-++++ Testing: - Ran core job - Ran a perf-AB-test job Change-Id: I5010a56bf8202ccb3f1710425002f81587fd412b Reviewed-on: http://gerrit.cloudera.org:8080/19773 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/util/compress.cc M be/src/util/compress.h M be/src/util/decompress.cc 3 files changed, 26 insertions(+), 5 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/19773 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I5010a56bf8202ccb3f1710425002f81587fd412b Gerrit-Change-Number: 19773 Gerrit-PatchSet: 6 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-12076: Use ZSTD interfaces with reusable context
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/19773 ) Change subject: IMPALA-12076: Use ZSTD interfaces with reusable context .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19773 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5010a56bf8202ccb3f1710425002f81587fd412b Gerrit-Change-Number: 19773 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Fri, 06 Oct 2023 20:06:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12076: Use ZSTD interfaces with reusable context
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19773 ) Change subject: IMPALA-12076: Use ZSTD interfaces with reusable context .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9792/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19773 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5010a56bf8202ccb3f1710425002f81587fd412b Gerrit-Change-Number: 19773 Gerrit-PatchSet: 5 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Fri, 06 Oct 2023 20:06:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12076: Use ZSTD interfaces with reusable context
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19773 ) Change subject: IMPALA-12076: Use ZSTD interfaces with reusable context .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19773 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5010a56bf8202ccb3f1710425002f81587fd412b Gerrit-Change-Number: 19773 Gerrit-PatchSet: 5 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Fri, 06 Oct 2023 20:06:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12076: Use ZSTD interfaces with reusable context
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19773 ) Change subject: IMPALA-12076: Use ZSTD interfaces with reusable context .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19773 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5010a56bf8202ccb3f1710425002f81587fd412b Gerrit-Change-Number: 19773 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Fri, 25 Aug 2023 22:08:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12076: Use ZSTD interfaces with reusable context
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19773 ) Change subject: IMPALA-12076: Use ZSTD interfaces with reusable context .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9636/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/19773 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5010a56bf8202ccb3f1710425002f81587fd412b Gerrit-Change-Number: 19773 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Fri, 25 Aug 2023 17:52:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12076: Use ZSTD interfaces with reusable context
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/19773 ) Change subject: IMPALA-12076: Use ZSTD interfaces with reusable context .. Patch Set 4: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/19773 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5010a56bf8202ccb3f1710425002f81587fd412b Gerrit-Change-Number: 19773 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Fri, 25 Aug 2023 17:52:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12076: Use ZSTD interfaces with reusable context
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19773 ) Change subject: IMPALA-12076: Use ZSTD interfaces with reusable context .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9622/ -- To view, visit http://gerrit.cloudera.org:8080/19773 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5010a56bf8202ccb3f1710425002f81587fd412b Gerrit-Change-Number: 19773 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Wed, 23 Aug 2023 05:15:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12076: Use ZSTD interfaces with reusable context
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19773 ) Change subject: IMPALA-12076: Use ZSTD interfaces with reusable context .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9622/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/19773 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5010a56bf8202ccb3f1710425002f81587fd412b Gerrit-Change-Number: 19773 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Wed, 23 Aug 2023 00:59:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12076: Use ZSTD interfaces with reusable context
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/19773 ) Change subject: IMPALA-12076: Use ZSTD interfaces with reusable context .. Patch Set 2: > Ah, we already did this for streaming decompression. Right, the compressors/decompressors are single threaded and this was something that already required for streaming ZSTD. -- To view, visit http://gerrit.cloudera.org:8080/19773 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5010a56bf8202ccb3f1710425002f81587fd412b Gerrit-Change-Number: 19773 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Wed, 23 Aug 2023 00:57:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12076: Use ZSTD interfaces with reusable context
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/19773 ) Change subject: IMPALA-12076: Use ZSTD interfaces with reusable context .. Patch Set 2: Ah, we already did this for streaming decompression. -- To view, visit http://gerrit.cloudera.org:8080/19773 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5010a56bf8202ccb3f1710425002f81587fd412b Gerrit-Change-Number: 19773 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 10 Aug 2023 16:20:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12076: Use ZSTD interfaces with reusable context
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/19773 ) Change subject: IMPALA-12076: Use ZSTD interfaces with reusable context .. Patch Set 2: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/19773/2/be/src/util/compress.h File be/src/util/compress.h: http://gerrit.cloudera.org:8080/#/c/19773/2/be/src/util/compress.h@153 PS2, Line 153: ZSTD_CCtx* stream_ = nullptr; Pretty sure instances of these classes aren't shared across threads. Gzip has a similar construct. Main uses are tmp-file-mgr and hdfs-parquet-table-writer, which create an instance for their particular use (writing a temporary file block or sequentially to a file). -- To view, visit http://gerrit.cloudera.org:8080/19773 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5010a56bf8202ccb3f1710425002f81587fd412b Gerrit-Change-Number: 19773 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 10 Aug 2023 16:18:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12076: Use ZSTD interfaces with reusable context
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19773 ) Change subject: IMPALA-12076: Use ZSTD interfaces with reusable context .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19773 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5010a56bf8202ccb3f1710425002f81587fd412b Gerrit-Change-Number: 19773 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 10 Aug 2023 07:06:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12076: Use ZSTD interfaces with reusable context
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19773 ) Change subject: IMPALA-12076: Use ZSTD interfaces with reusable context .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13734/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19773 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5010a56bf8202ccb3f1710425002f81587fd412b Gerrit-Change-Number: 19773 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 10 Aug 2023 03:17:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12076: Use ZSTD interfaces with reusable context
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19773 ) Change subject: IMPALA-12076: Use ZSTD interfaces with reusable context .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9583/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/19773 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5010a56bf8202ccb3f1710425002f81587fd412b Gerrit-Change-Number: 19773 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 10 Aug 2023 02:50:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12076: Use ZSTD interfaces with reusable context
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19773 to look at the new patch set (#2). Change subject: IMPALA-12076: Use ZSTD interfaces with reusable context .. IMPALA-12076: Use ZSTD interfaces with reusable context For repeated compression/decompression, ZSTD recommends constructing a context once via ZSTD_createCCtx()/ZSTD_createDCtx() and using the set of interfaces that passes in the context explicitly to avoid constructing the context on each call. This follows the recommendation and allocates the ZSTD context once for each compressor / decompressor and reuses it for the lifetime of the compressor / decompressor. This gets a minor speedup for small-scale ZSTD TPC-H: +--++-++++ | Workload | File Format| Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) | +--++-++++ | TPCH(42) | parquet / zstd / block | 3.55| -1.40% | 2.52 | -1.63% | +--++-++++ Testing: - Ran core job - Ran a perf-AB-test job Change-Id: I5010a56bf8202ccb3f1710425002f81587fd412b --- M be/src/util/compress.cc M be/src/util/compress.h M be/src/util/decompress.cc 3 files changed, 26 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/19773/2 -- To view, visit http://gerrit.cloudera.org:8080/19773 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5010a56bf8202ccb3f1710425002f81587fd412b Gerrit-Change-Number: 19773 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins
