[
https://issues.apache.org/jira/browse/HUDI-2029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17380034#comment-17380034
]
ASF GitHub Bot commented on HUDI-2029:
--------------------------------------
nsivabalan commented on a change in pull request #3128:
URL: https://github.com/apache/hudi/pull/3128#discussion_r668901207
##########
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
##########
@@ -290,6 +290,11 @@
.defaultValue(ExternalSpillableMap.DiskMapType.BITCASK)
.withDocumentation("Enable usage of either BITCASK or ROCKS_DB as disk
map for External Spillable Map");
+ public static final ConfigProperty<Boolean>
DISK_MAP_BITCASK_COMPRESSION_ENABLED = ConfigProperty
+ .key("hoodie.diskmap.bitcask.enabled")
+ .defaultValue(false)
Review comment:
sorry, may I know why the property name does not have "compression"? I
was expecting something like "hoodie.diskmap.bitcask.compression.enabled"
##########
File path:
hudi-common/src/main/java/org/apache/hudi/common/util/collection/BitCaskDiskMap.java
##########
@@ -399,4 +419,47 @@ public int compareTo(ValueMetadata o) {
return Long.compare(this.offsetOfValue, o.offsetOfValue);
}
}
+
+ private static class CompressionHandler implements Serializable {
+ private static final int DISK_COMPRESSION_INITIAL_BUFFER_SIZE = 1048576;
+ private static final int DECOMPRESS_INTERMEDIATE_BUFFER_SIZE = 8192;
+
+ // Caching ByteArrayOutputStreams to avoid recreating it for every
operation
+ private final ByteArrayOutputStream compressBaos;
+ private final ByteArrayOutputStream decompressBaos;
+ private final byte[] decompressIntermediateBuffer;
+
+ CompressionHandler() {
+ compressBaos = new
ByteArrayOutputStream(DISK_COMPRESSION_INITIAL_BUFFER_SIZE);
+ decompressBaos = new
ByteArrayOutputStream(DISK_COMPRESSION_INITIAL_BUFFER_SIZE);
+ decompressIntermediateBuffer = new
byte[DECOMPRESS_INTERMEDIATE_BUFFER_SIZE];
+ }
+
+ private byte[] compressBytes(final byte[] value) throws IOException {
+ compressBaos.reset();
+ Deflater deflater = new Deflater(Deflater.BEST_COMPRESSION);
+ DeflaterOutputStream dos = new DeflaterOutputStream(compressBaos,
deflater);
+ try {
+ dos.write(value);
+ } finally {
+ dos.close();
+ deflater.end();
+ }
+ return compressBaos.toByteArray();
+ }
+
+ private byte[] decompressBytes(final byte[] bytes) throws IOException {
+ decompressBaos.reset();
+ InputStream in = new InflaterInputStream(new
ByteArrayInputStream(bytes));
+ try {
+ int len;
+ while ((len = in.read(decompressIntermediateBuffer)) > 0) {
Review comment:
is decompressIntermediateBuffer overwritten everytime is it?
##########
File path:
hudi-common/src/main/java/org/apache/hudi/common/util/collection/ExternalSpillableMap.java
##########
@@ -108,7 +116,7 @@ public ExternalSpillableMap(Long maxInMemorySizeInBytes,
String baseFilePath, Si
break;
case BITCASK:
default:
- diskBasedMap = new BitCaskDiskMap<>(baseFilePath);
+ diskBasedMap = isCompressionEnabled ? (new
BitCaskDiskMap<>(baseFilePath, true)) : (new BitCaskDiskMap<>(baseFilePath,
false));
Review comment:
new BitCaskDiskMap<>(baseFilePath, isCompressionEnabled)
##########
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
##########
@@ -290,6 +290,11 @@
.defaultValue(ExternalSpillableMap.DiskMapType.BITCASK)
.withDocumentation("Enable usage of either BITCASK or ROCKS_DB as disk
map for External Spillable Map");
+ public static final ConfigProperty<Boolean>
DISK_MAP_BITCASK_COMPRESSION_ENABLED = ConfigProperty
+ .key("hoodie.diskmap.bitcask.enabled")
+ .defaultValue(false)
Review comment:
also, why not enable by default
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Implement compression for DiskBasedMap in Spillable Map
> -------------------------------------------------------
>
> Key: HUDI-2029
> URL: https://issues.apache.org/jira/browse/HUDI-2029
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Performance
> Reporter: Rajesh Mahindra
> Assignee: Rajesh Mahindra
> Priority: Major
> Labels: pull-request-available
>
> Implement compression for DiskBasedMap in Spillable Map
> Without compression, DiskBasedMap is causing more spilling to disk than
> RockDb.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
