Re: [PR] [FLINK-32562][connectors/filesystem] Handle compressed streams correctly in ConcatFileCompactor [flink]

2024-08-12 Thread via GitHub
gaborgsomogyi commented on code in PR #24914: URL: https://github.com/apache/flink/pull/24914#discussion_r1713838756 ## flink-connectors/flink-connector-files/src/test/java/org/apache/flink/connector/file/sink/utils/TestBucketWriter.java: ## @@ -0,0 +1,160 @@ +/* + * Licensed to

Re: [PR] [FLINK-32562][connectors/filesystem] Handle compressed streams correctly in ConcatFileCompactor [flink]

2024-08-08 Thread via GitHub
gaborgsomogyi commented on code in PR #24914: URL: https://github.com/apache/flink/pull/24914#discussion_r1709562167 ## flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/compactor/operator/CompactService.java: ## @@ -173,4 +203,36 @@ priva

Re: [PR] [FLINK-32562][connectors/filesystem] Handle compressed streams correctly in ConcatFileCompactor [flink]

2024-06-17 Thread via GitHub
gaborgsomogyi commented on PR #24914: URL: https://github.com/apache/flink/pull/24914#issuecomment-2172892978 Seems like the actual change set is working for particular compression types and not working for others (an example is snappy). Solving the problem partially would generate bad UX f

Re: [PR] [FLINK-32562][connectors/filesystem] Handle compressed streams correctly in ConcatFileCompactor [flink]

2024-06-11 Thread via GitHub
ferenc-csaky commented on code in PR #24914: URL: https://github.com/apache/flink/pull/24914#discussion_r1634721788 ## flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/compactor/operator/CompactService.java: ## @@ -154,23 +158,27 @@ priva

Re: [PR] [FLINK-32562][connectors/filesystem] Handle compressed streams correctly in ConcatFileCompactor [flink]

2024-06-11 Thread via GitHub
ferenc-csaky commented on code in PR #24914: URL: https://github.com/apache/flink/pull/24914#discussion_r1634721788 ## flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/compactor/operator/CompactService.java: ## @@ -154,23 +158,27 @@ priva

Re: [PR] [FLINK-32562][connectors/filesystem] Handle compressed streams correctly in ConcatFileCompactor [flink]

2024-06-11 Thread via GitHub
ferenc-csaky commented on code in PR #24914: URL: https://github.com/apache/flink/pull/24914#discussion_r1634722169 ## flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/compactor/operator/CompactService.java: ## @@ -154,23 +158,27 @@ priva

Re: [PR] [FLINK-32562][connectors/filesystem] Handle compressed streams correctly in ConcatFileCompactor [flink]

2024-06-11 Thread via GitHub
gaborgsomogyi commented on code in PR #24914: URL: https://github.com/apache/flink/pull/24914#discussion_r1634351465 ## flink-connectors/flink-file-sink-common/src/main/java/org/apache/flink/streaming/api/functions/sink/filesystem/OutputStreamBasedPartFileWriter.java: ## @@ -50,

Re: [PR] [FLINK-32562][connectors/filesystem] Handle compressed streams correctly in ConcatFileCompactor [flink]

2024-06-10 Thread via GitHub
ferenc-csaky commented on code in PR #24914: URL: https://github.com/apache/flink/pull/24914#discussion_r1633282273 ## flink-connectors/flink-file-sink-common/src/main/java/org/apache/flink/streaming/api/functions/sink/filesystem/OutputStreamBasedPartFileWriter.java: ## @@ -50,7

Re: [PR] [FLINK-32562][connectors/filesystem] Handle compressed streams correctly in ConcatFileCompactor [flink]

2024-06-10 Thread via GitHub
gaborgsomogyi commented on code in PR #24914: URL: https://github.com/apache/flink/pull/24914#discussion_r1633243846 ## flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/compactor/operator/CompactService.java: ## @@ -154,23 +158,27 @@ priv

Re: [PR] [FLINK-32562][connectors/filesystem] Handle compressed streams correctly in ConcatFileCompactor [flink]

2024-06-07 Thread via GitHub
flinkbot commented on PR #24914: URL: https://github.com/apache/flink/pull/24914#issuecomment-2154960488 ## CI report: * e92f70c52167e49c2df5d00544407dd933c8d18e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure`

[PR] [FLINK-32562][connectors/filesystem] Handle compressed streams correctly in ConcatFileCompactor [flink]

2024-06-07 Thread via GitHub
ferenc-csaky opened a new pull request, #24914: URL: https://github.com/apache/flink/pull/24914 ## What is the purpose of the change Provides a way to handle output stream based compaction in case the data stream is compressed via `ConcatFileCompactor`. The problem is described well