ZTE-EBASE opened a new pull request, #1887: URL: https://github.com/apache/cloudberry/pull/1887
<!-- Thank you for your contribution to Apache Cloudberry (Incubating)! --> Fixes #ISSUE_Number ### What does this PR do? This PR adds **LZO compressed file format read/write support** to the `gpfdist` utility in Apache Cloudberry. ### Type of Change - [ ] New feature (non-breaking change) ### Breaking Changes <!-- Remove if not applicable. If yes, explain impact and migration path --> ### Test Plan <!-- How did you test these changes? --> - [ ] Unit tests added/updated - [ ] Integration tests added/updated - [ ] Passed `make installcheck` - [ ] Passed `make -C src/test installcheck-cbdb-parallel` ### Impact <!-- Remove sections that don't apply --> **Performance:** #### Scenario 1: High-Volume Data Loading LZO is a lightweight compression algorithm optimized for decompression speed (~500MB/s), making it ideal for scenarios requiring rapid data ingestion. Compared to BZip2 (~10MB/s) and GZIP (~100MB/s), LZO provides the fastest decompression while maintaining a reasonable compression ratio. **Why this matters**: For time-critical ETL jobs, LZO can reduce load times by 2-5x compared to heavier compression algorithms, while still saving ~50% storage space compared to uncompressed files. #### Scenario 2: Hadoop Ecosystem Integration Many Hadoop ecosystem tools natively support LZO compression: 1. **Hive**: Supports LZO-compressed text and sequence files 2. **Spark**: Can write LZO-compressed output (via hadoop-lzo library) 3. **Sqoop**: Commonly uses LZO for bulk imports from RDBMS to HDFS #### Scenario 3: Storage Efficiency with Parallel Performance LZO is **splittable**, meaning multiple gpfdist instances (or segments) can read different portions of the same file simultaneously. This enables near-linear scalability with additional segments. **User-facing changes:** <!-- Any changes visible to users? --> **Dependencies:** <!-- New dependencies or version changes? --> https://github.com/apache/cloudberry/discussions/1886#discussioncomment-17929325 ### Checklist - [ ] Followed [contribution guide](https://cloudberry.apache.org/contribute/code) - [ ] Added/updated documentation - [ ] Reviewed code for security implications - [ ] This PR contains AI-assisted code generation - [ ] Requested review from [cloudberry committers](https://github.com/orgs/apache/teams/cloudberry-committers) ## References ### Technical Documentation - [LZO Official Website](https://www.oberhumer.com/opensource/lzo/) - [LZOP File Format Specification](http://www.lzop.org/) - [Hadoop LZO Compression Guide](https://github.com/twitter/lzo-guide) ### Cloudberry Documentation - [GPFDIST Official Docs](https://cloudberry.apache.org/docs/admin_guide/gpfdist.html) - [External Tables Guide](https://cloudberry.apache.org/docs/admin_guide/external-tables.html) ### Related Projects - [Apache Cloudberry GitHub](https://github.com/apache/cloudberry) - [Greenplum Database (upstream)](https://github.com/greenplum-db/gpdb) ### Additional Context <!-- Any other information that would help reviewers? Remove if none --> ### CI Skip Instructions <!-- To skip CI builds, add the appropriate CI skip identifier to your PR title. The identifier must: - Be in square brackets [] - Include the word "ci" and either "skip" or "no" - Only use for documentation-only changes or when absolutely necessary --> --- <!-- Join our community: - Mailing list: [[email protected]](https://lists.apache.org/[email protected]) (subscribe: [email protected]) - Discussions: https://github.com/apache/cloudberry/discussions --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
