Hi all
I want to make a proposal to support QATCodec [1] into CarbonData. QAT Codec
project provides compression and decompression library for Apache Hadoop/Spark
to make use of the Intel(r) QuickAssist Technology (Abbrev. QAT) [2] for
compression/decompression. This project has been open source
Hi all
I want to make a proposal to support QATCodec [1] into CarbonData. QAT Codec
project provides compression and decompression library for Apache Hadoop/Spark
to make use of the Intel(r) QuickAssist Technology (Abbrev. QAT) [2] for
compression/decompression. This project has been open source
Xu, Cheng A would like to recall the message, "Proposal to integrate QATCodec
into Carbondata".
Thanks Chuanyin. This PR looks cool. Allowing customized codec is a good
option. Comparing with the existing built-in Snappy codec in CarbonData, I
think QATCodec with better performance and better compression ratio is also a
good candidate for built-in support. Any thoughts?
Thanks
Ferdinand X
This would further enhance the performance of queries where io is the bottleneck
Regards
Raghu
On Fri, 12 Oct 2018, 12:18 pm Xu, Cheng A, wrote:
> Thanks Chuanyin. This PR looks cool. Allowing customized codec is a
> good option. Comparing with the existing built-in Snappy co
depends on any hardware to
achieve the performance improvement you have mentioned?
Is there any link for QATCodec project or source code?
Regards,
Jacky
> 在 2018年10月12日，上午10:40，Xu, Cheng A 写道：
>
> Hi all
> I want to make a proposal to support QATCodec [1] into CarbonData. QAT Code
Yes, we have the performance number against Snappy. It's included in our
proposal. The performance various depending on workloads.
> For the sort workload (input, intermediate data, output are all
> compression-enabled, 3TB data scale, 5 workers, 2 replica for data) with Map
> Reduce, using QAT
