That's great news. Will be keeping an eye on the release. On Mon, Aug 8, 2016 at 10:12 AM Jacob Maes <jm...@linkedin.com.invalid> wrote:
> Hey David, > > I think that behavior is meant to prevent an issue on the Kafka 0.8 > Brokers. Samza 10.1 allows compression on log compacted topics, but you'll > need to make sure you're using Kafka 0.9 or higher on the Brokers. > > -Jake > > On Fri, Aug 5, 2016 at 10:57 PM, David Yu <david...@optimizely.com> wrote: > > > I guess this might be the problem: > > > > 2016-08-06 05:23:23,622 [main ] WARN o.a.s.s.kafka.KafkaSystemFactory$ - > > System name 'kafka' is being used as a changelog. Disabling compression > > since Kafka does not support compression for log compacted topics. > > > > > > Is this a 0.8.x.x Kafka producer limitation? > > > > On Fri, Aug 5, 2016 at 4:25 PM Jacob Maes <jacob.m...@gmail.com> wrote: > > > > > Hey David, > > > > > > If you check your container logs Kafka should print the producer config > > > > > > > 2016-08-05 19:18:30.134 [main] ProducerConfig [INFO] ProducerConfig > > > values: > > > > compression.type = gzip > > > > > > ... > > > > > > If you see the correct compression type in that config, then the Kafka > > > producer definitely has compression enabled. > > > > > > > > > If it's still not compressing after confirming that setting then it > > > means snappy wasn't able to compress your messages. I talked to our > > > Kafka team and they said that's not uncommon. You might want to try > > > gzip or some other compression. > > > > > > > > > On Fri, Aug 5, 2016 at 12:10 PM, David Yu <david...@optimizely.com> > > wrote: > > > > > > > I'm reporting back my observations after enabling compression. > > > > > > > > Looks like compression is not doing anything. I'm still seeing > > > > "compression-rate-avg=1.0" and the same "record-size-avg" from JMX > > > > "kafka.producer" metrics. > > > > > > > > I did set the following: > > > > systems.kafka.producer.compression.type=snappy > > > > > > > > Am I missing anything? > > > > > > > > Thanks, > > > > David > > > > > > > > On Wed, Aug 3, 2016 at 1:48 PM David Yu <david...@optimizely.com> > > wrote: > > > > > > > > > Great. Thx. > > > > > > > > > > On Wed, Aug 3, 2016 at 1:42 PM Jacob Maes <jacob.m...@gmail.com> > > > wrote: > > > > > > > > > >> Hey David, > > > > >> > > > > >> what gets written to the changelog topic > > > > >> > > > > >> The changelog gets the same value as the store, which is the > > > serialized > > > > >> form of the key and value. The serdes for the store are configured > > > with > > > > >> the > > > > >> properties: > > > > >> stores.store-name.key.serde > > > > >> stores.store-name.msg.serde > > > > >> > > > > >> If I want to compress the changelog topic, do I enable that from > the > > > > >> > producer? > > > > >> > > > > >> Yes. When you specify the changelog for your store, you specify it > > in > > > > >> terms > > > > >> of a SystemStream (typically a Kafka topic). In the part of the > > config > > > > >> where you define the Kafka system, you can pass any Kafka producer > > > > config > > > > >> <http://kafka.apache.org/documentation.html#newproducerconfigs>. > So > > > to > > > > >> configure compression you should configure the following property. > > > > >> systems.system-name.producer.compression.type > > > > >> > > > > >> Hope this helps. > > > > >> -Jake > > > > >> > > > > >> > > > > >> > > > > >> On Wed, Aug 3, 2016 at 11:16 AM, David Yu < > david...@optimizely.com> > > > > >> wrote: > > > > >> > > > > >> > I'm trying to understand what gets written to the changelog > topic. > > > Is > > > > it > > > > >> > just the serialized value of the particular state store entry? > If > > I > > > > >> want to > > > > >> > compress the changelog topic, do I enable that from the > producer? > > > > >> > > > > > >> > The reason I'm asking is that, we are seeing producer throughput > > > > issues > > > > >> and > > > > >> > suspected that writing to changelog takes up most of the network > > > > >> bandwidth. > > > > >> > > > > > >> > Thanks, > > > > >> > David > > > > >> > > > > > >> > > > > > > > > > > > > > > >
