Re: Log twice ProducerConfig & ConsumerConfig values
Hi, And on a side note, it's logged _many_ times. I had to suppress some logging at package level :-/ Anybody else experiencing the same ? Cheers, Francesco On 20 February 2017 at 00:04, Simon Teles wrote: > Hello, > > I'm curious to know why, when the producer/consumer are creating, the > ProducerConfig, ConsumerConfig are logged twice ? Is that normal ? > > Example : > > 10:52:08.963 INFO [o.a.k.s.p.i.StreamThread||l.170] ~~ Creating > producer client for stream thread [StreamThread-1] > 10:52:08.969 INFO [o.a.k.c.p.ProducerConfig|logAll|l.178] ~~ > ProducerConfig values: > metric.reporters = [] > metadata.max.age.ms = 30 > reconnect.backoff.ms = 50 > sasl.kerberos.ticket.renew.window.factor = 0.8 > bootstrap.servers = [kafka:9092] > ssl.keystore.type = JKS > sasl.mechanism = GSSAPI > max.block.ms = 6 > interceptor.classes = null > ssl.truststore.password = null > client.id = test-stream-2-StreamThread-1-producer > ssl.endpoint.identification.algorithm = null > request.timeout.ms = 3 > acks = 1 > receive.buffer.bytes = 32768 > ssl.truststore.type = JKS > retries = 0 > ssl.truststore.location = null > ssl.keystore.password = null > send.buffer.bytes = 131072 > compression.type = none > metadata.fetch.timeout.ms = 6 > retry.backoff.ms = 100 > sasl.kerberos.kinit.cmd = /usr/bin/kinit > buffer.memory = 33554432 > timeout.ms = 3 > key.serializer = class org.apache.kafka.common.serial > ization.ByteArraySerializer > sasl.kerberos.service.name = null > sasl.kerberos.ticket.renew.jitter = 0.05 > ssl.trustmanager.algorithm = PKIX > block.on.buffer.full = false > ssl.key.password = null > sasl.kerberos.min.time.before.relogin = 6 > connections.max.idle.ms = 54 > max.in.flight.requests.per.connection = 5 > metrics.num.samples = 2 > ssl.protocol = TLS > ssl.provider = null > ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] > batch.size = 16384 > ssl.keystore.location = null > ssl.cipher.suites = null > security.protocol = PLAINTEXT > max.request.size = 1048576 > value.serializer = class org.apache.kafka.common.serial > ization.ByteArraySerializer > ssl.keymanager.algorithm = SunX509 > metrics.sample.window.ms = 3 > partitioner.class = class org.apache.kafka.clients.produ > cer.internals.DefaultPartitioner > linger.ms = 100 > > 10:52:08.996 INFO [o.a.k.c.p.ProducerConfig|logAll|l.178] ~~ > ProducerConfig values: > metric.reporters = [] > metadata.max.age.ms = 30 > reconnect.backoff.ms = 50 > sasl.kerberos.ticket.renew.window.factor = 0.8 > bootstrap.servers = [kafka:9092] > ssl.keystore.type = JKS > sasl.mechanism = GSSAPI > max.block.ms = 6 > interceptor.classes = null > ssl.truststore.password = null > client.id = test-stream-2-StreamThread-1-producer > ssl.endpoint.identification.algorithm = null > request.timeout.ms = 3 > acks = 1 > receive.buffer.bytes = 32768 > ssl.truststore.type = JKS > retries = 0 > ssl.truststore.location = null > ssl.keystore.password = null > send.buffer.bytes = 131072 > compression.type = none > metadata.fetch.timeout.ms = 6 > retry.backoff.ms = 100 > sasl.kerberos.kinit.cmd = /usr/bin/kinit > buffer.memory = 33554432 > timeout.ms = 3 > key.serializer = class org.apache.kafka.common.serial > ization.ByteArraySerializer > sasl.kerberos.service.name = null > sasl.kerberos.ticket.renew.jitter = 0.05 > ssl.trustmanager.algorithm = PKIX > block.on.buffer.full = false > ssl.key.password = null > sasl.kerberos.min.time.before.relogin = 6 > connections.max.idle.ms = 54 > max.in.flight.requests.per.connection = 5 > metrics.num.samples = 2 > ssl.protocol = TLS > ssl.provider = null > ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] > batch.size = 16384 > ssl.keystore.location = null > ssl.cipher.suites = null > security.protocol = PLAINTEXT > max.request.size = 1048576 > value.serializer = class org.apache.kafka.common.serial > ization.ByteArraySerializer > ssl.keymanager.algorithm = SunX509 > metrics.sample.window.ms = 3 > partitioner.class = class org.apache.kafka.clients.produ > cer.internals.DefaultPartitioner > linger.ms = 100 > > It's the same for ConsumerConfig and for "Creating restore consumer > client". > > Thanks, > > Simon > >
Re: Kafka and compression
+1 On 11 November 2016 at 06:36, Vincent Dautremont < vincent.dautrem...@olamobile.com> wrote: > Hi, > Can anyone explain me in more detail how Kakfa works with compression ? > I've read the doc but it's not all clear to me. > > > - There are compression settings on the broker, the topic of a broker, a > producer. > Are they all the same setting and one takes precedence on another ? > > - Is there a difference between compression for transfer and compression > for storage ? > > - Is the compression done per message or per topic ? > For example, my message which will transit from / to kafka are in a JSON > format (so with similar structure). will compression compress a chunk of > several message (which I guess would be more efficient in my case > considering my bottle neck is the transport link) or will it compress every > message separately ? > > > The documentation would be great if illustrated with few examples like : > Broker has compression type GZIP > Topic-A has compression Snappy > Topic-B has no compression setting > Producer-X has compression GZIP > Producer-Y has no compression settings. > > explain what happens (on the wire, and on disk) when > X Produces to A > X Produces to B > Y Produces to A > Y Produces to B > + explain if some settings n-uple are nonsense / imcompatible / could > result in errors. > > Thank you. > > -- > The information transmitted is intended only for the person or entity to > which it is addressed and may contain confidential and/or privileged > material. Any review, retransmission, dissemination or other use of, or > taking of any action in reliance upon, this information by persons or > entities other than the intended recipient is prohibited. If you received > this in error, please contact the sender and delete the material from any > computer. > -- <http://www.openbet.com/> Francesco laTorre Senior Developer T: +44 208 742 1600 +44 203 249 8394 E: francesco.lato...@openbet.com W: www.openbet.com OpenBet Ltd Chiswick Park Building 9 566 Chiswick High Rd London W4 5XT <https://www.openbet.com/email_promo> This message is confidential and intended only for the addressee. If you have received this message in error, please immediately notify the postmas...@openbet.com and delete it from your system as well as any copies. The content of e-mails as well as traffic data may be monitored by OpenBet for employment and security purposes. To protect the environment please do not print this e-mail unless necessary. OpenBet Ltd. Registered Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4 5XT, United Kingdom. A company registered in England and Wales. Registered no. 3134634. VAT no. GB927523612
Re: log compaction
Hi Becket, I can confirm : 1. All messages are keyed 2. Generous memory settings already in place 3. The topic is not compacted 4. Luckily we haven't experienced this issue. We are testing 0.10.0 and hope to make the jump shortly. Thanks you very much for the support, very much appreciated :) Cheers, Francesco On 4 November 2016 at 01:38, Becket Qin wrote: > Hi Francesco, > > There are a few things to think about before turning on log compaction for > a topic. > > 1. Does the topic have non-keyed message? Log compaction only works if all > the messages have a key. > 2. The log cleaner needs some memory to build the offset map for log > compaction, so the memory consumption may be higher. > > Given you are still running on 0.8, there are a few additional things to be > aware of: > 3. Log compaction doesn't work with compressed topics until Kafka 0.9.0. > 4. Some other potential issues that may caused by log compaction. (e.g. > KAFKA-2024). > > Jiangjie (Becket) Qin > > > > On Wed, Nov 2, 2016 at 5:55 AM, Francesco laTorre < > francesco.lato...@openbet.com> wrote: > > > Hi, > > > > We want to enable log compaction on an existing topic (in production). > > Is it a safe operation or there are things to take into consideration ? > > > > Kafka version 0.8 > > > > Cheers, > > Francesco > > > > -- > > <http://www.openbet.com/> Francesco laTorre > > Senior Developer > > T: +44 208 742 1600 > > +44 203 249 8394 > > > > E: francesco.lato...@openbet.com > > W: www.openbet.com > > OpenBet Ltd > > Chiswick Park Building 9 > > 566 Chiswick High Rd > > London > > W4 5XT > > <https://www.openbet.com/email_promo> > > This message is confidential and intended only for the addressee. If you > > have received this message in error, please immediately notify the > > postmas...@openbet.com and delete it from your system as well as any > > copies. The content of e-mails as well as traffic data may be monitored > by > > OpenBet for employment and security purposes. To protect the environment > > please do not print this e-mail unless necessary. OpenBet Ltd. Registered > > Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4 5XT, > > United Kingdom. A company registered in England and Wales. Registered no. > > 3134634. VAT no. GB927523612 > > > -- <http://www.openbet.com/> Francesco laTorre Senior Developer T: +44 208 742 1600 +44 203 249 8394 E: francesco.lato...@openbet.com W: www.openbet.com OpenBet Ltd Chiswick Park Building 9 566 Chiswick High Rd London W4 5XT <https://www.openbet.com/email_promo> This message is confidential and intended only for the addressee. If you have received this message in error, please immediately notify the postmas...@openbet.com and delete it from your system as well as any copies. The content of e-mails as well as traffic data may be monitored by OpenBet for employment and security purposes. To protect the environment please do not print this e-mail unless necessary. OpenBet Ltd. Registered Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4 5XT, United Kingdom. A company registered in England and Wales. Registered no. 3134634. VAT no. GB927523612
log compaction
Hi, We want to enable log compaction on an existing topic (in production). Is it a safe operation or there are things to take into consideration ? Kafka version 0.8 Cheers, Francesco -- <http://www.openbet.com/> Francesco laTorre Senior Developer T: +44 208 742 1600 +44 203 249 8394 E: francesco.lato...@openbet.com W: www.openbet.com OpenBet Ltd Chiswick Park Building 9 566 Chiswick High Rd London W4 5XT <https://www.openbet.com/email_promo> This message is confidential and intended only for the addressee. If you have received this message in error, please immediately notify the postmas...@openbet.com and delete it from your system as well as any copies. The content of e-mails as well as traffic data may be monitored by OpenBet for employment and security purposes. To protect the environment please do not print this e-mail unless necessary. OpenBet Ltd. Registered Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4 5XT, United Kingdom. A company registered in England and Wales. Registered no. 3134634. VAT no. GB927523612
Re: no kafka controllers in zookeeper
Hi, *get* /controller is the way to go, thought there should have been children. Cheers, Francesco On 13 September 2016 at 15:29, Francesco laTorre < francesco.lato...@openbet.com> wrote: > Hi, > > Anyone else finding this weird ? > > Cheers, > Francesco > > On 12 September 2016 at 15:53, Francesco laTorre < > francesco.lato...@openbet.com> wrote: > >> Hi guys, >> >> Pretty new to kafka and having sporadic issues with replication. >> Running Kafka *0.8.2.1 *: >> >> $> find ./libs/ -name \*kafka_\* | head -1 | grep -o '\kafka[^\n]*' >> kafka_2.11-0.8.2.1-scaladoc.jar >> >> on a cluster of *3 brokers* : >> >> $> ./zookeeper-shell.sh localhost:19374 >> Connecting to localhost:19374 >> >> >> Welcome to ZooKeeper! >> >> >> JLine support is disabled >> >> >> >> >> >> WATCHER:: >> >> >> >> >> >> WatchedEvent state:SyncConnected type:None path:null >> >> >> *ls /brokers/ids >> >> * >> *[1, 2, 3] * >> >> is it normal that zookeeper reports no controllers : >> >> [...] >> >> *ls /controller * >> >> >> *[] * >> >> Or am I missing anything in the configuration ? >> >> Cheers, >> Francesco >> >> -- >> >> >> > > > -- > <http://www.openbet.com/> Francesco laTorre > Senior Developer > T: +44 208 742 1600 > +44 203 249 8394 > > E: francesco.lato...@openbet.com > W: www.openbet.com > OpenBet Ltd > Chiswick Park Building 9 > 566 Chiswick High Rd > London > W4 5XT > <https://www.openbet.com/email_promo> > This message is confidential and intended only for the addressee. If you > have received this message in error, please immediately notify the > postmas...@openbet.com and delete it from your system as well as any > copies. The content of e-mails as well as traffic data may be monitored by > OpenBet for employment and security purposes. To protect the environment > please do not print this e-mail unless necessary. OpenBet Ltd. Registered > Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4 5XT, > United Kingdom. A company registered in England and Wales. Registered no. > 3134634. VAT no. GB927523612 > > -- <http://www.openbet.com/> Francesco laTorre Senior Developer T: +44 208 742 1600 +44 203 249 8394 E: francesco.lato...@openbet.com W: www.openbet.com OpenBet Ltd Chiswick Park Building 9 566 Chiswick High Rd London W4 5XT <https://www.openbet.com/email_promo> This message is confidential and intended only for the addressee. If you have received this message in error, please immediately notify the postmas...@openbet.com and delete it from your system as well as any copies. The content of e-mails as well as traffic data may be monitored by OpenBet for employment and security purposes. To protect the environment please do not print this e-mail unless necessary. OpenBet Ltd. Registered Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4 5XT, United Kingdom. A company registered in England and Wales. Registered no. 3134634. VAT no. GB927523612
Re: no kafka controllers in zookeeper
Hi, Anyone else finding this weird ? Cheers, Francesco On 12 September 2016 at 15:53, Francesco laTorre < francesco.lato...@openbet.com> wrote: > Hi guys, > > Pretty new to kafka and having sporadic issues with replication. > Running Kafka *0.8.2.1 *: > > $> find ./libs/ -name \*kafka_\* | head -1 | grep -o '\kafka[^\n]*' > kafka_2.11-0.8.2.1-scaladoc.jar > > on a cluster of *3 brokers* : > > $> ./zookeeper-shell.sh localhost:19374 > Connecting to localhost:19374 > > > Welcome to ZooKeeper! > > > JLine support is disabled > > > > > > WATCHER:: > > > > > > WatchedEvent state:SyncConnected type:None path:null > > > *ls /brokers/ids > >* > *[1, 2, 3] * > > is it normal that zookeeper reports no controllers : > > [...] > > *ls /controller * > > > *[] * > > Or am I missing anything in the configuration ? > > Cheers, > Francesco > > -- > > > -- <http://www.openbet.com/> Francesco laTorre Senior Developer T: +44 208 742 1600 +44 203 249 8394 E: francesco.lato...@openbet.com W: www.openbet.com OpenBet Ltd Chiswick Park Building 9 566 Chiswick High Rd London W4 5XT <https://www.openbet.com/email_promo> This message is confidential and intended only for the addressee. If you have received this message in error, please immediately notify the postmas...@openbet.com and delete it from your system as well as any copies. The content of e-mails as well as traffic data may be monitored by OpenBet for employment and security purposes. To protect the environment please do not print this e-mail unless necessary. OpenBet Ltd. Registered Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4 5XT, United Kingdom. A company registered in England and Wales. Registered no. 3134634. VAT no. GB927523612
no kafka controllers in zookeeper
Hi guys, Pretty new to kafka and having sporadic issues with replication. Running Kafka *0.8.2.1 *: $> find ./libs/ -name \*kafka_\* | head -1 | grep -o '\kafka[^\n]*' kafka_2.11-0.8.2.1-scaladoc.jar on a cluster of *3 brokers* : $> ./zookeeper-shell.sh localhost:19374 Connecting to localhost:19374 Welcome to ZooKeeper! JLine support is disabled WATCHER:: WatchedEvent state:SyncConnected type:None path:null *ls /brokers/ids * *[1, 2, 3] * is it normal that zookeeper reports no controllers : [...] *ls /controller * *[] * Or am I missing anything in the configuration ? Cheers, Francesco --
