Re: Single-replication topic behavior on failures

2017-04-12 Thread Manikumar 
I think you will get LeaderNotAvailable exception and producer requests
will fail.

On Thu, Apr 13, 2017 at 10:23 AM, Timur Fayruzov 
wrote:

> Hello,
>
> We have a number of topics with replication factor of 1 and a single
> partition. I want to update the cluster from 0.10.0 to 0.10.1 by taking
> nodes down one by one, but it is not clear to me what is the behavior will
> be for these topics.
>
> Brokers have topic auto-create setting set to true. When a broker that
> stores singly-replicated topics goes down, what happens when some producer
> decides to publish to that topic? Is a new topic created or publishing
> fails?
>
> I understand that replication factor 1 is a bad practice that can incur
> data loss and I'm ok with that for the kind of data stored there.
>
> Thanks,
> Timur
>

Single-replication topic behavior on failures

2017-04-12 Thread Timur Fayruzov 
Hello,

We have a number of topics with replication factor of 1 and a single
partition. I want to update the cluster from 0.10.0 to 0.10.1 by taking
nodes down one by one, but it is not clear to me what is the behavior will
be for these topics.

Brokers have topic auto-create setting set to true. When a broker that
stores singly-replicated topics goes down, what happens when some producer
decides to publish to that topic? Is a new topic created or publishing
fails?

I understand that replication factor 1 is a bad practice that can incur
data loss and I'm ok with that for the kind of data stored there.

Thanks,
Timur

Re: Kafka best practice on bare metal hardware

2017-04-12 Thread Marcos Juarez 
Ali,

I don't know of proper benchmarks out there, but I've done some work in
this area, when trying to determine what hardware to get for particular use
cases.  My answers are in-line:

On Mon, Apr 10, 2017 at 7:05 PM, Ali Nazemian  wrote:

> Hi all,
>
> I was wondering if there is any benchmark or any recommendation for having
> physical HW vs virtual for the Kafka Brokers. I am trying to calculate the
> HW requirements for a Kafka Cluster with a hard SLA. My questions are as
> follows.
>
> - What is the effect of OS disk caching for a Kafka-Broker? How much
> on-heap and off-heap memory would be required per node?
>

If by "OS disk caching" you mean page cache, then it's huge.  Kafka relies
on it to serve as much data as possible from memory directly.  The actual
on-heap vs off-heap RAM requirements will be completely dependent on your
scenario.  I've run Kafka brokers with heaps as little as 8GB of RAM, and
as much as 32GB of RAM each, using CMS GC with some customized tuning.  The
boxes usually have much more physical RAM than that (64GB of RAM and 128GB
respectively), for page cache.

I think the Confluent recommendation these days is to use G1 GC, but I've
got no experience using that.


>
> - Since Kafka read-write workload is pretty sequential which of the
> following spinning disks would be recommended? SATA 7.2k, SAS 10k, SAS 15k?
>

Again, that will depend on your use case.  Are your consumers mostly
consuming up-to-the-second messages, or are they always connecting and
consuming from arbitrary offsets, or even from the beginning of topics?  If
your consumers are always consuming latest, then actual disk IO will be
extremely low, since almost everything will be served out of page
cache/memory, and most of the activity will be fsync's every ~5 seconds or
so.  On the other hand, if most of your consumption will be of old data,
then the brokers will need to read a large amount of data from disk
sequentially whenever consumers request data.  That's when you'd benefit
from something like SAS 10k or SAS 15k.  You'd need to run some custom
benchmarks to figure out what would work on workload that mimics your use
case.


>
> - Since Kafka is not CPU-intensive, how bad would be to coexist
> Kafka-Broker and a CPU-intensive workload like STORM?
>

I wouldn't recommend that at all.  I've always run Kafka on bare metal with
nothing else running on that box.  Otherwise, you won't be able to identify
bottlenecks when you run into them, and both systems would be impacting
each other all the time, potentially.  Hardware is cheap, not worth it to
spend time chasing issues caused by "noisy neighbors" on the boxes.


>
> Regards,
> Ali
>

Marcos

[VOTE] 0.10.2.1 RC1

2017-04-12 Thread Gwen Shapira 
Hello Kafka users, developers, client-developers, friends, romans,
citizens, etc,

This is the second candidate for release of Apache Kafka 0.10.2.1.

This is a bug fix release and it includes fixes and improvements from 24 JIRAs
(including a few critical bugs).

Release notes for the 0.10.2.1 release:
http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc1/RELEASE_NOTES.html

*** Please download, test and vote by Monday, April 17, 5:30 pm PT

Kafka's KEYS file containing PGP keys we use to sign the release:
http://kafka.apache.org/KEYS

Your help in validating this bugfix release is super valuable, so
please take the time to test and vote!

Suggested tests:
 * Grab the source archive and make sure it compiles
 * Grab one of the binary distros and run the quickstarts against them
 * Extract and verify one of the site docs jars
 * Build a sample against jars in the staging repo
 * Validate GPG signatures on at least one file
 * Validate the javadocs look ok
 * The 0.10.2 documentation was updated for this bugfix release
(especially upgrade, streams and connect portions) - please make sure
it looks ok: http://kafka.apache.org/documentation.html

* Release artifacts to be voted upon (source and binary):
http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc1/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/

* Javadoc:
http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc1/javadoc/

* Tag to be voted upon (off 0.10.2 branch) is the 0.10.2.1 tag:
https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=e133f2ca57670e77f8114cc72dbc2f91a48e3a3b

* Documentation:
http://kafka.apache.org/0102/documentation.html

* Protocol:
http://kafka.apache.org/0102/protocol.html

/**

Thanks,

Gwen Shapira

Re: [VOTE] 0.10.2.1 RC0

2017-04-12 Thread Gwen Shapira 
Thank you for reporting Eno, and thank you for everyone who tested the
release. This is really appreciated.

I am rolling out a new RC right now. Expect a new vote thread soon :)

On Wed, Apr 12, 2017 at 10:01 AM, Eno Thereska  wrote:
> Hi Gwen,
>
> Unfortunately we found (and just fixed) a bug: 
> https://issues.apache.org/jira/browse/KAFKA-5038 
> . The PR just went in 
> 0.10.2 and trunk. It'd be great to do another RC since this problem is in the 
> general space of streams lock issues and we intended to fix them all for this 
> bug fix release.
>
> Thanks
> Eno
>
>> On 12 Apr 2017, at 08:37, Swen Moczarski  wrote:
>>
>> Hi Gwen,
>> thanks for the release candidate. Did a quick test, used the RC in my
>> recent project on client side, integration test against server version
>> 0.10.1.1 worked well.
>>
>> +1 (non binding)
>>
>> 2017-04-12 11:31 GMT+02:00 Mickael Maison > >:
>>
>>> I ran the quickstart steps against the 2.11 binary. Everything worked fine
>>> +1
>>>
>>> On Wed, Apr 12, 2017 at 8:53 AM, Michal Borowiecki <
>>> michal.borowie...@openbet.com> wrote:
>>>
 FWIW, I upgraded without issue and noticed the speedup from
 KAFKA-4851/KAFKA-4876.

 +1 from me (non-binding)

 On 12/04/17 02:06, Gwen Shapira wrote:

 Wrong link :)http://kafka.apache.org/documentation/#upgrade
 andhttp://kafka.apache.org/documentation/streams#streams_api_changes_0102

 On Tue, Apr 11, 2017 at 5:57 PM, Gwen Shapira >>> > mailto:g...@confluent.io>> 
 wrote:

 FYI: I just updated the upgrade notes with Streams 
 changes:http://kafka.apache.org/documentation/#gettingStarted 
 

 On Fri, Apr 7, 2017 at 5:12 PM, Gwen Shapira >>> > mailto:g...@confluent.io>> 
 wrote:

 Hello Kafka users, developers and client-developers,

 This is the first candidate for the release of Apache Kafka 0.10.2.1. This
 is a bug fix release and it includes fixes and improvements from 24 JIRAs
 (including a few critical bugs). See the release notes for more details:
 http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc0/RELEASE_NOTES.html 
 

 *** Please download, test and vote by Thursday, 13 April, 8am PT ***

 Your help in validating this bugfix release is super valuable, so
 please take the time to test and vote!

 Few notes:
 1. There are missing "Notable Changes" in the 
 docs:https://github.com/apache/kafka/pull/2824 
 
 I will review, merge and update the docs by Monday.
 2. The last commit (KAFKA-4943 chery-pick) did not pass system tests
 yet. We may need another RC if system tests fail tonight.

 Suggested tests:
 * Grab the source archive and make sure it compiles
 * Grab one of the binary distros and run the quickstarts against them
 * Extract and verify one of the site docs jars
 * Build a sample against jars in the staging repo
 * Validate GPG signatures on at least one file
 * Validate the javadocs look ok

 *

 Kafka's KEYS file containing PGP keys we use to sign the 
 release:http://kafka.apache.org/KEYS 

 * Release artifacts to be voted upon (source and 
 binary):http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc0/ 
 

 * Maven artifacts to be voted 
 upon:https://repository.apache.org/content/groups/staging 
 

 * Javadoc:http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc0/javadoc/ 
 

 * Tag to be voted upon (off 0.10.0 branch) is the 0.10.0.1-rc0 
 tag:https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=d08115f05da0e39c7f75b45e05d6d14ad5baf71d
  
 

 * Documentation:http://kafka.apache.org/0102/documentation.html 
 

 * Protocol:http://kafka.apache.org/0102/protocol.html 
 

 Thanks,
 Gwen Shapira


 --
 Gwen Shapira
 Product Manager | Confluent650.450.2760 <(650)%20450-2760> | @gwenshap
 Follow us: Twitter | blog


 --
 > Michal Borowiecki
 Senior Software Engineer L4
 T: +44 208 742 1600 <+44%2020%208742%201600>


 +44 203 249 8448 <+44%2020%203249%208448>



 E: michal.borowie...@openbet.com

Re: Subscription to this list

2017-04-12 Thread Matthias J. Sax 
It's self-service: http://kafka.apache.org/contact


-Matthias

On 4/11/17 11:17 PM, Vidya Priyadarshni Narayan wrote:



signature.asc
Description: OpenPGP digital signature

Java stdin producer loosing logs

2017-04-12 Thread Milind Vaidya 
Hi

Background :

I have following set up

Apache server >> Apache Kafka Producer >> Apache Kafka Cluster >> Apache
Storm

As a normal scenario, front end boxes run the apache server and populate
the  log files. The requirement is to read every log and send it to kafka
cluster.

The java producer reads the logs from stdin and transfer to cluster.

Zero loss criteria definition : contents of error log files should match
the data received by Kafka cluster per hour eventually per day.

The error_log files get rotated per hour.

There are couple of ways already tried to connect log files and the producer

1. Custom startup script to start, stop and check status of the server :
  tail -n0 -F /var/log/httpd/error_log /var/log/httpd/ssl_error_log |
java consumer
2. Hooking up directly to  apache using httpd.conf setting  :
  ErrorLog "| /usr/bin/tee -a /var/log/httpd/error_log |  java consumer"


In case 1 loss of logs was observed but that reduced significantly in case
2, where apache restarts the process the data is piped to if it crashes and
restarts it along with server restart as well. Now the loss is seen across
the restart of apache server.

Questions :

1. what is appropriate way to interface apache httpd and kafka ?
2. Is there way to gracefully shut down the kafka producer so that the
pending buffers are flushed before the process dies ?

Corrupted message in master partition, but valid in replica

2017-04-12 Thread Dhyan Muralidharan 
Hi,
  I see issue in kafka 0.9 consumer where messages are corrupted in
partitions master node . I have details explained in stack-overflow .
http://stackoverflow.com/questions/43379306/kafka-message-corrupted-in-master-but-replica-looks-good
. Can someone help ?

--Dhyan

Consumer offsets resetting

2017-04-12 Thread Lawrence Weikum 
We’re seeing a strange error on our Kafka cluster. Users have reported their 
consumer offsets seem to periodically reset, which makes downstream processes 
handle a massive amount of duplicates.

Our Kafka cluster is currently running 0.10.1.1. Previously the cluster had 
been at 0.10.0.1. It was upgraded to 0.10.1.0, but we ran into multiple 
deadlocks so we downgraded to 0.10.0.1. We upgrade from 0.10.0.1 to 0.10.1.1.

This is the error we see. It only comes up every few weeks and only on some 
topics.

[2017-04-12 08:41:14,313] ERROR [ReplicaFetcherThread-1-8], Error for partition 
[email_priority,10] to broker 
8:org.apache.kafka.common.errors.UnknownServerException: The server experienced 
an unexpected error when processing the request 
(kafka.server.ReplicaFetcherThread)

Has anyone seen this behavior before? Is there any way to prevent this from 
happening?

Thank you all!


Lawrence Weikum | Sr. Software Engineer | Pandora
1426 Pearl Street, Suite 100, Boulder CO 80302
m 720.203.1578 | lwei...@pandora.com

Kafka Support as Service

2017-04-12 Thread Diego Paes Ramalho Pereira 
Hello,

I work for a Stock Exchange in Brazil and We are looking for a company that can 
provide us Kafka support as a service.

Do you have companies that provide this kind of service?

Regards,
Diego Paes Ramalho Pereira
Senior Infrastructure Analyst
Phone.: + 55 11 2565-5004 (Team)
Phone.: + 55 11 2565-5049


[Logotipo_B3_Positivo]



'


Esta mensagem pode conter informa??o confidencial e/ou privilegiada e ? 
destinada exclusivamente ao indiv?duo ou ? entidade para a qual ? endere?ada. O 
remetente utiliza o correio eletr?nico no exerc?cio do seu trabalho ou em raz?o 
dele, eximindo a Companhia e as demais institui??es integrantes de seu grupo 
econ?mico de qualquer responsabilidade por sua utiliza??o indevida. Se voc? n?o 
for o destinat?rio ou a pessoa autorizada a receber esta mensagem, n?o dever? 
utilizar, armazenar, copiar, alterar e/ou divulgar a mensagem ou qualquer 
informa??o nela contida, ou tomar qualquer a??o baseada nessas informa??es. Se 
voc? recebeu esta mensagem por engano, por favor, avise imediatamente o 
remetente, apagando a mensagem em seguida. Agradecemos sua coopera??o.

This message may contain confidential and/or privileged information and is 
intended to be received solely by the person or entity it is addressed to. The 
sender uses its e-mail box as a working tool, being the Company and the other 
entities that are part of its corporate group not liable for any misuse. If you 
are not the addressee or authorized to receive this for the addressee, you must 
not use, retain, copy, disclose, change and/or take any action based on this 
message or any information herein. If you have received this message in error, 
please inform the sender immediately and delete this message. Thank you for 
your cooperation.
'

Consumer offsets resetting

2017-04-12 Thread Lawrence Weikum 
We’re seeing a strange error on our Kafka cluster. Users have reported their 
consumer offsets seem to periodically reset, which makes downstream processes 
handle a massive amount of duplicates.

Our Kafka cluster is currently running 0.10.1.1. Previously the cluster had 
been at 0.10.0.1. It was upgraded to 0.10.1.0, but we ran into multiple 
deadlocks so we downgraded to 0.10.0.1. We upgrade from 0.10.0.1 to 0.10.1.1.

This is the error we see. It only comes up every few weeks and only on some 
topics.

[2017-04-12 08:41:14,313] ERROR [ReplicaFetcherThread-1-8], Error for partition 
[email_priority,10] to broker 
8:org.apache.kafka.common.errors.UnknownServerException: The server experienced 
an unexpected error when processing the request 
(kafka.server.ReplicaFetcherThread)

Has anyone seen this behavior before? Is there any way to prevent this from 
happening?

Thank you all!


Lawrence Weikum | Sr. Software Engineer | Pandora
1426 Pearl Street, Suite 100, Boulder CO 80302
m 720.203.1578 | lwei...@pandora.com

Re: Kafka MTLS Support?

2017-04-12 Thread Hans Jespersen 

Are you asking about Multiplexed Transport Layer Security (MTLS) - 
https://en.wikipedia.org/wiki/Multiplexed_Transport_Layer_Security 
  or Mutual 
TLS authentication (mTLS) https://en.wikipedia.org/wiki/Mutual_authentication 
 and can you provide more 
information about the motivation for your question.

-hans




> On Apr 12, 2017, at 1:50 AM, Sriram Srinivasaraghavan (srirsri2) 
>  wrote:
> 
> Team,
> 
> Would like to know if MTLS is supported as a security protocol, for 
> inter-broker communications and clients to broker communications?
> Could someone throw some light?
> 
> Regards,
> Sriram S

Re: How to get metrics from Kafka?

2017-04-12 Thread Sachin Mittal 
check this
http://kafka.apache.org/documentation.html#monitoring

You can use any jmx client to hook up to the running server jvm.
https://cwiki.apache.org/confluence/display/KAFKA/JMX+Reporters or use
jconsole.

I use https://cwiki.apache.org/confluence/display/KAFKA/jmxterm+quickstart
and works well for me.

Thanks
Sachin


On Wed, Apr 12, 2017 at 10:51 PM, Sriram Srinivasaraghavan (srirsri2) <
srirs...@cisco.com> wrote:

> Team,
>
> I am new to Kafka.
> I would like to know how to get general metrics to monitor a kafka cluster.
> For ex:
>
> 1.  CPU, Memory details of individual brokers
>
> 2.  Number of messages produced by a Producer vs written to the broker.
>
> 3.  Number of messages consumed by the consumer vs lost
>
> Any help would be highly appreciated.
>
> Regards,
> Sriram S
>

How to get metrics from Kafka?

2017-04-12 Thread Sriram Srinivasaraghavan (srirsri2) 
Team,

I am new to Kafka.
I would like to know how to get general metrics to monitor a kafka cluster.
For ex:

1.  CPU, Memory details of individual brokers

2.  Number of messages produced by a Producer vs written to the broker.

3.  Number of messages consumed by the consumer vs lost

Any help would be highly appreciated.

Regards,
Sriram S

Re: [VOTE] 0.10.2.1 RC0

2017-04-12 Thread Michal Borowiecki 
FWIW, I upgraded without issue and noticed the speedup from 
KAFKA-4851/KAFKA-4876.


+1 from me (non-binding)


On 12/04/17 02:06, Gwen Shapira wrote:

Wrong link :)
http://kafka.apache.org/documentation/#upgrade
and
http://kafka.apache.org/documentation/streams#streams_api_changes_0102

On Tue, Apr 11, 2017 at 5:57 PM, Gwen Shapira  wrote:

FYI: I just updated the upgrade notes with Streams changes:
http://kafka.apache.org/documentation/#gettingStarted

On Fri, Apr 7, 2017 at 5:12 PM, Gwen Shapira  wrote:

Hello Kafka users, developers and client-developers,

This is the first candidate for the release of Apache Kafka 0.10.2.1. This
is a bug fix release and it includes fixes and improvements from 24 JIRAs
(including a few critical bugs). See the release notes for more details:

http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc0/RELEASE_NOTES.html

*** Please download, test and vote by Thursday, 13 April, 8am PT ***

Your help in validating this bugfix release is super valuable, so
please take the time to test and vote!

Few notes:
1. There are missing "Notable Changes" in the docs:
https://github.com/apache/kafka/pull/2824
I will review, merge and update the docs by Monday.
2. The last commit (KAFKA-4943 chery-pick) did not pass system tests
yet. We may need another RC if system tests fail tonight.

Suggested tests:
  * Grab the source archive and make sure it compiles
  * Grab one of the binary distros and run the quickstarts against them
  * Extract and verify one of the site docs jars
  * Build a sample against jars in the staging repo
  * Validate GPG signatures on at least one file
  * Validate the javadocs look ok

*

Kafka's KEYS file containing PGP keys we use to sign the release:
http://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc0/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging

* Javadoc:
http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc0/javadoc/

* Tag to be voted upon (off 0.10.0 branch) is the 0.10.0.1-rc0 tag:
https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=d08115f05da0e39c7f75b45e05d6d14ad5baf71d

* Documentation:
http://kafka.apache.org/0102/documentation.html

* Protocol:
http://kafka.apache.org/0102/protocol.html

Thanks,
Gwen Shapira



--
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog





--
Signature
 Michal Borowiecki
Senior Software Engineer L4
T:  +44 208 742 1600


+44 203 249 8448



E:  michal.borowie...@openbet.com
W:  www.openbet.com 


OpenBet Ltd

Chiswick Park Building 9

566 Chiswick High Rd

London

W4 5XT

UK




This message is confidential and intended only for the addressee. If you 
have received this message in error, please immediately notify the 
postmas...@openbet.com  and delete it 
from your system as well as any copies. The content of e-mails as well 
as traffic data may be monitored by OpenBet for employment and security 
purposes. To protect the environment please do not print this e-mail 
unless necessary. OpenBet Ltd. Registered Office: Chiswick Park Building 
9, 566 Chiswick High Road, London, W4 5XT, United Kingdom. A company 
registered in England and Wales. Registered no. 3134634. VAT no. 
GB927523612

Kafka MTLS Support?

2017-04-12 Thread Sriram Srinivasaraghavan (srirsri2) 
Team,

Would like to know if MTLS is supported as a security protocol, for 
inter-broker communications and clients to broker communications?
Could someone throw some light?

Regards,
Sriram S

Subscription

2017-04-12 Thread Jaikishan Khatwani 
Please subscribe to user questions news letter. Regards,  -Jaikishan

Apache Kafka SSL Deployment

2017-04-12 Thread IT Consultant 
Hi All ,



How can I avoid using password for keystore creation ?

We are currently passing keystore password while accessing TLS enabled
Kafka instance .

I would like to use either passwordless keystore or avoid password for
clients accessing Kafka .