I agree with Todd, the major concern I have is still the complexity on
broker which can kill the performance - which a key advantage of Kafka. I
think there are two separate issues here:
1. Key management
2. the actual encryption/decryption work.
Personally I think it might be OK to have [1]
[
https://issues.apache.org/jira/browse/KAFKA-2143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14650146#comment-14650146
]
Jiangjie Qin commented on KAFKA-2143:
-
Hmm, with current design when the follower got
[
https://issues.apache.org/jira/browse/KAFKA-2334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangjie Qin reassigned KAFKA-2334:
---
Assignee: Jiangjie Qin
Prevent HW from going back during leader failover
[
https://issues.apache.org/jira/browse/KAFKA-2143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangjie Qin reassigned KAFKA-2143:
---
Assignee: Jiangjie Qin
Replicas get ahead of leader and fail
I think the abstraction of processor would be useful. It is not quite clear
to me yet though which grid in the following API analysis chart this
processor is trying to satisfy.
https://cwiki.apache.org/confluence/display/KAFKA/New+consumer+API+change+proposal
For example, in current proposal. It
1 - Yes, authorization combined with encryption does get us most of the way
there. However, depending on the auditor it might not be good enough. The
problem is that if you are encrypting at the broker, then by definition
anyone who has access to the broker (i.e. operations staff) have access to
Hi Jun,
This is still debatable, but I think it makes the most sense to keep
pause/resume independent of assignment. Otherwise we still get into the
weird ordering problems that we were trying to resolve before. To me,
pause/resume expresses clearly the intent to suppress consumption from a
set
Hi all,
With the initial patch of of the Copycat APIs under review, I want to get
started on the data API. I wrote up some notes with a rough proposal for
what it should cover and a few requirements here:
https://cwiki.apache.org/confluence/display/KAFKA/Copycat+Data+API
Since they are very
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33620/#review93734
---
On July 31, 2015, 4:32 p.m., Sriharsha Chintalapani wrote:
core/src/test/scala/integration/kafka/api/SSLConsumerTest.scala, line 218
https://reviews.apache.org/r/33620/diff/13/?file=1022004#file1022004line218
If we want to enforce this coding convention.Lets open up a new JIRA
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33620/#review93735
---
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33620/#review93736
---
core/src/test/scala/integration/kafka/api/SSLConsumerTest.scala
[
https://issues.apache.org/jira/browse/KAFKA-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649475#comment-14649475
]
Ismael Juma commented on KAFKA-2364:
Yes, dev@kafka.apache.org. The forum hasn't been
[
https://issues.apache.org/jira/browse/KAFKA-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649472#comment-14649472
]
Grant Henke commented on KAFKA-2111:
Should we consider also adding a --version option
[
https://issues.apache.org/jira/browse/KAFKA-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flavio Junqueira reassigned KAFKA-2385:
---
Assignee: Flavio Junqueira
zookeeper-shell does not work
[
https://issues.apache.org/jira/browse/KAFKA-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649430#comment-14649430
]
Aseem Bansal commented on KAFKA-2364:
-
It says Create a patch that applies cleanly
[
https://issues.apache.org/jira/browse/KAFKA-2337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649421#comment-14649421
]
Jun Rao commented on KAFKA-2337:
Sorry for the late comment. It seems that in
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33620/#review93729
---
[
https://issues.apache.org/jira/browse/KAFKA-2393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649438#comment-14649438
]
Grant Henke commented on KAFKA-2393:
Makes this scenario more likely to occur.
On July 31, 2015, 4:29 p.m., Sriharsha Chintalapani wrote:
clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java,
line 162
https://reviews.apache.org/r/33620/diff/13/?file=1021979#file1021979line162
hasRemaining doesn't work here. Hence the reason to go
[
https://issues.apache.org/jira/browse/KAFKA-2337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649427#comment-14649427
]
Grant Henke commented on KAFKA-2337:
Since this is already committed, lets create a
[
https://issues.apache.org/jira/browse/KAFKA-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649439#comment-14649439
]
Ismael Juma commented on KAFKA-2364:
That's right, this is because the website is
[
https://issues.apache.org/jira/browse/KAFKA-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649447#comment-14649447
]
Aseem Bansal commented on KAFKA-2364:
-
You mean dev@kafka.apache.org? I found that on
[
https://issues.apache.org/jira/browse/KAFKA-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangjie Qin resolved KAFKA-2385.
-
Resolution: Not A Problem
zookeeper-shell does not work
-
[
https://issues.apache.org/jira/browse/KAFKA-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649470#comment-14649470
]
Jiangjie Qin commented on KAFKA-2385:
-
[~ewencp] [~fpj], Ah, you are right. It
[
https://issues.apache.org/jira/browse/KAFKA-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649370#comment-14649370
]
Flavio Junqueira commented on KAFKA-2385:
-
I'm also unclear on what the problem is
Grant Henke created KAFKA-2393:
--
Summary: Correctly Handle InvalidTopicException in
KafkaApis.getTopicMetadata()
Key: KAFKA-2393
URL: https://issues.apache.org/jira/browse/KAFKA-2393
Project: Kafka
Hi
When discussing on KAFKA-2364 migrating docs from svn to git came up. That
would make contributing to docs much easier. I have contributed to
groovy/grails via github so I think having mirror on github could be useful.
Also I think unless there is some good reason it should be a separate
[
https://issues.apache.org/jira/browse/KAFKA-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-2386:
-
Issue Type: Sub-task (was: Bug)
Parent: KAFKA-2054
Transient test failure:
[
https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649503#comment-14649503
]
Ismael Juma commented on KAFKA-1690:
[~sourabh0612], I think we need to figure out how
[
https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649523#comment-14649523
]
Ismael Juma commented on KAFKA-1690:
[~harsha_ch], I just meant whether it should be
I was thinking a little bit this morning about the subscription API and I
have a few ideas on how to address some of the concerns about intuitiveness
and exception handling.
1. Split the current notion of topic/partition subscription into
subscription of topics and assignment of partitions. These
[
https://issues.apache.org/jira/browse/KAFKA-2300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-2300:
-
Fix Version/s: 0.8.3
Status: Patch Available (was: Open)
Error in controller log
I like the idea as well. That is much clearer.
Also agree with Jay on the naming.
Thanks, Jason. I'll update the Jira ticket.
Jiangjie (Becket) Qin
On Fri, Jul 31, 2015 at 12:19 PM, Jay Kreps j...@confluent.io wrote:
I like all these ideas.
Our convention is to keep method names declarative
On Fri, Jul 31, 2015 at 8:37 PM, Jay Kreps j...@confluent.io wrote:
The issue last time was that Apache has special infrastructure for web
hosting built around svn called svnpubsub. This is what takes the content
changes and pushes them live to the site. They didn't yet have a gitpubsub
at
I think that Hadoop and Cassandra do [1] (Transparent Encryption)
We're doing [2] (on a side note, for [2] you still need authentication on
the producer side - you don't want an unauthorized user writing garbage).
Right now we have the 'user' doing the encryption and submitting raw bytes
to the
[
https://issues.apache.org/jira/browse/KAFKA-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Onur Karaman updated KAFKA-2388:
Description:
According to the mailing list discussion. The interface will be changed to:
{code}
I like all these ideas.
Our convention is to keep method names declarative so it should probably be
subscribe(ListString topics, Callback c)
assign(ListTopicPartition)
The javadoc would obviously have to clarify the relationship between a
subscribed topic and assigned partitions. Presumably
I think the goal here is to make the actual message stored on broker to be
encrypted, because after we have SSL, the transmission would be encrypted.
In general there might be tow approaches:
1. Broker do the encryption/decryption
2. Client do the encryption/decryption
From performance point of
Currently, we have a KafkaPreCommit and a kafka-trunk job on Jenkins. They
seem identical. kafka-trunk seems to be more stable. I am disabling
KafkaPreCommit job for now.
Thanks,
Jun
On Wed, Jul 29, 2015 at 9:02 AM, Ismael Juma ism...@juma.me.uk wrote:
On Wed, Jul 29, 2015 at 4:52 PM,
[
https://issues.apache.org/jira/browse/KAFKA-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangjie Qin updated KAFKA-2388:
Description:
According to the mailing list discussion. The interface will be changed to:
void
[
https://issues.apache.org/jira/browse/KAFKA-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Onur Karaman updated KAFKA-2388:
Description:
According to the mailing list discussion, the consumer interface will replace:
{code}
The issue last time was that Apache has special infrastructure for web
hosting built around svn called svnpubsub. This is what takes the content
changes and pushes them live to the site. They didn't yet have a gitpubsub
at the time. If they've fixed that then we should be unblocked to switch.
[
https://issues.apache.org/jira/browse/KAFKA-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangjie Qin reassigned KAFKA-2389:
---
Assignee: Jiangjie Qin
CommitType seems not necessary in commit().
[
https://issues.apache.org/jira/browse/KAFKA-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Onur Karaman updated KAFKA-2388:
Description:
According to the mailing list discussion on the consumer interface, we'll
replace:
Just a quick ping, that regardless of the name of the thing, I'm still
interested in answers to my questions :)
On Tue, Jul 28, 2015 at 3:07 PM, Gwen Shapira gshap...@cloudera.com wrote:
Thanks Guazhang! Much clearer now, at least for me.
Few comments / questions:
1. Perhaps punctuate(int
Good points :)
1) Kafka already (pending commit) has an authorization layer, so
theoretically we are good for SOX, HIPAA, PCI, etc. Transparent broker
encryption will support PCI
never-let-unencrypted-card-number-hit-disk.
2) Agree on Key Management being complete PITA. It may better to
On July 27, 2015, 1:32 p.m., Ismael Juma wrote:
clients/src/main/java/org/apache/kafka/common/config/SSLConfigs.java, line
29
https://reviews.apache.org/r/33620/diff/13/?file=1021968#file1021968line29
SSL is deprecated
I've seen interest in HDFS-like encryption zones in Kafka.
This has the advantage of magically encrypting data at rest regardless
of which client is used as a producer.
Adding it on the client side limits the feature to the java client.
Gwen
On Fri, Jul 31, 2015 at 1:20 PM, eugene miretsky
For those who are interested in the ticket:
https://issues.apache.org/jira/browse/KAFKA-2388
On Fri, Jul 31, 2015 at 1:14 PM, Jiangjie Qin j...@linkedin.com wrote:
I like the idea as well. That is much clearer.
Also agree with Jay on the naming.
Thanks, Jason. I'll update the Jira ticket.
Great ideas Jason!
On Fri, Jul 31, 2015 at 12:19 PM, Jay Kreps j...@confluent.io wrote:
I like all these ideas.
Our convention is to keep method names declarative so it should probably be
subscribe(ListString topics, Callback c)
assign(ListTopicPartition)
The javadoc would obviously
[
https://issues.apache.org/jira/browse/KAFKA-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648890#comment-14648890
]
Ewen Cheslack-Postava commented on KAFKA-2385:
--
What exactly doesn't work
Jason Gustafson created KAFKA-2394:
--
Summary: Use RollingFileAppender by default in log4j.properties
Key: KAFKA-2394
URL: https://issues.apache.org/jira/browse/KAFKA-2394
Project: Kafka
Jason,
I guess that with the new setAssignment() api, we will also be getting rid
of pause() and resume()?
Thanks,
Jun
On Fri, Jul 31, 2015 at 11:29 AM, Jason Gustafson ja...@confluent.io
wrote:
I was thinking a little bit this morning about the subscription API and I
have a few ideas on
It does limit it to clients that have an implementation for encryption,
however encryption on the client side is better from an auditing point of
view (whether that is SOX, HIPAA, PCI, or something else). Most of those
types of standards are based around allowing visibility of data to just the
Buğra Gedik created KAFKA-2392:
--
Summary: Kafka Server does not accept 0 as a port
Key: KAFKA-2392
URL: https://issues.apache.org/jira/browse/KAFKA-2392
Project: Kafka
Issue Type: Bug
56 matches
Mail list logo
