RE: Security for individual partitions

[赖剑清]

Hi, Yash

I don't know the version and language of consumer client you use.
Java client [kafka-clients-0.9.0.0] provides a method [public void 
assign(List partitions)] in KafkaConsumer.class to manually 
assign a list of partition to this consumer.

This may be what you want.

>-Original Message-
>From: Yash Ganthe [mailto:yas...@gmail.com]
>Sent: Monday, June 25, 2018 7:50 PM
>To: users@kafka.apache.org
>Subject: Security for individual partitions
>
>Hi,
>
>If I have a topic with 10 partitions, I would like each partition to be 
>accessible
>to only certain consumers. Consumer 1 should be able to read from partition 1
>but no other partition and so on. Is this possible in Kafka?
>
>-Yash

Re: NPE in low level Kafka topology

[Guozhang Wang]

Hi Frank,

One issue that I can remember is this one:
https://issues.apache.org/jira/browse/KAFKA-4324

Since I have not seen your full topology building code including
com.dexels.kafka.streams.remotejoin.StoreProcessor,
com.dexels.kafka.streams.remotejoin.PreJoinProcessor and
com.dexels.kafka.streams.remotejoin.ranged.OneToManyGroupedProcessor
implementations I cannot tell if it is related to your issue.

If you believe they are different, do you mind sharing the full chunk of
your topology building code with the above three classes?



Guozhang


On Wed, Jun 20, 2018 at 5:25 AM, Frank Lyaruu  wrote:

> I didn't get much further. When I run with the 1.1.0 release version the
> stacktrace looks slightly different, but still a very similar NPE, after
> the same amount of time.
> One observation is that I use a few different processors, and it seems
> random which one gets caught in the stack trace.
>
> I've put the description of the topology in another gist:
>
> https://gist.github.com/flyaruu/39fba78aec562ae5d6f11d3add6a0881
>
> I've captured the last second or so before the NPE at debug level here:
>
> https://gist.github.com/flyaruu/43d31de10e03af160b20e9534f13830e
>
> I've increased the heap size (in case of a silent OOM exception), doesn't
> seem to matter
>
> I'm kinda out of ideas.
>
>
>
> On Tue, Jun 19, 2018 at 11:02 AM Frank Lyaruu  wrote:
>
> > We've tried running a fresh version with yesterday morning's trunk
> > version, with the same result.
> > We're running +- 15 KafkaStreams instances, and the one that fails is
> ithe
> > biggest one, with >150 processors.
> > We haven't been able to reproduce this error with smaller sub-sets.
> >
> > I'm now going to try this with the Kafka 1.1.0 release version.
> >
> > regards, Frank
> >
> > On Tue, Jun 19, 2018 at 1:18 AM Guozhang Wang 
> wrote:
> >
> >> Hello Frank,
> >>
> >> Your OneToManyGroupedProcessor.java looks fine to me.
> >>
> >> Is it consistently re-producible? What if you restart from fresh using
> the
> >> trunk version?
> >>
> >>
> >> Guozhang
> >>
> >> On Mon, Jun 18, 2018 at 9:03 AM, Frank Lyaruu 
> wrote:
> >>
> >> > Yes, here it is:
> >> >
> >> > https://gist.github.com/flyaruu/84b65d52a01987946b07d21bafe0d5f1
> >> >
> >> > It ran completely fine for the last year (and still does), it just
> does
> >> not
> >> > seem to enjoy the upgrade of Kafka Streams.
> >> >
> >> > regards, Frank
> >> >
> >> > On Mon, Jun 18, 2018 at 4:49 PM Ted Yu  wrote:
> >> >
> >> > > Can you show the related code from OneToManyGroupedProcessor ?
> >> > >
> >> > > Thanks
> >> > >
> >> > > On Mon, Jun 18, 2018 at 4:29 AM, Frank Lyaruu 
> >> wrote:
> >> > >
> >> > > > Hi, I've upgraded our 0.11 based stream application to the trunk
> >> > version,
> >> > > > and I get an intermittent NPE. It's is quite a big topology, and I
> >> > > haven't
> >> > > > succeeded in reproducing it on a simpler topology.
> >> > > > It builds the topology, starts Kafka Streams, runs for about 20s.,
> >> and
> >> > > then
> >> > > > it terminates
> >> > > > It seems that the 'currentNode' in the ProcessorContext is null.
> >> > > >
> >> > > > Does this ring a bell for anyone?
> >> > > >
> >> > > > [lowlevel-develop-generation-7aa-lowlevel-e23137ae-d94b-
> >> > > > 4f17-a684-995320fd426d-StreamThread-12]
> >> > > > ERROR
> >> org.apache.kafka.streams.processor.internals.AssignedStreamsTasks
> >> > -
> >> > > > stream-thread
> >> > > > [lowlevel-develop-generation-7aa-lowlevel-e23137ae-d94b-
> >> > > > 4f17-a684-995320fd426d-StreamThread-12]
> >> > > > Failed to process stream task 0_0 due to the following error:
> >> > > > java.lang.NullPointerException
> >> > > > at
> >> > > >
> >> > > org.apache.kafka.streams.processor.internals.
> >> > ProcessorContextImpl.forward(
> >> > > > ProcessorContextImpl.java:114)
> >> > > > at
> >> > > >
> >> > > org.apache.kafka.streams.processor.internals.
> >> > ProcessorContextImpl.forward(
> >> > > > ProcessorContextImpl.java:90)
> >> > > > at
> >> > > >
> >> com.dexels.kafka.streams.remotejoin.ranged.OneToManyGroupedProcessor.
> >> > > > forwardMessage(OneToManyGroupedProcessor.java:125)
> >> > > > at
> >> > > >
> >> com.dexels.kafka.streams.remotejoin.ranged.OneToManyGroupedProcessor.
> >> > > > forwardJoin(OneToManyGroupedProcessor.java:101)
> >> > > > at
> >> > > >
> >> com.dexels.kafka.streams.remotejoin.ranged.OneToManyGroupedProcessor.
> >> > > > process(OneToManyGroupedProcessor.java:70)
> >> > > > at
> >> > > >
> >> com.dexels.kafka.streams.remotejoin.ranged.OneToManyGroupedProcessor.
> >> > > > process(OneToManyGroupedProcessor.java:1)
> >> > > > at
> >> > > > org.apache.kafka.streams.processor.internals.ProcessorNode$1.run(
> >> > > > ProcessorNode.java:50)
> >> > > > at
> >> > > > org.apache.kafka.streams.processor.internals.ProcessorNode.
> >> > > > runAndMeasureLatency(ProcessorNode.java:244)
> >> > > > at
> >> > > > org.apache.kafka.streams.processor.internals.
> ProcessorNode.process(
> >> > > > Pr

Re: ConsumerConnector and 0.10 message format

[Ismael Juma]

The performance impact happens if the consumer doesn't support the message
format defined by log.message.format.version. The old consumer in Kafka
0.10 supports the message format introduced in 0.10 (including timestamps
and the ability to avoid recompression in the produce path) so what you
propose is fine.

However, the old consumer does _not_ support the message format introduced
in 0.11.0 (no matter the version) so you will eventually have to deal with
down conversion if that application cannot be upgraded to use the Java
consumer. Kafka 2.0.0 will contain down conversion efficiency improvements
and you may consider testing if the down conversion costs are bearable for
that application. If they are, the 0.11.0+ message format version is a good
path forward as it is necessary for using headers, transactions and the
idempotent producer.

Ismael

On Thu, Jun 21, 2018 at 11:30 PM Jeff Pollard 
wrote:

> Hello all,
>
> We are in the process of upgrading all our consumers to version 0.10+. One
> of our pre-0.10 consumers still uses the deprecated ConsumerConnector API
> (I believe colloquially called the "old consumer").
>
> We're unfortunately not able to upgrade this consumer to the new consumer
> API, as the consumer is baked into a larger piece of software we did not
> write, and they have not offered a version of it with the new consumer.
> They have, however, released a version using Kafka 0.10, which we have
> upgraded to.
>
> We also have upgraded our *brokers* to 0.10+, but per the potential
> performance impact
> <
> http://kafka.apache.org/0102/documentation.html#upgrade_10_performance_impact
> >
> noted
> in documentation have set the log.message.format.version to 0.8.2. This has
> worked well for us, as we've slowly been upgrading our consumers to 0.10+.
>
> My question is about using the old ConsumerConnector API with Kafka 0.10+
> client and a 0.10+ broker. From my reading of the documentation, it seems
> the performance impact is translating the message format by the broker, and
> is independent of the kind (old or new) consumer. I believe both the old
> and new consumer both use the same fetch API, but I hadn't verified that
> yet.
>
> We just wanted to make sure this was correct, as to avoid any unexpected
> performance impacts when we finish our upgrade and change the on-broker
> message format to the 0.10+ one.
>
> Thanks again. Happy to answer or clarify any points of this email as
> needed.
>

Re: [VOTE] 2.0.0 RC0

[Thomas Crayford]

+1 (non-binding) Heroku has run our usual set of upgrade and performance
tests, and we haven't found any notable issues through that.

On Sat, Jun 23, 2018 at 12:30 AM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> +1 (non-binding)
>
> Built from source and ran quickstart successfully on Ubuntu (with Java 8
> and Java 9).
>
> Thanks Rajini!
> --Vahid
>
>

Re: [kafka-clients] Re: [VOTE] 1.1.1 RC1

[Manikumar]

+1 (non-binding)  Ran tests,  Verified quick start,  producer/consumer perf
tests


On Sat, Jun 23, 2018 at 8:11 AM Dong Lin  wrote:

> Thank you for testing and voting the release!
>
> I noticed that the date for 1.1.1-rc1 is wrong. Please kindly test and
> vote by Tuesday, June 26, 12 pm PT.
>
> Thanks,
> Dong
>
> On Fri, Jun 22, 2018 at 10:09 AM, Dong Lin  wrote:
>
>> Hello Kafka users, developers and client-developers,
>>
>> This is the second candidate for release of Apache Kafka 1.1.1.
>>
>> Apache Kafka 1.1.1 is a bug-fix release for the 1.1 branch that was
>> first released with 1.1.0 about 3 months ago. We have fixed about 25 issues
>> since that release. A few of the more significant fixes include:
>>
>> KAFKA-6925  - Fix
>> memory leak in StreamsMetricsThreadImpl
>> KAFKA-6937  - In-sync
>> replica delayed during fetch if replica throttle is exceeded
>> KAFKA-6917  - Process
>> txn completion asynchronously to avoid deadlock
>> KAFKA-6893  - Create
>> processors before starting acceptor to avoid ArithmeticException
>> KAFKA-6870  -
>> Fix ConcurrentModificationException in SampledStat
>> KAFKA-6878  - Fix
>> NullPointerException when querying global state store
>> KAFKA-6879  - Invoke
>> session init callbacks outside lock to avoid Controller deadlock
>> KAFKA-6857  - Prevent
>> follower from truncating to the wrong offset if undefined leader epoch is
>> requested
>> KAFKA-6854  - Log
>> cleaner fails with transaction markers that are deleted during clean
>> KAFKA-6747  - Check
>> whether there is in-flight transaction before aborting transaction
>> KAFKA-6748  - Double
>> check before scheduling a new task after the punctuate call
>> KAFKA-6739  -
>> Fix IllegalArgumentException when down-converting from V2 to V0/V1
>> KAFKA-6728  -
>> Fix NullPointerException when instantiating the HeaderConverter
>>
>> Kafka 1.1.1 release plan:
>> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+1.1.1
>>
>> Release notes for the 1.1.1 release:
>> http://home.apache.org/~lindong/kafka-1.1.1-rc1/RELEASE_NOTES.html
>>
>> *** Please download, test and vote by Thursday, Jun 22, 12pm PT ***
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> http://kafka.apache.org/KEYS
>>
>> * Release artifacts to be voted upon (source and binary):
>> http://home.apache.org/~lindong/kafka-1.1.1-rc1/
>>
>> * Maven artifacts to be voted upon:
>> https://repository.apache.org/content/groups/staging/
>>
>> * Javadoc:
>> http://home.apache.org/~lindong/kafka-1.1.1-rc1/javadoc/
>>
>> * Tag to be voted upon (off 1.1 branch) is the 1.1.1-rc1 tag:
>> https://github.com/apache/kafka/tree/1.1.1-rc1
>>
>> * Documentation:
>> http://kafka.apache.org/11/documentation.html
>>
>> * Protocol:
>> http://kafka.apache.org/11/protocol.html
>>
>> * Successful Jenkins builds for the 1.1 branch:
>> Unit/integration tests: *https://builds.apache.org/job/kafka-1.1-jdk7/152/
>> *
>> System tests:
>> https://jenkins.confluent.io/job/system-test-kafka-branch-builder/1817
>>
>>
>> Please test and verify the release artifacts and submit a vote for this
>> RC,
>> or report any issues so we can fix them and get a new RC out ASAP.
>> Although
>> this release vote requires PMC votes to pass, testing, votes, and bug
>> reports are valuable and appreciated from everyone.
>>
>> Cheers,
>> Dong
>>
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "kafka-clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kafka-clients+unsubscr...@googlegroups.com.
> To post to this group, send email to kafka-clie...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kafka-clients.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kafka-clients/CAAaarBZCqdUPK8asaZS0ws0yr_vjFw0o8RxFcdRv07%3Df_7g%3DkQ%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

Re: Kafka Streams Session store fetch latency very high with caching turned on

[Guozhang Wang]

Hello Sam,

That is an interesting find. My reasoning is similar to yours: since you
have 1K / sec input traffic, it means 3600K / hour. Since you mean there
are about 500K / hour unique keys, it means each key will be updated
roughly about 7 times per hour. Assuming the traffic is even not skewed,
and your cache size is not large (by default it is only 50Mb) then it may
not help too much.

About the caching space, we had some optimizations along with KIP-155
implementations before (https://github.com/apache/kafka/pull/3027), to
reduce the search space inside cache to also corresponding segments in the
underlying store, for the same reason that `TreeMap#get()` is less
efficient with large key space. For session windows, there is no fixed
window length and the segment interval is purely dependent on the retention
period, my suspicion is that if your retention period is set to be small,
then `sessionStore#findSessions()`'s range will still span over almost all
segments which will not help reducing the search key space.

So for your case, I think disabling caching for that session store is a
good idea. At the same time we should consider further improving our
caching implementations to have sth. better than TreeMap.



Guozhang




On Sun, Jun 24, 2018 at 5:47 PM, Matthias J. Sax 
wrote:

> Sam,
>
> Thanks for your email. This is a very interesting find. I did not double
> check the code but your reasoning makes sense to me. Note, that caching
> was _not_ introduced to reduce the writes to RocksDB, but to reduce the
> write the the changelog topic and to reduce the number of records send
> downstream.
>
> Because, you don't want to have a fault-tolerant store and disabled
> caching, I see no reason why disabling caching would be a bad idea for
> your use case.
>
> From a performance point of view, there should be no difference between
> DSL and Processor API. Note, that the DSL sits on top of Processor API
> and at runtime, we don't even know if DSL was used or not. Caching is
> enabled by default to reduce the downstream load -- we have many
> discussion if this is the best default behavior. The latest conclusion
> was, that it is... :)
>
>
>
> -Matthias
>
>
>
> On 6/22/18 12:11 PM, Sam Lendle wrote:
> > I am using a session store in a kafka streams application. With caching
> turned on, average fetch latency was very high, about 200 ms after running
> for about 1 hour. With caching turned off, it was about 100 μs. We seem to
> be running fine without caching, but I am very curious as to why caching
> performance is so bad in our case. Any insight into what might be going on
> would be helpful.
> >
> >
> > Setup/config
> >
> >   *   I'm using a custom transformer, but the implementation is almost
> identical to this section of KStreamSessionWindowAggregate
> https://github.com/apache/kafka/blob/fdcf75ea326b8e07d8d7c4f5cc14f1
> e70451bcd4/streams/src/main/java/org/apache/kafka/streams/
> kstream/internals/KStreamSessionWindowAggregate.java#L94-L113 The main
> difference is I'm forwarding something other than the updated session
> downstream
> >   *   Logging is turned off, so updates are not pushed to a change log
> topic. The store starts empty whenever streams is initialized
> >   *   I don't think I'm setting any potentially related configs. Rocksdb
> config is the default.
> >
> > We're receiving about 1000 messages/second in a topic w/ five
> partitions. With caching turned on, this custom transformer is the
> bottleneck and processing rate is much lower, 100-200 ops per second. With
> caching turned off the volume is no problem. There are about 500k unique
> keys per hour.
> >
> > Using a sampling profiler I saw that most time was spent in TreeMap
> operations. Unfortunately I don't have a copy of the profile data anymore,
> but I think the map in question is the `cache` field in the NamedCache
> class.
> >
> > If I look at a plot of fetch latency vs time since starting, it looks to
> me like latency is about O(log(time)). I think what's going on is the size
> of the map is increasing linearly in time, particularly for the first few
> minutes that streams is running, because almost all keys will be unique. So
> the latency is almost entirely spent in TreeMap#get.
> >
> > Questions:
> > 1) Does my theory make sense?
> > 2) Could the issue be related to the fact that I'm using a state store
> with the transformer/processor API vs the dsl? I know that caching is
> turned on by default for state stores in the dsl but not in the processor
> API, but I don't understand why.
> > 3) My understanding is that streams side state store caching is an
> optimization to reduce the number of writes to the underlying rocksdb
> store. In that case, because I have so many unique keys, and the same keys
> usually show up a few minutes apart, it makes sense that caching wouldn't
> do much for me. Is that correct?
> > 4) Given that things seem to work fine with caching turned off, could
> there be any advantage to having it 

Re: Retries

[Michael Eugene]

Well it does matter just because the variable wasn’t even in the 1.1 build, 
because as you probably know there was already a Jira opened that the variable 
was set as private and thus couldn’t be used. It was fixed but if you want to 
keep your build aligned with official releases then you can’t use the variable 
until the next one. 

Well as far as the string being used, there must be something else going I 
guess because I am doing that already.  I already have gone through it enough 
that it doesn’t really make sense to me why though. I have peeled my 
application so there isn’t that many moving pieces anymore so I’m not really 
sure. If you could just post a sample piece of code of when you got it to work 
that would be amazing. I know that’s asking a lot but I think it just doesn’t 
make sense at this point. 

Sent from my iPhone

> On Jun 25, 2018, at 11:22 AM, Matthias J. Sax  wrote:
> 
> `default.production.exception.handler` is the correct parameter.
> 
> It should not make any difference if you pass in a plain String or
> `StreamConfig#DEFAULT_PRODUCTION_EXCEPTION_HANDLER_CLASS_CONFIG` -- the
> variable is also just a String...
> 
> When you start the application, it should the config in the logs. Can
> you double check if it did pick up the handler there?
> 
> -Matthias
> 
>> On 6/24/18 6:42 PM, Michael Eugene wrote:
>> The thing about that is, when I try to register the handler, it doesn’t 
>> work.  It’s easy to register the deserialization handler because there is a 
>> static final constant variable I can pass. But when I pass the string 
>> "default.production.exception.handler” it doesn’t work. (That actually might 
>> not be the exact string but I did get the string from the source code on 
>> GitHub.) Has anyone actually used this?
>> 
>> Sent from my iPhone
>> 
>>> On Jun 24, 2018, at 8:03 PM, Matthias J. Sax  wrote:
>>> 
>>> Michael,
>>> 
>>> It depends on the semantics you want to get. About retries in general,
>>> as long as a producer retries internally, you would not even notice.
>>> Only after retries are exhausted, an exception is thrown.
>>> 
>>> Kafka Streams allows you to implement a handler for this (cf
>>> https://kafka.apache.org/11/documentation/streams/developer-guide/config-streams.html#default-production-exception-handler)
>>> that allows you to react to the error as you wish.
>>> 
>>> You can either use a provided handler or implement a custom one. You can
>>> either skip over a record or let Kafka Streams stop processing.
>>> 
>>> It might make sense to write the record to a "retry topic", but it
>>> depends on the error. If the whole cluster is down, of course you cannot
>>> write to the retry topic either. If the output topic is currently
>>> under-repliated and does not allow for new write, the "retry topic"
>>> might be available thought.
>>> 
>>> 
>>> For exactly-once, producer retries are set to MAX_VALUE and thus the
>>> application would re-try practically forever.
>>> 
>>> 
>>> -Matthias
>>> 
 On 6/16/18 1:52 PM, Michael Eugene wrote:
 Hi I am trying to understand when to retry sending messages to topics and 
 when to start trying to send to "retry" topics.  The scenario is basically
 
 1. A KafkaStreams application is consuming from a topic and sending to a 
 topic.  The "retries" is set at the default of 10.
 
 2a. After 10 retries, does it make sense to then try to publish to another 
 "retry topic"?
 2a1. What mechanism is there to know its the 10th retry, and to then start 
 sending to a "retry topic" after the 10th?
 
 2b. Or after 10 retries - for that message if its not successful its just 
 done. Since there is no real difference between sending to a "retry topic" 
 and sending to a non-retry topic, why not just set retries to a high level 
 (like 100).
 
 3. On an implementation level (Ive read the kafka docs, i find it a bit 
 high level) can someone throw a nugget out there about how exactly-once 
 semantics would erase the need for an "retry topic"?
 
 If u have time to answer any part of the above question, thank you in 
 advance.
 
 
 
 
 
 Get Outlook for Android
 
 
>>> 
> 

Re: Retries

[Matthias J. Sax]

`default.production.exception.handler` is the correct parameter.

It should not make any difference if you pass in a plain String or
`StreamConfig#DEFAULT_PRODUCTION_EXCEPTION_HANDLER_CLASS_CONFIG` -- the
variable is also just a String...

When you start the application, it should the config in the logs. Can
you double check if it did pick up the handler there?

-Matthias

On 6/24/18 6:42 PM, Michael Eugene wrote:
> The thing about that is, when I try to register the handler, it doesn’t work. 
>  It’s easy to register the deserialization handler because there is a static 
> final constant variable I can pass. But when I pass the string 
> "default.production.exception.handler” it doesn’t work. (That actually might 
> not be the exact string but I did get the string from the source code on 
> GitHub.) Has anyone actually used this?
> 
> Sent from my iPhone
> 
>> On Jun 24, 2018, at 8:03 PM, Matthias J. Sax  wrote:
>>
>> Michael,
>>
>> It depends on the semantics you want to get. About retries in general,
>> as long as a producer retries internally, you would not even notice.
>> Only after retries are exhausted, an exception is thrown.
>>
>> Kafka Streams allows you to implement a handler for this (cf
>> https://kafka.apache.org/11/documentation/streams/developer-guide/config-streams.html#default-production-exception-handler)
>> that allows you to react to the error as you wish.
>>
>> You can either use a provided handler or implement a custom one. You can
>> either skip over a record or let Kafka Streams stop processing.
>>
>> It might make sense to write the record to a "retry topic", but it
>> depends on the error. If the whole cluster is down, of course you cannot
>> write to the retry topic either. If the output topic is currently
>> under-repliated and does not allow for new write, the "retry topic"
>> might be available thought.
>>
>>
>> For exactly-once, producer retries are set to MAX_VALUE and thus the
>> application would re-try practically forever.
>>
>>
>> -Matthias
>>
>>> On 6/16/18 1:52 PM, Michael Eugene wrote:
>>> Hi I am trying to understand when to retry sending messages to topics and 
>>> when to start trying to send to "retry" topics.  The scenario is basically
>>>
>>> 1. A KafkaStreams application is consuming from a topic and sending to a 
>>> topic.  The "retries" is set at the default of 10.
>>>
>>> 2a. After 10 retries, does it make sense to then try to publish to another 
>>> "retry topic"?
>>>  2a1. What mechanism is there to know its the 10th retry, and to then start 
>>> sending to a "retry topic" after the 10th?
>>>
>>> 2b. Or after 10 retries - for that message if its not successful its just 
>>> done. Since there is no real difference between sending to a "retry topic" 
>>> and sending to a non-retry topic, why not just set retries to a high level 
>>> (like 100).
>>>
>>> 3. On an implementation level (Ive read the kafka docs, i find it a bit 
>>> high level) can someone throw a nugget out there about how exactly-once 
>>> semantics would erase the need for an "retry topic"?
>>>
>>> If u have time to answer any part of the above question, thank you in 
>>> advance.
>>>
>>>
>>>
>>>
>>>
>>> Get Outlook for Android
>>>
>>>
>>



signature.asc
Description: OpenPGP digital signature

[VOTE] KIP-312: Add Overloaded StreamsBuilder Build Method to Accept java.util.Properties

[Bill Bejeck]

All,
I'd like to start a vote for this KIP now.

Thanks,
Bill

Re: Security for individual partitions

[Hans Jespersen]

Kafka ACLs are at the topic level, not partition level.

Probably better to make 10 topics of 1 partition each and use topic ACLs to 
control access.

-hans

> On Jun 25, 2018, at 9:50 PM, Yash Ganthe  wrote:
> 
> Hi,
> 
> If I have a topic with 10 partitions, I would like each partition to be
> accessible to only certain consumers. Consumer 1 should be able to read
> from partition 1 but no other partition and so on. Is this possible in
> Kafka?
> 
> -Yash

Re: Security for individual partitions

[Sönke Liebau]

Hi Yash,

I'm afraid this is not easily possible with existing functionality.
Even if you created your own authorizer, I'm fairly certain that the
partition is not available as part of the resource that is being
accessed.

Is there any specific reason why you can't create more than one topic
and give every customer access to a dedicated topic instead of trying
to manage this by partition?

Best regards,
Sönke

On Mon, Jun 25, 2018 at 1:50 PM, Yash Ganthe  wrote:
> Hi,
>
> If I have a topic with 10 partitions, I would like each partition to be
> accessible to only certain consumers. Consumer 1 should be able to read
> from partition 1 but no other partition and so on. Is this possible in
> Kafka?
>
> -Yash



-- 
Sönke Liebau
Partner
Tel. +49 179 7940878
OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Is possible to have an infinite delete.retention.ms?

[David Espinosa]

Hi all,

I would like to setup a compaction policy on a topic where a message can be
deleted (GDPR..) using a tombstone with the same key that the message to be
removed. My problem is that I would like to use empty payload messages also
for identifying that an entity has been deleted, but these last messages
got deleted by compaction. So my whole data integrity goes to trash.

Mi idea is to modify the property delete.retention.ms in a way that
tombstones are not deleted. My questions are:

   - Can I set an infinite value for delete.rentention.ms the same way a
   can do it with retention.ms=-1?
   - Does this property (delete.rentention.ms) has any consequence on the
   compaction interval?

Thanks in advance,
David

Security for individual partitions

[Yash Ganthe]

Hi,

If I have a topic with 10 partitions, I would like each partition to be
accessible to only certain consumers. Consumer 1 should be able to read
from partition 1 but no other partition and so on. Is this possible in
Kafka?

-Yash