Re: [DISCUSS] KIP-983: Full speed async processing during rebalance

2023-10-15 Thread Erik van Oosten
am still reading to KIP and your previous KIP to see if I can make more constructive suggestions here. P On Fri, Oct 13, 2023 at 11:54 PM Erik van Oosten wrote: Hello David, Thanks, I am happy to hear we agree on the problem. All the tiny details of an implementation are less important. I

Re: [DISCUSS] KIP-983: Full speed async processing during rebalance

2023-10-14 Thread Erik van Oosten
. 2023 à 20:44, Erik van Oosten a écrit : Thanks Philip, No worries, I am not in a hurry. Knowing this is not forgotten is enough for me. If there is anything I can do to help the process please let me know. Kind regards, Erik. Op 13-10-2023 om 20:29 schreef Philip Nee: Hi Erik, Sorry

Re: [DISCUSS] KIP-983: Full speed async processing during rebalance

2023-10-13 Thread Erik van Oosten
reviewing the KIP, but I also have not forgotten about it! In general, KIP review process can be lengthy, so I think mailing list is the best bet to get the committer's attention. P On Fri, Oct 13, 2023 at 10:55 AM Erik van Oosten wrote: Hi client developers, The text is updated so that it is more

Re: [DISCUSS] KIP-983: Full speed async processing during rebalance

2023-10-13 Thread Erik van Oosten
. Is async processing not something that the kafka client wants to support? Kind regards,     Erik. Op 25-09-2023 om 18:17 schreef Erik van Oosten: Hi Viktor, Good questions! 1. Auto-commits would only work with approach 1 in the KIP. Any async solution is incompatible with auto-commits. Do you

Re: [DISCUSS] KIP-983: Full speed async processing during rebalance

2023-09-25 Thread Erik van Oosten
but how would you change work in case of async offset commits? There can be a few caveats there as you may not know whether a commit is finished or not until your callback is called. Thanks, Viktor On Sat, Sep 23, 2023 at 4:00 PM Erik van Oosten wrote: Hi all, I would like to start

[DISCUSS] KIP-983: Full speed async processing during rebalance

2023-09-23 Thread Erik van Oosten
/confluence/display/KAFKA/KIP-983%3A+Full+speed+async+processing+during+rebalance -- Erik van Oosten e.vanoos...@grons.nl https://day-to-day-stuff.blogspot.com

Re: [slf4j-user] Release of SLF4J version 2.0.9

2023-09-05 Thread Erik van Oosten via slf4j-user
-user: Moreover, the "slf4j.provider" system property bypasses the service loader mechanism for finding providers and may shorten SLF4J initialization. -- Erik van Oosten e.vanoos...@grons.nl https://day-to-day-stuff.blogspot.com ___ slf4j-us

[jira] [Resolved] (KAFKA-14972) Make KafkaConsumer usable in async runtimes

2023-08-05 Thread Erik van Oosten (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik van Oosten resolved KAFKA-14972. - Resolution: Won't Fix > Make KafkaConsumer usable in async runti

[jira] [Resolved] (KAFKA-14972) Make KafkaConsumer usable in async runtimes

2023-08-05 Thread Erik van Oosten (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik van Oosten resolved KAFKA-14972. - Resolution: Won't Fix > Make KafkaConsumer usable in async runti

[jira] [Commented] (KAFKA-14972) Make KafkaConsumer usable in async runtimes

2023-08-05 Thread Erik van Oosten (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751330#comment-17751330 ] Erik van Oosten commented on KAFKA-14972: - I am closing this task as won't fix as the committers

[jira] [Assigned] (KAFKA-14972) Make KafkaConsumer usable in async runtimes

2023-08-05 Thread Erik van Oosten (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik van Oosten reassigned KAFKA-14972: --- Assignee: (was: Erik van Oosten) > Make KafkaConsumer usable in async runti

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

2023-08-03 Thread Erik van Oosten
my energy elsewhere. Kind regards,     Erik. Op 24-07-2023 om 18:12 schreef Erik van Oosten: Hello Xiangyuan LI, I am not familiar with coroutines, nor with Kotlin. You will have to work with the documentation: https://kotlinlang.org/docs/coroutines-overview.html However, I am familiar

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

2023-07-24 Thread Erik van Oosten
aybe you need a more elaborate description to demonstrate how coroutine code run. Erik van Oosten 于2023年7月23日周日 17:47写道: Hi David, > Could you elaborate a bit more on why the callbacks must be ran in another thread vs in the invoker thread? I have been thinking on how to explai

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

2023-07-23 Thread Erik van Oosten
KIP based on the discussion. Le sam. 22 juil. 2023 à 18:34, Erik van Oosten a écrit : Colin, Matthias, Chris, I have expanded the use case description in KIP-944. I hope it is more clear what we're trying to achieve. https://cwiki.apache.org/confluence/x/chw0Dw Kind regards, Erik. Op 22

[DISCUSS] KIP-957 Support async runtimes in consumer

2023-07-22 Thread Erik van Oosten
. The wiki page for KIP-957 contains more details https://cwiki.apache.org/confluence/x/lY6zDw This is a call for discussion. If possible I would like to include this change in Kafka 3.6. Any questions, comments, ideas and other additions are welcome! Kind regards,     Erik. -- Erik van Oosten

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

2023-07-22 Thread Erik van Oosten
Colin, Matthias, Chris, I have expanded the use case description in KIP-944. I hope it is more clear what we're trying to achieve. https://cwiki.apache.org/confluence/x/chw0Dw Kind regards,     Erik. Op 22-07-2023 om 17:23 schreef Erik van Oosten: Hello Chris, Thanks for elaborating

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

2023-07-22 Thread Erik van Oosten
the visibility of these methods from private to protected, it would be possible for subclasses to implement the proposal in KIP-944, without any KIPs or other changes to the official Java clients library. Best, Chris On Sat, Jul 22, 2023 at 4:24 AM Erik van Oosten wrote: Hi Matthias, I am getting a bit

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

2023-07-22 Thread Erik van Oosten
sal in KIP-944, without any KIPs or other changes to the official Java clients library. Best, Chris On Sat, Jul 22, 2023 at 4:24 AM Erik van Oosten wrote: Hi Matthias, I am getting a bit frustrated here. All the concerns and questions I have seen so far are addressed in KIP-944. Please let me k

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

2023-07-22 Thread Erik van Oosten
the radio silence. It gets like that sometimes, and I apologize for my lack of feedback. I’d personally like to see this lively exchange move over to the DISCUSS thread you’d created before. Thanks, Kirk On Jul 14, 2023, at 1:33 AM, Erik van Oosten wrote: Hi Colin, The way I understood

Re: [DISCUSS] KIP-944 Support async runtimes in consumer

2023-07-16 Thread Erik van Oosten
that are described in the KIP. Kind regards,     Erik. Op 30-06-2023 om 07:56 schreef Erik van Oosten: [This is a resend with the correct KIP number.] Hello developers of the Java based consumer, I submitted https://github.com/apache/kafka/pull/13914 to fix a long standing problem that the Kafka

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

2023-07-14 Thread Erik van Oosten
any unsafe access. I think this can be demonstrated via diagrams and some code in the KIP. Thanks, P On Sat, Jul 8, 2023 at 12:28 PM Erik van Oosten wrote: Hello Colin, >> In KIP-944, the callback thread can only delegate to another thread after reading from and writing to a thre

Re: KafkaConsumer refactor proposal

2023-07-13 Thread Erik van Oosten
-944) because, concurrent invocations are simply no longer needed. (Of course, if backward compatibility is a goal, not all of these things can be done.) Curious to your thoughts and kind regards,     Erik. -- Erik van Oosten e.vanoos...@grons.nl https://day-to-day-stuff.blogspot.com

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

2023-07-12 Thread Erik van Oosten
-Threadsafety Kind regards,     Erik. Op 12-07-2023 om 13:24 schreef Erik van Oosten: Thanks Philip, > I think this can be demonstrated via diagrams and some code in the KIP. There are some diagrams in KIP-944. How can they be improved? I will add some code to address the concerns around mem

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

2023-07-12 Thread Erik van Oosten
we are running inside the callback of the rebalance listener. The only way to execute something and also have a timeout on it is to run the something on another thread. Kind regards, Erik. Op 08-07-2023 om 19:17 schreef Colin McCabe: On Sat, Jul 8, 2023, at 02:41, Erik van Oosten wrote: Hi C

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

2023-07-08 Thread Erik van Oosten
t we are running inside the callback of the rebalance listener. The only way to execute something and also have a timeout on it is to run the something on another thread. Kind regards,     Erik. Op 08-07-2023 om 19:17 schreef Colin McCabe: On Sat, Jul 8, 2023, at 02:41, Erik van Oosten wrote: H

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

2023-07-08 Thread Erik van Oosten
not get a quick response. However, overall I think this one needs a lot more discussion before getting anywhere near a vote. I will leave a -1 just as a procedural step. Maybe some of the people working in the client area can also chime in. best, Colin On Thu, Jul 6, 2023, at 12:02, Erik v

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

2023-07-06 Thread Erik van Oosten
-2023 om 11:38 schreef Erik van Oosten: Hello all, I'd like to call a vote on KIP-944 Support async runtimes in consumer. It has has been 'under discussion' for 7 days now. 'Under discussion' between quotes, because there were 0 comments so far. I hope the KIP is clear! KIP description: https

[VOTE] KIP-944 Support async runtimes in consumer

2023-07-05 Thread Erik van Oosten
Hello all, I'd like to call a vote on KIP-944 Support async runtimes in consumer. It has has been 'under discussion' for 7 days now. 'Under discussion' between quotes, because there were 0 comments so far. I hope the KIP is clear! KIP description:

[DISCUSS] KIP-944 Support async runtimes in consumer

2023-06-29 Thread Erik van Oosten
section. As far as I am aware there are no tests for the current behavior. Any help in this area would be appreciated. Kind regards,     Erik. -- Erik van Oosten e.vanoos...@grons.nl https://day-to-day-stuff.blogspot.com

[jira] [Updated] (KAFKA-14972) Make KafkaConsumer usable in async runtimes

2023-06-28 Thread Erik van Oosten (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik van Oosten updated KAFKA-14972: Description: KafkaConsumer contains a check that rejects nested invocations from

[jira] [Updated] (KAFKA-14972) Make KafkaConsumer usable in async runtimes

2023-06-28 Thread Erik van Oosten (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik van Oosten updated KAFKA-14972: Description: KafkaConsumer contains a check that rejects nested invocations from

[jira] [Commented] (KAFKA-14972) Make KafkaConsumer usable in async runtimes

2023-06-28 Thread Erik van Oosten (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738014#comment-17738014 ] Erik van Oosten commented on KAFKA-14972: - KIP-944 https://cwiki.apache.org/confluence/x/chw0Dw

Re: [DISCUSS] KIP-944 Support async runtimes in consumer

2023-06-28 Thread Erik van Oosten
It seems KIP-941 was already taken. Updated to: KIP-944. Op 28-06-2023 om 10:11 schreef Erik van Oosten: Hello developers of the Java based consumer, I submitted https://github.com/apache/kafka/pull/13914 to fix a long standing problem that the Kafka consumer on the JVM is not usable from

[DISCUSS] KIP-941 Support async runtimes in consumer

2023-06-28 Thread Erik van Oosten
for the current behavior. Any help in this area would be appreciated. Kind regards,     Erik. -- Erik van Oosten e.vanoos...@grons.nl https://day-to-day-stuff.blogspot.com

[jira] [Commented] (KAFKA-14972) Make KafkaConsumer usable in async runtimes

2023-06-27 Thread Erik van Oosten (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17737711#comment-17737711 ] Erik van Oosten commented on KAFKA-14972: - I will complete the KIP tomorrow. > M

Permissions to contribute to Apache Kafka

2023-06-26 Thread Erik van Oosten
Dear reader, I would like to create a KIP and understand I need to request permissions for that. my wiki username: e.vanoos...@chello.nl  (note, this is /not/ my email address) my Jira username: erikvanoosten Kind regards,     Erik. -- Erik van Oosten e.vanoos...@grons.nl https://day

Review request for Java Kafka consumer

2023-06-26 Thread Erik van Oosten
Dear Kafka developers, I submitted https://github.com/apache/kafka/pull/13914 to fix a long standing problem that the Kafka consumer on the JVM is not usable from asynchronous runtimes such as Kotlin co-routines and ZIO. Your review is much appreciated. Kind regards,     Erik. -- Erik van

[jira] [Commented] (KAFKA-10337) Wait for pending async commits in commitSync() even if no offsets are specified

2023-06-08 Thread Erik van Oosten (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17730665#comment-17730665 ] Erik van Oosten commented on KAFKA-10337: - Thanks for your PR [~thomaslee]. It has been merged

[jira] [Created] (KAFKA-14972) Make KafkaConsumer usable in async runtimes

2023-05-07 Thread Erik van Oosten (Jira)
Erik van Oosten created KAFKA-14972: --- Summary: Make KafkaConsumer usable in async runtimes Key: KAFKA-14972 URL: https://issues.apache.org/jira/browse/KAFKA-14972 Project: Kafka Issue Type

[jira] [Created] (KAFKA-14972) Make KafkaConsumer usable in async runtimes

2023-05-07 Thread Erik van Oosten (Jira)
Erik van Oosten created KAFKA-14972: --- Summary: Make KafkaConsumer usable in async runtimes Key: KAFKA-14972 URL: https://issues.apache.org/jira/browse/KAFKA-14972 Project: Kafka Issue Type

Re: Kafka client needs KAFKA-10337 to cover async commit use case

2023-05-06 Thread Erik van Oosten
`. Is that an acceptable change? Kind regards,     Erik. Op 05-05-2023 om 20:20 schreef Philip Nee: Hey Eric, Maybe its more straightforward to open a new PR. Thanks! P On Fri, May 5, 2023 at 9:36 AM Erik van Oosten wrote: If I were to rebase the old pull request and re-open KAFKA-10337, would

[jira] [Commented] (KAFKA-10337) Wait for pending async commits in commitSync() even if no offsets are specified

2023-05-06 Thread Erik van Oosten (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720114#comment-17720114 ] Erik van Oosten commented on KAFKA-10337: - [~thomaslee] when we use commitAsync from

[jira] [Commented] (KAFKA-10337) Wait for pending async commits in commitSync() even if no offsets are specified

2023-05-06 Thread Erik van Oosten (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720113#comment-17720113 ] Erik van Oosten commented on KAFKA-10337: - Opened [~thomaslee] 's PR again: https://github.com

Re: Kafka client needs KAFKA-10337 to cover async commit use case

2023-05-05 Thread Erik van Oosten
If I were to rebase the old pull request and re-open KAFKA-10337, would it be considered for merging? Kind regards,     Erik. Op 03-05-2023 om 09:21 schreef Erik van Oosten: Hi Philip, Firstly, could you explain the situation in that you would prefer to invoke commitAsync over commitSync

Re: Kafka client needs KAFKA-10337 to cover async commit use case

2023-05-03 Thread Erik van Oosten
, which leads to my second comment/question. is it your concern that we currently don't have a way to invoke the callback, and the user won't be to correctly handle these failed/successful async commits? Thanks, P On Tue, May 2, 2023 at 12:22 PM Erik van Oosten wrote: Dear developers of the Kafka

Kafka client needs KAFKA-10337 to cover async commit use case

2023-05-02 Thread Erik van Oosten
be very happy to hear about that because it means I do not have to wait for a release cycle. If you agree these arguments are sound, I would be happy to make the pull request mergable again. Curious to your thoughts and kind regards,     Erik. -- Erik van Oosten e.vanoos...@grons.nl https

How to get async commit callbacks in the rebalance listener?

2023-04-26 Thread Erik van Oosten
d should I invoke (from the onPartitionsRevoked callback), to make the Kafka client invoke the callback of earlier async commits? Some context: I am working on zio-kafka; a completely async library that provides a concurrent streaming layer on top of the Java client. Thanks,     Erik. -

Re: [scala-functional] Re: java.lang.NullPointerException with JDK 11

2020-06-03 Thread 'Erik van Oosten' via scala-functional
If you want to run on JDK 11 you will need to make sure you have a reasonably recent scala version installed. These work: 2.11.12, 2.12.11 and of course 2.13.2. It wouldn't hurt to use a recent sbt version as well. Kind regards,     Erik. -- Erik van Oosten https://day-to-day

[jira] [Resolved] (SPARK-27025) Speed up toLocalIterator

2019-03-05 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik van Oosten resolved SPARK-27025. - Resolution: Incomplete > Speed up toLocalItera

[jira] [Commented] (SPARK-27025) Speed up toLocalIterator

2019-03-05 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784854#comment-16784854 ] Erik van Oosten commented on SPARK-27025: - If there is no obvious way to improve Spark, then its

[jira] [Comment Edited] (SPARK-27025) Speed up toLocalIterator

2019-03-04 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783220#comment-16783220 ] Erik van Oosten edited comment on SPARK-27025 at 3/4/19 10:36 AM

[jira] [Commented] (SPARK-27025) Speed up toLocalIterator

2019-03-04 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783220#comment-16783220 ] Erik van Oosten commented on SPARK-27025: - [~hyukjin.kwon] maybe I misunderstood Sean's comment

[jira] [Commented] (SPARK-27025) Speed up toLocalIterator

2019-03-04 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783110#comment-16783110 ] Erik van Oosten commented on SPARK-27025: - Thanks Sean, that is very useful. In my use case

[jira] [Comment Edited] (SPARK-27025) Speed up toLocalIterator

2019-03-02 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782322#comment-16782322 ] Erik van Oosten edited comment on SPARK-27025 at 3/2/19 8:43 AM

[jira] [Commented] (SPARK-27025) Speed up toLocalIterator

2019-03-02 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782322#comment-16782322 ] Erik van Oosten commented on SPARK-27025: - I have a program in which several steps need

[jira] [Created] (SPARK-27025) Speed up toLocalIterator

2019-03-01 Thread Erik van Oosten (JIRA)
Erik van Oosten created SPARK-27025: --- Summary: Speed up toLocalIterator Key: SPARK-27025 URL: https://issues.apache.org/jira/browse/SPARK-27025 Project: Spark Issue Type: Wish

Opportunity to speed up toLocalIterator?

2019-02-28 Thread Erik van Oosten
. -- Erik van Oosten http://www.day-to-day-stuff.blogspot.com/ - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Flink Streaming Job Task is getting cancelled silently and causing job to restart

2019-01-27 Thread Erik van Oosten
Hi sohimankotia, My advise from also having to sub-class BucketingSink: * rebase your changes on the BucketingSink that comes with the Flink version you are using * use the same super completely ugly hack I had to deploy as described here:

[jira] [Commented] (KAFKA-960) Upgrade Metrics to 3.x

2018-08-20 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16585713#comment-16585713 ] Erik van Oosten commented on KAFKA-960: --- Metrics 4.x was released not so long ago. The core

[jira] [Commented] (FLINK-5633) ClassCastException: X cannot be cast to X when re-submitting a job.

2017-11-09 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-5633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245474#comment-16245474 ] Erik van Oosten commented on FLINK-5633: bq. Just curious, why are you creating a new reader

[jira] [Commented] (FLINK-5633) ClassCastException: X cannot be cast to X when re-submitting a job.

2017-11-07 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-5633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241701#comment-16241701 ] Erik van Oosten commented on FLINK-5633: [~StephanEwen] We need to process 130K msg/s, I guess

Re: Exception in BucketingSink when cancelling Flink job

2017-10-12 Thread Erik van Oosten
Hi Wangsan, We were struggling with this for many days as well. In the end we found a work around. Well work-around, this for sure qualifies as one of the ugliest hacks I have ever contemplated. Our work-around for Flink immediately interrupting the close, is to continue closing on another

[jira] [Commented] (AVRO-2076) Combine already serialized Avro records to an Avro file

2017-09-15 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/AVRO-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167422#comment-16167422 ] Erik van Oosten commented on AVRO-2076: --- Awesome! Thanks Doug. Somehow I missed that method

[jira] [Resolved] (AVRO-2076) Combine already serialized Avro records to an Avro file

2017-09-15 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/AVRO-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik van Oosten resolved AVRO-2076. --- Resolution: Not A Problem > Combine already serialized Avro records to an Avro f

[jira] [Commented] (FLINK-4796) Add new Sink interface with access to more meta data

2017-09-14 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166443#comment-16166443 ] Erik van Oosten commented on FLINK-4796: I am not sure why this is marked as a duplicate

[jira] [Created] (AVRO-2076) Combine already serialized Avro records to an Avro file

2017-09-14 Thread Erik van Oosten (JIRA)
Erik van Oosten created AVRO-2076: - Summary: Combine already serialized Avro records to an Avro file Key: AVRO-2076 URL: https://issues.apache.org/jira/browse/AVRO-2076 Project: Avro Issue

[jira] [Commented] (FLINK-4796) Add new Sink interface with access to more meta data

2017-09-11 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-4796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16160810#comment-16160810 ] Erik van Oosten commented on FLINK-4796: A workaround is to override {{setRuntimeContext}} (make

[jira] [Commented] (FLINK-1390) java.lang.ClassCastException: X cannot be cast to X

2017-06-21 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-1390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058708#comment-16058708 ] Erik van Oosten commented on FLINK-1390: See https://issues.apache.org/jira/browse/FLINK-5633

[jira] [Commented] (FLINK-5633) ClassCastException: X cannot be cast to X when re-submitting a job.

2017-06-21 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-5633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058706#comment-16058706 ] Erik van Oosten commented on FLINK-5633: In case you need throughput (like we do), the caching

[jira] [Comment Edited] (FLINK-6928) Kafka sink: default topic should not need to exist

2017-06-15 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-6928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050925#comment-16050925 ] Erik van Oosten edited comment on FLINK-6928 at 6/15/17 6:39 PM: - In my

[jira] [Commented] (FLINK-6928) Kafka sink: default topic should not need to exist

2017-06-15 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-6928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050925#comment-16050925 ] Erik van Oosten commented on FLINK-6928: In my ideal world method {{getTargetTopic}} would

[jira] [Updated] (FLINK-6928) Kafka sink: default topic should not need to exist

2017-06-15 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-6928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik van Oosten updated FLINK-6928: --- Description: When using a Kafka sink, the defaultTopic needs to exist even when it is never

[jira] [Updated] (FLINK-6928) Kafka sink: default topic should not need to exist

2017-06-15 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-6928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik van Oosten updated FLINK-6928: --- Description: When using a Kafka sink, the defaultTopic needs to exist even when it is never

[jira] [Updated] (FLINK-6928) Kafka sink: default topic should not need to exist

2017-06-15 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-6928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik van Oosten updated FLINK-6928: --- Summary: Kafka sink: default topic should not need to exist (was: Kafka source: default

[jira] [Updated] (FLINK-6928) Kafka sink: default topic should not need to exist

2017-06-15 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-6928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik van Oosten updated FLINK-6928: --- Description: When using a Kafka sink, the defaultTopic needs to exist even when it is never

[jira] [Updated] (FLINK-6928) Kafka source: default topic should not need to exist

2017-06-15 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/FLINK-6928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik van Oosten updated FLINK-6928: --- Summary: Kafka source: default topic should not need to exist (was: Kafka source: default

[jira] [Created] (FLINK-6928) Kafka source: default topic needs to exist

2017-06-15 Thread Erik van Oosten (JIRA)
Erik van Oosten created FLINK-6928: -- Summary: Kafka source: default topic needs to exist Key: FLINK-6928 URL: https://issues.apache.org/jira/browse/FLINK-6928 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-6928) Kafka source: default topic needs to exist

2017-06-15 Thread Erik van Oosten (JIRA)
Erik van Oosten created FLINK-6928: -- Summary: Kafka source: default topic needs to exist Key: FLINK-6928 URL: https://issues.apache.org/jira/browse/FLINK-6928 Project: Flink Issue Type: Bug

[jira] [Commented] (AVRO-2022) IDL does not allow `schema` as identifier

2017-05-23 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/AVRO-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021085#comment-16021085 ] Erik van Oosten commented on AVRO-2022: --- After working with this change for some time we decided

[jira] [Resolved] (AVRO-2022) IDL does not allow `schema` as identifier

2017-05-23 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/AVRO-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik van Oosten resolved AVRO-2022. --- Resolution: Invalid > IDL does not allow `schema` as identif

[jira] [Updated] (AVRO-2022) IDL does not allow `schema` as identifier

2017-04-11 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/AVRO-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik van Oosten updated AVRO-2022: -- Description: The keyword {{schema}} is now allowed as escaped identifier in IDL. E.g

[jira] [Created] (AVRO-2022) IDL does not allow `schema` as identifier

2017-04-11 Thread Erik van Oosten (JIRA)
Erik van Oosten created AVRO-2022: - Summary: IDL does not allow `schema` as identifier Key: AVRO-2022 URL: https://issues.apache.org/jira/browse/AVRO-2022 Project: Avro Issue Type: Bug

[jira] [Issue Comment Deleted] (THRIFT-3867) Specify BinaryProtocol and CompactProtocol

2016-06-29 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/THRIFT-3867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik van Oosten updated THRIFT-3867: Comment: was deleted (was: Pull request in https://github.com/apache/thrift/pull/1036

[jira] [Commented] (THRIFT-3867) Specify BinaryProtocol and CompactProtocol

2016-06-29 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/THRIFT-3867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15355043#comment-15355043 ] Erik van Oosten commented on THRIFT-3867: - Pull request in https://github.com/apache/thrift/pull

[jira] [Created] (SPARK-6878) Sum on empty RDD fails with exception

2015-04-13 Thread Erik van Oosten (JIRA)
Erik van Oosten created SPARK-6878: -- Summary: Sum on empty RDD fails with exception Key: SPARK-6878 URL: https://issues.apache.org/jira/browse/SPARK-6878 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-6878) Sum on empty RDD fails with exception

2015-04-13 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14492302#comment-14492302 ] Erik van Oosten commented on SPARK-6878: Ah, yes. I now see that fold also first

[jira] [Commented] (SPARK-6878) Sum on empty RDD fails with exception

2015-04-13 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14492336#comment-14492336 ] Erik van Oosten commented on SPARK-6878: Pull request: https://github.com/apache

[jira] [Commented] (SPARK-6878) Sum on empty RDD fails with exception

2015-04-13 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14492282#comment-14492282 ] Erik van Oosten commented on SPARK-6878: The answer is only defined because

[jira] [Updated] (SPARK-6878) Sum on empty RDD fails with exception

2015-04-13 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik van Oosten updated SPARK-6878: --- Flags: Patch Sum on empty RDD fails with exception

[Bug 1388955] Re: POODLE vulnerability because SSL3 can not be disabled

2015-01-03 Thread Erik van Oosten
The problem has not gone away. I guess this package could use a new maintainer. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1388955 Title: POODLE vulnerability because SSL3 can not be disabled

[jira] [Commented] (KAFKA-960) Upgrade Metrics to 3.x

2014-11-03 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14194990#comment-14194990 ] Erik van Oosten commented on KAFKA-960: --- If 2.20 and 2.1.5 are indeed binary

[Bug 1388955] [NEW] POODLE vulnerability because SSL3 can not be disabled

2014-11-03 Thread Erik van Oosten
Public bug reported: Please upgrade pound package to the latest mainstream release (2.7d) so that SSL3 can be disabled. SSL3 needs to be disabled because of the POODLE vulnerability. More details around other options on http://www.apsis.ch/pound/pound_list/archive/2014/2014-10/1414097953000

Re: MBeans, dashes, underscores, and KAFKA-1481

2014-11-02 Thread Erik van Oosten
? So, instead of kafka.server:type=BrokerTopicMetrics,name=topic-1-BytesInPerSec we will have kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=topic-1 Thanks, Jun -- Erik van Oosten http://www.day-to-day-stuff.blogspot.com/

Re: How to produce and consume events in 2 DCs?

2014-10-22 Thread Erik van Oosten
schreef op 22-10-14 om 01:48: I think it doesn't have to be two more clusters. can be just two more topics. MirrorMaker can copy from source topics in both regions into one aggregate topic. On Tue, Oct 21, 2014 at 1:54 AM, Erik van oosten e.vanoos...@grons.nl.invalid wrote: Thanks Neha

Re: How to produce and consume events in 2 DCs?

2014-10-21 Thread Erik van oosten
. The mirror maker copies data from both the DC's local clusters into the aggregate clusters. So if you want access to a topic with data from both DC's, you subscribe to the aggregate cluster. Thanks, Neha On Mon, Oct 20, 2014 at 7:07 AM, Erik van oosten e.vanoos...@grons.nl.invalid wrote: Hi

Re: refactoring ZK so it is plugable, would this make sense?

2014-10-20 Thread Erik van oosten
You can run with a single node zookeeper cluster also. See http://zookeeper.apache.org/doc/r3.3.4/zookeeperStarted.html#sc_InstallingSingleMode Cheers, Erik. Op 9 okt. 2014, om 22:52 heeft S Ahmed sahmed1...@gmail.com het volgende geschreven: I want kafka features (w/o the redundancy)

How to produce and consume events in 2 DCs?

2014-10-20 Thread Erik van oosten
mirroring to a topic with another name. Is there another tool we could use? Or, is there another approach for producing and consuming from 2 DCs? Kind regards, Erik. — Erik van Oosten http://www.day-to-day-stuff.blogspot.nl/

Spark on secure HDFS

2014-10-15 Thread Erik van oosten
Hi, We really would like to use Spark but we can’t because we have a secure HDFS environment (Cloudera). I understood https://issues.apache.org/jira/browse/SPARK-2541 contains a patch. Can one of the committers please take a look? Thanks! Erik. — Erik van Oosten http://www.day-to-day

subscribe

2014-09-11 Thread Erik van oosten
- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org

Re: running on scala 2.11

2014-07-05 Thread Erik van Oosten
The message is *NoClassDefFoundError: scala/collection/GenTraversableOnce$class* Thanks Laszlo -- -- Guozhang -- Erik van Oosten http://www.day-to-day-stuff.blogspot.com/

  1   2   3   4   5   6   7   8   9   10   >