[jira] [Created] (KAFKA-12859) Kafka Server Repeated Responses

[Lea (Jira)]

Lea created KAFKA-12859:
---

 Summary: Kafka Server Repeated Responses
 Key: KAFKA-12859
 URL: https://issues.apache.org/jira/browse/KAFKA-12859
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 1.1.0
Reporter: Lea


When the client producer data, the Kafka server repeatedly responds with the 
same correlationId. As a result, all requests subsequent to the socket fail to 
be sent.

异常如下
{code:java}
//
java.lang.IllegalStateException: Correlation id for response (30205) does not 
match request (30211), request header: RequestHeader(apiKey=PRODUCE, 
apiVersion=5, clientId=producer-1, correlationId=30211)
at 
org.apache.kafka.clients.NetworkClient.correlate(NetworkClient.java:943) 
~[kafka-clients-2.4.0.jar!/:na]
at 
org.apache.kafka.clients.NetworkClient.parseStructMaybeUpdateThrottleTimeMetrics(NetworkClient.java:713)
 ~[kafka-clients-2.4.0.jar!/:na]
at 
org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:836)
 ~[kafka-clients-2.4.0.jar!/:na]
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:549) 
~[kafka-clients-2.4.0.jar!/:na]
at 
org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:335) 
~[kafka-clients-2.4.0.jar!/:na]
at 
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:244) 
~[kafka-clients-2.4.0.jar!/:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_141]


{code}
 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12858) dynamically update the ssl certificates of kafka connect worker without restarting connect process.

[kaushik srinivas (Jira)]

kaushik srinivas created KAFKA-12858:


 Summary: dynamically update the ssl certificates of kafka connect 
worker without restarting connect process.
 Key: KAFKA-12858
 URL: https://issues.apache.org/jira/browse/KAFKA-12858
 Project: Kafka
  Issue Type: Improvement
Reporter: kaushik srinivas


Hi,

 

We are trying to update the ssl certificates of kafka connect worker which is 
due for expiry. Is there any way to dynamically update the ssl certificate of 
connet worker as it is possible in kafka using kafka-configs.sh script ?

If not, what is the recommended way to update the ssl certificates of kafka 
connect worker without disrupting the existing traffic ?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Puzzle about InterBrokerSendThread.scala

[李铁根]

Hi team,
   
I'm leaning the source code recently，the method logic seems wants to collect 
all timeout request, but it acutually just remove the first expire request of 
each broker, is it a deliberate design?  will it lose some callbacks ?  Looking 
forward to your reply！

Source file:  kafka.common.InterBrokerSendThread.scala
def removeAllTimedOut(now: Long): Collection[ClientRequest] = {
  val expiredRequests = new ArrayList[ClientRequest]
  for (requests <- unsent.values.asScala) {
val requestIterator = requests.iterator
var foundExpiredRequest = false 
while (requestIterator.hasNext && !foundExpiredRequest) {
  val request = requestIterator.next
  val elapsedMs = Math.max(0, now - request.createdTimeMs)
  if (elapsedMs > request.requestTimeoutMs) {
expiredRequests.add(request)
requestIterator.remove()
foundExpiredRequest = true
  }
}
  }
  expiredRequests
}



Best
tiegen

   

[jira] [Created] (KAFKA-12857) Using Connect Sink with CooperativeStickyAssignor results in commit offsets failure

[Oliver Hsu (Jira)]

Oliver Hsu created KAFKA-12857:
--

 Summary: Using Connect Sink with CooperativeStickyAssignor results 
in commit offsets failure
 Key: KAFKA-12857
 URL: https://issues.apache.org/jira/browse/KAFKA-12857
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Affects Versions: 2.7.1
 Environment: Linux
Reporter: Oliver Hsu


We are attempting to use a Kafka Connect Sink Connector with 
{{CooperativeStickyAssignor}} assignment strategy.  When we use 
{{CooperativeStickyAssignor}} offset commits sometimes fail with 

{{[2021-05-26 22:03:36,435] WARN WorkerSinkTask\{id=sink-connector-7} Ignoring 
invalid task provided offset mytopic-0/OffsetAndMetadata\{offset=16305575, 
leaderEpoch=null, metadata=''} – partition not assigned, assignment=[mytopic-0] 
(org.apache.kafka.connect.runtime.WorkerSinkTask:434)}}

Note that the partition that the warning message says is not assigned matches 
the partition assignment.

*Config changes*

{{consumer.partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor}}

*Cooperative vs Eager Assignment Strategy background*
 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-429%3A+Kafka+Consumer+Incremental+Rebalance+Protocol#KIP429:KafkaConsumerIncrementalRebalanceProtocol-ConsumerRebalanceListenerandConsumerPartitionAssignorSemantics]

With eager assignment:
{quote}Listener#onPartitionsAssigned: called on the full set of assigned 
partitions (may have overlap with the partitions passed to #onPartitionsRevoked
{quote}
With cooperative assignment:
{quote}Listener#onPartitionsAssigned: called on the subset of assigned 
partitions that were not previously owned before this rebalance. There should 
be no overlap with the revoked partitions (if any). This will always be called, 
even if there are no new partitions being assigned to a given member.
{quote}
This means with cooperative assignment, `onPartitionsAssigned` may be called 
with an empty collection.

However, the 
[WorkerSinkTask.HandleRebalance|https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L669-L680]
 class makes the assumption that `onPartitionsAssigned` is called with the full 
set of assigned partitions which was true for eager but not coooperative.
{code:java|title=WorkerSinkTask.HandleRebalance.java|borderStyle=solid}
public void onPartitionsAssigned(Collection partitions) 
{
log.debug("{} Partitions assigned {}", WorkerSinkTask.this, 
partitions);
lastCommittedOffsets = new HashMap<>();
currentOffsets = new HashMap<>();
for (TopicPartition tp : partitions) {
long pos = consumer.position(tp);
lastCommittedOffsets.put(tp, new OffsetAndMetadata(pos));
currentOffsets.put(tp, new OffsetAndMetadata(pos));
log.debug("{} Assigned topic partition {} with offset {}", 
WorkerSinkTask.this, tp, pos);
}
{code}
The {{onPartitionsAssigned}} creates a new empty {{HashMap}} and puts the 
offsets of the {{partitions}} in that {{HashMap}}.

In the logs we see
{{[2021-05-26 22:02:09,785] DEBUG WorkerSinkTask}}

{{{id=sink-connector-7} Partitions assigned 
[scalyr-logstaging-v1-21050_1129447869326033792-AFulBCBPQjUB-0] 
(org.apache.kafka.connect.runtime.WorkerSinkTask:677)}}
{{ [2021-05-26 22:02:13,063] DEBUG WorkerSinkTask\{id=sink-connector-7}}}

{{Partitions assigned [] (org.apache.kafka.connect.runtime.WorkerSinkTask:677)}}
{{ [2021-05-26 22:02:16,074] DEBUG WorkerSinkTask}}

{{{id=sink-connector-7} }}{{Partitions assigned [] 
(org.apache.kafka.connect.runtime.WorkerSinkTask:677)}}



These logs show that the {{CooperativeStickyAssignor}} calls 
{{onPartitionsAssigned}} first with the partition assigned to it followed by 
additional calls with an empty {{partitions}} collection.

The {{HandleRebalance.onPartitionsAssigned}} code will then initialize 
{{lastCommittedOffsets}} to an empty {{HashMap}}.

Inside 
[WorkerSinkTask.commitOffsets|https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L415-L419],
 the current {{committableOffsets}} are based on the {{lastCommittedOffsets}}, 
which is an empty {{HashMap}}:
{code:java|title=WorkerSinkTask.java|borderStyle=solid}
private void commitOffsets(long now, boolean closing) {
...
final Map commitableOffsets = new 
HashMap<>(lastCommittedOffsets);
for (Map.Entry 
taskProvidedOffsetEntry : taskProvidedOffsets.entrySet()) {
final TopicPartition partition = taskProvidedOffsetEntry.getKey();
final OffsetAndMetadata taskProvidedOffset = 
taskProvidedOffsetEntry.getValue();
if (commitableOffsets.containsKey(partition)) {
long taskOffset = taskProvidedOffset.offset();
  

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #172

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

[Gunnar Morling]

Chris,

One follow-up question after thinking some more about this; is there any
limit in terms of duration or size of in-flight, connector-controlled
transactions? In case of Debezium for instance, there may be cases where we
tail the TX log from an upstream source database, not knowing whether the
events we receive belong to a committed or aborted transaction. Would it be
valid to emit all these events via a transactional task, and in case we
receive a ROLLBACK event eventually, to abort the pending Kafka
transaction? Such source transactions could be running for a long time
potentially, e.g. hours or days (at least in theory). Or would this sort of
usage not be considered a reasonable one?

Thanks,

--Gunnar


Am Do., 27. Mai 2021 um 23:15 Uhr schrieb Gunnar Morling <
gunnar.morl...@googlemail.com>:

> Chris, all,
>
> I've just read KIP-618, and let me congratulate you first of all for this
> impressive piece of work! Here's a few small suggestions and questions I
> had while reading:
>
> * TransactionContext: What's the use case for the methods accepting a
> source record (commitTransaction(SourceRecord
> record), abortTransaction(SourceRecord record))?
> * SourceTaskContext: Typo in "when the sink connector is deployed" ->
> source task
> * SourceTaskContext: Instead of guarding against NSME, is there a way for
> a connector to query the KC version and thus derive its capabilities? Going
> forward, a generic API for querying capabilities could be nice, so a
> connector can query for capabilities of the runtime in a safe and
> compatible way.
> * SourceConnector: exactlyOnceSupport() -> false return value doesn't match
> * SourceConnector: Would it make sense to merge the two methods perhaps
> and return one enum of { SUPPORTED, NOT_SUPPORTED,
> SUPPORTED_WITH_BOUNDARIES }? Or, alternatively return an enum
> from canDefineTransactionBoundaries(), too; even if it only has two values
> now, that'd allow for extension in the future
>
> And one general question: in Debezium, we have some connectors that
> produce records "out-of-bands" to a schema history topic via their own
> custom producer. Is there any way envisionable where such a producer would
> participate in the transaction managed by the KC runtime environment?
>
> Thanks a lot,
>
> --Gunnar
>
>
> Am Mo., 24. Mai 2021 um 18:45 Uhr schrieb Chris Egerton
> :
>
>> Hi all,
>>
>> Wanted to note here that I've updated the KIP document to include the
>> changes discussed recently. They're mostly located in the "Public
>> Interfaces" section. I suspect discussion hasn't concluded yet and there
>> will probably be a few more changes to come, but wanted to take the
>> opportunity to provide a snapshot of what the current design looks like.
>>
>> Cheers,
>>
>> Chris
>>
>> On Fri, May 21, 2021 at 4:32 PM Chris Egerton 
>> wrote:
>>
>> > Hi Tom,
>> >
>> > Wow, I was way off base! I was thinking that the intent of the fencible
>> > producer was to employ it by default with 3.0, as opposed to only after
>> the
>> > worker-level
>> > "exactly.once.source.enabled" property was flipped on. You are correct
>> > that with the case you were actually describing, there would be no
>> > heightened ACL requirements, and that it would leave room in the future
>> for
>> > exactly-once to be disabled on a per-connector basis (as long as all the
>> > workers in the cluster already had "exactly.once.source.enabled" set to
>> > "true") with no worries about breaking changes.
>> >
>> > I agree that this is something for another KIP; even if we could squeeze
>> > it in in time for this release, it might be a bit much for new users to
>> > take in all at once. But I can add it to the doc as "future work" since
>> > it's a promising idea that could prove valuable to someone who might
>> need
>> > per-connector granularity in the future.
>> >
>> > Thanks for clearing things up; in retrospect your comments make a lot
>> more
>> > sense now, and I hope I've sufficiently addressed them by now.
>> >
>> > PSA for you and everyone else--I plan on updating the doc next week with
>> > the new APIs for connector-defined transaction boundaries,
>> > user-configurable transaction boundaries (i.e., poll vs. interval vs.
>> > connectors), and preflight checks for exactly-once validation (required
>> vs.
>> > requested).
>> >
>> > Cheers,
>> >
>> > Chris
>> >
>> > On Fri, May 21, 2021 at 7:14 AM Tom Bentley 
>> wrote:
>> >
>> >> Hi Chris,
>> >>
>> >> Thanks for continuing to entertain some of these ideas.
>> >>
>> >> On Fri, May 14, 2021 at 5:06 PM Chris Egerton
>> > >> >
>> >> wrote:
>> >>
>> >> > [...]
>> >> >
>> >> That's true, but we do go from three static ACLs (write/describe on a
>> >> fixed
>> >> > transactional ID, and idempotent write on a fixed cluster) to a
>> dynamic
>> >> > collection of ACLs.
>> >> >
>> >>
>> >> I'm not quite sure I follow, maybe I've lost track. To be clear, I was
>> >> suggesting the use of a 'fencing producer' only in clusters with
>> >> exactly.once.source.ena

Jenkins build is back to normal : Kafka » kafka-2.7-jdk8 #157

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

[Gunnar Morling]

Chris, all,

I've just read KIP-618, and let me congratulate you first of all for this
impressive piece of work! Here's a few small suggestions and questions I
had while reading:

* TransactionContext: What's the use case for the methods accepting a
source record (commitTransaction(SourceRecord
record), abortTransaction(SourceRecord record))?
* SourceTaskContext: Typo in "when the sink connector is deployed" ->
source task
* SourceTaskContext: Instead of guarding against NSME, is there a way for a
connector to query the KC version and thus derive its capabilities? Going
forward, a generic API for querying capabilities could be nice, so a
connector can query for capabilities of the runtime in a safe and
compatible way.
* SourceConnector: exactlyOnceSupport() -> false return value doesn't match
* SourceConnector: Would it make sense to merge the two methods perhaps and
return one enum of { SUPPORTED, NOT_SUPPORTED, SUPPORTED_WITH_BOUNDARIES }?
Or, alternatively return an enum from canDefineTransactionBoundaries(),
too; even if it only has two values now, that'd allow for extension in the
future

And one general question: in Debezium, we have some connectors that produce
records "out-of-bands" to a schema history topic via their own custom
producer. Is there any way envisionable where such a producer would
participate in the transaction managed by the KC runtime environment?

Thanks a lot,

--Gunnar


Am Mo., 24. Mai 2021 um 18:45 Uhr schrieb Chris Egerton
:

> Hi all,
>
> Wanted to note here that I've updated the KIP document to include the
> changes discussed recently. They're mostly located in the "Public
> Interfaces" section. I suspect discussion hasn't concluded yet and there
> will probably be a few more changes to come, but wanted to take the
> opportunity to provide a snapshot of what the current design looks like.
>
> Cheers,
>
> Chris
>
> On Fri, May 21, 2021 at 4:32 PM Chris Egerton  wrote:
>
> > Hi Tom,
> >
> > Wow, I was way off base! I was thinking that the intent of the fencible
> > producer was to employ it by default with 3.0, as opposed to only after
> the
> > worker-level
> > "exactly.once.source.enabled" property was flipped on. You are correct
> > that with the case you were actually describing, there would be no
> > heightened ACL requirements, and that it would leave room in the future
> for
> > exactly-once to be disabled on a per-connector basis (as long as all the
> > workers in the cluster already had "exactly.once.source.enabled" set to
> > "true") with no worries about breaking changes.
> >
> > I agree that this is something for another KIP; even if we could squeeze
> > it in in time for this release, it might be a bit much for new users to
> > take in all at once. But I can add it to the doc as "future work" since
> > it's a promising idea that could prove valuable to someone who might need
> > per-connector granularity in the future.
> >
> > Thanks for clearing things up; in retrospect your comments make a lot
> more
> > sense now, and I hope I've sufficiently addressed them by now.
> >
> > PSA for you and everyone else--I plan on updating the doc next week with
> > the new APIs for connector-defined transaction boundaries,
> > user-configurable transaction boundaries (i.e., poll vs. interval vs.
> > connectors), and preflight checks for exactly-once validation (required
> vs.
> > requested).
> >
> > Cheers,
> >
> > Chris
> >
> > On Fri, May 21, 2021 at 7:14 AM Tom Bentley  wrote:
> >
> >> Hi Chris,
> >>
> >> Thanks for continuing to entertain some of these ideas.
> >>
> >> On Fri, May 14, 2021 at 5:06 PM Chris Egerton
>  >> >
> >> wrote:
> >>
> >> > [...]
> >> >
> >> That's true, but we do go from three static ACLs (write/describe on a
> >> fixed
> >> > transactional ID, and idempotent write on a fixed cluster) to a
> dynamic
> >> > collection of ACLs.
> >> >
> >>
> >> I'm not quite sure I follow, maybe I've lost track. To be clear, I was
> >> suggesting the use of a 'fencing producer' only in clusters with
> >> exactly.once.source.enabled=true where I imagined the key difference
> >> between the exactly once and fencing cases was how the producer was
> >> configured/used (transactional vs this new fencing semantic). I think
> the
> >> ACL requirements for connector producer principals would therefore be
> the
> >> same as currently described in the KIP. The same is true for the worker
> >> principals (which is the only breaking change you give in the KIP). So I
> >> don't think the fencing idea changes the backwards compatibility story
> >> that's already in the KIP, just allows a safe per-connector
> >> exactly.once=disabled option to be supported (with required as requested
> >> as
> >> we already discussed).
> >>
> >> But I'm wondering whether I've overlooked something.
> >>
> >> Ultimately I think it may behoove us to err on the side of reducing the
> >> > breaking changes here for now and saving them for 4.0 (or some later
> >> major
> >> > release), but would be interes

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #171

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 341892 lines...]
[2021-05-27T20:48:18.208Z] > Task :clients:jar UP-TO-DATE
[2021-05-27T20:48:18.208Z] > Task :server-common:compileJava UP-TO-DATE
[2021-05-27T20:48:18.208Z] > Task :storage:api:compileJava UP-TO-DATE
[2021-05-27T20:48:18.208Z] > Task :connect:api:compileJava UP-TO-DATE
[2021-05-27T20:48:18.208Z] > Task :streams:compileJava UP-TO-DATE
[2021-05-27T20:48:18.208Z] > Task :streams:classes UP-TO-DATE
[2021-05-27T20:48:18.208Z] > Task :connect:api:classes UP-TO-DATE
[2021-05-27T20:48:18.208Z] > Task :raft:compileJava UP-TO-DATE
[2021-05-27T20:48:18.208Z] > Task :connect:json:compileJava UP-TO-DATE
[2021-05-27T20:48:18.208Z] > Task :connect:json:classes UP-TO-DATE
[2021-05-27T20:48:18.208Z] > Task :connect:json:javadoc SKIPPED
[2021-05-27T20:48:18.208Z] > Task :storage:compileJava UP-TO-DATE
[2021-05-27T20:48:18.208Z] > Task :connect:json:javadocJar
[2021-05-27T20:48:18.208Z] > Task :clients:compileTestJava UP-TO-DATE
[2021-05-27T20:48:18.208Z] > Task :clients:testClasses UP-TO-DATE
[2021-05-27T20:48:18.208Z] > Task :streams:test-utils:compileJava UP-TO-DATE
[2021-05-27T20:48:18.208Z] > Task :metadata:compileJava UP-TO-DATE
[2021-05-27T20:48:18.208Z] > Task :core:compileJava NO-SOURCE
[2021-05-27T20:48:18.208Z] > Task :connect:json:compileTestJava UP-TO-DATE
[2021-05-27T20:48:18.208Z] > Task :connect:json:testClasses UP-TO-DATE
[2021-05-27T20:48:18.208Z] > Task :connect:json:testJar
[2021-05-27T20:48:18.208Z] > Task :connect:json:testSrcJar
[2021-05-27T20:48:18.208Z] > Task 
:clients:generateMetadataFileForMavenJavaPublication
[2021-05-27T20:48:18.208Z] > Task :streams:copyDependantLibs
[2021-05-27T20:48:19.148Z] > Task :streams:jar UP-TO-DATE
[2021-05-27T20:48:19.149Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2021-05-27T20:48:23.945Z] > Task :connect:api:javadoc
[2021-05-27T20:48:23.945Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2021-05-27T20:48:23.945Z] > Task :connect:api:jar UP-TO-DATE
[2021-05-27T20:48:23.945Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2021-05-27T20:48:23.945Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2021-05-27T20:48:23.945Z] > Task :connect:json:jar UP-TO-DATE
[2021-05-27T20:48:23.945Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2021-05-27T20:48:23.945Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2021-05-27T20:48:23.945Z] > Task :connect:json:publishToMavenLocal
[2021-05-27T20:48:23.945Z] > Task :connect:api:javadocJar
[2021-05-27T20:48:23.945Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2021-05-27T20:48:23.945Z] > Task :connect:api:testClasses UP-TO-DATE
[2021-05-27T20:48:23.945Z] > Task :connect:api:testJar
[2021-05-27T20:48:23.945Z] > Task :connect:api:testSrcJar
[2021-05-27T20:48:23.945Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2021-05-27T20:48:23.945Z] > Task :connect:api:publishToMavenLocal
[2021-05-27T20:48:26.587Z] > Task :streams:javadoc
[2021-05-27T20:48:26.587Z] > Task :streams:javadocJar
[2021-05-27T20:48:29.270Z] > Task :clients:javadoc
[2021-05-27T20:48:30.211Z] > Task :clients:javadocJar
[2021-05-27T20:48:31.642Z] > Task :clients:testJar
[2021-05-27T20:48:31.642Z] > Task :clients:testSrcJar
[2021-05-27T20:48:31.642Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2021-05-27T20:48:31.642Z] > Task :clients:publishToMavenLocal
[2021-05-27T20:48:48.234Z] > Task :core:compileScala
[2021-05-27T20:50:52.826Z] > Task :core:classes
[2021-05-27T20:50:52.826Z] > Task :core:compileTestJava NO-SOURCE
[2021-05-27T20:51:12.018Z] > Task :core:compileTestScala
[2021-05-27T20:52:45.704Z] > Task :core:testClasses
[2021-05-27T20:53:16.455Z] > Task :streams:compileTestJava
[2021-05-27T20:55:37.100Z] > Task :streams:testClasses
[2021-05-27T20:55:37.100Z] > Task :streams:testJar
[2021-05-27T20:55:37.100Z] > Task :streams:testSrcJar
[2021-05-27T20:55:37.100Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2021-05-27T20:55:37.100Z] > Task :streams:publishToMavenLocal
[2021-05-27T20:55:37.100Z] 
[2021-05-27T20:55:37.100Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 8.0.
[2021-05-27T20:55:37.100Z] Use '--warning-mode all' to show the individual 
deprecation warnings.
[2021-05-27T20:55:37.100Z] See 
https://docs.gradle.org/7.0.2/userguide/command_line_interface.html#sec:command_line_warnings
[2021-05-27T20:55:37.100Z] 
[2021-05-27T20:55:37.100Z] Execution optimizations have been disabled for 2 
invalid unit(s) of work during this build to ensure correctness.
[2021-05-27T20:55:37.100Z] Please consult deprecation warnings for more details.
[2021-05-27T20:55:37.100Z] 
[2021-05-27T20:55:37.100Z] BUILD SUCCESSFUL in 7m 32s
[2021-05-27T20:55:37.100Z] 71 actionable tasks: 37 executed, 34 up-to-date
[Pipeline] sh
[2021-05-27T20:55:40.027Z] + gr

Re: [DISCUSS] KIP-744: Migrate TaskMetadata to interface with internal implementation

[Josep Prat]

Hi Sophie,

Thanks for the feedback, I'll update the KIP tomorrow with your feedback.
They are all good points, and you are right, my phrasing could be
misleading.


Best,

On Thu, May 27, 2021 at 10:02 PM Sophie Blee-Goldman
 wrote:

> Thanks for the KIP! I'm on board with the overall proposal, just a few
> comments:
>
> 1) The motivation section says
>
> TaskMetadata should have never been a class available for the general
> > public, but more of an internal class
>
>
> which is a bit misleading as it seems to imply that TaskMetadata itself was
> never meant to be part of the public API
> at all. It might be better to phrase this as "TaskMetadata was never
> intended to be a public class that a user might
> need to instantiate, but rather an API for exposing metadata which is
> better served as an interface" --- or something
> to that effect.
>
> 2) You touch on this in a later section, but it would be good to call out
> directly in the *Public Interfaces* section that
> you are proposing to remove the `public TaskId getTaskId()` method that we
> added in KIP-740. Also I just want to
> note that to do so will require getting this KIP into 3.0, otherwise we'll
> need to go through a deprecation cycle for
> that API. I don't anticipate this being a problem as KIP freeze is still
> two weeks away, but it would be good to clarify.
>
> 3) nit: we should put the new internal implementation class under
> the org.apache.kafka.streams.processor.internals
> package instead of under org.apache.kafka.streams.internals. But this is an
> implementation detail and as such
> doesn't need to be covered by the KIP in the first place.
>
> - Sophie
>
> On Thu, May 27, 2021 at 1:55 AM Josep Prat 
> wrote:
>
> > I deliberately picked the most conservative approach of creating a new
> > Interface, instead of transforming the current class into an interface.
> > Feedback is most welcome!
> >
> > Best,
> >
> > On Thu, May 27, 2021 at 10:26 AM Josep Prat  wrote:
> >
> > > Hi there,
> > > I would like to propose KIP-744, to introduce TaskMetadata as an
> > > interface, to keep the its implementation as internal use.
> > > This KIP can be seen as a spin-off of KIP-740.
> > >
> > > https://cwiki.apache.org/confluence/x/XIrOCg
> > >
> > > Best,
> > > --
> > >
> > > Josep Prat
> > >
> > > *Aiven Deutschland GmbH*
> > >
> > > Immanuelkirchstraße 26, 10405 Berlin
> > >
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >
> > > *m:* +491715557497
> > >
> > > *w:* aiven.io
> > >
> > > *e:* josep.p...@aiven.io
> > >
> >
> >
> > --
> >
> > Josep Prat
> >
> > *Aiven Deutschland GmbH*
> >
> > Immanuelkirchstraße 26, 10405 Berlin
> >
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >
> > *m:* +491715557497
> >
> > *w:* aiven.io
> >
> > *e:* josep.p...@aiven.io
> >
>


-- 

Josep Prat

*Aiven Deutschland GmbH*

Immanuelkirchstraße 26, 10405 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

*m:* +491715557497

*w:* aiven.io

*e:* josep.p...@aiven.io

Re: [DISCUSS] KIP-744: Migrate TaskMetadata to interface with internal implementation

[Sophie Blee-Goldman]

Thanks for the KIP! I'm on board with the overall proposal, just a few
comments:

1) The motivation section says

TaskMetadata should have never been a class available for the general
> public, but more of an internal class


which is a bit misleading as it seems to imply that TaskMetadata itself was
never meant to be part of the public API
at all. It might be better to phrase this as "TaskMetadata was never
intended to be a public class that a user might
need to instantiate, but rather an API for exposing metadata which is
better served as an interface" --- or something
to that effect.

2) You touch on this in a later section, but it would be good to call out
directly in the *Public Interfaces* section that
you are proposing to remove the `public TaskId getTaskId()` method that we
added in KIP-740. Also I just want to
note that to do so will require getting this KIP into 3.0, otherwise we'll
need to go through a deprecation cycle for
that API. I don't anticipate this being a problem as KIP freeze is still
two weeks away, but it would be good to clarify.

3) nit: we should put the new internal implementation class under
the org.apache.kafka.streams.processor.internals
package instead of under org.apache.kafka.streams.internals. But this is an
implementation detail and as such
doesn't need to be covered by the KIP in the first place.

- Sophie

On Thu, May 27, 2021 at 1:55 AM Josep Prat 
wrote:

> I deliberately picked the most conservative approach of creating a new
> Interface, instead of transforming the current class into an interface.
> Feedback is most welcome!
>
> Best,
>
> On Thu, May 27, 2021 at 10:26 AM Josep Prat  wrote:
>
> > Hi there,
> > I would like to propose KIP-744, to introduce TaskMetadata as an
> > interface, to keep the its implementation as internal use.
> > This KIP can be seen as a spin-off of KIP-740.
> >
> > https://cwiki.apache.org/confluence/x/XIrOCg
> >
> > Best,
> > --
> >
> > Josep Prat
> >
> > *Aiven Deutschland GmbH*
> >
> > Immanuelkirchstraße 26, 10405 Berlin
> >
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >
> > *m:* +491715557497
> >
> > *w:* aiven.io
> >
> > *e:* josep.p...@aiven.io
> >
>
>
> --
>
> Josep Prat
>
> *Aiven Deutschland GmbH*
>
> Immanuelkirchstraße 26, 10405 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> *m:* +491715557497
>
> *w:* aiven.io
>
> *e:* josep.p...@aiven.io
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » 2.8 #29

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 463539 lines...]
[2021-05-27T18:55:44.124Z] AuthorizerIntegrationTest > testCommitWithNoAccess() 
STARTED
[2021-05-27T18:55:46.100Z] 
[2021-05-27T18:55:46.100Z] AuthorizerIntegrationTest > testCommitWithNoAccess() 
PASSED
[2021-05-27T18:55:46.100Z] 
[2021-05-27T18:55:46.100Z] AuthorizerIntegrationTest > 
testUnauthorizedCreatePartitions() STARTED
[2021-05-27T18:55:48.328Z] 
[2021-05-27T18:55:48.328Z] AuthorizerIntegrationTest > 
testUnauthorizedCreatePartitions() PASSED
[2021-05-27T18:55:48.328Z] 
[2021-05-27T18:55:48.328Z] AuthorizerIntegrationTest > 
testDeleteGroupOffsetsWithDeleteAclWithoutTopicAcl() STARTED
[2021-05-27T18:55:50.288Z] 
[2021-05-27T18:55:50.288Z] AuthorizerIntegrationTest > 
testDeleteGroupOffsetsWithDeleteAclWithoutTopicAcl() PASSED
[2021-05-27T18:55:50.288Z] 
[2021-05-27T18:55:50.288Z] AuthorizerIntegrationTest > 
testConsumeWithoutTopicDescribeAccess() STARTED
[2021-05-27T18:55:52.361Z] 
[2021-05-27T18:55:52.361Z] AuthorizerIntegrationTest > 
testConsumeWithoutTopicDescribeAccess() PASSED
[2021-05-27T18:55:52.361Z] 
[2021-05-27T18:55:52.361Z] AuthorizerIntegrationTest > testClusterId() STARTED
[2021-05-27T18:55:55.393Z] 
[2021-05-27T18:55:55.393Z] AuthorizerIntegrationTest > testClusterId() PASSED
[2021-05-27T18:55:55.393Z] 
[2021-05-27T18:55:55.393Z] AuthorizerIntegrationTest > 
shouldThrowTransactionalIdAuthorizationExceptionWhenNoTransactionAccessOnEndTransaction()
 STARTED
[2021-05-27T18:55:57.440Z] 
[2021-05-27T18:55:57.440Z] AuthorizerIntegrationTest > 
shouldThrowTransactionalIdAuthorizationExceptionWhenNoTransactionAccessOnEndTransaction()
 PASSED
[2021-05-27T18:55:57.440Z] 
[2021-05-27T18:55:57.440Z] AuthorizerIntegrationTest > 
shouldThrowTransactionalIdAuthorizationExceptionWhenNoTransactionAccessOnSendOffsetsToTxn()
 STARTED
[2021-05-27T18:55:59.578Z] 
[2021-05-27T18:55:59.578Z] AuthorizerIntegrationTest > 
shouldThrowTransactionalIdAuthorizationExceptionWhenNoTransactionAccessOnSendOffsetsToTxn()
 PASSED
[2021-05-27T18:55:59.578Z] 
[2021-05-27T18:55:59.578Z] AuthorizerIntegrationTest > 
testCommitWithNoGroupAccess() STARTED
[2021-05-27T18:56:01.924Z] 
[2021-05-27T18:56:01.924Z] AuthorizerIntegrationTest > 
testCommitWithNoGroupAccess() PASSED
[2021-05-27T18:56:01.924Z] 
[2021-05-27T18:56:01.924Z] AuthorizerIntegrationTest > 
testTransactionalProducerInitTransactionsNoDescribeTransactionalIdAcl() STARTED
[2021-05-27T18:56:03.863Z] 
[2021-05-27T18:56:03.863Z] AuthorizerIntegrationTest > 
testTransactionalProducerInitTransactionsNoDescribeTransactionalIdAcl() PASSED
[2021-05-27T18:56:03.863Z] 
[2021-05-27T18:56:03.863Z] AuthorizerIntegrationTest > 
testAuthorizeByResourceTypeDenyTakesPrecedence() STARTED
[2021-05-27T18:56:05.821Z] 
[2021-05-27T18:56:05.821Z] AuthorizerIntegrationTest > 
testAuthorizeByResourceTypeDenyTakesPrecedence() PASSED
[2021-05-27T18:56:05.821Z] 
[2021-05-27T18:56:05.821Z] AuthorizerIntegrationTest > 
testUnauthorizedDeleteRecordsWithDescribe() STARTED
[2021-05-27T18:56:08.908Z] 
[2021-05-27T18:56:08.908Z] AuthorizerIntegrationTest > 
testUnauthorizedDeleteRecordsWithDescribe() PASSED
[2021-05-27T18:56:08.908Z] 
[2021-05-27T18:56:08.908Z] AuthorizerIntegrationTest > 
testCreateTopicAuthorizationWithClusterCreate() STARTED
[2021-05-27T18:56:10.102Z] 
[2021-05-27T18:56:10.102Z] AuthorizerIntegrationTest > 
testCreateTopicAuthorizationWithClusterCreate() PASSED
[2021-05-27T18:56:10.102Z] 
[2021-05-27T18:56:10.102Z] AuthorizerIntegrationTest > 
testOffsetFetchWithTopicAndGroupRead() STARTED
[2021-05-27T18:56:12.435Z] 
[2021-05-27T18:56:12.435Z] AuthorizerIntegrationTest > 
testOffsetFetchWithTopicAndGroupRead() PASSED
[2021-05-27T18:56:12.435Z] 
[2021-05-27T18:56:12.435Z] AuthorizerIntegrationTest > 
testCommitWithTopicDescribe() STARTED
[2021-05-27T18:56:15.461Z] 
[2021-05-27T18:56:15.461Z] AuthorizerIntegrationTest > 
testCommitWithTopicDescribe() PASSED
[2021-05-27T18:56:15.461Z] 
[2021-05-27T18:56:15.461Z] AuthorizerIntegrationTest > 
testAuthorizationWithTopicExisting() STARTED
[2021-05-27T18:56:25.721Z] 
[2021-05-27T18:56:25.721Z] AuthorizerIntegrationTest > 
testAuthorizationWithTopicExisting() PASSED
[2021-05-27T18:56:25.721Z] 
[2021-05-27T18:56:25.721Z] AuthorizerIntegrationTest > 
testUnauthorizedDeleteRecordsWithoutDescribe() STARTED
[2021-05-27T18:56:27.638Z] 
[2021-05-27T18:56:27.638Z] AuthorizerIntegrationTest > 
testUnauthorizedDeleteRecordsWithoutDescribe() PASSED
[2021-05-27T18:56:27.638Z] 
[2021-05-27T18:56:27.638Z] AuthorizerIntegrationTest > 
testMetadataWithTopicDescribe() STARTED
[2021-05-27T18:56:29.516Z] 
[2021-05-27T18:56:29.516Z] AuthorizerIntegrationTest > 
testMetadataWithTopicDescribe() PASSED
[2021-05-27T18:56:29.516Z] 
[2021-05-27T18:56:29.516Z] AuthorizerIntegrationTest > 
testProduceWithTopicDescribe() STARTED
[2021-05-27T18:56:31.562Z] 
[2021-05-27T18:56:31.562Z] Authoriz

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #170

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 410879 lines...]
[Pipeline] sh
[2021-05-27T18:32:44.105Z] 
[2021-05-27T18:32:44.105Z] PlaintextConsumerTest > testMaxPollRecords() PASSED
[2021-05-27T18:32:44.105Z] 
[2021-05-27T18:32:44.105Z] PlaintextConsumerTest > testAutoOffsetReset() STARTED
[2021-05-27T18:32:45.444Z] + mvn clean install -Dgpg.skip
[2021-05-27T18:32:46.379Z] [INFO] Scanning for projects...
[2021-05-27T18:32:46.380Z] [INFO] 

[2021-05-27T18:32:46.380Z] [INFO] Reactor Build Order:
[2021-05-27T18:32:46.380Z] [INFO] 
[2021-05-27T18:32:46.380Z] [INFO] Kafka Streams :: Quickstart   
 [pom]
[2021-05-27T18:32:46.380Z] [INFO] streams-quickstart-java   
 [maven-archetype]
[2021-05-27T18:32:46.380Z] [INFO] 
[2021-05-27T18:32:46.380Z] [INFO] < 
org.apache.kafka:streams-quickstart >-
[2021-05-27T18:32:46.380Z] [INFO] Building Kafka Streams :: Quickstart 
3.0.0-SNAPSHOT[1/2]
[2021-05-27T18:32:46.380Z] [INFO] [ pom 
]-
[2021-05-27T18:32:46.380Z] [INFO] 
[2021-05-27T18:32:46.380Z] [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ streams-quickstart ---
[2021-05-27T18:32:47.314Z] [INFO] 
[2021-05-27T18:32:47.314Z] [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ streams-quickstart ---
[2021-05-27T18:32:47.314Z] [INFO] 
[2021-05-27T18:32:47.314Z] [INFO] --- maven-site-plugin:3.5.1:attach-descriptor 
(attach-descriptor) @ streams-quickstart ---
[2021-05-27T18:32:49.011Z] [INFO] 
[2021-05-27T18:32:49.011Z] [INFO] --- maven-gpg-plugin:1.6:sign 
(sign-artifacts) @ streams-quickstart ---
[2021-05-27T18:32:49.011Z] [INFO] 
[2021-05-27T18:32:49.011Z] [INFO] --- maven-install-plugin:2.5.2:install 
(default-install) @ streams-quickstart ---
[2021-05-27T18:32:49.011Z] [INFO] Installing 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/quickstart/pom.xml
 to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart/3.0.0-SNAPSHOT/streams-quickstart-3.0.0-SNAPSHOT.pom
[2021-05-27T18:32:49.011Z] [INFO] 
[2021-05-27T18:32:49.011Z] [INFO] --< 
org.apache.kafka:streams-quickstart-java >--
[2021-05-27T18:32:49.011Z] [INFO] Building streams-quickstart-java 
3.0.0-SNAPSHOT[2/2]
[2021-05-27T18:32:49.011Z] [INFO] --[ maven-archetype 
]---
[2021-05-27T18:32:49.011Z] [INFO] 
[2021-05-27T18:32:49.011Z] [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ streams-quickstart-java ---
[2021-05-27T18:32:49.011Z] [INFO] 
[2021-05-27T18:32:49.011Z] [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ streams-quickstart-java ---
[2021-05-27T18:32:49.086Z] 
[2021-05-27T18:32:49.086Z] PlaintextConsumerTest > testAutoOffsetReset() PASSED
[2021-05-27T18:32:49.086Z] 
[2021-05-27T18:32:49.086Z] PlaintextConsumerTest > 
testPerPartitionLagWithMaxPollRecords() STARTED
[2021-05-27T18:32:49.947Z] [INFO] 
[2021-05-27T18:32:49.947Z] [INFO] --- maven-resources-plugin:2.7:resources 
(default-resources) @ streams-quickstart-java ---
[2021-05-27T18:32:49.947Z] [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
[2021-05-27T18:32:49.947Z] [INFO] Copying 6 resources
[2021-05-27T18:32:49.947Z] [INFO] Copying 3 resources
[2021-05-27T18:32:49.947Z] [INFO] 
[2021-05-27T18:32:49.947Z] [INFO] --- maven-resources-plugin:2.7:testResources 
(default-testResources) @ streams-quickstart-java ---
[2021-05-27T18:32:49.947Z] [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
[2021-05-27T18:32:49.947Z] [INFO] Copying 2 resources
[2021-05-27T18:32:49.947Z] [INFO] Copying 3 resources
[2021-05-27T18:32:49.947Z] [INFO] 
[2021-05-27T18:32:49.947Z] [INFO] --- maven-archetype-plugin:2.2:jar 
(default-jar) @ streams-quickstart-java ---
[2021-05-27T18:32:49.947Z] [INFO] Building archetype jar: 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/quickstart/java/target/streams-quickstart-java-3.0.0-SNAPSHOT
[2021-05-27T18:32:49.947Z] [INFO] 
[2021-05-27T18:32:49.947Z] [INFO] --- maven-site-plugin:3.5.1:attach-descriptor 
(attach-descriptor) @ streams-quickstart-java ---
[2021-05-27T18:32:49.947Z] [INFO] 
[2021-05-27T18:32:49.947Z] [INFO] --- 
maven-archetype-plugin:2.2:integration-test (default-integration-test) @ 
streams-quickstart-java ---
[2021-05-27T18:32:49.947Z] [INFO] 
[2021-05-27T18:32:49.947Z] [INFO] --- maven-gpg-plugin:1.6:sign 
(sign-artifacts) @ streams-quickstart-java ---
[2021-05-27T18:32:49.947Z] [INFO] 
[2021-05-27T18:32:49.947Z] [INFO] --- maven-install-plugin:2.5.2:install 
(default-install) @ streams-quickstart-java ---
[2021-05-27T18:32:49.947Z] [INFO] Installing 
/home/jenkins/jenkins-

Re: Kafka getting down every week due to log file deletion.

[Ran Lupovich]

The main purpose of the /*tmp* directory is to temporarily store *files* when
installing an OS or software. If any *files* in the /*tmp* directory have
not been accessed for a while, they will be automatically *deleted* from
the system

בתאריך יום ה׳, 27 במאי 2021, 19:04, מאת Ran Lupovich ‏:

> Seems you log dir is sending your data to tmp folder, if I am bot mistken
> this dir automatically removing files from itself, causing the log deletuon
> procedure of the kafka internal to fail and shutdown broker on file not
> found
>
> בתאריך יום ה׳, 27 במאי 2021, 17:52, מאת Neeraj Gulia ‏<
> neeraj.gu...@opsworld.in>:
>
>> Hi team,
>>
>> Our Kafka is getting down almost once or twice a month due to log file
>> deletion failure.
>>
>>
>> There is single node kafka broker is running in our system and gets down
>> every time it tires to delete the log files as cleanup and fails.
>>
>> Sharing the Error Logs, we need a robust solution for this so that our
>> kafka broker doesn't gets down like this every time.
>>
>> Regards,
>> Neeraj Gulia
>>
>> Caused by: java.io.FileNotFoundException:
>> /tmp/kafka-logs/dokutopic-0/.index (No such file or
>> directory)
>> at java.base/java.io.RandomAccessFile.open0(Native Method)
>> at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:345)
>> at java.base/java.io.RandomAccessFile.(RandomAccessFile.java:259)
>> at java.base/java.io.RandomAccessFile.(RandomAccessFile.java:214)
>> at kafka.log.AbstractIndex.$anonfun$resize$1(AbstractIndex.scala:183)
>> at kafka.log.AbstractIndex.resize(AbstractIndex.scala:176)
>> at
>>
>> kafka.log.AbstractIndex.$anonfun$trimToValidSize$1(AbstractIndex.scala:242)
>> at kafka.log.AbstractIndex.trimToValidSize(AbstractIndex.scala:242)
>> at kafka.log.LogSegment.onBecomeInactiveSegment(LogSegment.scala:508)
>> at kafka.log.Log.$anonfun$roll$8(Log.scala:1954)
>> at kafka.log.Log.$anonfun$roll$2(Log.scala:1954)
>> at kafka.log.Log.roll(Log.scala:2387)
>> at kafka.log.Log.$anonfun$deleteSegments$2(Log.scala:1749)
>> at kafka.log.Log.deleteSegments(Log.scala:2387)
>> at kafka.log.Log.deleteRetentionMsBreachedSegments(Log.scala:1737)
>> at kafka.log.Log.deleteOldSegments(Log.scala:1806)
>> at kafka.log.LogManager.$anonfun$cleanupLogs$3(LogManager.scala:1074)
>> at
>> kafka.log.LogManager.$anonfun$cleanupLogs$3$adapted(LogManager.scala:1071)
>> at scala.collection.immutable.List.foreach(List.scala:431)
>> at kafka.log.LogManager.cleanupLogs(LogManager.scala:1071)
>> at kafka.log.LogManager.$anonfun$startup$2(LogManager.scala:409)
>> at
>> kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:114)
>> at
>>
>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>> at
>> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>> at
>>
>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>> at
>>
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> at
>>
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> at java.base/java.lang.Thread.run(Thread.java:829)
>> [2021-05-27 09:34:07,972] WARN [ReplicaManager broker=0] Broker 0 stopped
>> fetcher for partitions
>>
>> __consumer_offsets-22,__consumer_offsets-30,__consumer_offsets-8,__consumer_offsets-21,__consumer_offsets-4,__consumer_offsets-27,__consumer_offsets-7,__consumer_offsets-9,__consumer_offsets-46,fliptopic-0,__consumer_offsets-25,webhook-events-0,__consumer_offsets-35,__consumer_offsets-41,__consumer_offsets-33,__consumer_offsets-23,__consumer_offsets-49,__consumer_offsets-47,__consumer_offsets-16,__consumer_offsets-28,dokutopic-0,__consumer_offsets-31,__consumer_offsets-36,__consumer_offsets-42,__consumer_offsets-3,post_payment_topic-0,__consumer_offsets-18,__consumer_offsets-37,topic-0,events-0,__consumer_offsets-15,__consumer_offsets-24,__consumer_offsets-38,__consumer_offsets-17,__consumer_offsets-48,__consumer_offsets-19,__consumer_offsets-11,__consumer_offsets-13,__consumer_offsets-2,__consumer_offsets-43,__consumer_offsets-6,__consumer_offsets-14,__consumer_offsets-20,__consumer_offsets-0,__consumer_offsets-44,disbursementtopic-0,__consumer_offsets-39,__consumer_offsets-12,__consumer_offsets-45,__consumer_offsets-1,__consumer_offsets-5,__consumer_offsets-26,__consumer_offsets-29,__consumer_offsets-34,__consumer_offsets-10,__consumer_offsets-32,__consumer_offsets-40,faspaytopic-0
>> and stopped moving logs for partitions because they are in the failed log
>> directory /tmp/kafka-logs. (kafka.server.ReplicaManager)
>> [2021-05-27 09:34:07,974] WARN Stopping serving logs in dir
>> /tmp/kafka-logs
>> (kafka.log.LogManager)
>> [2021-05-27 09:34:07,983] ERROR Shutdown broker because all log dirs in
>> /tmp/kafka-logs have failed (kafka.log.LogManager)
>>
>

[jira] [Resolved] (KAFKA-12856) Upgrade Jackson to 2.12.3

[Ismael Juma (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-12856.
-
Fix Version/s: 3.0.0
   Resolution: Fixed

> Upgrade Jackson to 2.12.3
> -
>
> Key: KAFKA-12856
> URL: https://issues.apache.org/jira/browse/KAFKA-12856
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ismael Juma
>Assignee: Ismael Juma
>Priority: Major
> Fix For: 3.0.0
>
>
> 2.10.x is no longer supported, so we should move to 2.12 for the 3.0 release.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-741: Change default serde to be null

[Leah Thomas]

Thanks all, I think we can go ahead and close this now.

KIP-741 passes with 4 binding votes (Guozhang, Sophie, John, Bruno) and 3
non-binding (Walker, Lee, and myself).

On Tue, May 25, 2021 at 8:32 AM Dongjin Lee  wrote:

> Hi Bruno,
>
> Oh, thanks for pointing out my mistake. Yes, the KIP is not passed yet and
> here is the updated status:
>
> - Binding: Guozhang Wang, Sophie Blee-Goldman, John Roesler, Bruno Cadonna
> (+4)
> - Non-binding: Walker Carlson, Lee Dongjin (+2)
>
> Thanks,
> Dongjin
>
> On Tue, May 25, 2021 at 9:12 PM Bruno Cadonna  wrote:
>
> > Hi Leah,
> >
> > Thanks for the KIP!
> >
> > +1 (binding)
> >
> > Best,
> > Bruno
> >
> > On 25.05.21 14:02, Bruno Cadonna wrote:
> > > Hi Dongjin,
> > >
> > > voting for a KIP needs to remain open for at least 72 hours [1].
> > > According to the date and time of the first message to this thread 72
> > > hours havn't passed, yet. Theoretically, there could still be -1 votes
> > > coming in.
> > >
> > > Best,
> > > Bruno
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-Process
> > >
> > >
> > >
> > > On 25.05.21 13:13, Dongjin Lee wrote:
> > >> +1 (non-binding).
> > >>
> > >> As of present:
> > >>
> > >> - Binding: Guozhang Wang, Sophie Blee-Goldman, John Roesler (+3)
> > >> - Non-binding: Walker Carlson, Lee Dongjin (+2)
> > >>
> > >> This KIP is now passed.
> > >>
> > >> Thanks,
> > >> Dongjin
> > >>
> > >> On Tue, May 25, 2021 at 10:12 AM John Roesler 
> > >> wrote:
> > >>
> > >>> +1 (binding) from me. Thanks for the KIP!
> > >>> -John
> > >>>
> > >>> On Mon, May 24, 2021, at 18:10, Sophie Blee-Goldman wrote:
> >  +1 binding
> > 
> >  thanks for the KIP
> >  -Sophie
> > 
> >  On Mon, May 24, 2021 at 2:02 PM Walker Carlson
> >   wrote:
> > 
> > > +1 (non-binding) from me, Leah
> > >
> > > Walker
> > >
> > > On Mon, May 24, 2021 at 1:51 PM Leah Thomas
> > >>> 
> > > wrote:
> > >
> > >> Hi,
> > >>
> > >> I'd like to kick-off voting for KIP-741: Change default serde to
> be
> > >>> null.
> > >> <
> > >>
> > >
> > >>>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-741%3A+Change+default+serde+to+be+null
> > >>>
> > >>>
> > >> The
> > >> discussion is linked on the KIP for context.
> > >>
> > >> Cheers,
> > >> Leah
> > >>
> > >
> > 
> > >>>
> > >>
> > >>
> >
> > --
> > *Dongjin Lee*
> >
> > *A hitchhiker in the mathematical world.*
> >
> >
> >
> > *github:  github.com/dongjinleekr
> > keybase:
> https://keybase.io/dongjinleekr
> > linkedin:
> kr.linkedin.com/in/dongjinleekr
> > speakerdeck:
> speakerdeck.com/dongjin
> > *
> >
>

Re: Kafka getting down every week due to log file deletion.

[Ran Lupovich]

Seems you log dir is sending your data to tmp folder, if I am bot mistken
this dir automatically removing files from itself, causing the log deletuon
procedure of the kafka internal to fail and shutdown broker on file not
found

בתאריך יום ה׳, 27 במאי 2021, 17:52, מאת Neeraj Gulia ‏<
neeraj.gu...@opsworld.in>:

> Hi team,
>
> Our Kafka is getting down almost once or twice a month due to log file
> deletion failure.
>
>
> There is single node kafka broker is running in our system and gets down
> every time it tires to delete the log files as cleanup and fails.
>
> Sharing the Error Logs, we need a robust solution for this so that our
> kafka broker doesn't gets down like this every time.
>
> Regards,
> Neeraj Gulia
>
> Caused by: java.io.FileNotFoundException:
> /tmp/kafka-logs/dokutopic-0/.index (No such file or
> directory)
> at java.base/java.io.RandomAccessFile.open0(Native Method)
> at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:345)
> at java.base/java.io.RandomAccessFile.(RandomAccessFile.java:259)
> at java.base/java.io.RandomAccessFile.(RandomAccessFile.java:214)
> at kafka.log.AbstractIndex.$anonfun$resize$1(AbstractIndex.scala:183)
> at kafka.log.AbstractIndex.resize(AbstractIndex.scala:176)
> at
> kafka.log.AbstractIndex.$anonfun$trimToValidSize$1(AbstractIndex.scala:242)
> at kafka.log.AbstractIndex.trimToValidSize(AbstractIndex.scala:242)
> at kafka.log.LogSegment.onBecomeInactiveSegment(LogSegment.scala:508)
> at kafka.log.Log.$anonfun$roll$8(Log.scala:1954)
> at kafka.log.Log.$anonfun$roll$2(Log.scala:1954)
> at kafka.log.Log.roll(Log.scala:2387)
> at kafka.log.Log.$anonfun$deleteSegments$2(Log.scala:1749)
> at kafka.log.Log.deleteSegments(Log.scala:2387)
> at kafka.log.Log.deleteRetentionMsBreachedSegments(Log.scala:1737)
> at kafka.log.Log.deleteOldSegments(Log.scala:1806)
> at kafka.log.LogManager.$anonfun$cleanupLogs$3(LogManager.scala:1074)
> at
> kafka.log.LogManager.$anonfun$cleanupLogs$3$adapted(LogManager.scala:1071)
> at scala.collection.immutable.List.foreach(List.scala:431)
> at kafka.log.LogManager.cleanupLogs(LogManager.scala:1071)
> at kafka.log.LogManager.$anonfun$startup$2(LogManager.scala:409)
> at kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:114)
> at
>
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> at
>
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at
>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> [2021-05-27 09:34:07,972] WARN [ReplicaManager broker=0] Broker 0 stopped
> fetcher for partitions
>
> __consumer_offsets-22,__consumer_offsets-30,__consumer_offsets-8,__consumer_offsets-21,__consumer_offsets-4,__consumer_offsets-27,__consumer_offsets-7,__consumer_offsets-9,__consumer_offsets-46,fliptopic-0,__consumer_offsets-25,webhook-events-0,__consumer_offsets-35,__consumer_offsets-41,__consumer_offsets-33,__consumer_offsets-23,__consumer_offsets-49,__consumer_offsets-47,__consumer_offsets-16,__consumer_offsets-28,dokutopic-0,__consumer_offsets-31,__consumer_offsets-36,__consumer_offsets-42,__consumer_offsets-3,post_payment_topic-0,__consumer_offsets-18,__consumer_offsets-37,topic-0,events-0,__consumer_offsets-15,__consumer_offsets-24,__consumer_offsets-38,__consumer_offsets-17,__consumer_offsets-48,__consumer_offsets-19,__consumer_offsets-11,__consumer_offsets-13,__consumer_offsets-2,__consumer_offsets-43,__consumer_offsets-6,__consumer_offsets-14,__consumer_offsets-20,__consumer_offsets-0,__consumer_offsets-44,disbursementtopic-0,__consumer_offsets-39,__consumer_offsets-12,__consumer_offsets-45,__consumer_offsets-1,__consumer_offsets-5,__consumer_offsets-26,__consumer_offsets-29,__consumer_offsets-34,__consumer_offsets-10,__consumer_offsets-32,__consumer_offsets-40,faspaytopic-0
> and stopped moving logs for partitions because they are in the failed log
> directory /tmp/kafka-logs. (kafka.server.ReplicaManager)
> [2021-05-27 09:34:07,974] WARN Stopping serving logs in dir /tmp/kafka-logs
> (kafka.log.LogManager)
> [2021-05-27 09:34:07,983] ERROR Shutdown broker because all log dirs in
> /tmp/kafka-logs have failed (kafka.log.LogManager)
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #169

[Apache Jenkins Server]

See 

Kafka getting down every week due to log file deletion.

[Neeraj Gulia]

Hi team,

Our Kafka is getting down almost once or twice a month due to log file
deletion failure.


There is single node kafka broker is running in our system and gets down
every time it tires to delete the log files as cleanup and fails.

Sharing the Error Logs, we need a robust solution for this so that our
kafka broker doesn't gets down like this every time.

Regards,
Neeraj Gulia

Caused by: java.io.FileNotFoundException:
/tmp/kafka-logs/dokutopic-0/.index (No such file or
directory)
at java.base/java.io.RandomAccessFile.open0(Native Method)
at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:345)
at java.base/java.io.RandomAccessFile.(RandomAccessFile.java:259)
at java.base/java.io.RandomAccessFile.(RandomAccessFile.java:214)
at kafka.log.AbstractIndex.$anonfun$resize$1(AbstractIndex.scala:183)
at kafka.log.AbstractIndex.resize(AbstractIndex.scala:176)
at
kafka.log.AbstractIndex.$anonfun$trimToValidSize$1(AbstractIndex.scala:242)
at kafka.log.AbstractIndex.trimToValidSize(AbstractIndex.scala:242)
at kafka.log.LogSegment.onBecomeInactiveSegment(LogSegment.scala:508)
at kafka.log.Log.$anonfun$roll$8(Log.scala:1954)
at kafka.log.Log.$anonfun$roll$2(Log.scala:1954)
at kafka.log.Log.roll(Log.scala:2387)
at kafka.log.Log.$anonfun$deleteSegments$2(Log.scala:1749)
at kafka.log.Log.deleteSegments(Log.scala:2387)
at kafka.log.Log.deleteRetentionMsBreachedSegments(Log.scala:1737)
at kafka.log.Log.deleteOldSegments(Log.scala:1806)
at kafka.log.LogManager.$anonfun$cleanupLogs$3(LogManager.scala:1074)
at
kafka.log.LogManager.$anonfun$cleanupLogs$3$adapted(LogManager.scala:1071)
at scala.collection.immutable.List.foreach(List.scala:431)
at kafka.log.LogManager.cleanupLogs(LogManager.scala:1071)
at kafka.log.LogManager.$anonfun$startup$2(LogManager.scala:409)
at kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:114)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at
java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
[2021-05-27 09:34:07,972] WARN [ReplicaManager broker=0] Broker 0 stopped
fetcher for partitions
__consumer_offsets-22,__consumer_offsets-30,__consumer_offsets-8,__consumer_offsets-21,__consumer_offsets-4,__consumer_offsets-27,__consumer_offsets-7,__consumer_offsets-9,__consumer_offsets-46,fliptopic-0,__consumer_offsets-25,webhook-events-0,__consumer_offsets-35,__consumer_offsets-41,__consumer_offsets-33,__consumer_offsets-23,__consumer_offsets-49,__consumer_offsets-47,__consumer_offsets-16,__consumer_offsets-28,dokutopic-0,__consumer_offsets-31,__consumer_offsets-36,__consumer_offsets-42,__consumer_offsets-3,post_payment_topic-0,__consumer_offsets-18,__consumer_offsets-37,topic-0,events-0,__consumer_offsets-15,__consumer_offsets-24,__consumer_offsets-38,__consumer_offsets-17,__consumer_offsets-48,__consumer_offsets-19,__consumer_offsets-11,__consumer_offsets-13,__consumer_offsets-2,__consumer_offsets-43,__consumer_offsets-6,__consumer_offsets-14,__consumer_offsets-20,__consumer_offsets-0,__consumer_offsets-44,disbursementtopic-0,__consumer_offsets-39,__consumer_offsets-12,__consumer_offsets-45,__consumer_offsets-1,__consumer_offsets-5,__consumer_offsets-26,__consumer_offsets-29,__consumer_offsets-34,__consumer_offsets-10,__consumer_offsets-32,__consumer_offsets-40,faspaytopic-0
and stopped moving logs for partitions because they are in the failed log
directory /tmp/kafka-logs. (kafka.server.ReplicaManager)
[2021-05-27 09:34:07,974] WARN Stopping serving logs in dir /tmp/kafka-logs
(kafka.log.LogManager)
[2021-05-27 09:34:07,983] ERROR Shutdown broker because all log dirs in
/tmp/kafka-logs have failed (kafka.log.LogManager)

[jira] [Created] (KAFKA-12856) Upgrade Jackson to 2.12.3

[Ismael Juma (Jira)]

Ismael Juma created KAFKA-12856:
---

 Summary: Upgrade Jackson to 2.12.3
 Key: KAFKA-12856
 URL: https://issues.apache.org/jira/browse/KAFKA-12856
 Project: Kafka
  Issue Type: Bug
Reporter: Ismael Juma
Assignee: Ismael Juma


2.10.x is no longer supported, so we should move to 2.12 for the 3.0 release.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #168

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-12855) Update ssl certificates of kafka connect worker runtime without restarting the worker process.

[kaushik srinivas (Jira)]

kaushik srinivas created KAFKA-12855:


 Summary: Update ssl certificates of kafka connect worker runtime 
without restarting the worker process.
 Key: KAFKA-12855
 URL: https://issues.apache.org/jira/browse/KAFKA-12855
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect
Reporter: kaushik srinivas


Is there a possibility to update the ssl certificates of kafka connect worker 
dynamically something similar to kafka-configs script for kafka ? Or the only 
way to update the certificates is to restart the worker processes and update 
the certificates ?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-744: Migrate TaskMetadata to interface with internal implementation

[Josep Prat]

I deliberately picked the most conservative approach of creating a new
Interface, instead of transforming the current class into an interface.
Feedback is most welcome!

Best,

On Thu, May 27, 2021 at 10:26 AM Josep Prat  wrote:

> Hi there,
> I would like to propose KIP-744, to introduce TaskMetadata as an
> interface, to keep the its implementation as internal use.
> This KIP can be seen as a spin-off of KIP-740.
>
> https://cwiki.apache.org/confluence/x/XIrOCg
>
> Best,
> --
>
> Josep Prat
>
> *Aiven Deutschland GmbH*
>
> Immanuelkirchstraße 26, 10405 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> *m:* +491715557497
>
> *w:* aiven.io
>
> *e:* josep.p...@aiven.io
>


-- 

Josep Prat

*Aiven Deutschland GmbH*

Immanuelkirchstraße 26, 10405 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

*m:* +491715557497

*w:* aiven.io

*e:* josep.p...@aiven.io

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #167

[Apache Jenkins Server]

See 

[DISCUSS] KIP-744: Migrate TaskMetadata to interface with internal implementation

[Josep Prat]

Hi there,
I would like to propose KIP-744, to introduce TaskMetadata as an interface,
to keep the its implementation as internal use.
This KIP can be seen as a spin-off of KIP-740.

https://cwiki.apache.org/confluence/x/XIrOCg

Best,
-- 

Josep Prat

*Aiven Deutschland GmbH*

Immanuelkirchstraße 26, 10405 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

*m:* +491715557497

*w:* aiven.io

*e:* josep.p...@aiven.io