Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #957

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-13942) LogOffsetTest occasionally hangs during Jenkins build

[David Arthur (Jira)]

David Arthur created KAFKA-13942:


 Summary: LogOffsetTest occasionally hangs during Jenkins build
 Key: KAFKA-13942
 URL: https://issues.apache.org/jira/browse/KAFKA-13942
 Project: Kafka
  Issue Type: Bug
  Components: unit tests
Reporter: David Arthur


[~hachikuji] parsed the log output of one of the recent stalled Jenkins builds 
and singled out LogOffsetTest as a likely culprit for not completing.

I looked closely at the following build which appeared to be stuck and found 
this test case had STARTED but not PASSED or FAILED.

15:19:58  LogOffsetTest > 
testFetchOffsetByTimestampForMaxTimestampWithUnorderedTimestamps(String) > 
kafka.server.LogOffsetTest.testFetchOffsetByTimestampForMaxTimestampWithUnorderedTimestamps(String)[2]
 STARTED




--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (KAFKA-13941) Re-enable ARM builds following INFRA-23305

[David Arthur (Jira)]

David Arthur created KAFKA-13941:


 Summary: Re-enable ARM builds following INFRA-23305
 Key: KAFKA-13941
 URL: https://issues.apache.org/jira/browse/KAFKA-13941
 Project: Kafka
  Issue Type: Task
  Components: build
Reporter: David Arthur


Once https://issues.apache.org/jira/browse/INFRA-23305 is resolved, we should 
re-enable ARM builds in the Jenkinsfile.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API

[Jun Rao]

Hi, Igor,

Thanks for the reply.

I agree that this KIP could be useful for improving the tool for moving
data across disks. It would be useful to clarify on the main motivation of
the KIP. Also, DescribeLogDirsResponse already includes the size of each
partition on a disk. So, it seems that UsableBytes is redundant since it's
derivable.

Thanks,

Jun

On Thu, May 26, 2022 at 3:30 AM Igor Soarez  wrote:

> Hi,
>
> This can also be quite useful to make better use of existing functionality
> in the Kafka API — moving replicas between log directories via
> ALTER_REPLICA_LOG_DIRS. If usable space information is also available the
> caller can make better decisions using the same API. It means a more
> consistent way of interacting with Kafka to manage replicas locations
> within a broker without having to correlate Kafka metrics with information
> from the Kafka API.
>
> --
> Igor
>
> On Wed, May 25, 2022, at 8:16 PM, Jun Rao wrote:
> > Hi, Mickael,
> >
> > Thanks for the KIP.  Since this is mostly for monitoring and alerting,
> > could we expose them as metrics instead of as part of the API? We already
> > have a size metric per log. Perhaps we could extend that to add
> used/total
> > metrics per disk?
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, May 19, 2022 at 10:21 PM Raman Verma  >
> > wrote:
> >
> >> Hello Mikael,
> >>
> >> Thanks for the KIP.
> >>
> >> I see that the API response contains some information about each
> partition.
> >> ```
> >> { "name": "PartitionSize", "type": "int64", "versions": "0+",
> >>   "about": "The size of the log segments in this partition in bytes." }
> >> ```
> >> Can this be summed up to provide a used space in a `log.dir`
> >> This will also be specific to a `log.dir` (for the case where multiple
> >> log.dir are hosted on the same underlying device)
> >>
> >> On Thu, May 19, 2022 at 10:21 AM Cong Ding 
> >> wrote:
> >> >
> >> > Hey Mickael,
> >> >
> >> > Great KIP!
> >> >
> >> > I have one question:
> >> >
> >> > You mentioned "DescribeLogDirs is usually a low volume API. This
> change
> >> > should not
> >> > significantly affect the latency of this API." and "That would allow
> to
> >> > easily validate whether disk operations (like a resize), or topic
> >> deletion
> >> > (log deletion only happen after a short delay) have completed." I
> wonder
> >> if
> >> > there is an existing metric/API that can allow administrators to
> >> determine
> >> > whether we need to resize? If administrators use this API to determine
> >> > whether we need a resize, would this API become a high-volume API? I
> >> > understand we don't want this API to be a high-volume one because the
> API
> >> > is already costly by returning `"name": "Topics"`.
> >> >
> >> > Cong
> >> >
> >> > On Thu, Apr 7, 2022 at 2:17 AM Mickael Maison <
> mickael.mai...@gmail.com>
> >> > wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > > I wrote a small KIP to expose the total and usable space of logdirs
> >> > > via the DescribeLogDirs API:
> >> > >
> >> > >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API
> >> > >
> >> > > Please take a look and let me know if you have any feedback.
> >> > >
> >> > > Thanks,
> >> > > Mickael
> >> > >
> >>
> >>
> >>
> >> --
> >> Best Regards,
> >> Raman Verma
> >>
>

RE: Re: KIP 678: New Kafka Connect SMT for plainText => Struct(or Map) with Regex

[최GJ]

Hi Chris,

Thank you for your positive feed back,
And sorry about late reply ( I didn’t recognize your reply email… TT )

And I update KIP and PR with your review
https://cwiki.apache.org/confluence/display/KAFKA/KIP+678%3A+New+Kafka+Connect+SMT+for+plainText+%3D%3E+Struct%28or+Map%29+with+Regex
 

1. typecast function removed
2. struct.field removed
3. Proposed Interface changed 


I will wait for you second feed back

Thanks~


On 2021/07/15 15:57:14 Chris Egerton wrote:
> Hi whsoul,
> 
> Thanks for the KIP. The two use cases you identified seem quite appealing,
> especially the first one: parsing structured log messages.
> 
> Here are my initial thoughts on the proposed design:
> 
> 1. I wonder if it's necessary to include support for type casting with this
> SMT. We already have a Cast SMT (
> https://github.com/apache/kafka/blob/trunk/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/Cast.java)
> that can parse multiple fields of a structured record value with differing
> types. Would it be enough for your new SMT to only produce string values
> for its structured data, and then allow users to perform casting logic
> using the Cast SMT afterward?
> 
> 2. It seems like the "struct.field" property is similar; based on the
> examples, it looks like when the SMT is configured with a value for that
> property, it will first pull out a field from a structured record value
> (for example, it would pull out the value "
> https://kafka.apache.org/documentation/#connect; from a map of {"url": "
> https://kafka.apache.org/documentation/#connect"}), then parse that field's
> value, and replace the entire record value (or key) with the result of the
> parsing stage. It seems like this could be accomplished using the
> ExtractField SMT (
> https://github.com/apache/kafka/blob/trunk/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/ExtractField.java)
> as a preliminary step before passing it to your new SMT. Is this correct?
> And if so, could we simplify the interface for your SMT by removing the
> "struct.field" property in favor of the existing ExtractField SMT?
> 
> 3. (Nit) I believe the property names in the table beneath the "Proposed
> Interfaces" section should have the "transforms.RegexTransform." prefix
> removed, since users will be able to configure the SMT with any name, not
> just "RegexTransform". This keeps in line with similar KIPs such as
> KIP-437, which added a new property to the MaskField SMT (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-437%3A+Custom+replacement+for+MaskField+SMT
> ).
> 
> Cheers,
> 
> Chris
> 
> On Thu, Jul 15, 2021 at 7:56 AM gyejun choi  wrote:
> 
> > is there someone who any feed back about this?
> >
> > 2020년 10월 23일 (금) 오후 2:56, gyejun choi 님이 작성:
> >
> > > Hi,
> > >
> > > I've opened KIP-678 which is intended to provide a new SMT in Kafka
> > Connect.
> > > I'd be grateful for any
> > > feedback:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP+678%3A+New+Kafka+Connect+SMT+for+plainText+%3D%3E+Struct%28or+Map%29+with+Regex==
> > >
> > > thanks,
> > >
> > > whsoul
> > >
> > >
> >
> 

Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API

[Igor Soarez]

Hi,

This can also be quite useful to make better use of existing functionality in 
the Kafka API — moving replicas between log directories via 
ALTER_REPLICA_LOG_DIRS. If usable space information is also available the 
caller can make better decisions using the same API. It means a more consistent 
way of interacting with Kafka to manage replicas locations within a broker 
without having to correlate Kafka metrics with information from the Kafka API.

--
Igor

On Wed, May 25, 2022, at 8:16 PM, Jun Rao wrote:
> Hi, Mickael,
>
> Thanks for the KIP.  Since this is mostly for monitoring and alerting,
> could we expose them as metrics instead of as part of the API? We already
> have a size metric per log. Perhaps we could extend that to add used/total
> metrics per disk?
>
> Thanks,
>
> Jun
>
> On Thu, May 19, 2022 at 10:21 PM Raman Verma 
> wrote:
>
>> Hello Mikael,
>>
>> Thanks for the KIP.
>>
>> I see that the API response contains some information about each partition.
>> ```
>> { "name": "PartitionSize", "type": "int64", "versions": "0+",
>>   "about": "The size of the log segments in this partition in bytes." }
>> ```
>> Can this be summed up to provide a used space in a `log.dir`
>> This will also be specific to a `log.dir` (for the case where multiple
>> log.dir are hosted on the same underlying device)
>>
>> On Thu, May 19, 2022 at 10:21 AM Cong Ding 
>> wrote:
>> >
>> > Hey Mickael,
>> >
>> > Great KIP!
>> >
>> > I have one question:
>> >
>> > You mentioned "DescribeLogDirs is usually a low volume API. This change
>> > should not
>> > significantly affect the latency of this API." and "That would allow to
>> > easily validate whether disk operations (like a resize), or topic
>> deletion
>> > (log deletion only happen after a short delay) have completed." I wonder
>> if
>> > there is an existing metric/API that can allow administrators to
>> determine
>> > whether we need to resize? If administrators use this API to determine
>> > whether we need a resize, would this API become a high-volume API? I
>> > understand we don't want this API to be a high-volume one because the API
>> > is already costly by returning `"name": "Topics"`.
>> >
>> > Cong
>> >
>> > On Thu, Apr 7, 2022 at 2:17 AM Mickael Maison 
>> > wrote:
>> >
>> > > Hi,
>> > >
>> > > I wrote a small KIP to expose the total and usable space of logdirs
>> > > via the DescribeLogDirs API:
>> > >
>> > >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API
>> > >
>> > > Please take a look and let me know if you have any feedback.
>> > >
>> > > Thanks,
>> > > Mickael
>> > >
>>
>>
>>
>> --
>> Best Regards,
>> Raman Verma
>>

Re: [DISCUSS] KIP-845: HasField predicate for kafka connect

[Sagar]

Hi Kumud,

Thanks for that. I don't have any other comments at this point on the KIP.
LGTM overall.

Thanks!
Sagar.

On Wed, May 25, 2022 at 5:14 PM Sagar  wrote:

> Thanks for the KIP Kumud.
>
> Can you please add a couple of examples on how this would behave with
> different combinations. I think that way it would be easier to understand.
>
> Thanks!
> Sagar.
>
> On Wed, May 25, 2022 at 4:59 PM Kumud Kumar Srivatsava Tirupati <
> kumudkumartirup...@gmail.com> wrote:
>
>> Hi all,
>> I have written a KIP for having a new HasField predicate for kafka connect
>> transforms and would like to start a discussion. Please share your
>> thoughts
>> on the same.
>>
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-845%3A+%27HasField%27+predicate+for+kafka+connect
>>
>> *---*
>> *Thanks and Regards,*
>> *Kumud Kumar Srivatsava Tirupati*
>>
>

Re: [DISCUSS] KIP-840: Config file option for MessageReader/MessageFormatter in ConsoleProducer/ConsoleConsumer

[Alexandre Garnier]

Hello everyone,

Any feedback on this KIP https://cwiki.apache.org/confluence/x/bBqhD?
It is a straightforward improvement without any impact on existing users,
so not much to discuss besides maybe the option name.

-- 
Alex


Le mer. 18 mai 2022 à 10:44, Alexandre Garnier  a écrit :

> Hi everyone,
>
> I created a KIP to add a config file option of reader/formatter for
> kafka-console-(consumer|producer).sh tools.
> https://cwiki.apache.org/confluence/x/bBqhD
>
> Thanks for your feedback,
> --
> Alex
>