Re: [VOTE] KIP-864: Add End-To-End Latency Metrics to Connectors

[Yash Mayya]

Hi Jorge,

Thanks for the KIP.

+1 (non-binding)

Thanks,
Yash

On Mon, Sep 12, 2022 at 4:17 PM Jorge Esteban Quilcate Otoya <
quilcate.jo...@gmail.com> wrote:

> Pressed send to soon. Updating subject.
>
> On Mon, 12 Sept 2022 at 11:45, Jorge Esteban Quilcate Otoya <
> quilcate.jo...@gmail.com> wrote:
>
> > Hi everyone,
> >
> > Thank you all for the positive discussion about KIP-864.
> >
> > I would like to start a voting thread, to introduce these new metrics for
> > Connector tasks
> >
> > KIP: https://cwiki.apache.org/confluence/x/6I5rDQ
> >
> > Thanks,
> > Jorge
> >
> >
>

Re: [VOTE] KIP-792: Add "generation" field into consumer protocol

[David Jacot]

Hi Luke,

Are you still interested in implementing this KIP? We need it for
KIP-848. If you are not, we could find someone to take it over.

Thanks,
David

On Thu, Mar 3, 2022 at 10:04 AM Luke Chen  wrote:
>
> Thanks, Ziming!
>
> So, now, this KIP vote passed with 3 binding +1 votes (David, Tom,
> Guozhang) and 1 non-binding +1 vote (Ziming).
> The vote will be closed.
>
> Thanks again.
> Luke
>
> On Thu, Mar 3, 2022 at 5:00 PM deng ziming  wrote:
>
> > Thank you Luke for this work,
> > I’m +1(non-binding)
> >
> > --
> > Best,
> > Ziming Deng
> >
> > > On Dec 1, 2021, at 8:36 AM, Luke Chen  wrote:
> > >
> > > Hi all,
> > >
> > > I'd like to start the vote for KIP-792: Add "generation" field into
> > > consumer protocol.
> > >
> > > The goal of this KIP is to allow the assignor/consumer coordinator to
> > have
> > > a way to identify the out-of-date members/assignments, to avoid rebalance
> > > stuck issues in current protocol.
> > >
> > > Detailed description can be found here:
> > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191336614
> > >
> > > Any feedback is welcome.
> > >
> > > Thank you.
> > > Luke
> >
> >

[jira] [Created] (KAFKA-14255) Fetching from follower should be disallowed if fetch from follower is disabled

[David Jacot (Jira)]

David Jacot created KAFKA-14255:
---

 Summary: Fetching from follower should be disallowed if fetch from 
follower is disabled
 Key: KAFKA-14255
 URL: https://issues.apache.org/jira/browse/KAFKA-14255
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.4.0
Reporter: David Jacot
Assignee: David Jacot


There are clients out there that have implemented KIP-392 (Fetch From Follower) 
and thus use FetchRequest >= 11. However, they have not implemented KIP-320 
which add the leader epoch to the FetchRequest in version 9. Without KIP-320, 
it is not safe to fetch from the follower. If a client does it by mistake – 
e.g. based on stale metadata – that could lead to offset out of range.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-792: Add "generation" field into consumer protocol

[Luke Chen]

Hi David,

Sorry for the delay.
I'll complete it in v3.4.0.

Thank you.
Luke

On Thu, Sep 22, 2022 at 3:52 PM David Jacot 
wrote:

> Hi Luke,
>
> Are you still interested in implementing this KIP? We need it for
> KIP-848. If you are not, we could find someone to take it over.
>
> Thanks,
> David
>
> On Thu, Mar 3, 2022 at 10:04 AM Luke Chen  wrote:
> >
> > Thanks, Ziming!
> >
> > So, now, this KIP vote passed with 3 binding +1 votes (David, Tom,
> > Guozhang) and 1 non-binding +1 vote (Ziming).
> > The vote will be closed.
> >
> > Thanks again.
> > Luke
> >
> > On Thu, Mar 3, 2022 at 5:00 PM deng ziming 
> wrote:
> >
> > > Thank you Luke for this work,
> > > I’m +1(non-binding)
> > >
> > > --
> > > Best,
> > > Ziming Deng
> > >
> > > > On Dec 1, 2021, at 8:36 AM, Luke Chen  wrote:
> > > >
> > > > Hi all,
> > > >
> > > > I'd like to start the vote for KIP-792: Add "generation" field into
> > > > consumer protocol.
> > > >
> > > > The goal of this KIP is to allow the assignor/consumer coordinator to
> > > have
> > > > a way to identify the out-of-date members/assignments, to avoid
> rebalance
> > > > stuck issues in current protocol.
> > > >
> > > > Detailed description can be found here:
> > > >
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191336614
> > > >
> > > > Any feedback is welcome.
> > > >
> > > > Thank you.
> > > > Luke
> > >
> > >
>

Re: [VOTE] KIP-792: Add "generation" field into consumer protocol

[David Jacot]

Thanks, Luke. Feel free to ping me for reviews. I am happy to help on this one.

Cheers,
David

On Thu, Sep 22, 2022 at 11:00 AM Luke Chen  wrote:
>
> Hi David,
>
> Sorry for the delay.
> I'll complete it in v3.4.0.
>
> Thank you.
> Luke
>
> On Thu, Sep 22, 2022 at 3:52 PM David Jacot 
> wrote:
>
> > Hi Luke,
> >
> > Are you still interested in implementing this KIP? We need it for
> > KIP-848. If you are not, we could find someone to take it over.
> >
> > Thanks,
> > David
> >
> > On Thu, Mar 3, 2022 at 10:04 AM Luke Chen  wrote:
> > >
> > > Thanks, Ziming!
> > >
> > > So, now, this KIP vote passed with 3 binding +1 votes (David, Tom,
> > > Guozhang) and 1 non-binding +1 vote (Ziming).
> > > The vote will be closed.
> > >
> > > Thanks again.
> > > Luke
> > >
> > > On Thu, Mar 3, 2022 at 5:00 PM deng ziming 
> > wrote:
> > >
> > > > Thank you Luke for this work,
> > > > I’m +1(non-binding)
> > > >
> > > > --
> > > > Best,
> > > > Ziming Deng
> > > >
> > > > > On Dec 1, 2021, at 8:36 AM, Luke Chen  wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > I'd like to start the vote for KIP-792: Add "generation" field into
> > > > > consumer protocol.
> > > > >
> > > > > The goal of this KIP is to allow the assignor/consumer coordinator to
> > > > have
> > > > > a way to identify the out-of-date members/assignments, to avoid
> > rebalance
> > > > > stuck issues in current protocol.
> > > > >
> > > > > Detailed description can be found here:
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191336614
> > > > >
> > > > > Any feedback is welcome.
> > > > >
> > > > > Thank you.
> > > > > Luke
> > > >
> > > >
> >

Re: [DISCUSS] KIP-869: Improve Streams State Restoration Visibility

[Bruno Cadonna]

Hi Guozhang!

Thanks for the updates!

1.
Why do you distinguish between active and standby for the total amount 
of restored/updated records but not for the rate of restored/updated 
records?


2.
Regarding "standby-records-remaining" I am not sure how useful this 
metric is and I am not sure how hard it will be to record. I see the 
usefulness of monitoring the lag of a single standby state store, but I 
am not sure how useful it is to monitor the "lag" of a state updater 
thread that might potentially contain multiple state stores. 
Furthermore, do we need to issue a remote call to record this metric or 
do we get this information from each poll?


3.
Could you please give an example where "active-records-restored-total" 
and "standby-records-updated-total" are useful?


Best,
Bruno

On 20.09.22 22:45, Guozhang Wang wrote:

Hello Bruno/Nick,

I've updated the KIP wiki to reflect the incorporated comments from you,
please feel free to take another look and let me know what you think.


Guozhang

On Tue, Sep 20, 2022 at 9:37 AM Guozhang Wang  wrote:


Hi Nick,

Thanks for the reviews, and I think these are good suggestions. Note that
currently the `restore-records-total/rate` would include both restoring
active tasks as well as updating standby tasks, I think for your purposes
you'd be more interested in active restoring tasks since they could
complete, while updating standby tasks would not complete even if they have
caught up with the active. At the same time, the changelog reader would
only be restoring either active or standby at a given time, and active
tasks has a higher priority such that as long as there is at least one
active task still restoring, we would not restore any standby tasks. From
your suggestion, I'm thinking that maybe I should break up the rate / total
metric for active and standby tasks separately.

For deriving estimated time remaining though, the `total` metric may not
be helpful since they will not be "reset" after rebalances, i.e. they will
be an ever-increasing number and record the total number of records for the
lifetime of the app. But still, just the remaining records alone, with the
time elapsed monitored by the apps, should be sufficient to get the
estimated time remaining.


Guozhang


On Tue, Sep 20, 2022 at 3:10 AM Nick Telford 
wrote:


Hi Guozhang,

KIP looks great, I have one suggestion: in addition to
"restore-records-total", it would also be useful to track the number of
records *remaining*, that is, the records that have not yet been restored.
This is actually the metric I was attempting to implement in the
StateRestoreListener that bumped me up against KAFKA-10575 :-)

With both a "restore-records-total" and a "restore-remaining-total" (or
similar) metric, it's possible to derive useful information like the
estimated time remaining for restoration (by dividing the remaining total
by the restoration rate).

Regards,

Nick

On Mon, 19 Sept 2022 at 19:57, Guozhang Wang  wrote:


Hello Bruno,

Thanks for your comments!

1. Regarding the metrics group name: originally I put
"stream-state-metrics" as it's related to state store restorations, but
after a second thought I think I agree with you that this is quite
confusing and not right. About the metrics groups, right now I have two
ideas:

a) Still use the metric group name "stream-thread-metrics", but
differentiate with the processing threads on the thread id. The pros is
that we do not introduce a new group, the cons is that users who want to
separate processing from restoration/updating in the future needs to do
that on the thread id labels.
b) Introduce a new group name, for example

"stream-state-updater-metrics"

and still have a thread-id label. We would be introducing a new group

which

still have a thread-id, which may sound a bit weird (maybe if we do

that we

could change the existing "stream-thread-metrics" into
"stream-processor-metrics").

Right now I'm leaning towards b) and maybe in the future rename
"thread-metrics" to "processor-metrics", LMK what do you think.

2. Regarding the metric names: today we may also pause a standby tasks,

and

hence I'm trying to differentiate updating standbys from paused

standbys.

Right now I'm thinking we can change "restoring-standby-tasks" to
"updating-standby-tasks", and change all other metrics' "restore"

(except

the "restoring-active-tasks") to "state-update", a.k.a
"state-update-ratio", "state-update-records-total",
"updating-standby-tasks" etc.

3. Regarding the function name: yeah I think that's a valid concern, I

can

change it to "onRestoreSuspended" since "Aborted" may confuse people

that

previously called "onBatchRestored" are undone as part of the abortion.


Guozhang



On Mon, Sep 19, 2022 at 10:47 AM Bruno Cadonna 

wrote:



Hi Guozhang,

Thanks for the KIP! I think this KIP is a really nice addition to

better

understand what is going on in a Kafka Streams application.

1.
The metric names "paused-active-tasks" and "paused-standby-tasks"

Re: [kafka-clients] Re: [VOTE] 3.3.0 RC2

[David Arthur]

Josep, thanks for the note. We will mention the CVEs fixed in this release
in the announcement email. I believe we can also update the release notes
HTML after the vote is complete.

-David

On Wed, Sep 21, 2022 at 2:51 AM 'Josep Prat' via kafka-clients <
kafka-clie...@googlegroups.com> wrote:

> Hi David,
>
> Thanks for driving this. One question, should we include in the release
> notes the recently fixed CVE vulnerability? I understand this not being
> explicitly mentioned on the recently released versions to not cause an
> unintentional 0-day, but I think it could be mentioned for this release.
> What do you think?
>
> Best,
>
> On Wed, Sep 21, 2022 at 1:17 AM David Arthur 
> wrote:
>
>> Hello Kafka users, developers and client-developers,
>>
>> This is the second release candidate for Apache Kafka 3.3.0. Many new
>> features and bug fixes are included in this major release of Kafka. A
>> significant number of the issues in this release are related to KRaft,
>> which will be considered "production ready" as part of this release
>> (KIP-833)
>>
>> KRaft improvements:
>> * KIP-778: Online KRaft to KRaft Upgrades
>> * KIP-833: Mark KRaft as Production Ready
>> * KIP-835: Monitor Quorum health (many new KRaft metrics)
>> * KIP-836: Expose voter lag via kafka-metadata-quorum.sh
>> * KIP-841: Fenced replicas should not be allowed to join the ISR in KRaft
>> * KIP-859: Add Metadata Log Processing Error Related Metrics
>>
>> Other major improvements include:
>> * KIP-618: Exactly-Once Support for Source Connectors
>> * KIP-831: Add metric for log recovery progress
>> * KIP-827: Expose logdirs total and usable space via Kafka API
>> * KIP-834: Add ability to Pause / Resume KafkaStreams Topologies
>>
>> The full release notes are available here:
>> https://home.apache.org/~davidarthur/kafka-3.3.0-rc2/RELEASE_NOTES.html
>>
>> Please download, test and vote by Monday, Sep 26 at 5pm EDT
>>
>> Also, huge thanks to José for running the release so far. He has done
>> the vast majority of the work to prepare this rather large release :)
>>
>> -
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> https://kafka.apache.org/KEYS
>>
>> * Release artifacts to be voted upon (source and binary):
>> https://home.apache.org/~davidarthur/kafka-3.3.0-rc2/
>>
>> * Maven artifacts to be voted upon:
>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>>
>> * Javadoc: https://home.apache.org/~davidarthur/kafka-3.3.0-rc2/javadoc/
>>
>> * Tag to be voted upon (off 3.3 branch) is the 3.3.0 tag:
>> https://github.com/apache/kafka/releases/tag/3.3.0-rc2
>>
>> * Documentation:  https://kafka.apache.org/33/documentation.html
>>
>> * Protocol: https://kafka.apache.org/33/protocol.html
>>
>>
>>
>>
>> Successful Jenkins builds to follow in a future update to this email.
>>
>>
>> Thanks!
>> David Arthur
>>
>
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |
> 
>    
> *Aiven Deutschland GmbH*
> Immanuelkirchstraße 26, 10405 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B
>
> --
> You received this message because you are subscribed to the Google Groups
> "kafka-clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kafka-clients+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kafka-clients/CAOJ18G4DE9Q_DYyZTbDLF6J6MRj30WrCNj6njrYRV3SQeThs-w%40mail.gmail.com
> 
> .
>


-- 
-David

[jira] [Created] (KAFKA-14256) Update to Scala 2.13.9

[Matthew de Detrich (Jira)]

Matthew de Detrich created KAFKA-14256:
--

 Summary: Update to Scala 2.13.9
 Key: KAFKA-14256
 URL: https://issues.apache.org/jira/browse/KAFKA-14256
 Project: Kafka
  Issue Type: Task
Reporter: Matthew de Detrich






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-863: Reduce Fetcher#parseRecord() memory copy

[ShunKang Lin]

Hi Guozhang,

I've updated the "Motivation" section of the KIP, please take a look.

Thanks.
ShunKang

Guozhang Wang  于2022年9月21日周三 01:26写道：

> In this case, could you update the KIP to clarify the allocation savings
> more clearly in the "Motivation" section? Also you could mention that for
> user customizable serdes, if they could provide overwrites on the
> overloaded function that's also possible for optimize memory allocations.
>
> Guozhang
>
> On Tue, Sep 20, 2022 at 10:24 AM Guozhang Wang  wrote:
>
> > 1. Ack, thanks.
> > 2. Sounds good, thanks for clarifying.
> >
> > On Tue, Sep 20, 2022 at 9:50 AM ShunKang Lin 
> > wrote:
> >
> >> Hi Guozhang,
> >>
> >> Thanks for your comments!
> >>
> >> 1. We can reduce memory allocation if the key/value types happen to be
> >> ByteBuffer or String.
> >> 2. I would like to add `default ByteBuffer serializeToByteBuffer(String
> >> topic, Headers headers, T data)` in Serializer to reduce memory copy in
> >> `KafkaProducer#doSend(ProducerRecord, Callback)`, but this change is a
> bit
> >> big, I prefer to submit another one KIP to do the job.
> >>
> >> Thanks.
> >> ShunKang
> >>
> >> Guozhang Wang  于2022年9月20日周二 06:32写道：
> >>
> >> > Hello ShunKang,
> >> >
> >> > Thanks for filing the proposal, and sorry for the late reply!
> >> >
> >> > I looked over your KIP proposal and the PR, in general I think I agree
> >> that
> >> > adding an overloaded function with `ByteBuffer` param is beneficial,
> >> but I
> >> > have a meta question regarding it's impact on Kafka consumer: my
> >> > understanding from your PR is that, we can only save memory
> allocations
> >> if
> >> > the key/value types happen to be ByteBuffer as well, otherwise we
> would
> >> > still do the `return deserialize(topic, headers,
> Utils.toArray(data));`
> >> > from default impls unless the user customized deserializers is
> >> augmented to
> >> > handle ByteBuffer directly, right?
> >> >
> >> >
> >> > Guozhang
> >> >
> >> >
> >> >
> >> > On Sun, Aug 21, 2022 at 9:56 AM ShunKang Lin <
> linshunkang@gmail.com
> >> >
> >> > wrote:
> >> >
> >> > > Hi all，
> >> > >
> >> > > I'd like to start a discussion on KIP-863 which is Reduce
> >> > > Fetcher#parseRecord() memory copy. This KIP can reduce Kafka
> Consumer
> >> > > memory allocation by nearly 50% during fetch records.
> >> > >
> >> > > Please check
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035
> >> > > and https://github.com/apache/kafka/pull/12545 for more details.
> >> > >
> >> > > Any feedbacks and comments are welcomed.
> >> > >
> >> > > Thanks.
> >> > >
> >> >
> >> >
> >> > --
> >> > -- Guozhang
> >> >
> >>
> >
> >
> > --
> > -- Guozhang
> >
>
>
> --
> -- Guozhang
>

Re: [DISCUSS] KIP-863: Reduce Fetcher#parseRecord() memory copy

[Guozhang Wang]

Thanks ShunKang,

I made a few nit edits on the Motivation section as well. LGTM for me now.

On Thu, Sep 22, 2022 at 7:33 AM ShunKang Lin 
wrote:

> Hi Guozhang,
>
> I've updated the "Motivation" section of the KIP, please take a look.
>
> Thanks.
> ShunKang
>
> Guozhang Wang  于2022年9月21日周三 01:26写道：
>
> > In this case, could you update the KIP to clarify the allocation savings
> > more clearly in the "Motivation" section? Also you could mention that for
> > user customizable serdes, if they could provide overwrites on the
> > overloaded function that's also possible for optimize memory allocations.
> >
> > Guozhang
> >
> > On Tue, Sep 20, 2022 at 10:24 AM Guozhang Wang 
> wrote:
> >
> > > 1. Ack, thanks.
> > > 2. Sounds good, thanks for clarifying.
> > >
> > > On Tue, Sep 20, 2022 at 9:50 AM ShunKang Lin <
> linshunkang@gmail.com>
> > > wrote:
> > >
> > >> Hi Guozhang,
> > >>
> > >> Thanks for your comments!
> > >>
> > >> 1. We can reduce memory allocation if the key/value types happen to be
> > >> ByteBuffer or String.
> > >> 2. I would like to add `default ByteBuffer
> serializeToByteBuffer(String
> > >> topic, Headers headers, T data)` in Serializer to reduce memory copy
> in
> > >> `KafkaProducer#doSend(ProducerRecord, Callback)`, but this change is a
> > bit
> > >> big, I prefer to submit another one KIP to do the job.
> > >>
> > >> Thanks.
> > >> ShunKang
> > >>
> > >> Guozhang Wang  于2022年9月20日周二 06:32写道：
> > >>
> > >> > Hello ShunKang,
> > >> >
> > >> > Thanks for filing the proposal, and sorry for the late reply!
> > >> >
> > >> > I looked over your KIP proposal and the PR, in general I think I
> agree
> > >> that
> > >> > adding an overloaded function with `ByteBuffer` param is beneficial,
> > >> but I
> > >> > have a meta question regarding it's impact on Kafka consumer: my
> > >> > understanding from your PR is that, we can only save memory
> > allocations
> > >> if
> > >> > the key/value types happen to be ByteBuffer as well, otherwise we
> > would
> > >> > still do the `return deserialize(topic, headers,
> > Utils.toArray(data));`
> > >> > from default impls unless the user customized deserializers is
> > >> augmented to
> > >> > handle ByteBuffer directly, right?
> > >> >
> > >> >
> > >> > Guozhang
> > >> >
> > >> >
> > >> >
> > >> > On Sun, Aug 21, 2022 at 9:56 AM ShunKang Lin <
> > linshunkang@gmail.com
> > >> >
> > >> > wrote:
> > >> >
> > >> > > Hi all，
> > >> > >
> > >> > > I'd like to start a discussion on KIP-863 which is Reduce
> > >> > > Fetcher#parseRecord() memory copy. This KIP can reduce Kafka
> > Consumer
> > >> > > memory allocation by nearly 50% during fetch records.
> > >> > >
> > >> > > Please check
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035
> > >> > > and https://github.com/apache/kafka/pull/12545 for more details.
> > >> > >
> > >> > > Any feedbacks and comments are welcomed.
> > >> > >
> > >> > > Thanks.
> > >> > >
> > >> >
> > >> >
> > >> > --
> > >> > -- Guozhang
> > >> >
> > >>
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
> >
> > --
> > -- Guozhang
> >
>


-- 
-- Guozhang

Re: [DISCUSS] KIP-863: Reduce Fetcher#parseRecord() memory copy

[ShunKang Lin]

Hi Guozhang,

Thanks for your help! By the way, what should I do next?

Best,
ShunKang

Guozhang Wang  于2022年9月22日周四 23:21写道：

> Thanks ShunKang,
>
> I made a few nit edits on the Motivation section as well. LGTM for me now.
>
> On Thu, Sep 22, 2022 at 7:33 AM ShunKang Lin 
> wrote:
>
> > Hi Guozhang,
> >
> > I've updated the "Motivation" section of the KIP, please take a look.
> >
> > Thanks.
> > ShunKang
> >
> > Guozhang Wang  于2022年9月21日周三 01:26写道：
> >
> > > In this case, could you update the KIP to clarify the allocation
> savings
> > > more clearly in the "Motivation" section? Also you could mention that
> for
> > > user customizable serdes, if they could provide overwrites on the
> > > overloaded function that's also possible for optimize memory
> allocations.
> > >
> > > Guozhang
> > >
> > > On Tue, Sep 20, 2022 at 10:24 AM Guozhang Wang 
> > wrote:
> > >
> > > > 1. Ack, thanks.
> > > > 2. Sounds good, thanks for clarifying.
> > > >
> > > > On Tue, Sep 20, 2022 at 9:50 AM ShunKang Lin <
> > linshunkang@gmail.com>
> > > > wrote:
> > > >
> > > >> Hi Guozhang,
> > > >>
> > > >> Thanks for your comments!
> > > >>
> > > >> 1. We can reduce memory allocation if the key/value types happen to
> be
> > > >> ByteBuffer or String.
> > > >> 2. I would like to add `default ByteBuffer
> > serializeToByteBuffer(String
> > > >> topic, Headers headers, T data)` in Serializer to reduce memory copy
> > in
> > > >> `KafkaProducer#doSend(ProducerRecord, Callback)`, but this change
> is a
> > > bit
> > > >> big, I prefer to submit another one KIP to do the job.
> > > >>
> > > >> Thanks.
> > > >> ShunKang
> > > >>
> > > >> Guozhang Wang  于2022年9月20日周二 06:32写道：
> > > >>
> > > >> > Hello ShunKang,
> > > >> >
> > > >> > Thanks for filing the proposal, and sorry for the late reply!
> > > >> >
> > > >> > I looked over your KIP proposal and the PR, in general I think I
> > agree
> > > >> that
> > > >> > adding an overloaded function with `ByteBuffer` param is
> beneficial,
> > > >> but I
> > > >> > have a meta question regarding it's impact on Kafka consumer: my
> > > >> > understanding from your PR is that, we can only save memory
> > > allocations
> > > >> if
> > > >> > the key/value types happen to be ByteBuffer as well, otherwise we
> > > would
> > > >> > still do the `return deserialize(topic, headers,
> > > Utils.toArray(data));`
> > > >> > from default impls unless the user customized deserializers is
> > > >> augmented to
> > > >> > handle ByteBuffer directly, right?
> > > >> >
> > > >> >
> > > >> > Guozhang
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Sun, Aug 21, 2022 at 9:56 AM ShunKang Lin <
> > > linshunkang@gmail.com
> > > >> >
> > > >> > wrote:
> > > >> >
> > > >> > > Hi all，
> > > >> > >
> > > >> > > I'd like to start a discussion on KIP-863 which is Reduce
> > > >> > > Fetcher#parseRecord() memory copy. This KIP can reduce Kafka
> > > Consumer
> > > >> > > memory allocation by nearly 50% during fetch records.
> > > >> > >
> > > >> > > Please check
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035
> > > >> > > and https://github.com/apache/kafka/pull/12545 for more
> details.
> > > >> > >
> > > >> > > Any feedbacks and comments are welcomed.
> > > >> > >
> > > >> > > Thanks.
> > > >> > >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > -- Guozhang
> > > >> >
> > > >>
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
>
> --
> -- Guozhang
>

Re: [DISCUSS] KIP-863: Reduce Fetcher#parseRecord() memory copy

[Guozhang Wang]

Could you start a separate VOTE email thread calling for votes?

On Thu, Sep 22, 2022 at 9:19 AM ShunKang Lin 
wrote:

> Hi Guozhang,
>
> Thanks for your help! By the way, what should I do next?
>
> Best,
> ShunKang
>
> Guozhang Wang  于2022年9月22日周四 23:21写道：
>
> > Thanks ShunKang,
> >
> > I made a few nit edits on the Motivation section as well. LGTM for me
> now.
> >
> > On Thu, Sep 22, 2022 at 7:33 AM ShunKang Lin 
> > wrote:
> >
> > > Hi Guozhang,
> > >
> > > I've updated the "Motivation" section of the KIP, please take a look.
> > >
> > > Thanks.
> > > ShunKang
> > >
> > > Guozhang Wang  于2022年9月21日周三 01:26写道：
> > >
> > > > In this case, could you update the KIP to clarify the allocation
> > savings
> > > > more clearly in the "Motivation" section? Also you could mention that
> > for
> > > > user customizable serdes, if they could provide overwrites on the
> > > > overloaded function that's also possible for optimize memory
> > allocations.
> > > >
> > > > Guozhang
> > > >
> > > > On Tue, Sep 20, 2022 at 10:24 AM Guozhang Wang 
> > > wrote:
> > > >
> > > > > 1. Ack, thanks.
> > > > > 2. Sounds good, thanks for clarifying.
> > > > >
> > > > > On Tue, Sep 20, 2022 at 9:50 AM ShunKang Lin <
> > > linshunkang@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Hi Guozhang,
> > > > >>
> > > > >> Thanks for your comments!
> > > > >>
> > > > >> 1. We can reduce memory allocation if the key/value types happen
> to
> > be
> > > > >> ByteBuffer or String.
> > > > >> 2. I would like to add `default ByteBuffer
> > > serializeToByteBuffer(String
> > > > >> topic, Headers headers, T data)` in Serializer to reduce memory
> copy
> > > in
> > > > >> `KafkaProducer#doSend(ProducerRecord, Callback)`, but this change
> > is a
> > > > bit
> > > > >> big, I prefer to submit another one KIP to do the job.
> > > > >>
> > > > >> Thanks.
> > > > >> ShunKang
> > > > >>
> > > > >> Guozhang Wang  于2022年9月20日周二 06:32写道：
> > > > >>
> > > > >> > Hello ShunKang,
> > > > >> >
> > > > >> > Thanks for filing the proposal, and sorry for the late reply!
> > > > >> >
> > > > >> > I looked over your KIP proposal and the PR, in general I think I
> > > agree
> > > > >> that
> > > > >> > adding an overloaded function with `ByteBuffer` param is
> > beneficial,
> > > > >> but I
> > > > >> > have a meta question regarding it's impact on Kafka consumer: my
> > > > >> > understanding from your PR is that, we can only save memory
> > > > allocations
> > > > >> if
> > > > >> > the key/value types happen to be ByteBuffer as well, otherwise
> we
> > > > would
> > > > >> > still do the `return deserialize(topic, headers,
> > > > Utils.toArray(data));`
> > > > >> > from default impls unless the user customized deserializers is
> > > > >> augmented to
> > > > >> > handle ByteBuffer directly, right?
> > > > >> >
> > > > >> >
> > > > >> > Guozhang
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Sun, Aug 21, 2022 at 9:56 AM ShunKang Lin <
> > > > linshunkang@gmail.com
> > > > >> >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Hi all，
> > > > >> > >
> > > > >> > > I'd like to start a discussion on KIP-863 which is Reduce
> > > > >> > > Fetcher#parseRecord() memory copy. This KIP can reduce Kafka
> > > > Consumer
> > > > >> > > memory allocation by nearly 50% during fetch records.
> > > > >> > >
> > > > >> > > Please check
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035
> > > > >> > > and https://github.com/apache/kafka/pull/12545 for more
> > details.
> > > > >> > >
> > > > >> > > Any feedbacks and comments are welcomed.
> > > > >> > >
> > > > >> > > Thanks.
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >> > --
> > > > >> > -- Guozhang
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
> >
> > --
> > -- Guozhang
> >
>


-- 
-- Guozhang

[VOTE] KIP-863: Reduce Fetcher#parseRecord() memory copy

[ShunKang Lin]

Hi everyone,

I'd like to open the vote for KIP-863, which proposes to reduce memory
allocation and memory copying in Fetcher#parseRecord(TopicPartition,
RecordBatch, Record).

The proposal is here:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035

Thanks to all who reviewed the proposal, and thanks in advance for taking
the time to vote!

Best,
ShunKang

Re: [DISCUSS] KIP-863: Reduce Fetcher#parseRecord() memory copy

[ShunKang Lin]

Thanks Guozhang!

Best,
ShunKang

Guozhang Wang  于2022年9月23日周五 00:27写道：

> Could you start a separate VOTE email thread calling for votes?
>
> On Thu, Sep 22, 2022 at 9:19 AM ShunKang Lin 
> wrote:
>
> > Hi Guozhang,
> >
> > Thanks for your help! By the way, what should I do next?
> >
> > Best,
> > ShunKang
> >
> > Guozhang Wang  于2022年9月22日周四 23:21写道：
> >
> > > Thanks ShunKang,
> > >
> > > I made a few nit edits on the Motivation section as well. LGTM for me
> > now.
> > >
> > > On Thu, Sep 22, 2022 at 7:33 AM ShunKang Lin <
> linshunkang@gmail.com>
> > > wrote:
> > >
> > > > Hi Guozhang,
> > > >
> > > > I've updated the "Motivation" section of the KIP, please take a look.
> > > >
> > > > Thanks.
> > > > ShunKang
> > > >
> > > > Guozhang Wang  于2022年9月21日周三 01:26写道：
> > > >
> > > > > In this case, could you update the KIP to clarify the allocation
> > > savings
> > > > > more clearly in the "Motivation" section? Also you could mention
> that
> > > for
> > > > > user customizable serdes, if they could provide overwrites on the
> > > > > overloaded function that's also possible for optimize memory
> > > allocations.
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Tue, Sep 20, 2022 at 10:24 AM Guozhang Wang  >
> > > > wrote:
> > > > >
> > > > > > 1. Ack, thanks.
> > > > > > 2. Sounds good, thanks for clarifying.
> > > > > >
> > > > > > On Tue, Sep 20, 2022 at 9:50 AM ShunKang Lin <
> > > > linshunkang@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Hi Guozhang,
> > > > > >>
> > > > > >> Thanks for your comments!
> > > > > >>
> > > > > >> 1. We can reduce memory allocation if the key/value types happen
> > to
> > > be
> > > > > >> ByteBuffer or String.
> > > > > >> 2. I would like to add `default ByteBuffer
> > > > serializeToByteBuffer(String
> > > > > >> topic, Headers headers, T data)` in Serializer to reduce memory
> > copy
> > > > in
> > > > > >> `KafkaProducer#doSend(ProducerRecord, Callback)`, but this
> change
> > > is a
> > > > > bit
> > > > > >> big, I prefer to submit another one KIP to do the job.
> > > > > >>
> > > > > >> Thanks.
> > > > > >> ShunKang
> > > > > >>
> > > > > >> Guozhang Wang  于2022年9月20日周二 06:32写道：
> > > > > >>
> > > > > >> > Hello ShunKang,
> > > > > >> >
> > > > > >> > Thanks for filing the proposal, and sorry for the late reply!
> > > > > >> >
> > > > > >> > I looked over your KIP proposal and the PR, in general I
> think I
> > > > agree
> > > > > >> that
> > > > > >> > adding an overloaded function with `ByteBuffer` param is
> > > beneficial,
> > > > > >> but I
> > > > > >> > have a meta question regarding it's impact on Kafka consumer:
> my
> > > > > >> > understanding from your PR is that, we can only save memory
> > > > > allocations
> > > > > >> if
> > > > > >> > the key/value types happen to be ByteBuffer as well, otherwise
> > we
> > > > > would
> > > > > >> > still do the `return deserialize(topic, headers,
> > > > > Utils.toArray(data));`
> > > > > >> > from default impls unless the user customized deserializers is
> > > > > >> augmented to
> > > > > >> > handle ByteBuffer directly, right?
> > > > > >> >
> > > > > >> >
> > > > > >> > Guozhang
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > On Sun, Aug 21, 2022 at 9:56 AM ShunKang Lin <
> > > > > linshunkang@gmail.com
> > > > > >> >
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > Hi all，
> > > > > >> > >
> > > > > >> > > I'd like to start a discussion on KIP-863 which is Reduce
> > > > > >> > > Fetcher#parseRecord() memory copy. This KIP can reduce Kafka
> > > > > Consumer
> > > > > >> > > memory allocation by nearly 50% during fetch records.
> > > > > >> > >
> > > > > >> > > Please check
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035
> > > > > >> > > and https://github.com/apache/kafka/pull/12545 for more
> > > details.
> > > > > >> > >
> > > > > >> > > Any feedbacks and comments are welcomed.
> > > > > >> > >
> > > > > >> > > Thanks.
> > > > > >> > >
> > > > > >> >
> > > > > >> >
> > > > > >> > --
> > > > > >> > -- Guozhang
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -- Guozhang
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
>
> --
> -- Guozhang
>

Re: [DISCUSS] KIP-793: Sink Connectors: Support topic-mutating SMTs for async connectors (preCommit users)

[Randall Hauch]

Hi, Yash. Thanks for picking up this KIP and discussion.

The KIP includes this rejected alternative:

> 4. Update SinkTask.put in any way to pass the new information outside
> SinkRecord (e.g. a Map or a derived class)
>
>-
>
>Much more disruptive change without considerable pros
>
>
One advantage about doing this is that sink connector implementations can
more easily implement two different "put(...)" methods to handle running in
a variety of runtimes, without having to use try-catch logic around the
newer SinkRecord access methods. That latter logic can get quite ugly.

For example, the existing `put` method has this signature:

public abstract void put(Collection records);

If we added an overloaded method that passed in a map of the old
topic+partition for each record (and defined the absence of an entry as
having an unchanged topic and partition):

public void put(Collection records, Map updatedTopicPartitions) {
put(records);
}

then a `SinkTask` implementation that wants to use this new feature could
simply implement both methods:

public void put(Collection records) {
// Running in an older runtime, so no tracking of SMT-modified topic names
or partitions
put(records, Map.of());
}

public void put(Collection records, Map updatedTopicPartitions) {
// real logic here
}

This seems a lot easier than having to use try-catch logic, yet still
allows sink connectors to utilize the new functionality and still work with
older Connect runtimes.

WDYT?

Randall


On Thu, Sep 8, 2022 at 7:03 AM Yash Mayya  wrote:

> Hi all,
>
> I would like to (re)start a new discussion thread on KIP-793 (Kafka
> Connect) which proposes some additions to the public SinkRecord interface
> in order to support topic mutating SMTs for sink connectors that do their
> own offset tracking.
>
> Links:
>
> KIP:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191336830
>
> Older discussion thread:
> https://lists.apache.org/thread/00kcth6057jdcsyzgy1x8nb2s1cymy8h,
> https://lists.apache.org/thread/rzqkm0q5y5v3vdjhg8wqppxbkw7nyopj
>
> Jira: https://issues.apache.org/jira/browse/KAFKA-13431
>
>
> Thanks,
> Yash
>

Re: [VOTE] KIP-863: Reduce Fetcher#parseRecord() memory copy

[Guozhang Wang]

+1, thanks ShunKang.

Though its proposed motivation is on consumer fetcher's deserialization, I
think adding an overloaded method with ByteBuffer would help with other
serde places on the client side as well.


Guozhang

On Thu, Sep 22, 2022 at 9:41 AM ShunKang Lin 
wrote:

> Hi everyone,
>
> I'd like to open the vote for KIP-863, which proposes to reduce memory
> allocation and memory copying in Fetcher#parseRecord(TopicPartition,
> RecordBatch, Record).
>
> The proposal is here:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035
>
> Thanks to all who reviewed the proposal, and thanks in advance for taking
> the time to vote!
>
> Best,
> ShunKang
>


-- 
-- Guozhang

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1249

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-14257) Unexpected error INCONSISTENT_CLUSTER_ID in VOTE response

[jianbin.chen (Jira)]

jianbin.chen created KAFKA-14257:


 Summary: Unexpected error INCONSISTENT_CLUSTER_ID in VOTE response
 Key: KAFKA-14257
 URL: https://issues.apache.org/jira/browse/KAFKA-14257
 Project: Kafka
  Issue Type: Bug
  Components: kraft
Affects Versions: 3.2.3
Reporter: jianbin.chen


Please help me see why the error message is output indefinitely

broker1：
{code:java}
process.roles=broker,controller
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
node.id=1
listeners=PLAINTEXT://192.168.6.57:9092,CONTROLLER://192.168.6.57:9093
inter.broker.listener.name=PLAINTEXT
advertised.listeners=PLAINTEXT://192.168.6.57:9092
controller.listener.names=CONTROLLER
num.io.threads=8
num.network.threads=5
controller.quorum.voters=1@192.168.6.57:9093,2@192.168.6.56:9093,3@192.168.6.55:9093
log.dirs=/data01/kafka323-logs{code}
broker2
{code:java}
process.roles=broker,controller
controller.listener.names=CONTROLLER
num.io.threads=8
num.network.threads=5
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
node.id=2
listeners=PLAINTEXT://192.168.6.56:9092,CONTROLLER://192.168.6.56:9093
inter.broker.listener.name=PLAINTEXT
controller.quorum.voters=1@192.168.6.57:9093,2@192.168.6.56:9093,3@192.168.6.55:9093
log.dirs=/data01/kafka323-logs{code}
broker3
{code:java}
process.roles=broker,controller
controller.listener.names=CONTROLLER
num.io.threads=8
num.network.threads=5
node.id=3
listeners=PLAINTEXT://192.168.6.55:9092,CONTROLLER://192.168.6.55:9093
inter.broker.listener.name=PLAINTEXT
controller.quorum.voters=1@192.168.6.57:9093,2@192.168.6.56:9093,3@192.168.6.55:9093
log.dirs=/data01/kafka323-logs

{code}
error msg：
{code:java}
[2022-09-22 18:44:01,601] ERROR [RaftManager nodeId=2] Unexpected error 
INCONSISTENT_CLUSTER_ID in VOTE response: InboundResponse(correlationId=378, 
data=VoteResponseData(errorCode=104, topics=[]), sourceId=1) 
(org.apache.kafka.raft.KafkaRaftClient)
[2022-09-22 18:44:01,625] ERROR [RaftManager nodeId=2] Unexpected error 
INCONSISTENT_CLUSTER_ID in VOTE response: InboundResponse(correlationId=380, 
data=VoteResponseData(errorCode=104, topics=[]), sourceId=1) 
(org.apache.kafka.raft.KafkaRaftClient)
[2022-09-22 18:44:01,655] ERROR [RaftManager nodeId=2] Unexpected error 
INCONSISTENT_CLUSTER_ID in VOTE response: InboundResponse(correlationId=382, 
data=VoteResponseData(errorCode=104, topics=[]), sourceId=1) 
(org.apache.kafka.raft.KafkaRaftClient)
[2022-09-22 18:44:01,679] ERROR [RaftManager nodeId=2] Unexpected error 
INCONSISTENT_CLUSTER_ID in VOTE response: InboundResponse(correlationId=384, 
data=VoteResponseData(errorCode=104, topics=[]), sourceId=1) 
(org.apache.kafka.raft.KafkaRaftClient)
[2022-09-22 18:44:01,706] ERROR [RaftManager nodeId=2] Unexpected error 
INCONSISTENT_CLUSTER_ID in VOTE response: InboundResponse(correlationId=386, 
data=VoteResponseData(errorCode=104, topics=[]), sourceId=1) 
(org.apache.kafka.raft.KafkaRaftClient)
[2022-09-22 18:44:01,729] ERROR [RaftManager nodeId=2] Unexpected error 
INCONSISTENT_CLUSTER_ID in VOTE response: InboundResponse(correlationId=388, 
data=VoteResponseData(errorCode=104, topics=[]), sourceId=1) 
(org.apache.kafka.raft.KafkaRaftClient){code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-868 Metadata Transactions (new thread)

[deng ziming]

David,
Thanks for the feedback about #2 and #3, I'm OK with them.
I also mentioned the visibility in the MetadataShell in #1, do you have any
thoughts?

--
Best,
Ziming

On Wed, Sep 21, 2022 at 10:56 PM David Arthur  wrote:

> Ziming, thanks for the feedback! Let me know your thoughts on #2 and #3
>
> 1. Good idea. I consolidated all the details of record visibility into
> that section.
>
> 2. I'm not sure we can always know the number of records ahead of time
> for a transaction. One future use case is likely for the ZK data
> migration which will have an undetermined number of records. I would
> be okay with some short textual fields like "name" for the Begin
> record and "reason" for the Abort record. These could also be tagged
> fields if we don't want to always include them in the records.
>
> 3. The metadata records end up in org.apache.kafka.common.metadata, so
> maybe we can avoid Metadata in the name since it's kind of implicit.
> I'd be okay with [Begin|End|Abort]TransactionRecord.
>
> -David
>
> On Mon, Sep 19, 2022 at 10:58 PM deng ziming 
> wrote:
> >
> > Hello David,
> > Thanks for the KIP, certainly it makes sense, I left some minor
> questions.
> >
> > 1. In “Record Visibility” section you declare visibility in the
> controller, in “Broker Support” you mention visibility in the broker, we
> can put them together, and I think we can also describe visibility in the
> MetadataShell since it is also a public interface.
> >
> > 2. In “Public interfaces” section, I found that the “BeginMarkerRecord”
> has no fields, should we include some auxiliary attributes to help parse
> the transaction, for example, number of records in this transaction.
> >
> > 3. The record name seems vague, and we already have a
> `EndTransactionMarker` class in `org.apache.kafka.common.record`, how about
> `BeginMetadataTransactionRecord`?
> >
> > - -
> > Best,
> > Ziming
> >
> > > On Sep 10, 2022, at 1:13 AM, David Arthur 
> wrote:
> > >
> > > Starting a new thread to avoid issues with mail client threading.
> > >
> > > Original thread follows:
> > >
> > > Hey folks, I'd like to start a discussion on the idea of adding
> > > transactions in the KRaft controller. This will allow us to overcome
> > > the current limitation of atomic batch sizes in Raft which lets us do
> > > things like create topics with a huge number of partitions.
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-868+Metadata+Transactions
> > >
> > > Thanks!
> > >
> > > ---
> > >
> > > Colin McCabe said:
> > >
> > > Thanks for this KIP, David!
> > >
> > > In the "motivation" section, it might help to give a concrete example
> > > of an operation we want to be atomic. My favorite one is probably
> > > CreateTopics since it's easy to see that we want to create all of a
> > > topic or none of it, and a topic could be a potentially unbounded
> > > number of records (although hopefully people have reasonable create
> > > topic policy classes in place...)
> > >
> > > In "broker support", it would be good to clarify that we will buffer
> > > the records in the MetadataDelta and not publish a new MetadataImage
> > > until the transaction is over. This is an implementation detail, but
> > > it's a simple one and I think it will make it easier to understand how
> > > this works.
> > >
> > > In the "Raft Transactions" section of "Rejected Alternatives," I'd add
> > > that managing buffering in the Raft layer would be a lot less
> > > efficient than doing it in the controller / broker layer. We would end
> > > up accumulating big lists of records which would then have to be
> > > applied when the transaction completed, rather than building up a
> > > MetadataDelta (or updating the controller state) incrementally.
> > >
> > > Maybe we want to introduce the concept of "last stable offset" to be
> > > the last committed offset that is NOT part of an ongoing transaction?
> > > Just a nomenclature suggestion...
> > >
> > > best,
> > > Colin
> >
>
>
> --
> David Arthur
>