Fix slow processing rate in Kafka streams

[Nirmal Das]

Hi All,

My streams application is not processing more than 350 records/sec on a
high load of 3milliom records produced every 2-3 minutes.

My scenarios are as below -
I am on Kafka and streams version of 3.5.1 .
My key-value pair is in protobuf format .
I do a groupbykey followed by TimeWindow of 10 mins with grace period of 6
hours . It is then followed by a aggregate function which stores the first
and last offset of the record along with partition for that message key.

Am I doing something wrong? Am I doing something anti-pattern which is
throttling the system ? How can I improve this?

Regards,
Dev Lover

Re: production ready for zookeeper to kraft migration

[David Arthur]

Matthieu,

There are a few things to look out for during the migration.

1) Migration not starting. This is pretty much always going to be due to
mis-configuration. Controller KRaftMigrationDriver logs should reveal what
its waiting for
2) Migration of metadata fails. If some poison record in ZK causes the
metadata migration to fail, you'll see stacktraces and ERRORs in the
controller logs. The brokers will also not see an active controller and
won't be getting metadata updates.
3) Something in the dual-write path fails. If the write back to ZK fails
during dual-write mode (with ZK brokers or KRaft brokers), you'll see an
increasing lag (ZkWriteBehindLag metric)

So, in general: lack of an active controller, metadata not being propagated
(e.g., ISRs not getting updated), errors in the logs.

I'll try to get this written up in the docs soon.

Cheers,
David


On Thu, Apr 4, 2024 at 12:56 PM Matthieu Patou  wrote:

> Hey Luke,
>
> Thank you for the update.
> Out of curiosity if the migration is not working what are the symptoms ? is
> it just that the controller won't show that the migration is complete ? or
> could the controller claim (wrongfully) that the migration is complete when
> it's not ?
>
> Best.
> Matthieu
>
> On Wed, Apr 3, 2024 at 4:53 PM Luke Chen  wrote:
>
> > Hi Matthieu,
> >
> > Yes, the ZK migrating to KRaft feature is already GA in v3.6.0.
> > Sorry, we forgot to update the document in the Kafka-site repo.
> > I've filed a PR for it: https://github.com/apache/kafka-site/pull/594
> >
> > Thanks.
> > Luke
> >
> > On Thu, Apr 4, 2024 at 6:14 AM Matthieu Patou 
> wrote:
> >
> > > I looked at the notes for 3.7.x and the migration from ZK to Kraft is
> > still
> > > not marked as production ready.
> > >
> > > I'm wondering what are the issues that people could be facing during
> the
> > > migration.
> > >
> > > If 4.0 is still planned to be the full removal for ZK, is there a plan
> > for
> > > something after 3.7 to mark ZK migration as production ready ?
> > >
> > > Best.
> > >
> > > Matthieu.
> > >
> >
>


-- 
-David

Re: outerJoin confusion

[Matthias J. Sax]

Yeah, that is some quirk of KS runtime...

There is some internal config (for perf reasons) that delays emitting 
results... An alternative to advancing wall-clock time would be to set 
this internal config to zero, to disable the delay.


Maybe we should disable this config when topology test driver is used 
automatically... It's not the first time it did came up.


I opened a PR for it: https://github.com/apache/kafka/pull/15660


-Matthias



On 4/3/24 3:52 PM, Chad Preisler wrote:

Changing the code to this...

assertTrue(outputTopic.isEmpty());
 testDriver.advanceWallClockTime(Duration.ofMillis(2001));
 leftTopic.pipeInput("1", "test string 3", 4002L);
 testDriver.advanceWallClockTime(Duration.ofMillis(2001));
 leftTopic.pipeInput("1", "test string 4", 6004L);

Did appear to fix the issue. Output:

First join result:
Key: 1 Value: test string 1, null
Second join result:
Key: 1 Value: test string 2, null
Key: 1 Value: test string 3, null

Still a little strange that it works the first time without advancing the
wall clock.

On Wed, Apr 3, 2024 at 5:05 PM Shashwat Pandey <
shashwat.pandey@gmail.com> wrote:


I believe you need to advanceWallClockTime

https://kafka.apache.org/24/javadoc/org/apache/kafka/streams/TopologyTestDriver.html#advanceWallClockTime-java.time.Duration-


Regards,
Shashwat Pandey


On Wed, Apr 3, 2024 at 5:05 PM Chad Preisler 
wrote:


Seems like there is some issue with the TopologyTestDriver. I am able to
run the same stream against Kakfa and I'm getting the output I expect.

I'd

appreciate it if someone could confirm that there is an issue with the
TopologyTestDriver. If there is, any suggestions on how to test this type
of join?

On Wed, Apr 3, 2024 at 2:46 PM Chad Preisler 
wrote:


Hello,

I'm confused about the outerJoin and when records are produced with the
following code.

Topology buildTopology() {
 var builder = new StreamsBuilder();
 var leftStream = builder.stream("leftSecondsTopic",
Consumed.with(Serdes.String(), Serdes.String()));
 var rightStream = builder.stream("rightSecondsTopic",
Consumed.with(Serdes.String(), Serdes.String()));

 leftStream.outerJoin(rightStream, (left, right) -> left + ", "

+

right,

JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMillis(2000)))
 .to("outputTopicSeconds");

 return builder.build();
 }

Here is the test driver.

@Test
 public void testSecondsJoinDoesNotWork() {
 Properties props = new Properties();
 props.put(StreamsConfig.APPLICATION_ID_CONFIG, "testSeconds");
 props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.StringSerde.class);
 props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.StringSerde.class);
 var app = new KafkaStreamJoinTest();
 var serializer = new StringSerializer();

 try(var testDriver = new

TopologyTestDriver(app.buildTopology(),

 props)) {
 var leftTopic =
testDriver.createInputTopic("leftSecondsTopic",
 serializer, serializer, Instant.ofEpochMilli(0L),
Duration.ZERO);
 leftTopic.pipeInput("1", "test string 1", 0L);
 leftTopic.pipeInput("1", "test string 2", 2001L);

 var outputTopic =
testDriver.createOutputTopic("outputTopicSeconds",
 new StringDeserializer(), new

StringDeserializer());

 assertFalse(outputTopic.isEmpty());
 System.out.println("First join result:");
 outputTopic.readKeyValuesToList()
 .forEach((keyValue)->
 System.out.println("Key: " + keyValue.key + "

Value:

"

+ keyValue.value));

 assertTrue(outputTopic.isEmpty());

 leftTopic.pipeInput("1", "test string 3", 4002L);
 leftTopic.pipeInput("1", "test string 4", 6004L);

 System.out.println("Second join result:");
 outputTopic.readKeyValuesToList()
 .forEach((keyValue)->
 System.out.println("Key: " + keyValue.key + "

Value:

"

+ keyValue.value));

 }
 }

Here is the output:
First join result:
Key: 1 Value: test string 1, null
Second join result:

I would have expected a join to happen with "test string 2" and "test
string 3" being output with a null right value. Why didn't that happen?

Re: production ready for zookeeper to kraft migration

[Matthieu Patou]

Hey Luke,

Thank you for the update.
Out of curiosity if the migration is not working what are the symptoms ? is
it just that the controller won't show that the migration is complete ? or
could the controller claim (wrongfully) that the migration is complete when
it's not ?

Best.
Matthieu

On Wed, Apr 3, 2024 at 4:53 PM Luke Chen  wrote:

> Hi Matthieu,
>
> Yes, the ZK migrating to KRaft feature is already GA in v3.6.0.
> Sorry, we forgot to update the document in the Kafka-site repo.
> I've filed a PR for it: https://github.com/apache/kafka-site/pull/594
>
> Thanks.
> Luke
>
> On Thu, Apr 4, 2024 at 6:14 AM Matthieu Patou  wrote:
>
> > I looked at the notes for 3.7.x and the migration from ZK to Kraft is
> still
> > not marked as production ready.
> >
> > I'm wondering what are the issues that people could be facing during the
> > migration.
> >
> > If 4.0 is still planned to be the full removal for ZK, is there a plan
> for
> > something after 3.7 to mark ZK migration as production ready ?
> >
> > Best.
> >
> > Matthieu.
> >
>

Re: [EXTERNAL] Re: Kafka Streams 3.5.1 based app seems to get stalled

[Matthias J. Sax]

Glad to hear that the config change helped.

For continuous rebalancing, it might be expected for KS, as KS uses the 
protocol in advanced ways. If you see log lines saying "follow up 
rebalance requested" than there is nothing to worry about, and the group 
is stable.


If you see "no follow up rebalance request" but you actually do get 
rebalances, it would indicate an issue.



-Matthias

On 4/3/24 2:24 PM, Venkatesh Nagarajan wrote:

Apologies for the delay, Bruno. Thank you so much for the excellent link and 
for your inputs! Also, I would like to thank Matthias and yourself for the 
guidance on the stalling issue in the Kafka Streams client. After restoring the 
default value for the  METADATA_MAX_AGE_CONFIG, I haven’t seen the issue 
happening. Heavy rebalancing (as mentioned before) continues to happen. I will 
refer to the link which mentions about certain metrics which can give insights.

Thank you very much.

Kind regards,
Venkatesh

From: Bruno Cadonna 
Date: Friday, 22 March 2024 at 9:53 PM
To: users@kafka.apache.org 
Subject: Re: [EXTERNAL] Re: Kafka Streams 3.5.1 based app seems to get stalled
Hi Venkatesh,

The 1 core 1 stream thread recommendation is just s starting point. You
need to set the number of stream thread as it fits you by monitoring the
app.

Maybe this blog post might be interesting for you:
https://www.responsive.dev/blog/a-size-for-every-stream

Best,
Bruno


On 3/19/24 4:14 AM, Venkatesh Nagarajan wrote:

Thanks very much for sharing the links and for your important inputs, Bruno!


We recommend to use as many stream threads as cores on the compute node where 
the Kafka Streams client is run. How many Kafka Streams tasks do you have to 
distribute over the clients?


We use 1vCPU (probably 1 core) per Kafka Streams Client (ECS Task). Each 
client/ECS Task runs 10 streaming threads and the CPU utilisation is just 4% on 
an average. It increases when transient errors occur as they require retries 
and threads to be replaced.

We run a maximum of 6 clients/ECS Tasks when the offset lags are high. The 
input topics have 60 partitions each and this matches (total number of 
clients/ECS Tasks i.e. 6) * ( Streaming threads per client/ECS task i.e.10).

With the 1 streaming thread per core approach, we will need 60 vCPUs/cores. As 
I mentioned above, we have observed 10 threads using just 4% of 1 vCPU/core on 
an average. It may be difficult to justify provisioning more cores as it will 
be expensive and because Kafka Streams recovers from failures in acquiring 
locks.

Please feel free to correct me and/or share your thoughts.

Thank you.

Kind regards,
Venkatesh

From: Bruno Cadonna 
Date: Friday, 15 March 2024 at 8:47 PM
To: users@kafka.apache.org 
Subject: Re: [EXTERNAL] Re: Kafka Streams 3.5.1 based app seems to get stalled
Hi Venkatesh,

As you discovered, in Kafka Streams 3.5.1 there is no stop-the-world
rebalancing.

Static group member is helpful when Kafka Streams clients are restarted
as you pointed out.


ERROR org.apache.kafka.streams.processor.internals.StandbyTask -

stream-thread [-StreamThread-1] standby-task [1_32] Failed to
acquire lock while closing the state store for STANDBY task

This error (and some others about lock acquisition) happens when a
stream thread wants to lock the state directory for a task but the
stream thread on the same Kafka Streams client has not releases the lock
yet. And yes, Kafka Streams handles them.

30 and 60 stream threads is a lot for one Kafka Streams client. We
recommend to use as many stream threads as cores on the compute node
where the Kafka Streams client is run. How many Kafka Streams tasks do
you have to distribute over the clients?


Would you consider this level of rebalancing to be normal?


The rate of rebalance events seems high indeed. However, the log
messages you posted in one of your last e-mails are normal during a
rebalance and they have nothing to do with METADATA_MAX_AGE_CONFIG.

I do not know the metric SumOffsetLag. Judging from a quick search on
the internet, I think it is a MSK specific metric.
https://repost.aws/questions/QUthnU3gycT-qj3Mtb-ekmRA/msk-metric-sumoffsetlag-how-it-works>
Under the link you can also find some other metrics that you can use.

The following talk might help you debugging your rebalance issues:

Re: [VOTE] 3.6.2 RC2

[Manikumar]

Thanks all for voting.

I'm now closing the vote. The vote passes with
- 3 +1 bindings votes from Divij Vaidya, Justine, and Manikumar
- 3 +1 non-binding votes from Andrew, Jakub, and  Lianet
- 0 -1 votes

I'll go ahead and finish the release process.


Thanks,

On Thu, Apr 4, 2024 at 8:53 PM Manikumar  wrote:

> +1 (binding) from my side also.
>
> - verified the signatures, artifacts
> - ran the tests on the source archive
> - verified the quickstarts
>
>
> Thanks
>
> On Thu, Apr 4, 2024 at 7:46 PM Justine Olshan 
> wrote:
>
>> Thanks for clarifying Manikumar!
>>
>> +1 (binding) from me
>>
>> Justine
>>
>> On Thu, Apr 4, 2024 at 3:19 AM Divij Vaidya 
>> wrote:
>>
>> > Hi Manikumar
>> >
>> > I verified the following:
>> > - all non-minor commits in the branch are captured in release notes
>> > - signature on the artifact match the public signature of Manikumar
>> > - basic topic creation & produce / consumer works with JDK8 + ARM +
>> Kafka
>> > 3.6.2 + Scala 2.12 + ZK + compression (zstd)
>> >
>> > Things look good to me. We don't need another RC for fixing docs.
>> >
>> > +1 (binding) from me.
>> >
>> > --
>> > Divij Vaidya
>> >
>> >
>> >
>> > On Thu, Apr 4, 2024 at 10:04 AM Manikumar 
>> > wrote:
>> >
>> > > Hi Justine,
>> > >
>> > > Thanks for catching this. looks like we have missed updating
>> > > `docs/documentation.html` in kafka repo during 3.5 and 3.6 release.
>> > >
>> > > I will make sure to use the correct version when updating docs for
>> 3.6.2
>> > > release.
>> > > I will also update 3.5 and 3.6 branches with the correct heading and
>> also
>> > > update the release wiki.
>> > >
>> > > >what was expected: "fullDotVersion": "3.6.2-SNAPSHOT"
>> > > we will remove the "-SNAPSHOT" suffix while updating the website
>> docs. we
>> > > may need to automate this in the release script.
>> > >
>> > >
>> > > [1]
>> https://github.com/apache/kafka/blob/3.6/docs/documentation.html#L36
>> > > [2]
>> https://github.com/apache/kafka/blob/3.5/docs/documentation.html#L36
>> > >
>> > >
>> > > Thanks,
>> > >
>> > > On Thu, Apr 4, 2024 at 3:50 AM Justine Olshan
>> > > > > >
>> > > wrote:
>> > >
>> > > > Thanks for clarifying!
>> > > > I took a look at the documentation.html file in there, and it said
>> 3.4.
>> > > Is
>> > > > that expected?
>> > > >
>> > > > There are some files that request fullDot version and that seemed
>> > closer
>> > > to
>> > > > what was expected: "fullDotVersion": "3.6.2-SNAPSHOT"
>> > > > The upgrade.html file also looked ok.
>> > > >
>> > > > Thanks for running the release and answering my questions!
>> > > > Justine
>> > > >
>> > > > On Wed, Apr 3, 2024 at 10:21 AM Manikumar <
>> manikumar.re...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Hi Justine,
>> > > > >
>> > > > > Yes, it is intended. For bug fix releases website docs will be
>> > updated
>> > > > > during the final release process.
>> > > > > We can verify the site-docs artifacts here:
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://home.apache.org/~manikumar/kafka-3.6.2-rc2/kafka_2.12-3.6.2-site-docs.tgz
>> > > > > These site-docs artifacts will be used to update website docs.
>> > > > >
>> > > > >
>> > > > > Thanks,
>> > > > >
>> > > > > On Wed, Apr 3, 2024 at 10:30 PM Justine Olshan
>> > > > > 
>> > > > > wrote:
>> > > > >
>> > > > > > Hi Manikumar,
>> > > > > >
>> > > > > > I've verified the keys, scanned the artifacts, and other docs.
>> > > > > > I built from source and ran with a ZK cluster (since I saw that
>> we
>> > > > > updated
>> > > > > > ZK version in this release)
>> > > > > > I ran a few tests on this cluster.
>> > > > > >
>> > > > > > I also ran the 2.12 binary.
>> > > > > >
>> > > > > > I noticed the docs link (
>> > > > https://kafka.apache.org/36/documentation.html)
>> > > > > > mentions 3.6.1 as the latest. Is that intended?
>> > > > > > I will give my final vote when we figure this out.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Justine
>> > > > > >
>> > > > > > On Wed, Apr 3, 2024 at 7:25 AM Lianet M. 
>> > wrote:
>> > > > > >
>> > > > > > > Hi Manikumar, I did the following checks:
>> > > > > > >
>> > > > > > > - downloaded and built from src
>> > > > > > > - ran all unit test and integration test for clients
>> > > > > > > - ran quickstart with Kraft mode
>> > > > > > > - ran simple workloads with the console consumer/producer
>> > > > > > > - checked all links
>> > > > > > >
>> > > > > > > All looks good to me with this.
>> > > > > > >
>> > > > > > > +1 (non-binding)
>> > > > > > >
>> > > > > > > Thanks!
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > On Wed, Apr 3, 2024, 2:19 a.m. Manikumar <
>> maniku...@apache.org>
>> > > > wrote:
>> > > > > > >
>> > > > > > > > Gentle reminder. Please download, test and vote for the
>> > release.
>> > > > > > > >
>> > > > > > > > Thanks,
>> > > > > > > >
>> > > > > > > > On Fri, Mar 29, 2024 at 4:57 PM Manikumar <
>> > maniku...@apache.org>
>> > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Hi All,
>> > > > > > > > >
>> > > > > 

Re: [VOTE] 3.6.2 RC2

[Manikumar]

+1 (binding) from my side also.

- verified the signatures, artifacts
- ran the tests on the source archive
- verified the quickstarts


Thanks

On Thu, Apr 4, 2024 at 7:46 PM Justine Olshan 
wrote:

> Thanks for clarifying Manikumar!
>
> +1 (binding) from me
>
> Justine
>
> On Thu, Apr 4, 2024 at 3:19 AM Divij Vaidya 
> wrote:
>
> > Hi Manikumar
> >
> > I verified the following:
> > - all non-minor commits in the branch are captured in release notes
> > - signature on the artifact match the public signature of Manikumar
> > - basic topic creation & produce / consumer works with JDK8 + ARM + Kafka
> > 3.6.2 + Scala 2.12 + ZK + compression (zstd)
> >
> > Things look good to me. We don't need another RC for fixing docs.
> >
> > +1 (binding) from me.
> >
> > --
> > Divij Vaidya
> >
> >
> >
> > On Thu, Apr 4, 2024 at 10:04 AM Manikumar 
> > wrote:
> >
> > > Hi Justine,
> > >
> > > Thanks for catching this. looks like we have missed updating
> > > `docs/documentation.html` in kafka repo during 3.5 and 3.6 release.
> > >
> > > I will make sure to use the correct version when updating docs for
> 3.6.2
> > > release.
> > > I will also update 3.5 and 3.6 branches with the correct heading and
> also
> > > update the release wiki.
> > >
> > > >what was expected: "fullDotVersion": "3.6.2-SNAPSHOT"
> > > we will remove the "-SNAPSHOT" suffix while updating the website docs.
> we
> > > may need to automate this in the release script.
> > >
> > >
> > > [1]
> https://github.com/apache/kafka/blob/3.6/docs/documentation.html#L36
> > > [2]
> https://github.com/apache/kafka/blob/3.5/docs/documentation.html#L36
> > >
> > >
> > > Thanks,
> > >
> > > On Thu, Apr 4, 2024 at 3:50 AM Justine Olshan
> >  > > >
> > > wrote:
> > >
> > > > Thanks for clarifying!
> > > > I took a look at the documentation.html file in there, and it said
> 3.4.
> > > Is
> > > > that expected?
> > > >
> > > > There are some files that request fullDot version and that seemed
> > closer
> > > to
> > > > what was expected: "fullDotVersion": "3.6.2-SNAPSHOT"
> > > > The upgrade.html file also looked ok.
> > > >
> > > > Thanks for running the release and answering my questions!
> > > > Justine
> > > >
> > > > On Wed, Apr 3, 2024 at 10:21 AM Manikumar  >
> > > > wrote:
> > > >
> > > > > Hi Justine,
> > > > >
> > > > > Yes, it is intended. For bug fix releases website docs will be
> > updated
> > > > > during the final release process.
> > > > > We can verify the site-docs artifacts here:
> > > > >
> > > > >
> > > >
> > >
> >
> https://home.apache.org/~manikumar/kafka-3.6.2-rc2/kafka_2.12-3.6.2-site-docs.tgz
> > > > > These site-docs artifacts will be used to update website docs.
> > > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > On Wed, Apr 3, 2024 at 10:30 PM Justine Olshan
> > > > > 
> > > > > wrote:
> > > > >
> > > > > > Hi Manikumar,
> > > > > >
> > > > > > I've verified the keys, scanned the artifacts, and other docs.
> > > > > > I built from source and ran with a ZK cluster (since I saw that
> we
> > > > > updated
> > > > > > ZK version in this release)
> > > > > > I ran a few tests on this cluster.
> > > > > >
> > > > > > I also ran the 2.12 binary.
> > > > > >
> > > > > > I noticed the docs link (
> > > > https://kafka.apache.org/36/documentation.html)
> > > > > > mentions 3.6.1 as the latest. Is that intended?
> > > > > > I will give my final vote when we figure this out.
> > > > > >
> > > > > > Thanks,
> > > > > > Justine
> > > > > >
> > > > > > On Wed, Apr 3, 2024 at 7:25 AM Lianet M. 
> > wrote:
> > > > > >
> > > > > > > Hi Manikumar, I did the following checks:
> > > > > > >
> > > > > > > - downloaded and built from src
> > > > > > > - ran all unit test and integration test for clients
> > > > > > > - ran quickstart with Kraft mode
> > > > > > > - ran simple workloads with the console consumer/producer
> > > > > > > - checked all links
> > > > > > >
> > > > > > > All looks good to me with this.
> > > > > > >
> > > > > > > +1 (non-binding)
> > > > > > >
> > > > > > > Thanks!
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Apr 3, 2024, 2:19 a.m. Manikumar  >
> > > > wrote:
> > > > > > >
> > > > > > > > Gentle reminder. Please download, test and vote for the
> > release.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > On Fri, Mar 29, 2024 at 4:57 PM Manikumar <
> > maniku...@apache.org>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi All,
> > > > > > > > >
> > > > > > > > > System test runs are green. There were 13 test failures in
> > the
> > > > > first
> > > > > > > run.
> > > > > > > > > All the failed tests passed in the second run.
> > > > > > > > >
> > > > > > > > > System test results:
> > > > > > > > >
> > > > https://gist.github.com/omkreddy/17d23d3eb36ef840011f2494d65bbd4f
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > On Thu, Mar 28, 2024 at 3:21 PM Manikumar <
> > > maniku...@apache.org>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Hello 

Re: Kafka capabilities

[Kafka Life]

Dear Kafka experts , Could anyone having this data share the details please

On Wed, Apr 3, 2024 at 3:42 PM Kafka Life  wrote:

> Hi Kafka users
>
> Does any one have a document or ppt that showcases the capabilities of
> Kafka along with any cost management capability?
> i have a customer who is still using IBM MQM and rabbit MQ. I want the
> client to consider kafka for messaging and data streaming. I wanted to seek
> your expert help if you have any document or ppt i can propose it as an
> example. could you pls help.
>
> thanks and regards
> KrisG
>