Re: MM2 offset translations not working

[Samuel Cantero]

This is working for 2.5.1 but broken for trunk (or master).

On Mon, Oct 5, 2020 at 5:19 PM Samuel Cantero  wrote:

> oh no, actually I was wrong. We have the same problem.
>
> On Mon, Oct 5, 2020 at 5:17 PM Samuel Cantero  wrote:
>
>> It seems the problem is just between kafka 0.10.x --> kafka 2.x. It is
>> working fine for kafka 2.x --> 2.x.
>>
>> On Sat, Oct 3, 2020 at 4:50 PM Samuel Cantero 
>> wrote:
>>
>>> Hi all,
>>>
>>> It looks like the offsets returned by the
>>> RemoteClusterUtils.translateOffsets are always the same as the
>>> source/primary cluster, i.e, there's no actual translation.
>>>
>>> The upstream offsets are fine but the downstream are not (it is just the
>>> same as the upstream one). After a quick look, it seems the bug might be on
>>> the writer of the source-cluster-alias.checkpoints.internal topic
>>> rather than on this lib that decodes it.
>>>
>>> I have double checked the log-end-offset of the topic on the target
>>> cluster via
>>> ./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list
>>> KAFKA-CONN-STR --time -1 --topic TOPIC
>>>
>>> This gives me confirmation that there's no actual translation.
>>>
>>> To double check it is not just my app, I enabled
>>> sync.group.offsets.enabled
>>> 
>>>  and
>>> I can find the same problem. So I think there's a bug unless I'm doing
>>> something wrong.
>>>
>>> I'm using the latest changes from trunk. Anyone hitting the same issue?
>>>
>>> Best,
>>>
>>

Re: MM2 offset translations not working

[Samuel Cantero]

oh no, actually I was wrong. We have the same problem.

On Mon, Oct 5, 2020 at 5:17 PM Samuel Cantero  wrote:

> It seems the problem is just between kafka 0.10.x --> kafka 2.x. It is
> working fine for kafka 2.x --> 2.x.
>
> On Sat, Oct 3, 2020 at 4:50 PM Samuel Cantero  wrote:
>
>> Hi all,
>>
>> It looks like the offsets returned by the
>> RemoteClusterUtils.translateOffsets are always the same as the
>> source/primary cluster, i.e, there's no actual translation.
>>
>> The upstream offsets are fine but the downstream are not (it is just the
>> same as the upstream one). After a quick look, it seems the bug might be on
>> the writer of the source-cluster-alias.checkpoints.internal topic rather
>> than on this lib that decodes it.
>>
>> I have double checked the log-end-offset of the topic on the target
>> cluster via
>> ./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list
>> KAFKA-CONN-STR --time -1 --topic TOPIC
>>
>> This gives me confirmation that there's no actual translation.
>>
>> To double check it is not just my app, I enabled
>> sync.group.offsets.enabled
>> 
>>  and
>> I can find the same problem. So I think there's a bug unless I'm doing
>> something wrong.
>>
>> I'm using the latest changes from trunk. Anyone hitting the same issue?
>>
>> Best,
>>
>

Re: MM2 offset translations not working

[Samuel Cantero]

It seems the problem is just between kafka 0.10.x --> kafka 2.x. It is
working fine for kafka 2.x --> 2.x.

On Sat, Oct 3, 2020 at 4:50 PM Samuel Cantero  wrote:

> Hi all,
>
> It looks like the offsets returned by the
> RemoteClusterUtils.translateOffsets are always the same as the
> source/primary cluster, i.e, there's no actual translation.
>
> The upstream offsets are fine but the downstream are not (it is just the
> same as the upstream one). After a quick look, it seems the bug might be on
> the writer of the source-cluster-alias.checkpoints.internal topic rather
> than on this lib that decodes it.
>
> I have double checked the log-end-offset of the topic on the target
> cluster via
> ./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list
> KAFKA-CONN-STR --time -1 --topic TOPIC
>
> This gives me confirmation that there's no actual translation.
>
> To double check it is not just my app, I enabled
> sync.group.offsets.enabled
> 
>  and
> I can find the same problem. So I think there's a bug unless I'm doing
> something wrong.
>
> I'm using the latest changes from trunk. Anyone hitting the same issue?
>
> Best,
>

Re: Getting the suggested partition assignment via the Admin client

[Stig Rohde Døssing]

Thank you.

You are right about the limitations of Kafka's own assignments. Thank you
for letting me know about Cruise Control.

I think given that Kafka's own assignment is useful enough to be exposed
via the local reassign command, it should also be exposed on the Admin
client. It is likely sufficient for our use case.

Will look into creating a KIP for this.

Den man. 5. okt. 2020 kl. 11.34 skrev Tom Bentley :

> Hi Stig,
>
> AdminUtils is not a public API, so you're right to not want to depend on
> it. Unfortunately the Admin client API doesn't offer functionality
> equivalent to `kafka-reassign-partitions.sh --generate`. That is, while you
> can get the broker to generate the assignments for you when you create a
> topic or add partitions, you can't say "reassign these existing topics as
> if they were new".
>
> One of the problems with the naive assignments generated by
> AdminUtils.assignReplicasToBrokers is that the assignments it returns can
> result in an unbalanced cluster, where some brokers have much higher load
> than others. Many people running Kafka use external tools, such as Cruise
> Control, to manage replica assignments so that the cluster is balanced.
>
> You could try to open a KIP to provide the functionality. I can see it
> might be useful for clusters where it doesn't matter if the brokers have
> similar load. The risk of adding this functionality is that people end up
> using it inappropriately and ending up with very unbalanced clusters in
> production.
>
> Kind regards,
>
> Tom
>
> On Mon, Oct 5, 2020 at 10:05 AM Stig Rohde Døssing  >
> wrote:
>
> > Hi,
> >
> > Kafka can recommend an assignment via the ReassignPartitionsCommand (with
> > --generate), and it is also possible to get this assignment directly by
> > calling AdminUtils.assignReplicasToBrokers.
> >
> > We would like our application to be able to get the suggested partition
> > assignment and apply it to a topic. We're currently doing this by calling
> > AdminUtils directly and passing the result to
> > Admin.alterPartitionReassignments.
> >
> > Is there a way to do this solely using the kafka-clients Admin client?
> We'd
> > like to avoid depending directly on the Kafka server jar.
> >
>

Re: started getting TopologyException: Invalid topology after moving to streams-2.5.1

[John Roesler]

Hi Pushkar,

Sorry for the trouble. Can you share your config and
topology description?

If I read your error message correctly, it says that your
app is configured with no source topics and no threads. Is
that accurate?

Thanks,
-John

On Mon, 2020-10-05 at 15:04 +0530, Pushkar Deole wrote:
> Hi All,
> 
> After moving to kafka-streams-2.5.1 version, one of our services started
> failing with below exception. Any idea what this is about and why it was
> passing with 2.5.0? Any changes made in 2.5.1 that is breaking this?
> 
> Exception in thread "main"
> org.springframework.context.ApplicationContextException: Failed to start
> bean 'defaultKafkaStreamsBuilder'; nested exception is
> org.springframework.kafka.KafkaException: Could not start stream: ; nested
> exception is org.apache.kafka.streams.errors.TopologyException: Invalid
> topology: Topology has no stream threads and no global threads, must
> subscribe to at least one source topic or global table.
> 
> at org.springframework.context.support.DefaultLifecycleProcessor.doStart(
> DefaultLifecycleProcessor.java:185
> ;)
> 
> at org.springframework.context.support.DefaultLifecycleProcessor.access$200(
> DefaultLifecycleProcessor.java:53
> ;)
> 
> at org.springframework.context.support
> .DefaultLifecycleProcessor$LifecycleGroup.start(
> DefaultLifecycleProcessor.java:360
> ;)
> 
> at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(
> DefaultLifecycleProcessor.java:158
> ;)
> 
> at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(
> DefaultLifecycleProcessor.java:122
> ;)
> 
> at org.springframework.context.support
> .AbstractApplicationContext.finishRefresh(
> AbstractApplicationContext.java:895
> ;)
> 
> at org.springframework.context.support.AbstractApplicationContext.refresh(
> AbstractApplicationContext.java:554
> ;)
> 
> at
> org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(
> ServletWebServerApplicationContext.java:143
> ;)
> 
> at org.springframework.boot.SpringApplication.refresh(
> SpringApplication.java:758 ;)
> 
> at org.springframework.boot.SpringApplication.refresh(
> SpringApplication.java:750 ;)
> 
> at org.springframework.boot.SpringApplication.refreshContext(
> SpringApplication.java:397 ;)
> 
> at org.springframework.boot.SpringApplication.run
> (
> SpringApplication.java:315 ;)
> 
> at org.springframework.boot.SpringApplication.run
> (
> SpringApplication.java:1237 ;)
> 
> at org.springframework.boot.SpringApplication.run
> (
> SpringApplication.java:1226 ;)
> 
> at com.avaya.analytics.AnalyticsStreamsDataPublisherApplication.main(
> AnalyticsStreamsDataPublisherApplication.java:31
> ;)
> 
> Caused by: org.springframework.kafka.KafkaException: Could not start
> stream: ; nested exception is
> org.apache.kafka.streams.errors.TopologyException: Invalid topology:
> Topology has no stream threads and no global threads, must subscribe to at
> least one source topic or global table.
> 
> at org.springframework.kafka.config.StreamsBuilderFactoryBean.start(
> StreamsBuilderFactoryBean.java:326
> ;)
> 
> at org.springframework.context.support.DefaultLifecycleProcessor.doStart(
> DefaultLifecycleProcessor.java:182
> ;)
> 
> ... 14 more
> 
> Caused by: org.apache.kafka.streams.errors.TopologyException: Invalid
> topology: Topology has no stream threads and no global threads, must
> subscribe to at least one source topic or global table.
> 
> at org.apache.kafka.streams.KafkaStreams.(KafkaStreams.java:728
> ;)
> 
> at org.apache.kafka.streams.KafkaStreams.(KafkaStreams.java:587
> ;)
> 
> at org.springframework.kafka.config.StreamsBuilderFactoryBean.start(
> StreamsBuilderFactoryBean.java:311
> ;)
> 
> ... 15 more

Re: Apache Kafka Windows

[Himanshu Shukla]

You can probably look at maxwell by zendesk as well for CDC. They too have
a very simple producer for Kafka.

On Mon, 5 Oct 2020, 04:42 Naveen D,  wrote:

> Hi Guys,
>
> We have below scenario to achieve via KAfka windows stand alone
>
> 1. Download apache kafka for windows
> 2.Start Zookeeper
> 3. Basic consumer /producer communication should work.
>
> 4. use debezium SQL connector, uses CDC sql  push data to azure data bricks
>
> We achieved step 1 -3,actually it was easy
>
> Step 4:
>
> We downloaded   debezium SQL connector jar files, not sure what to do
> after that .
>
> I have seen videos etc/searched on the web, everything points to a
> confluent cloud would be easy, but we want to go with traditional apache
> kafka on windows.
>
>
> Step by Step instruction: on Source/Sink Connect setup would be helpful
> even links are fine too.
>
> Regards,
> Navin
>
> On Sun, Oct 4, 2020 at 6:09 PM Naveen D  wrote:
>
> > Hi Guys,
> >
> > We have below scenario to achieve via KAfka windows stand alone
> >
> > 1. Download apache kafka for windows
> > 2.Start Zookeeper
> > 3. Basic consumer /producer communication should work.
> >
> > 4. use debezium SQL connector, uses CDC sql  push data to azure data
> bricks
> >
> > We achieved step 1 -3,actually it was easy
> >
> > Step 4:
> >
> > We downloaded   debezium SQL connector jar files, not sure what to do
> > after that .
> >
> > I have seen videos etc/searched on the web, everything points to a
> > confluent cloud would be easy, but we want to go with traditional apache
> > kafka on windows.
> >
> >
> > Step by Step instruction: on Source/Sink Connect setup would be helpful
> > even links are fine too.
> >
> > Regards,
> > Navin
> >
> >
>
> --
> Regards
> Navin.d
>

KSQL should not silently overwrite Schemas in Registry with same name

[Rainer Schamm]

Hi all

See existing issue:

https://github.com/confluentinc/ksql/issues/2427 



Does anyone know if there is a work around for this? This problem is driving us 
crazy.
We really don't want to allow KSQL streams to constantly change the associated 
schema.

This is particularly painful for us because for some reason KSQL is not dealing 
correctly with date types.
We are creating the schema like this:

…
"type": {
"type": "int",
"connect.version": 1,
"connect.name": "org.apache.kafka.connect.data.Date",
"logicalType": "date"
  }
...

And KSQL is changing the type to this:

…
"type": {
"type": „int"
  }
...

And then the sink connector fails to insert the int value because the db 
expects a date value.

Any help would be greatly appreciated.

Regards
Rainer

Re: Getting the suggested partition assignment via the Admin client

[Tom Bentley]

Hi Stig,

AdminUtils is not a public API, so you're right to not want to depend on
it. Unfortunately the Admin client API doesn't offer functionality
equivalent to `kafka-reassign-partitions.sh --generate`. That is, while you
can get the broker to generate the assignments for you when you create a
topic or add partitions, you can't say "reassign these existing topics as
if they were new".

One of the problems with the naive assignments generated by
AdminUtils.assignReplicasToBrokers is that the assignments it returns can
result in an unbalanced cluster, where some brokers have much higher load
than others. Many people running Kafka use external tools, such as Cruise
Control, to manage replica assignments so that the cluster is balanced.

You could try to open a KIP to provide the functionality. I can see it
might be useful for clusters where it doesn't matter if the brokers have
similar load. The risk of adding this functionality is that people end up
using it inappropriately and ending up with very unbalanced clusters in
production.

Kind regards,

Tom

On Mon, Oct 5, 2020 at 10:05 AM Stig Rohde Døssing 
wrote:

> Hi,
>
> Kafka can recommend an assignment via the ReassignPartitionsCommand (with
> --generate), and it is also possible to get this assignment directly by
> calling AdminUtils.assignReplicasToBrokers.
>
> We would like our application to be able to get the suggested partition
> assignment and apply it to a topic. We're currently doing this by calling
> AdminUtils directly and passing the result to
> Admin.alterPartitionReassignments.
>
> Is there a way to do this solely using the kafka-clients Admin client? We'd
> like to avoid depending directly on the Kafka server jar.
>

started getting TopologyException: Invalid topology after moving to streams-2.5.1

[Pushkar Deole]

Hi All,

After moving to kafka-streams-2.5.1 version, one of our services started
failing with below exception. Any idea what this is about and why it was
passing with 2.5.0? Any changes made in 2.5.1 that is breaking this?

Exception in thread "main"
org.springframework.context.ApplicationContextException: Failed to start
bean 'defaultKafkaStreamsBuilder'; nested exception is
org.springframework.kafka.KafkaException: Could not start stream: ; nested
exception is org.apache.kafka.streams.errors.TopologyException: Invalid
topology: Topology has no stream threads and no global threads, must
subscribe to at least one source topic or global table.

at org.springframework.context.support.DefaultLifecycleProcessor.doStart(
DefaultLifecycleProcessor.java:185
)

at org.springframework.context.support.DefaultLifecycleProcessor.access$200(
DefaultLifecycleProcessor.java:53
)

at org.springframework.context.support
.DefaultLifecycleProcessor$LifecycleGroup.start(
DefaultLifecycleProcessor.java:360
)

at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(
DefaultLifecycleProcessor.java:158
)

at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(
DefaultLifecycleProcessor.java:122
)

at org.springframework.context.support
.AbstractApplicationContext.finishRefresh(
AbstractApplicationContext.java:895
)

at org.springframework.context.support.AbstractApplicationContext.refresh(
AbstractApplicationContext.java:554
)

at
org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(
ServletWebServerApplicationContext.java:143
)

at org.springframework.boot.SpringApplication.refresh(
SpringApplication.java:758 )

at org.springframework.boot.SpringApplication.refresh(
SpringApplication.java:750 )

at org.springframework.boot.SpringApplication.refreshContext(
SpringApplication.java:397 )

at org.springframework.boot.SpringApplication.run
(
SpringApplication.java:315 )

at org.springframework.boot.SpringApplication.run
(
SpringApplication.java:1237 )

at org.springframework.boot.SpringApplication.run
(
SpringApplication.java:1226 )

at com.avaya.analytics.AnalyticsStreamsDataPublisherApplication.main(
AnalyticsStreamsDataPublisherApplication.java:31
)

Caused by: org.springframework.kafka.KafkaException: Could not start
stream: ; nested exception is
org.apache.kafka.streams.errors.TopologyException: Invalid topology:
Topology has no stream threads and no global threads, must subscribe to at
least one source topic or global table.

at org.springframework.kafka.config.StreamsBuilderFactoryBean.start(
StreamsBuilderFactoryBean.java:326
)

at org.springframework.context.support.DefaultLifecycleProcessor.doStart(
DefaultLifecycleProcessor.java:182
)

... 14 more

Caused by: org.apache.kafka.streams.errors.TopologyException: Invalid
topology: Topology has no stream threads and no global threads, must
subscribe to at least one source topic or global table.

at org.apache.kafka.streams.KafkaStreams.(KafkaStreams.java:728
)

at org.apache.kafka.streams.KafkaStreams.(KafkaStreams.java:587
)

at org.springframework.kafka.config.StreamsBuilderFactoryBean.start(
StreamsBuilderFactoryBean.java:311
)

... 15 more

Getting the suggested partition assignment via the Admin client

[Stig Rohde Døssing]

Hi,

Kafka can recommend an assignment via the ReassignPartitionsCommand (with
--generate), and it is also possible to get this assignment directly by
calling AdminUtils.assignReplicasToBrokers.

We would like our application to be able to get the suggested partition
assignment and apply it to a topic. We're currently doing this by calling
AdminUtils directly and passing the result to
Admin.alterPartitionReassignments.

Is there a way to do this solely using the kafka-clients Admin client? We'd
like to avoid depending directly on the Kafka server jar.