Re: Where is replication factor stored?

[James Cheng]

> On Oct 16, 2015, at 1:19 PM, Guozhang Wang  wrote:
>
> Replication factor is stored as topic configs that are introduced since
> 0.8.1, you can find it in the wiki you mentioned.
>

Ah, I didn't notice the /config section.

But it still doesn't show the replication factor.

[zk: localhost:2181(CONNECTED) 3] get /config/topics/__consumer_offsets
{"version":1,"config":{"segment.bytes":"104857600","cleanup.policy":"compact"}}
cZxid = 0xc017a
ctime = Wed Aug 05 22:48:12 UTC 2015
mZxid = 0xc017a
mtime = Wed Aug 05 22:48:12 UTC 2015
pZxid = 0xc017a
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 79
numChildren = 0

I tried that for a number of different topics, and none of them have it.

-James


> Guozhang
>
> On Fri, Oct 16, 2015 at 12:33 PM, James Cheng  wrote:
>
>> Hi,
>>
>> Where is the replication factor for a topic stored? It isn't listed at
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper.
>> But the kafka-topics --describe command returns something. Where is it
>> finding that?
>>
>> Thanks,
>> -James
>>
>>
>> 
>>
>> This email and any attachments may contain confidential and privileged
>> material for the sole use of the intended recipient. Any review, copying,
>> or distribution of this email (or any attachments) by others is prohibited.
>> If you are not the intended recipient, please contact the sender
>> immediately and permanently delete this email and any attachments. No
>> employee or agent of TiVo Inc. is authorized to conclude any binding
>> agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
>> Inc. may only be made by a signed written agreement.
>>
>
>
>
> --
> -- Guozhang




This email and any attachments may contain confidential and privileged material 
for the sole use of the intended recipient. Any review, copying, or 
distribution of this email (or any attachments) by others is prohibited. If you 
are not the intended recipient, please contact the sender immediately and 
permanently delete this email and any attachments. No employee or agent of TiVo 
Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by 
email. Binding agreements with TiVo Inc. may only be made by a signed written 
agreement.

Re: Where is replication factor stored?

[Guozhang Wang]

The replica list can be from at /brokers/topics//
partitions//state

Guozhang

On Fri, Oct 16, 2015 at 2:06 PM, Edward Ribeiro 
wrote:

> Umm... the replica *assignment* gets stored under /brokers/topics/
> as "partitions" field, no? Under /brokers/topics//partitions there
> is a znode for each partition number with a 'state' as a sub-znode, right?
>
> James, by doing via zkCli.sh:
>
> get /brokers/topics/
>
> You get a json like below:
>
>
> {"version":1,"partitions":{"8":[1],"4":[1],"9":[1],"5":[1],"6":[1],"1":[1],"0":[1],"2":[1],"7":[1],"3":[1]}}
>
> As you can see above, the replica list has only one element ([1]) for each
> partition because ReplicationFactor = 1. You can infer the replication
> factor by getting the *size* of this list for any element.
>
>
> On Fri, Oct 16, 2015 at 5:51 PM, Gwen Shapira  wrote:
>
> > We don't store the replication factor per-se. When the topic is created,
> we
> > use the replication factor to generate replica-assignment, and the
> replica
> > assignment gets stored in ZK under:
> /brokers/topics//partitions/...
> >
> > This is what gets modified when we re-assign replicas.
> >
> > Hope this helps.
> >
> > Gwen
> >
> > On Fri, Oct 16, 2015 at 12:33 PM, James Cheng  wrote:
> >
> > > Hi,
> > >
> > > Where is the replication factor for a topic stored? It isn't listed at
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper
> > .
> > > But the kafka-topics --describe command returns something. Where is it
> > > finding that?
> > >
> > > Thanks,
> > > -James
> > >
> > >
> > > 
> > >
> > > This email and any attachments may contain confidential and privileged
> > > material for the sole use of the intended recipient. Any review,
> copying,
> > > or distribution of this email (or any attachments) by others is
> > prohibited.
> > > If you are not the intended recipient, please contact the sender
> > > immediately and permanently delete this email and any attachments. No
> > > employee or agent of TiVo Inc. is authorized to conclude any binding
> > > agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> > > Inc. may only be made by a signed written agreement.
> > >
> >
>



-- 
-- Guozhang

Where is replication factor stored?

[James Cheng]

Hi,

Where is the replication factor for a topic stored? It isn't listed at 
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper.
 But the kafka-topics --describe command returns something. Where is it finding 
that?

Thanks,
-James




This email and any attachments may contain confidential and privileged material 
for the sole use of the intended recipient. Any review, copying, or 
distribution of this email (or any attachments) by others is prohibited. If you 
are not the intended recipient, please contact the sender immediately and 
permanently delete this email and any attachments. No employee or agent of TiVo 
Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by 
email. Binding agreements with TiVo Inc. may only be made by a signed written 
agreement.

Re: Where is replication factor stored?

[Gwen Shapira]

We don't store the replication factor per-se. When the topic is created, we
use the replication factor to generate replica-assignment, and the replica
assignment gets stored in ZK under: /brokers/topics//partitions/...

This is what gets modified when we re-assign replicas.

Hope this helps.

Gwen

On Fri, Oct 16, 2015 at 12:33 PM, James Cheng  wrote:

> Hi,
>
> Where is the replication factor for a topic stored? It isn't listed at
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper.
> But the kafka-topics --describe command returns something. Where is it
> finding that?
>
> Thanks,
> -James
>
>
> 
>
> This email and any attachments may contain confidential and privileged
> material for the sole use of the intended recipient. Any review, copying,
> or distribution of this email (or any attachments) by others is prohibited.
> If you are not the intended recipient, please contact the sender
> immediately and permanently delete this email and any attachments. No
> employee or agent of TiVo Inc. is authorized to conclude any binding
> agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> Inc. may only be made by a signed written agreement.
>

Re: Where is replication factor stored?

[Edward Ribeiro]

Umm... the replica *assignment* gets stored under /brokers/topics/
as "partitions" field, no? Under /brokers/topics//partitions there
is a znode for each partition number with a 'state' as a sub-znode, right?

James, by doing via zkCli.sh:

get /brokers/topics/

You get a json like below:

{"version":1,"partitions":{"8":[1],"4":[1],"9":[1],"5":[1],"6":[1],"1":[1],"0":[1],"2":[1],"7":[1],"3":[1]}}

As you can see above, the replica list has only one element ([1]) for each
partition because ReplicationFactor = 1. You can infer the replication
factor by getting the *size* of this list for any element.


On Fri, Oct 16, 2015 at 5:51 PM, Gwen Shapira  wrote:

> We don't store the replication factor per-se. When the topic is created, we
> use the replication factor to generate replica-assignment, and the replica
> assignment gets stored in ZK under: /brokers/topics//partitions/...
>
> This is what gets modified when we re-assign replicas.
>
> Hope this helps.
>
> Gwen
>
> On Fri, Oct 16, 2015 at 12:33 PM, James Cheng  wrote:
>
> > Hi,
> >
> > Where is the replication factor for a topic stored? It isn't listed at
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper
> .
> > But the kafka-topics --describe command returns something. Where is it
> > finding that?
> >
> > Thanks,
> > -James
> >
> >
> > 
> >
> > This email and any attachments may contain confidential and privileged
> > material for the sole use of the intended recipient. Any review, copying,
> > or distribution of this email (or any attachments) by others is
> prohibited.
> > If you are not the intended recipient, please contact the sender
> > immediately and permanently delete this email and any attachments. No
> > employee or agent of TiVo Inc. is authorized to conclude any binding
> > agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> > Inc. may only be made by a signed written agreement.
> >
>

Re: Where is replication factor stored?

[Todd Palino]

Actually, be very careful with this. There are two different things stored
in Zookeeper, and depending on what you're interested in you want to make
sure you're looking at the right one.

If you want to know the replica assignment - that is, what brokers a given
partition is assigned to - you need to look at the following path:
/brokers/topics/(topic)

The data of that znode is JSON formatted, and the 'partitions' key is a
dictionary where they key is a string representation of the partition
number (i.e. it's not 0, it's "0") and the value is a list of the replicas
that the partition is assigned to. It's worth noting that this replica list
is also the order in which the preferred leader is selected (the first
replica in the list that is in sync will be selected as the leader).

If you want to know what the current in sync replicas are - that is, out of
the assigned replica list, which ones are currently considered to be in
sync, you need to look at the following path:
/brokers/topics/(topic)/partitions/(partition number)/state

The data of that znode is also JSON formatted, and the 'isr' key is a list
of the replicas that are currently considered to be in sync. The important
distinction here is that this list can be shorter than the actual assigned
replica list (from the znode above) if not all of the replicas are in sync.
The state znode also has a 'leader' key which holds the broker ID of the
replica that is currently the leader for that partition.

-Todd


On Fri, Oct 16, 2015 at 5:25 PM, Edward Ribeiro 
wrote:

> Hey, Guozhang,
>
> On Fri, Oct 16, 2015 at 6:20 PM, Guozhang Wang  wrote:
>
> > The replica list can be from at /brokers/topics//
> > partitions//state
> >
>
> Nice, good to know. Thanks! :)
>
> Regards,
> Edward​
>

Re: Where is replication factor stored?

[Guozhang Wang]

Gwen is right, I was wrong :P

kafka-topics --describe reads the ZK path that Gwen mentioned to return the
replica list, which gives you the idea about replication factor as well.

Thanks,
Guozhang

On Fri, Oct 16, 2015 at 1:51 PM, Gwen Shapira  wrote:

> We don't store the replication factor per-se. When the topic is created, we
> use the replication factor to generate replica-assignment, and the replica
> assignment gets stored in ZK under: /brokers/topics//partitions/...
>
> This is what gets modified when we re-assign replicas.
>
> Hope this helps.
>
> Gwen
>
> On Fri, Oct 16, 2015 at 12:33 PM, James Cheng  wrote:
>
> > Hi,
> >
> > Where is the replication factor for a topic stored? It isn't listed at
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper
> .
> > But the kafka-topics --describe command returns something. Where is it
> > finding that?
> >
> > Thanks,
> > -James
> >
> >
> > 
> >
> > This email and any attachments may contain confidential and privileged
> > material for the sole use of the intended recipient. Any review, copying,
> > or distribution of this email (or any attachments) by others is
> prohibited.
> > If you are not the intended recipient, please contact the sender
> > immediately and permanently delete this email and any attachments. No
> > employee or agent of TiVo Inc. is authorized to conclude any binding
> > agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> > Inc. may only be made by a signed written agreement.
> >
>



-- 
-- Guozhang

Re: Where is replication factor stored?

[Edward Ribeiro]

Umm... Reading the TopicCommand code
https://github.com/apache/kafka/blob/362613347371e9d74184e900ab80ba230940a5c8/core/src/main/scala/kafka/admin/TopicCommand.scala#L192
, it looks like the replication factor (for --describe option, at least) is
calculated by:

1) retrieving the "partitions" map from /brokers/topics/, where
the key is the partition number and the value is a list of replicas and

2) getting the first key-value pair (head) from the map retrieved above,
and from this kv pair then get the size of the replica list (._2.size), no?

When I got the data of get /config/topics/testTopic it came as
{"version":1,"config":{}}, that is, the config was empty (I am running from
a stale -- about two weeks -- trunk).

​What I am missing? :)​

Cheers,
Ed

On Fri, Oct 16, 2015 at 5:19 PM, Guozhang Wang  wrote:

> Replication factor is stored as topic configs that are introduced since
> 0.8.1, you can find it in the wiki you mentioned.
>
> Guozhang
>
> On Fri, Oct 16, 2015 at 12:33 PM, James Cheng  wrote:
>
> > Hi,
> >
> > Where is the replication factor for a topic stored? It isn't listed at
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper
> .
> > But the kafka-topics --describe command returns something. Where is it
> > finding that?
> >
> > Thanks,
> > -James
> >
> >
> > 
> >
> > This email and any attachments may contain confidential and privileged
> > material for the sole use of the intended recipient. Any review, copying,
> > or distribution of this email (or any attachments) by others is
> prohibited.
> > If you are not the intended recipient, please contact the sender
> > immediately and permanently delete this email and any attachments. No
> > employee or agent of TiVo Inc. is authorized to conclude any binding
> > agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> > Inc. may only be made by a signed written agreement.
> >
>
>
>
> --
> -- Guozhang
>

Re: Where is replication factor stored?

[Edward Ribeiro]

Hey, Guozhang,

On Fri, Oct 16, 2015 at 6:20 PM, Guozhang Wang  wrote:

> The replica list can be from at /brokers/topics//
> partitions//state
>

Nice, good to know. Thanks! :)

Regards,
Edward​

Re: Where is replication factor stored?

[Todd Palino]

Sorry, I forgot the tl;dr on that :)

If you want to know the replication factor for a given partition, you want
to check the length of the replica list in the /brokers/topic/(topic) data
for that partition. Note that all the partitions for a topic do not have to
have the same replication factor (you can use partition reassignment to
change it). But if they are not all the same, some of the tooling will
break (such as altering the partition count for the topic).

-Todd


On Fri, Oct 16, 2015 at 5:39 PM, Todd Palino  wrote:

> Actually, be very careful with this. There are two different things stored
> in Zookeeper, and depending on what you're interested in you want to make
> sure you're looking at the right one.
>
> If you want to know the replica assignment - that is, what brokers a given
> partition is assigned to - you need to look at the following path:
> /brokers/topics/(topic)
>
> The data of that znode is JSON formatted, and the 'partitions' key is a
> dictionary where they key is a string representation of the partition
> number (i.e. it's not 0, it's "0") and the value is a list of the replicas
> that the partition is assigned to. It's worth noting that this replica list
> is also the order in which the preferred leader is selected (the first
> replica in the list that is in sync will be selected as the leader).
>
> If you want to know what the current in sync replicas are - that is, out
> of the assigned replica list, which ones are currently considered to be in
> sync, you need to look at the following path:
> /brokers/topics/(topic)/partitions/(partition number)/state
>
> The data of that znode is also JSON formatted, and the 'isr' key is a list
> of the replicas that are currently considered to be in sync. The important
> distinction here is that this list can be shorter than the actual assigned
> replica list (from the znode above) if not all of the replicas are in sync.
> The state znode also has a 'leader' key which holds the broker ID of the
> replica that is currently the leader for that partition.
>
> -Todd
>
>
> On Fri, Oct 16, 2015 at 5:25 PM, Edward Ribeiro 
> wrote:
>
>> Hey, Guozhang,
>>
>> On Fri, Oct 16, 2015 at 6:20 PM, Guozhang Wang 
>> wrote:
>>
>> > The replica list can be from at /brokers/topics//
>> > partitions//state
>> >
>>
>> Nice, good to know. Thanks! :)
>>
>> Regards,
>> Edward​
>>
>
>

Re: Where is replication factor stored?

[Guozhang Wang]

Replication factor is stored as topic configs that are introduced since
0.8.1, you can find it in the wiki you mentioned.

Guozhang

On Fri, Oct 16, 2015 at 12:33 PM, James Cheng  wrote:

> Hi,
>
> Where is the replication factor for a topic stored? It isn't listed at
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper.
> But the kafka-topics --describe command returns something. Where is it
> finding that?
>
> Thanks,
> -James
>
>
> 
>
> This email and any attachments may contain confidential and privileged
> material for the sole use of the intended recipient. Any review, copying,
> or distribution of this email (or any attachments) by others is prohibited.
> If you are not the intended recipient, please contact the sender
> immediately and permanently delete this email and any attachments. No
> employee or agent of TiVo Inc. is authorized to conclude any binding
> agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> Inc. may only be made by a signed written agreement.
>



-- 
-- Guozhang

Re: ("offsets.storage","kafka") not working with Consumer API

[Stevo Slavić]

Hello Kiran,

Check how many brokers you have in the cluster. Consumer offsets topic
requires by default at least 3. In dev environment you could lower
replication factor for that topic (see broker config options).

Kind regards,
Stevo Slavic.

On Fri, Oct 16, 2015, 07:31 Kiran Singh  wrote:

> Hello
>
> I am trying to use ConsumerConnector API. But if i used
> ("offsets.storage","kafka") property
> "createJavaConsumerConnector" function will hang. And I removed this, API
> work fine.
>
> I am using kafka_2.10-0.8.2.2.jar
>
> Is there any issue or I am missing any thing.
>
> Thanks
> Kiran Singh
>

Kafka Synchronous Performance issue

[sugumar analysis]

Hi All,

We are using Kafka-Net(C# SDK) for our kafka messaging system, we tested
Producer to produce message to Kafka with Synchronous and Asynchronous
mode. But In Asynchronous mode it performs well than Synchronous.

Synchronous mode takes 2 minutes to produce 1000 message where as
Asynchronous mode takes 2 seconds to produce 1000 messages.

Is there any limitation using Asynchronous mode?

(*Like message order problem, Message lose*)

Can we use Asynchronous mode in Production environment ?

Please any one suggest us...

Thanks,
Sugumar

Re: ("offsets.storage","kafka") not working with Consumer API

[Kiran Singh]

Hi Stevo

I have 3 broker in my kafka cluster.

On Fri, Oct 16, 2015 at 12:21 PM, Stevo Slavić  wrote:

> Hello Kiran,
>
> Check how many brokers you have in the cluster. Consumer offsets topic
> requires by default at least 3. In dev environment you could lower
> replication factor for that topic (see broker config options).
>
> Kind regards,
> Stevo Slavic.
>
> On Fri, Oct 16, 2015, 07:31 Kiran Singh  wrote:
>
> > Hello
> >
> > I am trying to use ConsumerConnector API. But if i used
> > ("offsets.storage","kafka") property
> > "createJavaConsumerConnector" function will hang. And I removed this, API
> > work fine.
> >
> > I am using kafka_2.10-0.8.2.2.jar
> >
> > Is there any issue or I am missing any thing.
> >
> > Thanks
> > Kiran Singh
> >
>

Re: Getting error code 15 while connect to the offset manager.

[Kiran Singh]

Found following error in server.log file under log folder.

ERROR Closing socket for /x.x.x.x because of error (kafka.network.Processor)
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at kafka.utils.Utils$.read(Utils.scala:380)
at
kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
at kafka.network.Processor.read(SocketServer.scala:444)
at kafka.network.Processor.run(SocketServer.scala:340)
at java.lang.Thread.run(Thread.java:745)


On Fri, Oct 16, 2015 at 11:35 AM, Kiran Singh  wrote:

> Hi Mayuresh
>
> Can you please share how and from where i can collect logs.
>
> On Fri, Oct 16, 2015 at 9:53 AM, Kiran Singh 
> wrote:
>
>> Hi Mayuresh
>>
>> I am following link
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka
>>
>> for implementing commit and fech offset API and i am using
>> kafka_2.10-0.8.2.2 jar file.
>>
>> On Thu, Oct 15, 2015 at 9:40 PM, Mayuresh Gharat <
>> gharatmayures...@gmail.com> wrote:
>>
>>> This means that there is no coordinator available for the consumer group.
>>> Can you send the log. Also which consumer are you using. Is there a wiki
>>> you are referring to, to implement the commit and fetch offset API?
>>>
>>> Thanks,
>>>
>>> Mayuresh
>>>
>>> On Thu, Oct 15, 2015 at 7:17 AM, Kiran Singh 
>>> wrote:
>>>
>>> > Hi
>>> >
>>> > I am trying to implement fetch and commit offset API. But i am getting
>>> > error code 15 "ConsumerCoordinatorNotAvailableCode" while connecting to
>>> > kafka.
>>> >
>>> > Can any one explain this error?
>>> >
>>> > Thanks and Regards
>>> > Kiran Singh
>>> >
>>>
>>>
>>>
>>> --
>>> -Regards,
>>> Mayuresh R. Gharat
>>> (862) 250-7125
>>>
>>
>>
>

Re: Getting error code 15 while connect to the offset manager.

[Kiran Singh]

Hi Mayuresh

Can you please share how and from where i can collect logs.

On Fri, Oct 16, 2015 at 9:53 AM, Kiran Singh  wrote:

> Hi Mayuresh
>
> I am following link
>
> https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka
>
> for implementing commit and fech offset API and i am using
> kafka_2.10-0.8.2.2 jar file.
>
> On Thu, Oct 15, 2015 at 9:40 PM, Mayuresh Gharat <
> gharatmayures...@gmail.com> wrote:
>
>> This means that there is no coordinator available for the consumer group.
>> Can you send the log. Also which consumer are you using. Is there a wiki
>> you are referring to, to implement the commit and fetch offset API?
>>
>> Thanks,
>>
>> Mayuresh
>>
>> On Thu, Oct 15, 2015 at 7:17 AM, Kiran Singh 
>> wrote:
>>
>> > Hi
>> >
>> > I am trying to implement fetch and commit offset API. But i am getting
>> > error code 15 "ConsumerCoordinatorNotAvailableCode" while connecting to
>> > kafka.
>> >
>> > Can any one explain this error?
>> >
>> > Thanks and Regards
>> > Kiran Singh
>> >
>>
>>
>>
>> --
>> -Regards,
>> Mayuresh R. Gharat
>> (862) 250-7125
>>
>
>

Re: How to get group list in kafka?

[Kiran Singh]

By when we can get 0.9.0.0 version of kafka?

On Fri, Oct 16, 2015 at 10:07 AM, Kiran Singh  wrote:

> Hi Ashish
>
> I am using kafka_2.10-0.8.2.2.jar
>
> On Fri, Oct 16, 2015 at 9:25 AM, Ashish Singh  wrote:
>
>> Hello Kiran,
>>
>> kafka-consumer-groups.sh will be released as part of 0.9.0.0.
>>
>> What version of Kafka are you using? Info on ConsumerGroups is persisted
>> in
>> Zookeeper and you can write a simple tool like here
>>  to get this info.
>> ​
>>
>> On Thu, Oct 15, 2015 at 7:14 AM, Kiran Singh 
>> wrote:
>>
>> > Hello
>> >
>> > I am trying to list all group in my kafka but not able to get any info.
>> > There is one JIRA https://issues.apache.org/jira/browse/KAFKA-1476,
>> which
>> > talk about "kafka-consumer-groups.sh" script. Using this we can list all
>> > groups. But i am not able to find this script in my kafka tar file.
>> >
>> > In which version this script was introduced ?
>> >
>> > Thanks and Regards
>> >
>>
>>
>>
>> --
>>
>> Regards,
>> Ashish
>>
>
>

Spring xd stream - kafka

[bigdata hadoop]

Hi Kafka Users
I am new to streams in spring xd and trying to consumer data from a topic.
I downloaded the spring xd single node on my laptop and ran the following
command:

1. stream create myKafka --definition "kafka
--zkconnect=localhost:2181 --topic=mytopic | log" --deploy

I did not see the log where it deployed. I checked under/tmp and
/xd/logs I did not find any. Where do I need to look for log.

Saw in C:\tmp\kafka-logs\myKafka-kafka-offsets-0 but was not sure how
to read this log. I did more and filename but it was unreadable file.

2. stream create myKafkaFile --definition "kafka
--zkconnect=localhost:2181 --topic=mytopic | file" --deploy

Tried this statement then got the numbers format of file under /tmp/xd/output.

Tried adding

stream create myKafkaFile --definition "kafka
--zkconnect=localhost:2181 --topic=mytopic | file
--outputType=text/plain" --deploy

and

stream create myKafkaFile --definition "kafka
--zkconnect=localhost:2181 --topic=msi_client_topic | file --mode=ref
--dir=/tmp/xd/output --pattern=*.txt" --deploy

and --binary=false

but was throwing error and did not deploy my stream.

I have JSON String in topic, Can any one please help with this to read
as a JSON string instead of numbers.

Thanks

Sowjanya

kafka consumer shell scripts (and kafkacat) with respect to JSON pretty printing

[David Luu]

I was wondering, do the kafka consumer shell scripts (high and low level
ones) and kafkacat do any pre-processing of the topic messages before
outputting to stdout or does it just output "as is" in the format the
message originally came in through kafka from the producer?

Meaning pretty printed JSON produced is consumed as pretty printed JSON,
line delimited JSON blobs (not pretty printed) is consumed the same way.

I'm asking this as I notice some topics I'm consuming with the shell
scripts (and kafkacat) are pretty printed and some not. So just wanted to
confirm this while also checking with the developers of the topic producers
on whether they are pretty printing on their end or not.

In general, I'm assuming as best practice, it's better not to pretty print
the JSON as a producer to save on message size for network transmission and
file storage by kafka, since those extra newlines and spaces/tabs add up
over time?