Frequent SIGSEGV Errors on Kafka Brokers
We are running Kafka 2.0.1 and lately the brokers have been crashing with several different SIGSEGV errors as listed below. I was wondering if anybody has encountered similar errors or might have advice on what the problem could be? Thanks, David 1. # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x7f883195b06d, pid=5999, tid=0x7f7d9c1ee700 # # JRE version: Java(TM) SE Runtime Environment (8.0_192-b12) (build 1.8.0_192-b12) # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.192-b12 mixed mode linux-amd64 compressed oops) # Problematic frame: # J 21026 C2 org.apache.kafka.common.network.Selector.pollSelectionKeys(Ljava/util/Set;ZJ)V (543 bytes) @ 0x7f883195b06d [0x7f883195ab40+0x52d] # # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # Stack: [0x7f7d9c0ee000,0x7f7d9c1ef000], sp=0x7f7d9c1ed830, free space=1022k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) J 21026 C2 org.apache.kafka.common.network.Selector.pollSelectionKeys(Ljava/util/Set;ZJ)V (543 bytes) @ 0x7f883195b06d [0x7f883195ab40+0x52d] J 19626 C2 org.apache.kafka.common.network.Selector.poll(J)V (396 bytes) @ 0x7f883121f42c [0x7f883121eca0+0x78c] J 19419 C2 kafka.network.Processor.poll()V (66 bytes) @ 0x7f883001afa8 [0x7f883001af60+0x48] j kafka.network.Processor.run()V+31 j java.lang.Thread.run()V+11 v ~StubRoutines::call_stub V [libjvm.so+0x685d2b] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0xddb V [libjvm.so+0x6835f3] JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x263 V [libjvm.so+0x683bb7] JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x47 V [libjvm.so+0x6efb3c] thread_entry(JavaThread*, Thread*)+0x6c V [libjvm.so+0xa77a5b] JavaThread::thread_main_inner()+0xdb V [libjvm.so+0xa77d61] JavaThread::run()+0x2d1 V [libjvm.so+0x90a8b2] java_start(Thread*)+0x102 C [libpthread.so.0+0x7de5] start_thread+0xc5 2. # # JRE version: Java(TM) SE Runtime Environment (8.0_192-b12) (build 1.8.0_192-b12) # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.192-b12 mixed mode linux-amd64 compressed oops) # Problematic frame: # J 16655 C2 sun.nio.ch.FileChannelImpl.size()J (239 bytes) @ 0x7fda693c4c80 [0x7fda693c4c60+0x20] # # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # --- T H R E A D --- Current thread (0x7fda7d048800): JavaThread "kafka-network-thread-93732870-ListenerName(PLAINTEXT)-PLAINTEXT-4" [_thread_in_Java, id=15967, stack(0x7fcfb84fc000,0x7fcfb85fd000)] siginfo: si_signo: 11 (SIGSEGV), si_code: 2 (SEGV_ACCERR), si_addr: 0x7fda693c4c80 Stack: [0x7fcfb84fc000,0x7fcfb85fd000], sp=0x7fcfb85fb818, free space=1022k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) J 16655 C2 sun.nio.ch.FileChannelImpl.size()J (239 bytes) @ 0x7fda693c4c80 [0x7fda693c4c60+0x20] C 0x00055ea5b690 3. # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x7fbfc45c7080, pid=2548, tid=0x7fba8f24c700 # # JRE version: Java(TM) SE Runtime Environment (8.0_192-b12) (build 1.8.0_192-b12) # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.192-b12 mixed mode linux-amd64 compressed oops) # Problematic frame: # J 15764 C2 org.apache.kafka.common.record.FileRecords.writeTo(Ljava/nio/channels/GatheringByteChannel;JI)J (151 bytes) @ 0x7fbfc45c7080 [0x7fbfc45c7060+0x20] # # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # --- T H R E A D --- Current thread (0x7fbfd9b6): JavaThread "kafka-network-thread-93732870-ListenerName(PLAINTEXT)-PLAINTEXT-7" [_thread_in_Java, id=2679, stack(0x7fba8f14c000,0x7fba8f24d000)] siginfo: si_signo: 11 (SIGSEGV), si_code: 2 (SEGV_ACCERR), si_addr: 0x7fbfc45c7080 Stack: [0x7fba8f14c000,0x7fba8f24d000], sp=0x7fba8f24b838, free space=1022k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) J 15764 C2 org.apache.kafka.common.record.FileRecords.writeTo(Ljava/nio/channels/GatheringByteChannel;JI)J (151 bytes) @ 0x7fbfc45c7080 [0x7fbfc45c7060+0x20] C 0x0001
Re: SIGSEGV (0xb) on TransactionCoordinator
But G1GC is the default garbage collection algo used in kafka, it'll still be using G1GC. Did you add remove/add other parameters as well? Also, which kafka version are you using? On Fri, Jan 11, 2019 at 2:34 AM wenxing zheng wrote: > Previously we added G1GC option to the JVM options when starting the > Kafka, and now we remove this option and it works fine up to now. > > On 2019/01/08 07:58:01, Car Devops wrote: > > Hi folks :) > > > > Can you please Wenxing comfirm that removing G1GC really solved the > problem. > > Unfortunately I faced that issue last night. > > > > And what do you mean by " remove the G1GC"? How did you do that? > > > > Thanks! > > > > > > > > -Original Message- > > From: wenxing zheng [mailto:wenxing.zh...@gmail.com] > > Sent: 28 grudnia 2018 04:05 > > To: Peter Levart > > Cc: users@kafka.apache.org > > Subject: Re: SIGSEGV (0xb) on TransactionCoordinator > > > > > > > > Hi Peter, we didn't upgrade the JDK8, but just remove the G1GC. And now > it > > seemed stable, but we need to keep monitoring the status to finally > confirm. > > > > > > > > Kind Regards, Wenxing > > > > > > > > On Thu, Dec 27, 2018 at 9:43 PM Peter Levart > wrote: > > > > > > > > > Here's a report on the Jira with exactly the same crash and the > > > > > reported was using the same JDK 8u92... > > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-7625 > > > > > > > > > > ...the reporter upgraded to latest JDK 8 and it seems to be stable > > > > > since then. > > > > > > > > > > Regards, Peter > > > > > > > > > > On 12/27/18 11:29 AM, wenxing zheng wrote: > > > > > > Thanks to Peter. > > > > > > > > > > > > We did a lot of tests today, and found that the issue will happen > > > > > > after enabling G1GC. If we go with default settings, everything looks > > fine. > > > > > > > > > > > > On Thu, Dec 27, 2018 at 4:49 PM Peter Levart > > > > > > > > > > > wrote: > > > > > > > > > > > >> Hi, > > > > > >> > > > > > >> It looks like a JVM bug. If I were you, 1st thing I'd do is > > > > > >> upgrading the JDK to the latest JDK8u192. You're using JDK8u92 > > > > > >> which is quite old (2+ years)... > > > > > >> > > > > > >> Regards, Peter > > > > > >> > > > > > >> On 12/27/18 3:53 AM, wenxing zheng wrote: > > > > > >>> Dear all, > > > > > >>> > > > > > >>> We got a coredump with the following info last night, on this > > > > > >> environment, > > > > > >>> we enable the > > > > > >>> transaction. Please kindly advice what would be the problem here. > > > > > >>> > > > > > >>> # > > > > > # A fatal error has been detected by the Java Runtime Environment: > > > > > # > > > > > # SIGSEGV (0xb) at pc=0x7f546a857d0d, pid=13288, > > > > > tid=0x7f53701f9700 > > > > > # > > > > > # JRE version: Java(TM) SE Runtime Environment (8.0_92-b14) > > > > > (build > > > > > 1.8.0_92-b14) > > > > > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.92-b14 mixed > > > > > mode > > > > > linux-amd64 compressed oops) > > > > > # Problematic frame: > > > > > # J 9563 C1 > > > > > >> > > > > > *kafka.coordinator.transaction.TransactionCoordinator.$anonfun$handleE > > > > > ndTransaction$7(Lkafka/coordinator/transaction/TransactionCoordinator; > > > > > Ljava/lang/String;JSLorg/apache/kafka/common/requests/TransactionResul > > > > > t;Lkafka/coordinator/transaction/TransactionMetadata;)Lscala/util/Eith > > > > > er; > > > > > (518 bytes) @ 0x7f546a857d0d [0x7f546a856b40+0x11cd]* # # > > > > > Failed to write core dump. Core dumps have been disabled. To > > > > > enable > > > > > >> core > > > > > dumping, try "ulimit -c unlimited" before starting Java again # # > > > > > If you would like to submit a bug report, please visit: > > > > > # http://bugreport.java.com/bugreport/crash.jsp > > > > > # > > > > > --- T H R E A D --- Current thread > > > > > (0x7f547a29e800): JavaThread > > > > > >> "kafka-request-handler-5" > > > > > daemon [_thread_in_Java, id=13722, > > > > > stack(0x7f53700f9000,0x7f53701fa000)] > > > > > siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), > si_addr: > > > > > 0xdd310c13 > > > > > Registers: > > > > > RAX=0x0001, RBX=0x0006e9072fc8, > > > > > RCX=0x0688, > > > > > RDX=0x00075e026fc0 > > > > > RSP=0x7f53701f7f00, RBP=0x0006e98861f8, > > > > > RSI=0x7f53771a4238, > > > > > RDI=0x0006e9886098 > > > > > R8 =0x132d, R9 =0xdd310c13, > > > > > R10=0x0007c010bbb0, > > > > > R11=0xdd310c13 > > > > > R12=0x, R13=0xdd310b3d, > > > > > R14=0xdd310c0c, > > > > > R15=0x7f547a29e800 > > > > > RIP=0x7f546a857d0d, EFLAGS=0x00010202, > > > > > CSGSFS=0x002b0033, ERR=0x0004 > > > > > TRAPNO=0x
Re: SIGSEGV (0xb) on TransactionCoordinator
Previously we added G1GC option to the JVM options when starting the Kafka, and now we remove this option and it works fine up to now. On 2019/01/08 07:58:01, Car Devops wrote: > Hi folks :) > > Can you please Wenxing comfirm that removing G1GC really solved the problem. > Unfortunately I faced that issue last night. > > And what do you mean by " remove the G1GC"? How did you do that? > > Thanks! > > > > -Original Message- > From: wenxing zheng [mailto:wenxing.zh...@gmail.com] > Sent: 28 grudnia 2018 04:05 > To: Peter Levart > Cc: users@kafka.apache.org > Subject: Re: SIGSEGV (0xb) on TransactionCoordinator > > > > Hi Peter, we didn't upgrade the JDK8, but just remove the G1GC. And now it > seemed stable, but we need to keep monitoring the status to finally confirm. > > > > Kind Regards, Wenxing > > > > On Thu, Dec 27, 2018 at 9:43 PM Peter Levart wrote: > > > > > Here's a report on the Jira with exactly the same crash and the > > > reported was using the same JDK 8u92... > > > > > > https://issues.apache.org/jira/browse/KAFKA-7625 > > > > > > ...the reporter upgraded to latest JDK 8 and it seems to be stable > > > since then. > > > > > > Regards, Peter > > > > > > On 12/27/18 11:29 AM, wenxing zheng wrote: > > > > Thanks to Peter. > > > > > > > > We did a lot of tests today, and found that the issue will happen > > > > after enabling G1GC. If we go with default settings, everything looks > fine. > > > > > > > > On Thu, Dec 27, 2018 at 4:49 PM Peter Levart > > > > > > > wrote: > > > > > > > >> Hi, > > > >> > > > >> It looks like a JVM bug. If I were you, 1st thing I'd do is > > > >> upgrading the JDK to the latest JDK8u192. You're using JDK8u92 > > > >> which is quite old (2+ years)... > > > >> > > > >> Regards, Peter > > > >> > > > >> On 12/27/18 3:53 AM, wenxing zheng wrote: > > > >>> Dear all, > > > >>> > > > >>> We got a coredump with the following info last night, on this > > > >> environment, > > > >>> we enable the > > > >>> transaction. Please kindly advice what would be the problem here. > > > >>> > > > >>> # > > > # A fatal error has been detected by the Java Runtime Environment: > > > # > > > # SIGSEGV (0xb) at pc=0x7f546a857d0d, pid=13288, > > > tid=0x7f53701f9700 > > > # > > > # JRE version: Java(TM) SE Runtime Environment (8.0_92-b14) > > > (build > > > 1.8.0_92-b14) > > > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.92-b14 mixed > > > mode > > > linux-amd64 compressed oops) > > > # Problematic frame: > > > # J 9563 C1 > > > >> > > > *kafka.coordinator.transaction.TransactionCoordinator.$anonfun$handleE > > > ndTransaction$7(Lkafka/coordinator/transaction/TransactionCoordinator; > > > Ljava/lang/String;JSLorg/apache/kafka/common/requests/TransactionResul > > > t;Lkafka/coordinator/transaction/TransactionMetadata;)Lscala/util/Eith > > > er; > > > (518 bytes) @ 0x7f546a857d0d [0x7f546a856b40+0x11cd]* # # > > > Failed to write core dump. Core dumps have been disabled. To > > > enable > > > >> core > > > dumping, try "ulimit -c unlimited" before starting Java again # # > > > If you would like to submit a bug report, please visit: > > > # http://bugreport.java.com/bugreport/crash.jsp > > > # > > > --- T H R E A D --- Current thread > > > (0x7f547a29e800): JavaThread > > > >> "kafka-request-handler-5" > > > daemon [_thread_in_Java, id=13722, > > > stack(0x7f53700f9000,0x7f53701fa000)] > > > siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: > > > 0xdd310c13 > > > Registers: > > > RAX=0x0001, RBX=0x0006e9072fc8, > > > RCX=0x0688, > > > RDX=0x00075e026fc0 > > > RSP=0x7f53701f7f00, RBP=0x0006e98861f8, > > > RSI=0x7f53771a4238, > > > RDI=0x0006e9886098 > > > R8 =0x132d, R9 =0xdd310c13, > > > R10=0x0007c010bbb0, > > > R11=0xdd310c13 > > > R12=0x, R13=0xdd310b3d, > > > R14=0xdd310c0c, > > > R15=0x7f547a29e800 > > > RIP=0x7f546a857d0d, EFLAGS=0x00010202, > > > CSGSFS=0x002b0033, ERR=0x0004 > > > TRAPNO=0x000e > > > >>> Thanks, > > > >>> > > > >> > > > > > > >
Re: KTable.suppress(Suppressed.untilWindowCloses) does not suppress some non-final results when the kafka streams process is restarted
Hi Peter, Regarding retention, I was not referring to log retention, but to the window store retention. Since a new window is created every second (for example), there are in principle an unbounded number of windows (the longer the application runs, the more windows there are, with no end). However, we obviously can't store an infinite amount of data, so the window definition includes a retention period. By default, this is 24 hours. After the retention period elapses, all of the data for the window is purged to make room for new windows. So what I meant was that if you buffer some key "A" in window (Monday 09:00:00) and then get no further activity for A for over 24 hours, then when you do get that next event for A, say at (Tuesday 11:00:00), you'd do the scan but find nothing, since your buffered state would already have been purged from the store. The way I avoided this problem for Suppression was to organize the data by timestamp instead of by key, so on *every* update I can search for all the keys that are old enough and emit them. I also don't use a window store, so I don't have to worry about the retention time. To answer your question about the window store's topic, it configures a retention time the same length as the store's retention time, (and they keys are the full windowed key including the window start time), so it'll have roughly the same size bound as the store itself. Back to the process of figuring out what might be wrong with Suppression, I don't suppose you would be able to file a Jira and upload a repro program? If not, that's ok. I haven't been able to reproduce the bug yet, but it seems like it's happening somewhat consistently for you, so I should be able to get it to happen eventually. Thanks, and sorry again for the troubles. -John On Tue, Jan 8, 2019 at 6:48 AM Peter Levart wrote: > > > On 1/8/19 12:57 PM, Peter Levart wrote: > > Hi John, > > > > On 1/8/19 12:45 PM, Peter Levart wrote: > >>> I looked at your custom transfomer, and it looks almost correct to > >>> me. The > >>> only flaw seems to be that it only looks > >>> for closed windows for the key currently being processed, which > >>> means that > >>> if you have key "A" buffered, but don't get another event for it for a > >>> while after the window closes, you won't emit the final result. This > >>> might > >>> actually take longer than the window retention period, in which > >>> case, the > >>> data would be deleted without ever emitting the final result. > >> > >> So in DSL case, the suppression works by flushing *all* of the "ripe" > >> windows in the whole buffer whenever a singe event comes in with > >> recent enough timestamp regardless of the key of that event? > >> > >> Is the buffer shared among processing tasks or does each task > >> maintain its own private buffer that only contains its share of data > >> pertaining to assigned input partitions? In case the tasks are > >> executed on several processing JVM(s) the buffer can't really be > >> shared, right? In that case a single event can't flush all of the > >> "ripe" windows, but just those that are contained in the task's part > >> of buffer... > > > > Just a question about your comment above: > > > > /"This might actually take longer than the window retention period, in > > which case, the data would be deleted without ever emitting the final > > result"/ > > > > Are you talking about the buffer log topic retention? Aren't log > > topics configured to "compact" rather than "delete" messages? So the > > last "version" of the buffer entry for a particular key should stay > > forever? What are the keys in suppression buffer log topic? Are they a > > pair of (timestamp, key) ? Probably not since in that case the > > compacted log would grow indefinitely... > > > > Another question: > > > > What are the keys in WindowStore's log topic? If the input keys to the > > processor that uses such WindowStore consist of a bounded set of > > values (for example user ids), would compacted log of such WindowStore > > also be bounded? > > In case the key of WindowStore log topic is (timestamp, key) then would > explicitly deleting flushed entries from WindowStore (by putting null > value into the store) keep the compacted log bounded? In other words, > does WindowStore log topic support a special kind of "tombstone" message > that effectively removes the key from the compacted log? > > In that case, my custom processor could keep entries in its WindowStore > for as log as needed, depending on the activity of a particular input > key... > > > > > Regards, Peter > > > > > >
Re: Kafka data log timestamp
Thatns Robin. Can you send reference to it? Also can it be used for log
file stored locally (not in DB)?
Thanks,
Parth Gandhi
DevOps
On Thu, Jan 10, 2019 at 9:17 AM Robin Moffatt wrote:
> You can use kafkacat to examine the timestamp (and other metadata). Here's
> an example of calling it, and two sample output records:
>
> $ kafkacat -b localhost:9092 -t mysql_users -C -c2 -f '\nKey (%K bytes):
> %k\t\nValue (%S bytes): %s\nTimestamp: %T\tPartition: %p\tOffset: %o\n--\n'
>
> Key (1 bytes): 1
> Value (79 bytes):
> {"uid":1,"name":"Cliff","locale":"en_US","address_city":"St
> Louis","elite":"P"}
> Timestamp: 1520618381093 Partition: 0 Offset: 0
> --
>
> Key (1 bytes): 2
> Value (79 bytes):
> {"uid":2,"name":"Nick","locale":"en_US","address_city":"Palo
> Alto","elite":"G"}
> Timestamp: 1520618381093 Partition: 0 Offset: 1
> --
>
>
> --
>
> Robin Moffatt | Developer Advocate | ro...@confluent.io | @rmoff
>
>
> On Thu, 10 Jan 2019 at 10:13, Parth Gandhi <
> parth.gan...@excellenceinfonet.com> wrote:
>
> > Hi,
> >
> > Does kafka record the timestamp of the incoming message in its data log?
> I
> > checked one of the partition log and I can see the message without any
> > timestamp. Also there are few special characters in the message log. IS
> > that normal?
> >
> > Here is a sample log: pastebin.com/hStyCW13
>
> >
> > Thanks,
> > Parth Gandhi
> > DevOps
> >
> > Disclaimer
> >
> > The information contained in this communication from the sender is
> > confidential. It is intended solely for use by the recipient and others
> > authorized to receive it. If you are not the recipient, you are hereby
> > notified that any disclosure, copying, distribution or taking action in
> > relation of the contents of this information is strictly prohibited and
> may
> > be unlawful.
> >
> > This email has been scanned for viruses and malware, and may have been
> > automatically archived by Mimecast Ltd, an innovator in Software as a
> > Service (SaaS) for business. Providing a safer and more useful place for
> > your human generated data. Specializing in; Security, archiving and
> > compliance. To find out more visit the Mimecast website.
> >
>
Disclaimer
The information contained in this communication from the sender is
confidential. It is intended solely for use by the recipient and others
authorized to receive it. If you are not the recipient, you are hereby notified
that any disclosure, copying, distribution or taking action in relation of the
contents of this information is strictly prohibited and may be unlawful.
This email has been scanned for viruses and malware, and may have been
automatically archived by Mimecast Ltd, an innovator in Software as a Service
(SaaS) for business. Providing a safer and more useful place for your human
generated data. Specializing in; Security, archiving and compliance. To find
out more visit the Mimecast website.
Aw: Re: Doubts in Kafka
Okay, but what if one also needs to preserve the order of messages coming from a particular device? With Kafka, this is perhaps possible if all messages from a particular device go into the same partition. Would it be a good and efficient solution for this approach to set the key of each Kafka ProducerRecord to the unique ID of the Device AND deactivate the key-based log-cleaner on the broker so that it does not delete older records that have the same key? Sven Gesendet: Donnerstag, 10. Januar 2019 um 08:35 Uhr Von: "Peter Levart" An: users@kafka.apache.org, "aruna ramachandran" Betreff: Re: Doubts in Kafka Hi Aruna, On 1/10/19 8:19 AM, aruna ramachandran wrote: > I am using keyed partitions with 1000 partitions, so I need to create 1000 > consumers because consumers groups and re balancing concepts is not worked > in the case of manually assigned consumers.Is there any replacement for the > above problem. > What API are you using in the KafkaConsumer? Are you using subscribe(Collection topics) or are you using assign(Collection partitions) ? The 1st one (subscribe) is the one you should be using for your usecase. With that call, when you subscribe to a multi-partition topic and you have multiple KafkaConsumer(s) configured with the same consumer group id, then partitions of the topic are dynamically assigned (and possibly reassigned when consumers come or go) to a set of live consumers. Will this work for you (and why not)? Regards, Peter
Re: Kafka data log timestamp
You can use kafkacat to examine the timestamp (and other metadata). Here's
an example of calling it, and two sample output records:
$ kafkacat -b localhost:9092 -t mysql_users -C -c2 -f '\nKey (%K bytes):
%k\t\nValue (%S bytes): %s\nTimestamp: %T\tPartition: %p\tOffset: %o\n--\n'
Key (1 bytes): 1
Value (79 bytes):
{"uid":1,"name":"Cliff","locale":"en_US","address_city":"St
Louis","elite":"P"}
Timestamp: 1520618381093Partition: 0Offset: 0
--
Key (1 bytes): 2
Value (79 bytes):
{"uid":2,"name":"Nick","locale":"en_US","address_city":"Palo
Alto","elite":"G"}
Timestamp: 1520618381093Partition: 0Offset: 1
--
--
Robin Moffatt | Developer Advocate | ro...@confluent.io | @rmoff
On Thu, 10 Jan 2019 at 10:13, Parth Gandhi <
parth.gan...@excellenceinfonet.com> wrote:
> Hi,
>
> Does kafka record the timestamp of the incoming message in its data log? I
> checked one of the partition log and I can see the message without any
> timestamp. Also there are few special characters in the message log. IS
> that normal?
>
> Here is a sample log: pastebin.com/hStyCW13
>
> Thanks,
> Parth Gandhi
> DevOps
>
> Disclaimer
>
> The information contained in this communication from the sender is
> confidential. It is intended solely for use by the recipient and others
> authorized to receive it. If you are not the recipient, you are hereby
> notified that any disclosure, copying, distribution or taking action in
> relation of the contents of this information is strictly prohibited and may
> be unlawful.
>
> This email has been scanned for viruses and malware, and may have been
> automatically archived by Mimecast Ltd, an innovator in Software as a
> Service (SaaS) for business. Providing a safer and more useful place for
> your human generated data. Specializing in; Security, archiving and
> compliance. To find out more visit the Mimecast website.
>
Kafka data log timestamp
Hi, Does kafka record the timestamp of the incoming message in its data log? I checked one of the partition log and I can see the message without any timestamp. Also there are few special characters in the message log. IS that normal? Here is a sample log: pastebin.com/hStyCW13 Thanks, Parth Gandhi DevOps Disclaimer The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful. This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more visit the Mimecast website.
RE: Zookeeper timeout message in logs has value < configured timeout
Hi,
I think it's the zk client's problem.
1. Where the log producted:
if (to <= 0) {
String warnInfo;
warnInfo = "Client session timed out, have not heard from server in "
+ clientCnxnSocket.getIdleRecv()
+ "ms"
+ " for sessionid 0x"
+ Long.toHexString(sessionId);
LOG.warn(warnInfo);
throw new SessionTimeoutException(warnInfo);
}
2. And the 'to' value came from :
to = readTimeout - clientCnxnSocket.getIdleRecv();
3. Then the 'readTimeout' is defined as:
readTimeout = sessionTimeout * 2 / 3;
Thus, the 'actually' sessionTimeout is 1333ms while
config:zookeeper.session.timeout=2000ms
>-Original Message-
>From: Mark Anderson [mailto:manderso...@gmail.com]
>Sent: Wednesday, January 9, 2019 11:34 PM
>To: users@kafka.apache.org
>Subject: Zookeeper timeout message in logs has value < configured timeout
>
>Hi,
>
>I'm experimenting with the value of zookeeper.session.timeout.ms in Kafka
>2.0.1.
>
>In my broker logs I see the following message
>
>[2019-01-09 15:12:01,246] WARN Client session timed out, have not heard
>from server in 1369ms for sessionid 0x200d78d415e0002
>(org.apache.zookeeper.ClientCnxn)
>
>However, my zookeeper session timeout is configured as 2000ms.
>
>Why does the log file show a session timeout for a value less than what is
>configured?
>
>Thanks,
>Mark
User Activity Tracking
Hi everyone, we are planning to add some user activity tracking to an application and I wanted to ask around for your general experiences and best practices. Do you use one topic per application or more granular? Do you write directly from the application to Kafka for tracking purposes? How to best avoid blocking anything in the application in case of broker issues? Is it usually tolerable to lose some tracking information or do you use any caching and asynchronously produce the messages? Any learnings are welcome, especially things you would now do differently if you had to start again :-) Thanks in advance and best regards Patrik
