Hi, can you please file a JIRA ticket so this doesn't get lost?
Thanks,
Ismael
On 6 Dec 2016 5:06 pm, "Schumann,Robert" <roschum...@ebay.com> wrote:
> Hi all,
>
> we are facing an issue with latest kafka 0.10.1 and the log cleaner thread
> with regards to the timeindex files. From the log of the log-cleaner we see
> after startup that it tries to cleanup a topic called xdc_listing-status-v2
> [1]. The topic is setup with log compaction [2] and the kafka cluster
> configuration has log.cleaner enabled [3]. Looking at the log and the newly
> created file [4], the cleaner seems to refer to tombstones prior to
> epoch_time=0 - maybe because he finds messages, which don’t have a
> timestamp at all (?). All producers and consumers are using 0.10.1 and the
> topics have been created completely new, so I’m not sure, where this issue
> would come from. The original timeindex file [5] seems to show only valid
> timestamps for the mentioned offsets. I would also like to mention that the
> issue happened in two independent datacenters at the same time, so I would
> rather expect an application/producer issue instead of random disk
> failures. We didn’t have the problem with 0.10.0 for around half a year, it
> appeared shortly after the upgrade to 0.10.1.
>
> The puzzling message from the cleaner “cleaning prior to Fri Dec 02
> 16:35:50 CET 2016, discarding tombstones prior to Thu Jan 01 01:00:00 CET
> 1970” also confuses me a bit. Does that mean, it does not find any log
> segments which can be cleaned up or the last timestamp of the last log
> segment is somehow broken/missing?
>
> I’m also a bit wondering, why the log cleaner thread stops completely
> after an error with one topic. I would at least expect that it keeps on
> cleaning up other topics, but apparently it doesn’t do that, e.g. it’s not
> even cleaning the __consumer_offsets anymore.
>
> Does anybody have the same issues or can explain, what’s going on? Thanks
> for any help or suggestions.
>
> Cheers
> Robert
>
> [1]
> [2016-12-06 12:49:17,885] INFO Starting the log cleaner
> (kafka.log.LogCleaner)
> [2016-12-06 12:49:17,895] INFO [kafka-log-cleaner-thread-0], Starting
> (kafka.log.LogCleaner)
> [2016-12-06 12:49:17,947] INFO Cleaner 0: Beginning cleaning of log
> xdc_listing-status-v2-1. (kafka.log.LogCleaner)
> [2016-12-06 12:49:17,948] INFO Cleaner 0: Building offset map for
> xdc_listing-status-v2-1... (kafka.log.LogCleaner)
> [2016-12-06 12:49:17,989] INFO Cleaner 0: Building offset map for log
> xdc_listing-status-v2-1 for 1 segments in offset range [0, 194991).
> (kafka.log.LogCleaner)
> [2016-12-06 12:49:24,572] INFO Cleaner 0: Offset map for log
> xdc_listing-status-v2-1 complete. (kafka.log.LogCleaner)
> [2016-12-06 12:49:24,577] INFO Cleaner 0: Cleaning log
> xdc_listing-status-v2-1 (cleaning prior to Fri Dec 02 16:35:50 CET 2016,
> discarding tombstones prior to Thu Jan 01 01:00:00 CET 1970)...
> (kafka.log.LogCleaner)
> [2016-12-06 12:49:24,580] INFO Cleaner 0: Cleaning segment 0 in log
> xdc_listing-status-v2-1 (largest timestamp Fri Dec 02 16:35:50 CET 2016)
> into 0, retaining deletes. (kafka.log.LogCleaner)
> [2016-12-06 12:49:24,968] ERROR [kafka-log-cleaner-thread-0], Error due
> to (kafka.log.LogCleaner)
> kafka.common.InvalidOffsetException: Attempt to append an offset (-1) to
> slot 9 no larger than the last offset appended (11832) to
> /var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.
> timeindex.cleaned.
> at kafka.log.TimeIndex$$anonfun$maybeAppend$1.apply$mcV$sp(
> TimeIndex.scala:117)
> at kafka.log.TimeIndex$$anonfun$maybeAppend$1.apply(TimeIndex.
> scala:107)
> at kafka.log.TimeIndex$$anonfun$maybeAppend$1.apply(TimeIndex.
> scala:107)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.log.TimeIndex.maybeAppend(TimeIndex.scala:107)
> at kafka.log.LogSegment.append(LogSegment.scala:106)
> at kafka.log.Cleaner.cleanInto(LogCleaner.scala:518)
> at kafka.log.Cleaner$$anonfun$cleanSegments$1.apply(
> LogCleaner.scala:404)
> at kafka.log.Cleaner$$anonfun$cleanSegments$1.apply(
> LogCleaner.scala:400)
> at scala.collection.immutable.List.foreach(List.scala:381)
> at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:400)
> at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:364)
> at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:363)
> at scala.collection.immutable.List.foreach(List.scala:381)
> at kafka.log.Cleaner.clean(LogCleaner.scala:363)
> at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(
> LogCleaner.scala:239)
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:218)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> [2016-12-06 12:49:24,971] INFO [kafka-log-cleaner-thread-0], Stopped
> (kafka.log.LogCleaner)
>
> [2]
> $ kafka-configs-topics xdc_listing-status-v2 --describe
> Configs for topic 'xdc_listing-status-v2' are cleanup.policy=compact
>
> [3]
> broker.id=74
> listeners = PLAINTEXT://0.0.0.0:9092,TRACE://localhost:9093
> advertised.listeners = PLAINTEXT://broker2:9092
> num.network.threads=2
> num.io.threads=2
> inter.broker.protocol.version=0.10.1.0
> socket.send.buffer.bytes=1048576
> socket.receive.buffer.bytes=1048576
> socket.request.max.bytes=104857600
> log.dirs=/var/lib/kafka
> num.partitions=3
> default.replication.factor=2
> log.flush.interval.messages=10000
> log.flush.interval.ms=1000
> log.retention.hours=168
> log.segment.bytes=536870912
> log.cleanup.interval.mins=1
> zookeeper.connect=broker1:2181,broker2:2181,broker3:2181
> zookeeper.connection.timeout.ms=300000
> kafka.metrics.reporters=com.airbnb.kafka.KafkaStatsdMetricsReporter
> external.kafka.statsd.reporter.enabled=true
> external.kafka.statsd.host=localhost
> external.kafka.statsd.port=8125
> external.kafka.statsd.tag.enabled=false
> external.kafka.statsd.metrics.prefix=collectd.broker2
> kafka.metrics.polling.interval.secs=60
> external.kafka.statsd.dimension.enabled.meanRate=false
> external.kafka.statsd.dimension.enabled.rate15m=false
> external.kafka.statsd.dimension.enabled.min=false
> external.kafka.statsd.dimension.enabled.max=false
> external.kafka.statsd.dimension.enabled.median=false
> external.kafka.statsd.dimension.enabled.p75=false
> external.kafka.statsd.dimension.enabled.p98=false
> external.kafka.statsd.dimension.enabled.p999=false
> delete.topic.enable=true
> controlled.shutdown.enable=true
> auto.leader.rebalance.enable=true
> auto.create.topics.enable=false
> log.cleaner.enable=true
> offsets.topic.compression.codec=2
> offsets.topic.num.partitions=20
>
> [4]
> #
> ti=/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned;
> for e in $(xxd -c 12 -p $ti); do echo -n "file:[$ti] raw:[$e] "; d="$(date
> -d @$(echo -n $((16#${e:0:16})) | sed 's/.\{10\}/&./g'))"; echo -n
> "timestamp:[$d] "; echo "offset:[$((16#${e:16:24}))]";done | head -n 20
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000001589c490ae1ffffffff] timestamp:[Fri Nov 25 17:19:42 CET 2016]
> offset:[4294967295]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000001589c81f249ffffffff] timestamp:[Fri Nov 25 18:21:51 CET 2016]
> offset:[4294967295]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000001589d4b4ea7ffffffff] timestamp:[Fri Nov 25 22:01:48 CET 2016]
> offset:[4294967295]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000001589e04a783ffffffff] timestamp:[Sat Nov 26 01:24:14 CET 2016]
> offset:[4294967295]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000001589ede5af6ffffffff] timestamp:[Sat Nov 26 05:22:02 CET 2016]
> offset:[4294967295]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000001589f8ccc16ffffffff] timestamp:[Sat Nov 26 08:32:34 CET 2016]
> offset:[4294967295]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000001589ffb82bcffffffff] timestamp:[Sat Nov 26 10:33:30 CET 2016]
> offset:[4294967295]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[00000158a04fef7fffffffff] timestamp:[Sat Nov 26 12:05:43 CET 2016]
> offset:[4294967295]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[00000158a0b506eb00002e38] timestamp:[Sat Nov 26 13:56:08 CET 2016]
> offset:[11832]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
> offset:[0]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
> offset:[0]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
> offset:[0]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
> offset:[0]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
> offset:[0]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
> offset:[0]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
> offset:[0]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
> offset:[0]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
> offset:[0]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
> offset:[0]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
> raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
> offset:[0]
>
> [5]
> /var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex
> ...
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a06cbddc00002a21] timestamp:[Sat Nov 26 12:37:10 CET 2016]
> offset:[10785]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a06cded500002a29] timestamp:[Sat Nov 26 12:37:19 CET 2016]
> offset:[10793]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a06d25d400002a31] timestamp:[Sat Nov 26 12:37:37 CET 2016]
> offset:[10801]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a06dc34700002a39] timestamp:[Sat Nov 26 12:38:17 CET 2016]
> offset:[10809]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a06e3ccf00002a41] timestamp:[Sat Nov 26 12:38:48 CET 2016]
> offset:[10817]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07051ae00002a4a] timestamp:[Sat Nov 26 12:41:05 CET 2016]
> offset:[10826]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a071168100002a52] timestamp:[Sat Nov 26 12:41:55 CET 2016]
> offset:[10834]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a071603400002a5a] timestamp:[Sat Nov 26 12:42:14 CET 2016]
> offset:[10842]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07196c600002a62] timestamp:[Sat Nov 26 12:42:28 CET 2016]
> offset:[10850]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a072531500002a6a] timestamp:[Sat Nov 26 12:43:16 CET 2016]
> offset:[10858]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a072c5f700002a73] timestamp:[Sat Nov 26 12:43:46 CET 2016]
> offset:[10867]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07340a500002a7b] timestamp:[Sat Nov 26 12:44:17 CET 2016]
> offset:[10875]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0743e5a00002a85] timestamp:[Sat Nov 26 12:45:22 CET 2016]
> offset:[10885]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a074d37b00002a8d] timestamp:[Sat Nov 26 12:46:00 CET 2016]
> offset:[10893]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07551fe00002a95] timestamp:[Sat Nov 26 12:46:33 CET 2016]
> offset:[10901]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a075e21900002a9e] timestamp:[Sat Nov 26 12:47:09 CET 2016]
> offset:[10910]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a075faa900002aa6] timestamp:[Sat Nov 26 12:47:16 CET 2016]
> offset:[10918]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07619f900002aae] timestamp:[Sat Nov 26 12:47:24 CET 2016]
> offset:[10926]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a076447500002ab6] timestamp:[Sat Nov 26 12:47:35 CET 2016]
> offset:[10934]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a076728000002abf] timestamp:[Sat Nov 26 12:47:46 CET 2016]
> offset:[10943]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0768e7d00002ac8] timestamp:[Sat Nov 26 12:47:54 CET 2016]
> offset:[10952]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a076af9300002ad1] timestamp:[Sat Nov 26 12:48:02 CET 2016]
> offset:[10961]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a076d81500002ad9] timestamp:[Sat Nov 26 12:48:12 CET 2016]
> offset:[10969]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0773f2200002ae1] timestamp:[Sat Nov 26 12:48:39 CET 2016]
> offset:[10977]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a077babf00002ae9] timestamp:[Sat Nov 26 12:49:10 CET 2016]
> offset:[10985]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0792cee00002af1] timestamp:[Sat Nov 26 12:50:45 CET 2016]
> offset:[10993]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a079cb9b00002af9] timestamp:[Sat Nov 26 12:51:26 CET 2016]
> offset:[11001]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07a28a800002b02] timestamp:[Sat Nov 26 12:51:50 CET 2016]
> offset:[11010]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07a651500002b0a] timestamp:[Sat Nov 26 12:52:05 CET 2016]
> offset:[11018]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07c0d0800002b14] timestamp:[Sat Nov 26 12:53:54 CET 2016]
> offset:[11028]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07cc5cd00002b1c] timestamp:[Sat Nov 26 12:54:41 CET 2016]
> offset:[11036]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07d807300002b25] timestamp:[Sat Nov 26 12:55:29 CET 2016]
> offset:[11045]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07de02c00002b2d] timestamp:[Sat Nov 26 12:55:53 CET 2016]
> offset:[11053]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07e505c00002b35] timestamp:[Sat Nov 26 12:56:22 CET 2016]
> offset:[11061]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07e6c6e00002b3d] timestamp:[Sat Nov 26 12:56:29 CET 2016]
> offset:[11069]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07e8a2600002b45] timestamp:[Sat Nov 26 12:56:37 CET 2016]
> offset:[11077]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07ed26b00002b4d] timestamp:[Sat Nov 26 12:56:55 CET 2016]
> offset:[11085]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a07f729600002b55] timestamp:[Sat Nov 26 12:57:36 CET 2016]
> offset:[11093]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a080278400002b5e] timestamp:[Sat Nov 26 12:58:23 CET 2016]
> offset:[11102]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a08168e300002b67] timestamp:[Sat Nov 26 12:59:45 CET 2016]
> offset:[11111]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a08257c900002b6f] timestamp:[Sat Nov 26 13:00:46 CET 2016]
> offset:[11119]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a083f4b900002b77] timestamp:[Sat Nov 26 13:02:32 CET 2016]
> offset:[11127]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a084d01c00002b7f] timestamp:[Sat Nov 26 13:03:28 CET 2016]
> offset:[11135]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a086450500002b87] timestamp:[Sat Nov 26 13:05:03 CET 2016]
> offset:[11143]
> ...
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a09d4a7700002cff] timestamp:[Sat Nov 26 13:30:12 CET 2016]
> offset:[11519]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a09d783100002d07] timestamp:[Sat Nov 26 13:30:24 CET 2016]
> offset:[11527]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a09da2a100002d0f] timestamp:[Sat Nov 26 13:30:35 CET 2016]
> offset:[11535]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a09dcdf100002d17] timestamp:[Sat Nov 26 13:30:46 CET 2016]
> offset:[11543]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a09dfdb700002d1f] timestamp:[Sat Nov 26 13:30:58 CET 2016]
> offset:[11551]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a09e311700002d28] timestamp:[Sat Nov 26 13:31:11 CET 2016]
> offset:[11560]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a09eccaa00002d31] timestamp:[Sat Nov 26 13:31:51 CET 2016]
> offset:[11569]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a09f542300002d3a] timestamp:[Sat Nov 26 13:32:26 CET 2016]
> offset:[11578]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0a026c100002d43] timestamp:[Sat Nov 26 13:33:20 CET 2016]
> offset:[11587]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0a0d8c900002d4b] timestamp:[Sat Nov 26 13:34:05 CET 2016]
> offset:[11595]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0a163d400002d53] timestamp:[Sat Nov 26 13:34:41 CET 2016]
> offset:[11603]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0a310fe00002d5c] timestamp:[Sat Nov 26 13:36:31 CET 2016]
> offset:[11612]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0a3b0a300002d65] timestamp:[Sat Nov 26 13:37:11 CET 2016]
> offset:[11621]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0a3cdf100002d6d] timestamp:[Sat Nov 26 13:37:19 CET 2016]
> offset:[11629]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0a42ba200002d75] timestamp:[Sat Nov 26 13:37:43 CET 2016]
> offset:[11637]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0a4864900002d7e] timestamp:[Sat Nov 26 13:38:06 CET 2016]
> offset:[11646]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0a500e700002d86] timestamp:[Sat Nov 26 13:38:38 CET 2016]
> offset:[11654]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0a5a38200002d8e] timestamp:[Sat Nov 26 13:39:19 CET 2016]
> offset:[11662]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0a5f5a400002d96] timestamp:[Sat Nov 26 13:39:40 CET 2016]
> offset:[11670]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0a6e30400002d9e] timestamp:[Sat Nov 26 13:40:41 CET 2016]
> offset:[11678]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0a7e6b600002da7] timestamp:[Sat Nov 26 13:41:47 CET 2016]
> offset:[11687]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0aa276300002db1] timestamp:[Sat Nov 26 13:44:15 CET 2016]
> offset:[11697]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0ab040c00002dbb] timestamp:[Sat Nov 26 13:45:12 CET 2016]
> offset:[11707]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0ab6d2c00002dc3] timestamp:[Sat Nov 26 13:45:38 CET 2016]
> offset:[11715]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0aba59e00002dcb] timestamp:[Sat Nov 26 13:45:53 CET 2016]
> offset:[11723]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0acb9b200002dd4] timestamp:[Sat Nov 26 13:47:04 CET 2016]
> offset:[11732]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0ae188e00002ddc] timestamp:[Sat Nov 26 13:48:33 CET 2016]
> offset:[11740]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0ae682b00002de4] timestamp:[Sat Nov 26 13:48:54 CET 2016]
> offset:[11748]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0ae7a9b00002ded] timestamp:[Sat Nov 26 13:48:59 CET 2016]
> offset:[11757]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0ae9bed00002df5] timestamp:[Sat Nov 26 13:49:07 CET 2016]
> offset:[11765]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0aeb82200002dfd] timestamp:[Sat Nov 26 13:49:14 CET 2016]
> offset:[11773]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0aed16a00002e05] timestamp:[Sat Nov 26 13:49:21 CET 2016]
> offset:[11781]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0aedd8d00002e0d] timestamp:[Sat Nov 26 13:49:24 CET 2016]
> offset:[11789]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0aff17200002e16] timestamp:[Sat Nov 26 13:50:34 CET 2016]
> offset:[11798]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0b0d65e00002e1e] timestamp:[Sat Nov 26 13:51:33 CET 2016]
> offset:[11806]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0b1e31f00002e26] timestamp:[Sat Nov 26 13:52:42 CET 2016]
> offset:[11814]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0b3736300002e2e] timestamp:[Sat Nov 26 13:54:24 CET 2016]
> offset:[11822]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0b4648a00002e36] timestamp:[Sat Nov 26 13:55:26 CET 2016]
> offset:[11830]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0b7ad3d00002e40] timestamp:[Sat Nov 26 13:59:01 CET 2016]
> offset:[11840]
> file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
> raw:[00000158a0b7efe700002e49] timestamp:[Sat Nov 26 13:59:18 CET 2016]
> offset:[11849]
>
> --
> Robert Schumann | Lead DevOps Engineer | mobile.de GmbH
> T: + 49. 30. 8109. 7219
> M: +49.151. 5504. 8246
> F: +49. 30. 8109. 7131
> roschum...@ebay.com<mailto:roschum...@ebay.com>
> www.mobile.de
>
> Marktplatz 1 | 14532 Europarc Dreilinden | Germany
>
> Geschaeftsfuehrer: Malte Krueger
> HRB Nr.: 18517 P, AG Potsdam
> Sitz der Gesellschaft: Kleinmachnow
>
>
