Hi all,
we are facing an issue with latest kafka 0.10.1 and the log cleaner thread with
regards to the timeindex files. From the log of the log-cleaner we see after
startup that it tries to cleanup a topic called xdc_listing-status-v2 [1]. The
topic is setup with log compaction [2] and the kafka cluster configuration has
log.cleaner enabled [3]. Looking at the log and the newly created file [4], the
cleaner seems to refer to tombstones prior to epoch_time=0 - maybe because he
finds messages, which don’t have a timestamp at all (?). All producers and
consumers are using 0.10.1 and the topics have been created completely new, so
I’m not sure, where this issue would come from. The original timeindex file [5]
seems to show only valid timestamps for the mentioned offsets. I would also
like to mention that the issue happened in two independent datacenters at the
same time, so I would rather expect an application/producer issue instead of
random disk failures. We didn’t have the problem with 0.10.0 for around half a
year, it appeared shortly after the upgrade to 0.10.1.
The puzzling message from the cleaner “cleaning prior to Fri Dec 02 16:35:50
CET 2016, discarding tombstones prior to Thu Jan 01 01:00:00 CET 1970” also
confuses me a bit. Does that mean, it does not find any log segments which can
be cleaned up or the last timestamp of the last log segment is somehow
broken/missing?
I’m also a bit wondering, why the log cleaner thread stops completely after an
error with one topic. I would at least expect that it keeps on cleaning up
other topics, but apparently it doesn’t do that, e.g. it’s not even cleaning
the __consumer_offsets anymore.
Does anybody have the same issues or can explain, what’s going on? Thanks for
any help or suggestions.
Cheers
Robert
[1]
[2016-12-06 12:49:17,885] INFO Starting the log cleaner (kafka.log.LogCleaner)
[2016-12-06 12:49:17,895] INFO [kafka-log-cleaner-thread-0], Starting
(kafka.log.LogCleaner)
[2016-12-06 12:49:17,947] INFO Cleaner 0: Beginning cleaning of log
xdc_listing-status-v2-1. (kafka.log.LogCleaner)
[2016-12-06 12:49:17,948] INFO Cleaner 0: Building offset map for
xdc_listing-status-v2-1... (kafka.log.LogCleaner)
[2016-12-06 12:49:17,989] INFO Cleaner 0: Building offset map for log
xdc_listing-status-v2-1 for 1 segments in offset range [0, 194991).
(kafka.log.LogCleaner)
[2016-12-06 12:49:24,572] INFO Cleaner 0: Offset map for log
xdc_listing-status-v2-1 complete. (kafka.log.LogCleaner)
[2016-12-06 12:49:24,577] INFO Cleaner 0: Cleaning log xdc_listing-status-v2-1
(cleaning prior to Fri Dec 02 16:35:50 CET 2016, discarding tombstones prior to
Thu Jan 01 01:00:00 CET 1970)... (kafka.log.LogCleaner)
[2016-12-06 12:49:24,580] INFO Cleaner 0: Cleaning segment 0 in log
xdc_listing-status-v2-1 (largest timestamp Fri Dec 02 16:35:50 CET 2016) into
0, retaining deletes. (kafka.log.LogCleaner)
[2016-12-06 12:49:24,968] ERROR [kafka-log-cleaner-thread-0], Error due to
(kafka.log.LogCleaner)
kafka.common.InvalidOffsetException: Attempt to append an offset (-1) to slot 9
no larger than the last offset appended (11832) to
/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned.
at
kafka.log.TimeIndex$$anonfun$maybeAppend$1.apply$mcV$sp(TimeIndex.scala:117)
at kafka.log.TimeIndex$$anonfun$maybeAppend$1.apply(TimeIndex.scala:107)
at kafka.log.TimeIndex$$anonfun$maybeAppend$1.apply(TimeIndex.scala:107)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
at kafka.log.TimeIndex.maybeAppend(TimeIndex.scala:107)
at kafka.log.LogSegment.append(LogSegment.scala:106)
at kafka.log.Cleaner.cleanInto(LogCleaner.scala:518)
at
kafka.log.Cleaner$$anonfun$cleanSegments$1.apply(LogCleaner.scala:404)
at
kafka.log.Cleaner$$anonfun$cleanSegments$1.apply(LogCleaner.scala:400)
at scala.collection.immutable.List.foreach(List.scala:381)
at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:400)
at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:364)
at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:363)
at scala.collection.immutable.List.foreach(List.scala:381)
at kafka.log.Cleaner.clean(LogCleaner.scala:363)
at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:239)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:218)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2016-12-06 12:49:24,971] INFO [kafka-log-cleaner-thread-0], Stopped
(kafka.log.LogCleaner)
[2]
$ kafka-configs-topics xdc_listing-status-v2 --describe
Configs for topic 'xdc_listing-status-v2' are cleanup.policy=compact
[3]
broker.id=74
listeners = PLAINTEXT://0.0.0.0:9092,TRACE://localhost:9093
advertised.listeners = PLAINTEXT://broker2:9092
num.network.threads=2
num.io.threads=2
inter.broker.protocol.version=0.10.1.0
socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576
socket.request.max.bytes=104857600
log.dirs=/var/lib/kafka
num.partitions=3
default.replication.factor=2
log.flush.interval.messages=10000
log.flush.interval.ms=1000
log.retention.hours=168
log.segment.bytes=536870912
log.cleanup.interval.mins=1
zookeeper.connect=broker1:2181,broker2:2181,broker3:2181
zookeeper.connection.timeout.ms=300000
kafka.metrics.reporters=com.airbnb.kafka.KafkaStatsdMetricsReporter
external.kafka.statsd.reporter.enabled=true
external.kafka.statsd.host=localhost
external.kafka.statsd.port=8125
external.kafka.statsd.tag.enabled=false
external.kafka.statsd.metrics.prefix=collectd.broker2
kafka.metrics.polling.interval.secs=60
external.kafka.statsd.dimension.enabled.meanRate=false
external.kafka.statsd.dimension.enabled.rate15m=false
external.kafka.statsd.dimension.enabled.min=false
external.kafka.statsd.dimension.enabled.max=false
external.kafka.statsd.dimension.enabled.median=false
external.kafka.statsd.dimension.enabled.p75=false
external.kafka.statsd.dimension.enabled.p98=false
external.kafka.statsd.dimension.enabled.p999=false
delete.topic.enable=true
controlled.shutdown.enable=true
auto.leader.rebalance.enable=true
auto.create.topics.enable=false
log.cleaner.enable=true
offsets.topic.compression.codec=2
offsets.topic.num.partitions=20
[4]
#
ti=/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned;
for e in $(xxd -c 12 -p $ti); do echo -n "file:[$ti] raw:[$e] "; d="$(date -d
@$(echo -n $((16#${e:0:16})) | sed 's/.\{10\}/&./g'))"; echo -n "timestamp:[$d]
"; echo "offset:[$((16#${e:16:24}))]";done | head -n 20
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
raw:[000001589c490ae1ffffffff] timestamp:[Fri Nov 25 17:19:42 CET 2016]
offset:[4294967295]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
raw:[000001589c81f249ffffffff] timestamp:[Fri Nov 25 18:21:51 CET 2016]
offset:[4294967295]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
raw:[000001589d4b4ea7ffffffff] timestamp:[Fri Nov 25 22:01:48 CET 2016]
offset:[4294967295]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
raw:[000001589e04a783ffffffff] timestamp:[Sat Nov 26 01:24:14 CET 2016]
offset:[4294967295]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
raw:[000001589ede5af6ffffffff] timestamp:[Sat Nov 26 05:22:02 CET 2016]
offset:[4294967295]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
raw:[000001589f8ccc16ffffffff] timestamp:[Sat Nov 26 08:32:34 CET 2016]
offset:[4294967295]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
raw:[000001589ffb82bcffffffff] timestamp:[Sat Nov 26 10:33:30 CET 2016]
offset:[4294967295]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
raw:[00000158a04fef7fffffffff] timestamp:[Sat Nov 26 12:05:43 CET 2016]
offset:[4294967295]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
raw:[00000158a0b506eb00002e38] timestamp:[Sat Nov 26 13:56:08 CET 2016]
offset:[11832]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
offset:[0]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
offset:[0]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
offset:[0]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
offset:[0]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
offset:[0]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
offset:[0]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
offset:[0]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
offset:[0]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
offset:[0]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
offset:[0]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex.cleaned]
raw:[000000000000000000000000] timestamp:[Thu Jan 1 01:00:00 CET 1970]
offset:[0]
[5]
/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex
...
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a06cbddc00002a21] timestamp:[Sat Nov 26 12:37:10 CET 2016]
offset:[10785]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a06cded500002a29] timestamp:[Sat Nov 26 12:37:19 CET 2016]
offset:[10793]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a06d25d400002a31] timestamp:[Sat Nov 26 12:37:37 CET 2016]
offset:[10801]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a06dc34700002a39] timestamp:[Sat Nov 26 12:38:17 CET 2016]
offset:[10809]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a06e3ccf00002a41] timestamp:[Sat Nov 26 12:38:48 CET 2016]
offset:[10817]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a07051ae00002a4a] timestamp:[Sat Nov 26 12:41:05 CET 2016]
offset:[10826]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a071168100002a52] timestamp:[Sat Nov 26 12:41:55 CET 2016]
offset:[10834]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a071603400002a5a] timestamp:[Sat Nov 26 12:42:14 CET 2016]
offset:[10842]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a07196c600002a62] timestamp:[Sat Nov 26 12:42:28 CET 2016]
offset:[10850]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a072531500002a6a] timestamp:[Sat Nov 26 12:43:16 CET 2016]
offset:[10858]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a072c5f700002a73] timestamp:[Sat Nov 26 12:43:46 CET 2016]
offset:[10867]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a07340a500002a7b] timestamp:[Sat Nov 26 12:44:17 CET 2016]
offset:[10875]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0743e5a00002a85] timestamp:[Sat Nov 26 12:45:22 CET 2016]
offset:[10885]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a074d37b00002a8d] timestamp:[Sat Nov 26 12:46:00 CET 2016]
offset:[10893]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a07551fe00002a95] timestamp:[Sat Nov 26 12:46:33 CET 2016]
offset:[10901]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a075e21900002a9e] timestamp:[Sat Nov 26 12:47:09 CET 2016]
offset:[10910]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a075faa900002aa6] timestamp:[Sat Nov 26 12:47:16 CET 2016]
offset:[10918]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a07619f900002aae] timestamp:[Sat Nov 26 12:47:24 CET 2016]
offset:[10926]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a076447500002ab6] timestamp:[Sat Nov 26 12:47:35 CET 2016]
offset:[10934]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a076728000002abf] timestamp:[Sat Nov 26 12:47:46 CET 2016]
offset:[10943]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0768e7d00002ac8] timestamp:[Sat Nov 26 12:47:54 CET 2016]
offset:[10952]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a076af9300002ad1] timestamp:[Sat Nov 26 12:48:02 CET 2016]
offset:[10961]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a076d81500002ad9] timestamp:[Sat Nov 26 12:48:12 CET 2016]
offset:[10969]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0773f2200002ae1] timestamp:[Sat Nov 26 12:48:39 CET 2016]
offset:[10977]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a077babf00002ae9] timestamp:[Sat Nov 26 12:49:10 CET 2016]
offset:[10985]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0792cee00002af1] timestamp:[Sat Nov 26 12:50:45 CET 2016]
offset:[10993]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a079cb9b00002af9] timestamp:[Sat Nov 26 12:51:26 CET 2016]
offset:[11001]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a07a28a800002b02] timestamp:[Sat Nov 26 12:51:50 CET 2016]
offset:[11010]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a07a651500002b0a] timestamp:[Sat Nov 26 12:52:05 CET 2016]
offset:[11018]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a07c0d0800002b14] timestamp:[Sat Nov 26 12:53:54 CET 2016]
offset:[11028]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a07cc5cd00002b1c] timestamp:[Sat Nov 26 12:54:41 CET 2016]
offset:[11036]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a07d807300002b25] timestamp:[Sat Nov 26 12:55:29 CET 2016]
offset:[11045]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a07de02c00002b2d] timestamp:[Sat Nov 26 12:55:53 CET 2016]
offset:[11053]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a07e505c00002b35] timestamp:[Sat Nov 26 12:56:22 CET 2016]
offset:[11061]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a07e6c6e00002b3d] timestamp:[Sat Nov 26 12:56:29 CET 2016]
offset:[11069]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a07e8a2600002b45] timestamp:[Sat Nov 26 12:56:37 CET 2016]
offset:[11077]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a07ed26b00002b4d] timestamp:[Sat Nov 26 12:56:55 CET 2016]
offset:[11085]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a07f729600002b55] timestamp:[Sat Nov 26 12:57:36 CET 2016]
offset:[11093]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a080278400002b5e] timestamp:[Sat Nov 26 12:58:23 CET 2016]
offset:[11102]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a08168e300002b67] timestamp:[Sat Nov 26 12:59:45 CET 2016]
offset:[11111]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a08257c900002b6f] timestamp:[Sat Nov 26 13:00:46 CET 2016]
offset:[11119]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a083f4b900002b77] timestamp:[Sat Nov 26 13:02:32 CET 2016]
offset:[11127]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a084d01c00002b7f] timestamp:[Sat Nov 26 13:03:28 CET 2016]
offset:[11135]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a086450500002b87] timestamp:[Sat Nov 26 13:05:03 CET 2016]
offset:[11143]
...
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a09d4a7700002cff] timestamp:[Sat Nov 26 13:30:12 CET 2016]
offset:[11519]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a09d783100002d07] timestamp:[Sat Nov 26 13:30:24 CET 2016]
offset:[11527]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a09da2a100002d0f] timestamp:[Sat Nov 26 13:30:35 CET 2016]
offset:[11535]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a09dcdf100002d17] timestamp:[Sat Nov 26 13:30:46 CET 2016]
offset:[11543]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a09dfdb700002d1f] timestamp:[Sat Nov 26 13:30:58 CET 2016]
offset:[11551]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a09e311700002d28] timestamp:[Sat Nov 26 13:31:11 CET 2016]
offset:[11560]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a09eccaa00002d31] timestamp:[Sat Nov 26 13:31:51 CET 2016]
offset:[11569]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a09f542300002d3a] timestamp:[Sat Nov 26 13:32:26 CET 2016]
offset:[11578]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0a026c100002d43] timestamp:[Sat Nov 26 13:33:20 CET 2016]
offset:[11587]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0a0d8c900002d4b] timestamp:[Sat Nov 26 13:34:05 CET 2016]
offset:[11595]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0a163d400002d53] timestamp:[Sat Nov 26 13:34:41 CET 2016]
offset:[11603]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0a310fe00002d5c] timestamp:[Sat Nov 26 13:36:31 CET 2016]
offset:[11612]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0a3b0a300002d65] timestamp:[Sat Nov 26 13:37:11 CET 2016]
offset:[11621]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0a3cdf100002d6d] timestamp:[Sat Nov 26 13:37:19 CET 2016]
offset:[11629]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0a42ba200002d75] timestamp:[Sat Nov 26 13:37:43 CET 2016]
offset:[11637]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0a4864900002d7e] timestamp:[Sat Nov 26 13:38:06 CET 2016]
offset:[11646]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0a500e700002d86] timestamp:[Sat Nov 26 13:38:38 CET 2016]
offset:[11654]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0a5a38200002d8e] timestamp:[Sat Nov 26 13:39:19 CET 2016]
offset:[11662]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0a5f5a400002d96] timestamp:[Sat Nov 26 13:39:40 CET 2016]
offset:[11670]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0a6e30400002d9e] timestamp:[Sat Nov 26 13:40:41 CET 2016]
offset:[11678]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0a7e6b600002da7] timestamp:[Sat Nov 26 13:41:47 CET 2016]
offset:[11687]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0aa276300002db1] timestamp:[Sat Nov 26 13:44:15 CET 2016]
offset:[11697]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0ab040c00002dbb] timestamp:[Sat Nov 26 13:45:12 CET 2016]
offset:[11707]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0ab6d2c00002dc3] timestamp:[Sat Nov 26 13:45:38 CET 2016]
offset:[11715]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0aba59e00002dcb] timestamp:[Sat Nov 26 13:45:53 CET 2016]
offset:[11723]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0acb9b200002dd4] timestamp:[Sat Nov 26 13:47:04 CET 2016]
offset:[11732]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0ae188e00002ddc] timestamp:[Sat Nov 26 13:48:33 CET 2016]
offset:[11740]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0ae682b00002de4] timestamp:[Sat Nov 26 13:48:54 CET 2016]
offset:[11748]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0ae7a9b00002ded] timestamp:[Sat Nov 26 13:48:59 CET 2016]
offset:[11757]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0ae9bed00002df5] timestamp:[Sat Nov 26 13:49:07 CET 2016]
offset:[11765]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0aeb82200002dfd] timestamp:[Sat Nov 26 13:49:14 CET 2016]
offset:[11773]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0aed16a00002e05] timestamp:[Sat Nov 26 13:49:21 CET 2016]
offset:[11781]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0aedd8d00002e0d] timestamp:[Sat Nov 26 13:49:24 CET 2016]
offset:[11789]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0aff17200002e16] timestamp:[Sat Nov 26 13:50:34 CET 2016]
offset:[11798]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0b0d65e00002e1e] timestamp:[Sat Nov 26 13:51:33 CET 2016]
offset:[11806]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0b1e31f00002e26] timestamp:[Sat Nov 26 13:52:42 CET 2016]
offset:[11814]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0b3736300002e2e] timestamp:[Sat Nov 26 13:54:24 CET 2016]
offset:[11822]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0b4648a00002e36] timestamp:[Sat Nov 26 13:55:26 CET 2016]
offset:[11830]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0b7ad3d00002e40] timestamp:[Sat Nov 26 13:59:01 CET 2016]
offset:[11840]
file:[/var/lib/kafka/xdc_listing-status-v2-1/00000000000000000000.timeindex]
raw:[00000158a0b7efe700002e49] timestamp:[Sat Nov 26 13:59:18 CET 2016]
offset:[11849]
--
Robert Schumann | Lead DevOps Engineer | mobile.de GmbH
T: + 49. 30. 8109. 7219
M: +49.151. 5504. 8246
F: +49. 30. 8109. 7131
roschum...@ebay.com<mailto:roschum...@ebay.com>
www.mobile.de
Marktplatz 1 | 14532 Europarc Dreilinden | Germany
Geschaeftsfuehrer: Malte Krueger
HRB Nr.: 18517 P, AG Potsdam
Sitz der Gesellschaft: Kleinmachnow
