Yes, that would be useful. Thanks,
Jun On Tue, Jan 6, 2015 at 2:07 PM, Sa Li <[email protected]> wrote: > BTW, I found the the /kafka/logs also getting biger and bigger, like > controller.log and state-change.logs. should I launch a cron the clean them > up regularly or there is way to delete them regularly? > > thanks > > AL > > On Tue, Jan 6, 2015 at 2:01 PM, Sa Li <[email protected]> wrote: > > > Hi, All > > > > We fix the problem, I like to share the what the problem is in case > > someone come across the similar issues. We add the data drive for each > node > > /dev/sdb1 , but specify the wrong path in server.properties, which means > > the data was written into the wrong drive /dev/sda2, quickly eat up all > the > > space in sda2, now we change the path. The sdb1 has 15Tb, which allows us > > to store data for a while and will be deleted in 1/2 weeks as config > > mentioned. > > > > But I am kinda curious about David's comments, "... after having tuned > > retention bytes or retention (time?) incorrectly. .." How do you guys > set > > log.retention.bytes? I set log.retention.hours=336 (2 weeks), and should > > I set log.retention.bytes as default -1 or some other amount? > > > > thanks > > > > AL > > > > On Tue, Jan 6, 2015 at 12:43 PM, Sa Li <[email protected]> wrote: > > > >> Thanks the reply, the disk is not full: > >> > >> root@exemplary-birds:~# df -h > >> Filesystem Size Used Avail Use% Mounted on > >> /dev/sda2 133G 3.4G 123G 3% / > >> none 4.0K 0 4.0K 0% /sys/fs/cgroup > >> udev 32G 4.0K 32G 1% /dev > >> tmpfs 6.3G 764K 6.3G 1% /run > >> none 5.0M 0 5.0M 0% /run/lock > >> none 32G 0 32G 0% /run/shm > >> none 100M 0 100M 0% /run/user > >> /dev/sdb1 14T 15G 14T 1% /srv > >> > >> Neither the memory > >> > >> root@exemplary-birds:~# free > >> total used free shared buffers > cached > >> Mem: 65963372 9698380 56264992 776 170668 > 7863812 > >> -/+ buffers/cache: 1663900 64299472 > >> Swap: 997372 0 997372 > >> > >> thanks > >> > >> > >> On Tue, Jan 6, 2015 at 12:10 PM, David Birdsong < > [email protected] > >> > wrote: > >> > >>> I'm keen to hear about how to work one's way out of a filled partition > >>> since I've run into this many times after having tuned retention bytes > or > >>> retention (time?) incorrectly. The proper path to resolving this isn't > >>> obvious based on my many harried searches through documentation. > >>> > >>> I often end up stopping the particular broker, picking an unlucky > >>> topic/partition, deleting, modifying the any topics that consumed too > >>> much > >>> space by lowering their retention bytes, and restarting. > >>> > >>> On Tue, Jan 6, 2015 at 12:02 PM, Sa Li <[email protected]> wrote: > >>> > >>> > Continue this issue, when I restart the server, like > >>> > bin/kafka-server-start.sh config/server.properties > >>> > > >>> > it will fails to start the server, like > >>> > > >>> > [2015-01-06 20:00:55,441] FATAL Fatal error during KafkaServerStable > >>> > startup. Prepare to shutdown (kafka.server.KafkaServerStartable) > >>> > java.lang.InternalError: a fault occurred in a recent unsafe memory > >>> access > >>> > operation in compiled Java code > >>> > at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) > >>> > at java.nio.ByteBuffer.allocate(ByteBuffer.java:331) > >>> > at > >>> > kafka.log.FileMessageSet$$anon$1.makeNext(FileMessageSet.scala:188) > >>> > at > >>> > kafka.log.FileMessageSet$$anon$1.makeNext(FileMessageSet.scala:165) > >>> > at > >>> > > >>> > kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66) > >>> > at > >>> kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58) > >>> > at kafka.log.LogSegment.recover(LogSegment.scala:165) > >>> > at kafka.log.Log.recoverLog(Log.scala:179) > >>> > at kafka.log.Log.loadSegments(Log.scala:155) > >>> > at kafka.log.Log.<init>(Log.scala:64) > >>> > at > >>> > > >>> > > >>> > kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:118) > >>> > at > >>> > > >>> > > >>> > kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:113) > >>> > at > >>> > > >>> > > >>> > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) > >>> > at > >>> > scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:105) > >>> > at > >>> > kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:113) > >>> > at > >>> > kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:105) > >>> > at > >>> > > >>> > > >>> > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) > >>> > at > >>> > scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34) > >>> > at kafka.log.LogManager.loadLogs(LogManager.scala:105) > >>> > at kafka.log.LogManager.<init>(LogManager.scala:57) > >>> > at > >>> kafka.server.KafkaServer.createLogManager(KafkaServer.scala:275) > >>> > at kafka.server.KafkaServer.startup(KafkaServer.scala:72) > >>> > at > >>> > > >>> > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34) > >>> > at kafka.Kafka$.main(Kafka.scala:46) > >>> > at kafka.Kafka.main(Kafka.scala) > >>> > [2015-01-06 20:00:55,443] INFO [Kafka Server 100], shutting down > >>> > (kafka.server.KafkaServer) > >>> > [2015-01-06 20:00:55,444] INFO Terminate ZkClient event thread. > >>> > (org.I0Itec.zkclient.ZkEventThread) > >>> > [2015-01-06 20:00:55,446] INFO Session: 0x684a5ed9da3a1a0f closed > >>> > (org.apache.zookeeper.ZooKeeper) > >>> > [2015-01-06 20:00:55,446] INFO EventThread shut down > >>> > (org.apache.zookeeper.ClientCnxn) > >>> > [2015-01-06 20:00:55,447] INFO [Kafka Server 100], shut down > completed > >>> > (kafka.server.KafkaServer) > >>> > [2015-01-06 20:00:55,447] INFO [Kafka Server 100], shutting down > >>> > (kafka.server.KafkaServer) > >>> > > >>> > Any ideas > >>> > > >>> > On Tue, Jan 6, 2015 at 12:00 PM, Sa Li <[email protected]> wrote: > >>> > > >>> > > the complete error message: > >>> > > > >>> > > -su: cannot create temp file for here-document: No space left on > >>> device > >>> > > OpenJDK 64-Bit Server VM warning: Insufficient space for shared > >>> memory > >>> > > file: > >>> > > /tmp/hsperfdata_root/19721 > >>> > > Try using the -Djava.io.tmpdir= option to select an alternate temp > >>> > > location. > >>> > > [2015-01-06 19:50:49,244] FATAL (kafka.Kafka$) > >>> > > java.io.FileNotFoundException: conf (No such file or directory) > >>> > > at java.io.FileInputStream.open(Native Method) > >>> > > at java.io.FileInputStream.<init>(FileInputStream.java:146) > >>> > > at java.io.FileInputStream.<init>(FileInputStream.java:101) > >>> > > at kafka.utils.Utils$.loadProps(Utils.scala:144) > >>> > > at kafka.Kafka$.main(Kafka.scala:34) > >>> > > at kafka.Kafka.main(Kafka.scala) > >>> > > > >>> > > On Tue, Jan 6, 2015 at 11:58 AM, Sa Li <[email protected]> wrote: > >>> > > > >>> > >> > >>> > >> Hi, All > >>> > >> > >>> > >> I am doing performance test on our new kafka production server, > but > >>> > after > >>> > >> sending some messages (even faked message by using > >>> > bin/kafka-run-class.sh > >>> > >> org.apache.kafka.clients.tools.ProducerPerformance), it comes out > >>> the > >>> > error > >>> > >> of connection, and shut down the brokers, after that, I see such > >>> errors, > >>> > >> > >>> > >> conf-su: cannot create temp file for here-document: No space left > on > >>> > >> device > >>> > >> > >>> > >> How can I fix it, I am concerning that will happen when we start > to > >>> > >> publish real messages in kafka, and should I create some cron to > >>> > regularly > >>> > >> clean certain directories? > >>> > >> > >>> > >> thanks > >>> > >> > >>> > >> -- > >>> > >> > >>> > >> Alec Li > >>> > >> > >>> > > > >>> > > > >>> > > > >>> > > -- > >>> > > > >>> > > Alec Li > >>> > > > >>> > > >>> > > >>> > > >>> > -- > >>> > > >>> > Alec Li > >>> > > >>> > >> > >> > >> > >> -- > >> > >> Alec Li > >> > > > > > > > > -- > > > > Alec Li > > > > > > -- > > Alec Li >
