Yes, that would be useful.

Thanks,

Jun

On Tue, Jan 6, 2015 at 2:07 PM, Sa Li <[email protected]> wrote:

> BTW, I found the the /kafka/logs also getting biger and bigger, like
> controller.log and state-change.logs. should I launch a cron the clean them
> up regularly or there is way to delete them regularly?
>
> thanks
>
> AL
>
> On Tue, Jan 6, 2015 at 2:01 PM, Sa Li <[email protected]> wrote:
>
> > Hi, All
> >
> > We fix the problem, I like to share the what the problem is in case
> > someone come across the similar issues. We add the data drive for each
> node
> > /dev/sdb1 , but specify the wrong path in server.properties, which means
> > the data was written into the wrong drive /dev/sda2, quickly eat up all
> the
> > space in sda2, now we change the path. The sdb1 has 15Tb, which allows us
> > to store data for a while and will be deleted in 1/2 weeks as config
> > mentioned.
> >
> > But I am kinda curious about David's comments,  "... after having tuned
> > retention bytes or retention (time?) incorrectly. .."  How do you guys
> set
> > log.retention.bytes?  I set log.retention.hours=336 (2 weeks), and should
> > I set log.retention.bytes as default -1 or some other amount?
> >
> > thanks
> >
> > AL
> >
> > On Tue, Jan 6, 2015 at 12:43 PM, Sa Li <[email protected]> wrote:
> >
> >> Thanks the reply, the disk is not full:
> >>
> >> root@exemplary-birds:~# df -h
> >> Filesystem      Size  Used Avail Use% Mounted on
> >> /dev/sda2       133G  3.4G  123G   3% /
> >> none            4.0K     0  4.0K   0% /sys/fs/cgroup
> >> udev             32G  4.0K   32G   1% /dev
> >> tmpfs           6.3G  764K  6.3G   1% /run
> >> none            5.0M     0  5.0M   0% /run/lock
> >> none             32G     0   32G   0% /run/shm
> >> none            100M     0  100M   0% /run/user
> >> /dev/sdb1        14T   15G   14T   1% /srv
> >>
> >> Neither the memory
> >>
> >> root@exemplary-birds:~# free
> >>              total       used       free     shared    buffers
>  cached
> >> Mem:      65963372    9698380   56264992        776     170668
> 7863812
> >> -/+ buffers/cache:    1663900   64299472
> >> Swap:       997372          0     997372
> >>
> >> thanks
> >>
> >>
> >> On Tue, Jan 6, 2015 at 12:10 PM, David Birdsong <
> [email protected]
> >> > wrote:
> >>
> >>> I'm keen to hear about how to work one's way out of a filled partition
> >>> since I've run into this many times after having tuned retention bytes
> or
> >>> retention (time?) incorrectly. The proper path to resolving this isn't
> >>> obvious based on my many harried searches through documentation.
> >>>
> >>> I often end up stopping the particular broker, picking an unlucky
> >>> topic/partition, deleting, modifying the any topics that consumed too
> >>> much
> >>> space by lowering their retention bytes, and restarting.
> >>>
> >>> On Tue, Jan 6, 2015 at 12:02 PM, Sa Li <[email protected]> wrote:
> >>>
> >>> > Continue this issue, when I restart the server, like
> >>> > bin/kafka-server-start.sh config/server.properties
> >>> >
> >>> > it will fails to start the server, like
> >>> >
> >>> > [2015-01-06 20:00:55,441] FATAL Fatal error during KafkaServerStable
> >>> > startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
> >>> > java.lang.InternalError: a fault occurred in a recent unsafe memory
> >>> access
> >>> > operation in compiled Java code
> >>> >         at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
> >>> >         at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
> >>> >         at
> >>> > kafka.log.FileMessageSet$$anon$1.makeNext(FileMessageSet.scala:188)
> >>> >         at
> >>> > kafka.log.FileMessageSet$$anon$1.makeNext(FileMessageSet.scala:165)
> >>> >         at
> >>> >
> >>>
> kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
> >>> >         at
> >>> kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)
> >>> >         at kafka.log.LogSegment.recover(LogSegment.scala:165)
> >>> >         at kafka.log.Log.recoverLog(Log.scala:179)
> >>> >         at kafka.log.Log.loadSegments(Log.scala:155)
> >>> >         at kafka.log.Log.<init>(Log.scala:64)
> >>> >         at
> >>> >
> >>> >
> >>>
> kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:118)
> >>> >         at
> >>> >
> >>> >
> >>>
> kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:113)
> >>> >         at
> >>> >
> >>> >
> >>>
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> >>> >         at
> >>> > scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:105)
> >>> >         at
> >>> > kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:113)
> >>> >         at
> >>> > kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:105)
> >>> >         at
> >>> >
> >>> >
> >>>
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> >>> >         at
> >>> > scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
> >>> >         at kafka.log.LogManager.loadLogs(LogManager.scala:105)
> >>> >         at kafka.log.LogManager.<init>(LogManager.scala:57)
> >>> >         at
> >>> kafka.server.KafkaServer.createLogManager(KafkaServer.scala:275)
> >>> >         at kafka.server.KafkaServer.startup(KafkaServer.scala:72)
> >>> >         at
> >>> >
> >>>
> kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
> >>> >         at kafka.Kafka$.main(Kafka.scala:46)
> >>> >         at kafka.Kafka.main(Kafka.scala)
> >>> > [2015-01-06 20:00:55,443] INFO [Kafka Server 100], shutting down
> >>> > (kafka.server.KafkaServer)
> >>> > [2015-01-06 20:00:55,444] INFO Terminate ZkClient event thread.
> >>> > (org.I0Itec.zkclient.ZkEventThread)
> >>> > [2015-01-06 20:00:55,446] INFO Session: 0x684a5ed9da3a1a0f closed
> >>> > (org.apache.zookeeper.ZooKeeper)
> >>> > [2015-01-06 20:00:55,446] INFO EventThread shut down
> >>> > (org.apache.zookeeper.ClientCnxn)
> >>> > [2015-01-06 20:00:55,447] INFO [Kafka Server 100], shut down
> completed
> >>> > (kafka.server.KafkaServer)
> >>> > [2015-01-06 20:00:55,447] INFO [Kafka Server 100], shutting down
> >>> > (kafka.server.KafkaServer)
> >>> >
> >>> > Any ideas
> >>> >
> >>> > On Tue, Jan 6, 2015 at 12:00 PM, Sa Li <[email protected]> wrote:
> >>> >
> >>> > > the complete error message:
> >>> > >
> >>> > > -su: cannot create temp file for here-document: No space left on
> >>> device
> >>> > > OpenJDK 64-Bit Server VM warning: Insufficient space for shared
> >>> memory
> >>> > > file:
> >>> > >    /tmp/hsperfdata_root/19721
> >>> > > Try using the -Djava.io.tmpdir= option to select an alternate temp
> >>> > > location.
> >>> > > [2015-01-06 19:50:49,244] FATAL  (kafka.Kafka$)
> >>> > > java.io.FileNotFoundException: conf (No such file or directory)
> >>> > >         at java.io.FileInputStream.open(Native Method)
> >>> > >         at java.io.FileInputStream.<init>(FileInputStream.java:146)
> >>> > >         at java.io.FileInputStream.<init>(FileInputStream.java:101)
> >>> > >         at kafka.utils.Utils$.loadProps(Utils.scala:144)
> >>> > >         at kafka.Kafka$.main(Kafka.scala:34)
> >>> > >         at kafka.Kafka.main(Kafka.scala)
> >>> > >
> >>> > > On Tue, Jan 6, 2015 at 11:58 AM, Sa Li <[email protected]> wrote:
> >>> > >
> >>> > >>
> >>> > >> Hi, All
> >>> > >>
> >>> > >> I am doing performance test on our new kafka production server,
> but
> >>> > after
> >>> > >> sending some messages (even faked message by using
> >>> > bin/kafka-run-class.sh
> >>> > >> org.apache.kafka.clients.tools.ProducerPerformance), it comes out
> >>> the
> >>> > error
> >>> > >> of connection, and shut down the brokers, after that, I see such
> >>> errors,
> >>> > >>
> >>> > >> conf-su: cannot create temp file for here-document: No space left
> on
> >>> > >> device
> >>> > >>
> >>> > >> How can I fix it, I am concerning that will happen when we start
> to
> >>> > >> publish real messages in kafka, and should I create some cron to
> >>> > regularly
> >>> > >> clean certain directories?
> >>> > >>
> >>> > >> thanks
> >>> > >>
> >>> > >> --
> >>> > >>
> >>> > >> Alec Li
> >>> > >>
> >>> > >
> >>> > >
> >>> > >
> >>> > > --
> >>> > >
> >>> > > Alec Li
> >>> > >
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> >
> >>> > Alec Li
> >>> >
> >>>
> >>
> >>
> >>
> >> --
> >>
> >> Alec Li
> >>
> >
> >
> >
> > --
> >
> > Alec Li
> >
>
>
>
> --
>
> Alec Li
>

Reply via email to