Hey guys,
Thanks for the problem report. I've created an issue to track it at
http://tracker.newdream.net/issues/2687.
It looks like we just assume that if you're using a file, you've got
enough space for it. It shouldn't be a big deal to at least do some
startup checks which will fail gracefully.
-Greg
On Wed, Jun 20, 2012 at 1:57 PM, Matthew Roy wrote:
> I hit this a couple times and wondered the same thing. Why does the
> OSD need to bail when it runs out of journal space?
>
> On Wed, Jun 20, 2012 at 3:56 PM, Travis Rhoden wrote:
>> Not sure if this is a bug or not. It was definitely user error -- but
>> since the OSD process bailed, figured I would report it.
>>
>> I had /tmpfs mounted with 2.5GB of space:
>>
>> tmpfs on /tmpfs type tmpfs (rw,size=2560m)
>>
>> Then I decided to increase my journal size to 5G, but forgot to
>> increase the limit on /tmpfs. =)
>>
>> osd journal size = 5000
>>
>>
>> Predictably, things didn't go well when I ran a rados bench that
>> filled up the journal. I'm not sure if such a case can be handled
>> more gracefully:
>>
>>
>> -4> 2012-06-20 12:39:36.648773 7fc042a5f780 1 journal _open
>> /tmpfs/osd.2.journal fd 30: 524288 bytes, block size 4096 bytes,
>> directio = 0, aio = 0
>> -3> 2012-06-20 12:42:23.179164 7fc02e1ad700 1
>> CephxAuthorizeHandler::verify_authorizer isvalid=1
>> -2> 2012-06-20 12:42:46.643205 7fc0396cf700 -1 journal
>> FileJournal::write_bl : write_fd failed: (28) No space left on device
>> -1> 2012-06-20 12:42:46.643245 7fc0396cf700 -1 journal
>> FileJournal::do_write: write_bl(pos=2678079488) failed
>> 0> 2012-06-20 12:42:46.676991 7fc0396cf700 -1 os/FileJournal.cc:
>> In function 'void FileJournal::do_write(ceph::bufferlist&)' thread
>> 7fc0396cf700 time 2012-06-20 12:42:46.643315
>> os/FileJournal.cc: 994: FAILED assert(0)
>>
>> ceph version 0.47.2 (commit:8bf9fde89bd6ebc4b0645b2fe02dadb1c17ad372)
>> 1: (FileJournal::do_write(ceph::buffer::list&)+0xe22) [0x653082]
>> 2: (FileJournal::write_thread_entry()+0x735) [0x659545]
>> 3: (FileJournal::Writer::entry()+0xd) [0x5de41d]
>> 4: (()+0x7e9a) [0x7fc042434e9a]
>> 5: (clone()+0x6d) [0x7fc0409e94bd]
>> NOTE: a copy of the executable, or `objdump -rdS ` is
>> needed to interpret this.
>>
>> --- end dump of recent events ---
>> 2012-06-20 12:42:46.693963 7fc0396cf700 -1 *** Caught signal (Aborted) **
>> in thread 7fc0396cf700
>>
>> ceph version 0.47.2 (commit:8bf9fde89bd6ebc4b0645b2fe02dadb1c17ad372)
>> 1: /usr/bin/ceph-osd() [0x6eb32a]
>> 2: (()+0xfcb0) [0x7fc04243ccb0]
>> 3: (gsignal()+0x35) [0x7fc04092d445]
>> 4: (abort()+0x17b) [0x7fc040930bab]
>> 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fc04127b69d]
>> 6: (()+0xb5846) [0x7fc041279846]
>> 7: (()+0xb5873) [0x7fc041279873]
>> 8: (()+0xb596e) [0x7fc04127996e]
>> 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x282) [0x79dd02]
>> 10: (FileJournal::do_write(ceph::buffer::list&)+0xe22) [0x653082]
>> 11: (FileJournal::write_thread_entry()+0x735) [0x659545]
>> 12: (FileJournal::Writer::entry()+0xd) [0x5de41d]
>> 13: (()+0x7e9a) [0x7fc042434e9a]
>> 14: (clone()+0x6d) [0x7fc0409e94bd]
>> NOTE: a copy of the executable, or `objdump -rdS ` is
>> needed to interpret this.
>>
>> --- begin dump of recent events ---
>> 0> 2012-06-20 12:42:46.693963 7fc0396cf700 -1 *** Caught signal
>> (Aborted) **
>> in thread 7fc0396cf700
>>
>> ceph version 0.47.2 (commit:8bf9fde89bd6ebc4b0645b2fe02dadb1c17ad372)
>> 1: /usr/bin/ceph-osd() [0x6eb32a]
>> 2: (()+0xfcb0) [0x7fc04243ccb0]
>> 3: (gsignal()+0x35) [0x7fc04092d445]
>> 4: (abort()+0x17b) [0x7fc040930bab]
>> 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fc04127b69d]
>> 6: (()+0xb5846) [0x7fc041279846]
>> 7: (()+0xb5873) [0x7fc041279873]
>> 8: (()+0xb596e) [0x7fc04127996e]
>> 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x282) [0x79dd02]
>> 10: (FileJournal::do_write(ceph::buffer::list&)+0xe22) [0x653082]
>> 11: (FileJournal::write_thread_entry()+0x735) [0x659545]
>> 12: (FileJournal::Writer::entry()+0xd) [0x5de41d]
>> 13: (()+0x7e9a) [0x7fc042434e9a]
>> 14: (clone()+0x6d) [0x7fc0409e94bd]
>> NOTE: a copy of the executable, or `objdump -rdS ` is
>> needed to interpret this.
>>
>> --- end dump of recent events ---
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majord...@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html