I was check the code and i see these lines for that AIO timeout.
case qpid::linearstore::journal::RHM_IORES_PAGE_AIOWAIT:
if (++aio_sleep_cnt > MAX_AIO_SLEEPS)
THROW_STORE_EXCEPTION("Timeout waiting for AIO in
MessageStoreImpl::recoverMessages()");
::usleep(AIO_SLEEP_TIME_US);
break;
And these are the defaults
#define MAX_AIO_SLEEPS 100000 // tot: ~1 sec
#define AIO_SLEEP_TIME_US 10 // 0.01 ms
RHM_IORES_PAGE_AIOWAIT, ///< IO operation suspended - next page is
waiting for AIO.
So does page got blocked and its waiting for page availability?
Ram
On Mon, Nov 5, 2018 at 8:00 PM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:
>
> Actually we have upgraded from qpid-cpp 0.28 to 1.35 and after that we see
> this message
>
> 2018-10-27 18:58:25 [Store] warning Linear Store: Journal
> "<journal-name>": Bad record alignment found at fid=0x4605b offs=0x107680
> (likely journal overwrite boundary); 19 filler record(s) required.
> 2018-10-27 18:58:25 [Store] notice Linear Store: Journal "<journal-name>":
> Recover phase write: Wrote filler record: fid=0x4605b offs=0x107680
> 2018-10-27 18:58:25 [Store] notice Linear Store: Journal "<journal-name>":
> Recover phase write: Wr... few more Recover phase logs
>
> It worked fine for a day and started throwing this message:
>
> 2018-10-28 12:27:01 [Store] critical Linear Store: Journal "<name>":
> get_events() returned JERR_JCNTL_AIOCMPLWAIT; wmgr_status: wmgr: pi=25 pc=8
> po=0 aer=1 edac=TFFF ps=[-------------------------A------]
> 2018-10-28 12:27:01 [Broker] warning Exchange <name> cannot deliver to
> queue <queue-name>: Queue <queue-name>: MessageStoreImpl::store() failed:
> jexception 0x0202 jcntl::handle_aio_wait() threw JERR_JCNTL_AIOCMPLWAIT:
> Timeout waiting for AIOs to complete.
> (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211)
> 2018-10-28 12:27:01 [Broker] error Connection exception: framing-error:
> Queue <queue-name>: MessageStoreImpl::store() failed: jexception 0x0202
> jcntl::handle_aio_wait() threw JERR_JCNTL_AIOCMPLWAIT: Timeout waiting for
> AIOs to complete.
> (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211)
> 2018-10-28 12:27:01 [Protocol] error Connection
> qpid.server-ip:5672-client-ip:44457 closed by error: Queue <queue-name>:
> MessageStoreImpl::store() failed: jexception 0x0202
> jcntl::handle_aio_wait() threw JERR_JCNTL_AIOCMPLWAIT: Timeout waiting for
> AIOs to complete.
> (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211)(501)
> 2018-10-28 12:27:01 [Protocol] error Connection
> qpid.server-ip:5672-client-ip:44457 closed by error: illegal-argument:
> Value for replyText is too large(320)
>
> Thanks,
> Ram
>
>
> On Mon, Nov 5, 2018 at 3:34 PM rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
>> No, local disk.
>>
>> On Mon, Nov 5, 2018 at 3:26 PM Gordon Sim <g...@redhat.com> wrote:
>>
>>> On 05/11/18 22:58, rammohan ganapavarapu wrote:
>>> > Gordon,
>>> >
>>> > We are using java client 0.28 version and qpidd-cpp 1.35 version
>>> > (qpid-cpp-server-1.35.0-1.el7.x86_64), i dont know at what scenario its
>>> > happening but after i restart broker and if we wait for few days its
>>> > happening again. From the above logs do you have any pointers to check?
>>>
>>> Are you using NFS?
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
>>> For additional commands, e-mail: users-h...@qpid.apache.org
>>>
>>>
