On Mon, Oct 04, 2010 at 04:49:39PM +0200, Lars Ellenberg wrote:
> On Mon, Oct 04, 2010 at 02:42:22PM +0200, Dejan Muhamedagic wrote:
> > On Tue, Sep 28, 2010 at 05:01:21PM +0200, Lars Ellenberg wrote:
> > > # HG changeset patch
> > > # User Lars Ellenberg <l...@linbit.com>
> > > # Date 1285685263 -7200
> > > # Node ID beedb189b8a9c47c793f898755d0e301aacce4d4
> > > # Parent 94f4ee6025502c7ef710775d458ef766c51ad6db
> > > Medium: adjust socket buffers when adjusting ipc queue length
> > >
> > > If a channel adjusts its ipc queue length,
> > > it is probably a good idea to also change
> > > the socket buffers on the system level.
> > >
> > > The scaling qlen -> buffsize is somewhat arbitrary.
> > >
> > > diff -r 94f4ee602550 -r beedb189b8a9 lib/clplumbing/ipcsocket.c
> > > --- a/lib/clplumbing/ipcsocket.c Tue Sep 28 16:01:48 2010 +0200
> > > +++ b/lib/clplumbing/ipcsocket.c Tue Sep 28 16:47:43 2010 +0200
> > > @@ -1715,6 +1715,36 @@
> > > return socket_get_recv_fd(ch);
> > > }
> > >
> > > +static void
> > > +socket_adjust_buf(struct IPC_CHANNEL *ch, int optname, unsigned q_len)
> > > +{
> > > + const char *direction = optname == SO_SNDBUF ? "snd" : "rcv";
> > > + int fd = socket_get_send_fd(ch);
> > > + unsigned byte;
> > > +
> > > + /* Arbitrary scaling.
> > > + * DEFAULT_MAX_QLEN is 64, default socket buf is often 64k to 128k,
> > > + * at least on those linux I checked.
> > > + * Keep that ratio, and allow for some overhead. */
> > > + if (qlen == 0)
> > > + /* client does not want anything,
> > > + * reduce system buffers as well */
> > > + byte = 4096;
> > > + else if (q_len < 512)
> > > + byte = (32 + q_len) * 1024;
> > > + else
> > > + byte = q_len * 1024;
> > > +
> > > + if (0 == setsockopt(fd, SOL_SOCKET, optname, &byte, sizeof(byte))) {
> > > + cl_log(LOG_DEBUG, "adjusted %sbuf size to %u", direction,
> > > bytes);
> > > + } else {
> > > + /* If this fails, you may need to adjust net.core.rmem_max,
> > > + * ...wmem_max, or equivalent */
> > > + cl_log(LOG_DEBUG, "adjust %sbuf size to %u failed: %s",
> > > + direction, byte, strerror(errno));
> >
> > Why not make this a higher severity? When do you expect
> > setsockopt to fail? Would it keep failing very often?
>
> Ok, I'll make it WARN?
>
> > While we're at it, we'll need some kind of logging interface
> > which would issue a log message every now and then. Some messages
> > may under some circumstances be logged very often.
>
> hmmm.
> hb_media (that's heartbeat proper) have their "suppresserrs" thingy,
> which they can test for.
>
> but you probably would rather see a "rate_limit" thing,
> and then do cl_log_rate_limit(burst, timeout, args ...)
Right, something like that. Though at times the message may
differ slightly if it includes some timing information.
> Do you have a list of messages/location that may benefit from
> such a rate limit?
At times I saw bursts of various "took too long to execute" or
"scheduled at ... but started later" warnings, which happen when
the host can't keep up with the load. That's in GSource.c; there
are macros at the top of the file. Then I got a report for the
lrmd issuing tons of warnings when it can't keep up with running
resource actions (though that needs more/different fixing). There
could be more, but can't recall any other right now.
Cheers,
Dejan
> --
> : Lars Ellenberg
> : LINBIT | Your Way to High Availability
> : DRBD/HA support and consulting http://www.linbit.com
>
> DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
> _______________________________________________________
> Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
> Home Page: http://linux-ha.org/
_______________________________________________________
Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/
