https://github.com/wal-e/wal-e/pull/92
I think I found what might be a serious bug (but, seemingly not
often(?) in practice a problem) and optimization opportunities in
piper.py, as well as an explanation for an old problem documented in
the past.
Regrettably, it's a fairly old code and hasn't munched databases en
masse, so even a change to make it ostensibly better demands care and
review as there is not a copious suite -- including fault injection --
to test all strange code paths.
I hope some people will help.
Here's the commit message inlined for convenience:
"""
Redo questionable the non-blocking-pipe I/O routines
There are a few reasons to revisit this code.
Most distressingly is this:
buf = StringIO(data)
while bytes_written < bytes_total:
try:
# self._fp.write() doesn't return anything, so use
# os.write.
bytes_written += os.write(self._fp.fileno(),
buf.read(PIPE_BUF_BYTES))
except IOError, ex:
if ex[0] != errno.EAGAIN:
raise
sys.exc_clear()
In particular, the cursor-advancing side effect of "buf.read" (which
has side effects) wrapped inside a loop that might retry in event of
receiving EAGAIN from an "os.write". It has now been re-spelled to
avoid this.
Also changed is catching EnvironmentError rather than IOError, because
of a report from Art Fedorov <[email protected]> on Macintosh that came
from an OSError which is not caught and retried when it possesses the
errno EAGAIN as one might hope.
Also, the prior code tended to call gevent.socket.wait_read and
gevent.socket.wait_write too often, which may be vastly increasing the
number of system calls. Now those are only called when EAGAIN is
encountered.
Finally, I wonder of the possibility of mistaken increments on the
cursor on the CStringIO object above caused some of the reported
Macintosh deadlock problems, solved by reading the atomic pipe buffer
size from the operating system (see
0dd95420f354b4656d4b7ce5b1ba1720405b908c). If so, there may be a new
opportunity to use larger buffer sizes and reduce overhead.
"""
--
You received this message because you are subscribed to the Google Groups
"wal-e" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
