heftig added a comment.
In D8076#119526 <https://phab.mercurial-scm.org/D8076#119526>, @yuja wrote: > Might be better to optimize the common case `wrapped.read(size) == size`. I thought my code was already pretty optimized: We allocate the buffer to read into just once (no matter how many reads we issue) and give it to the unpickler without copying the data. > FWIW, if we don't mind issuing extra `read()` syscalls, maybe we can abuse > BufferedReader of `buffer_size=1`. My gut says that using just read(1) will be way worse when we try to transmit large objects. That said, I don't know if the Unpickler tries to do such large reads. > Another option is to rewrite the select loop to fully manage response buffer > by ourselves. > > for key, events in selector.select(): > ... > our_buffer.extend(key.fileobj.read()) > temp_io = BytesIO(our_buffer) > while ...: > try: > pickle.load(temp_io) > except ... > ... > del our_buffer[:temp_io.tell()] That would be an option, too, although `bytearray.extend` + `BytesIO.read` + `bytearray.__delitem__` looks a bit heavy on the memcpys/memmoves. We also have to restart the unpickling whenever the data was truncated. And remember when we're actually EOF. Wouldn't it look more like this? for key, events in selector.select(): try: read_data = key.fileobj.read() except IOError as e: if e.errno == errno.EINTR: continue raise: eof = not read_data if eof: selector.unregister(key.fileobj) key.fileobj.close() openpipes -= 1 buf = buffers[key.fileobj] buf.extend(read_data) temp_io = BytesIO(buf) pos = 0 while True: try: res = pickle.load(temp_io) except EOFError: break except UnpicklingError: if eof: raise break pos = temp_io.tell() if hasretval and res[0]: retval.update(res[1]) else: yield res del buf[:pos] This gets pretty convoluted. REPOSITORY rHG Mercurial CHANGES SINCE LAST ACTION https://phab.mercurial-scm.org/D8076/new/ REVISION DETAIL https://phab.mercurial-scm.org/D8076 To: heftig, #hg-reviewers, yuja Cc: yuja, mjpieters, mercurial-devel _______________________________________________ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
