[
https://issues.apache.org/jira/browse/THRIFT-5673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17647606#comment-17647606
]
Jens Geyer commented on THRIFT-5673:
------------------------------------
Why is that piece of code called at all? To me it looks like since poor {{def
read(self, sz):}} is instructed to read by someone to read sz bytes?
> TZlibTransport.py gets into infinite loop if underlying transport ends
> unexpectedly.
> ------------------------------------------------------------------------------------
>
> Key: THRIFT-5673
> URL: https://issues.apache.org/jira/browse/THRIFT-5673
> Project: Thrift
> Issue Type: Bug
> Components: Python - Library
> Reporter: Benjamin Mahler
> Priority: Major
>
> In python, when using a thrift client composed as follows, the
> TZlibTransport.py logic can enter an infinite loop when the response closes
> prematurely (e.g. disconnection):
> {code:java}
> class GzipTransport(TZlibTransport.TZlibTransport):
> def _init_zlib(self):
> """Override of internal method for setting up the zlib compression and
> decompression objects, to support 'gzip' content encoding.
> """
> self._zcomp_read = zlib.decompressobj(wbits=zlib.MAX_WBITS + 16)
> self._zcomp_write = zlib.compressobj(self.compresslevel,
> wbits=zlib.MAX_WBITS + 16)
> http_client.setCustomHeaders({"Content-Encoding": "gzip", "Accept-Encoding":
> "gzip"})
> transport = GzipTransport(http_client)
> transport = TTransport.TBufferedTransport(transport, rbuf_size=1024 * 1024)
> thriftClient = ReadOnlyScheduler.Client(protocol)
> httpClient.open()
> try:
> transport.open()
> return thriftClient.getFoo(FooQuery(...))
> finally:
> transport.close()
> {code}
> We observed that when the server shuts down at the same time as the response
> is being read, the program spins on the following stack trace:
> {code:java}
> File: "client.py", line 47, in query_scheduler_api
> return thrift_client_method(thrift_client, *args)
> File: "gen/apache/aurora/api/ReadOnlyScheduler.py", line 198, in
> getJobSummary
> return self.recv_getJobSummary()
> File: "gen/apache/aurora/api/ReadOnlyScheduler.py", line 210, in
> recv_getJobSummary
> (fname, mtype, rseqid) = iprot.readMessageBegin()
> File: "thrift/protocol/TJSONProtocol.py", line 417, in readMessageBegin
> self.readJSONArrayStart()
> File: "thrift/protocol/TJSONProtocol.py", line 405, in readJSONArrayStart
> self.readJSONSyntaxChar(LBRACKET)
> File: "thrift/protocol/TJSONProtocol.py", line 253, in readJSONSyntaxChar
> current = self.reader.read()
> File:
> "thrift-0.13.0-cp37-cp37m-linux_x86_64.whl/thrift/protocol/TJSONProtocol.py",
> line 164, in read
> self.data = self.protocol.trans.read(1)
> File: "thrift/transport/TTransport.py", line 164, in read
> self.__rbuf = BufferIO(self.__trans.read(max(sz, self.__rbuf_size)))
> File: "thrift/transport/TZlibTransport.py", line 191, in read
> if self.readComp(sz):
> File: "thrift/transport/TZlibTransport.py", line 208, in readComp
> return False
> {code}
> Essentially, it appears that when the json protocol asks to read 1 byte, and
> there's always 0 bytes returned from the http response (EOF), then the zlib
> transport will spin in a loop here:
> [https://github.com/apache/thrift/blob/v0.17.0/lib/py/src/transport/TZlibTransport.py#L190-L192]
> It seems like this is a bug, and the zlib transport should treat 0 bytes
> coming back from the underlying transport as EOF, at which point it closes
> its compression streams and tells the caller about the remaining data and EOF
> (possibly throwing an exception if the EOF does not cleanly finish the
> decompression stream).
> In our case, the underlying transport is ultimately the HttpClient's
> response, and that doesn't appear to throw an exception when the response is
> cut short, it just appears to return 0 bytes repeatedly.
> We use thrift 0.13.0 but I see the same logic in the latest release and the
> bug appears unfixed there as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
