Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/16318 )
Change subject: IMPALA-9867: Add Support for Spilling to S3: Milestone 1 ...................................................................... Patch Set 30: (1 comment) http://gerrit.cloudera.org:8080/#/c/16318/30//COMMIT_MSG Commit Message: PS30: > Following on from our discussion about a potential deadlock, I do think the The problem happens in the distributed environment. Suppose we have 5 queries A/B/C/D/E and 3 servers: Server 1: Waiting for buffer: A/B/C Partial Written Files: D/E Server 2: Waiting for buffer: C/D/E Partial Written Files: A/B Server 3: Waiting for buffer: A/C/E Partial Written Files: B/D The partial written files happen when all of the data of the query has been spilled, the buffer of partial written files are released when the query ends, however, in the above case, no query can be finished and the queries are waiting for each other, this creates a deadlock. The current solution adds a timeout waiting for the buffer in patch 35. The timeout value is passed by a start option "wait_for_spill_buffer_timeout_s", default is 60s. -- To view, visit http://gerrit.cloudera.org:8080/16318 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I419b1d5dbbfe35334d9f964c4b65e553579fdc89 Gerrit-Change-Number: 16318 Gerrit-PatchSet: 30 Gerrit-Owner: Yida Wu <wydbaggio...@gmail.com> Gerrit-Reviewer: Abhishek Rawat <ara...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Sahil Takiar <stak...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Reviewer: Yida Wu <wydbaggio...@gmail.com> Gerrit-Comment-Date: Tue, 09 Feb 2021 02:00:25 +0000 Gerrit-HasComments: Yes
