Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/13352 )
Change subject: IMPALA-8473: publish lineage info via hook ...................................................................... Patch Set 14: (1 comment) Thanks Radford, I think you'e making progress http://gerrit.cloudera.org:8080/#/c/13352/14/fe/src/main/java/org/apache/impala/service/Frontend.java File fe/src/main/java/org/apache/impala/service/Frontend.java: http://gerrit.cloudera.org:8080/#/c/13352/14/fe/src/main/java/org/apache/impala/service/Frontend.java@1532 PS14, Line 1532: // TODO: can we make use of the futures to implement better > Yes, whatever behavior we end up on needs to be clearly documented. These Currently the code is using a thread pool with a fixed number of threads and an unbounded queue. At present the thread execution has no timeout and so could block forever, which is what you what I think ("hook execution is not time-bound"). So if all the threads block then we will start queuing work forever, in a queue that cannot currently be observed. One idea would be to have thread pool with a fixed number of threads and an bounded queue. If the queue fills up then new work is rejected (which I think means thrown away) and logged. This means we are doing some sort of best effort to deliver work but not at the risk of consuming unlimited resources. And hopefully this will work well... But the next question is, when things go wrong how will you diagnose it? You can look at log messages but all they tell you is that work is being discarded. This might be because the worker threads are all stuck forever, but they might be just slow. So we want information to diagnose these issues, preferably observable in the webui/metrics as well as in log messages. -- To view, visit http://gerrit.cloudera.org:8080/13352 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23a896537a98bfef07fb27c70e9a87c105cd77a1 Gerrit-Change-Number: 13352 Gerrit-PatchSet: 14 Gerrit-Owner: radford nguyen <radford.ngu...@gmail.com> Gerrit-Reviewer: Andrew Sherman <asher...@cloudera.com> Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Austin Nobis <ano...@cloudera.com> Gerrit-Reviewer: Fredy Wijaya <fwij...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: radford nguyen <radford.ngu...@gmail.com> Gerrit-Comment-Date: Tue, 21 May 2019 16:28:08 +0000 Gerrit-HasComments: Yes