Andrew Sherman has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13352 )

Change subject: IMPALA-8473: publish lineage info via hook
......................................................................


Patch Set 14:

(1 comment)

Thanks Radford, I think you'e making progress

http://gerrit.cloudera.org:8080/#/c/13352/14/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:

http://gerrit.cloudera.org:8080/#/c/13352/14/fe/src/main/java/org/apache/impala/service/Frontend.java@1532
PS14, Line 1532:     // TODO: can we make use of the futures to implement better
> Yes, whatever behavior we end up on needs to be clearly documented.  These
Currently the code is using a thread pool with a fixed number of threads
and an unbounded queue. At present the thread execution has no timeout
and so could block forever, which is what you what I think ("hook
execution is not time-bound").

So if all the threads block then we will start queuing work forever, in
a queue that cannot currently be observed. One idea would be to have
thread pool with a fixed number of threads and an bounded queue. If the
queue fills up then new work is rejected (which I think means thrown
away) and logged. This means we are doing some sort of best effort to
deliver work but not at the risk of consuming unlimited resources. And
hopefully this will work well...

But the next question is, when things go wrong how will you diagnose it?
You can look at log messages but all they tell you is that work is being
discarded. This might be because the worker threads are all stuck
forever, but they might be just slow. So we want information to diagnose
these issues, preferably observable in the webui/metrics as well as in
log messages.



--
To view, visit http://gerrit.cloudera.org:8080/13352
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23a896537a98bfef07fb27c70e9a87c105cd77a1
Gerrit-Change-Number: 13352
Gerrit-PatchSet: 14
Gerrit-Owner: radford nguyen <radford.ngu...@gmail.com>
Gerrit-Reviewer: Andrew Sherman <asher...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (498)
Gerrit-Reviewer: Austin Nobis <ano...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fwij...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: radford nguyen <radford.ngu...@gmail.com>
Gerrit-Comment-Date: Tue, 21 May 2019 16:28:08 +0000
Gerrit-HasComments: Yes

Reply via email to