> On Nov. 19, 2019, 6:53 p.m., Slim Bouguerra wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java
> > Lines 274 (patched)
> > <https://reviews.apache.org/r/71784/diff/1/?file=2173907#file2173907line275>
> >
> > looking at the java code i see that it is using a bounded queue so not
> > sure what you mean by unbounded ?
> > can you please clarify ?
>
> Attila Magyar wrote:
> No, unfortunatelly it uses an unbounded DelayedWorkQueue internally and
> it cannot be changed.
```java
/**
* Specialized delay queue. To mesh with TPE declarations, this
* class must be declared as a BlockingQueue<Runnable> even though
* it can only hold RunnableScheduledFutures.
*/
static class DelayedWorkQueue extends AbstractQueue<Runnable>
implements BlockingQueue<Runnable> {
```
but it is blocking means that it should block and therefore we get the
RejectedException.
> On Nov. 19, 2019, 6:53 p.m., Slim Bouguerra wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java
> > Lines 279 (patched)
> > <https://reviews.apache.org/r/71784/diff/1/?file=2173907#file2173907line280>
> >
> > i am still not sure how this is going to work?
> > the original code was dropping events when the queue is full that is
> > the case where you see the `RejectedExecutionException`
>
> Attila Magyar wrote:
> RejectedExecutionException was never thrown with the original code
> because of the unbounded queue. The queue continued to be larger. In the heap
> dump there 17000 elements in the queue totally and it takes about 2.5g space.
Very strange,
As per the java doc it says ` Executor uses finite bounds for both maximum
threads and work queue capacity, and is saturated`
https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ThreadPoolExecutor.html
```
New tasks submitted in method execute(Runnable) will be rejected when the
Executor has been shut down, and also when the Executor uses finite bounds for
both maximum threads and work queue capacity, and is saturated. In either case,
the execute method invokes the
RejectedExecutionHandler.rejectedExecution(Runnable, ThreadPoolExecutor) method
of its RejectedExecutionHandler. Four predefined handler policies are provided:
In the default ThreadPoolExecutor.AbortPolicy, the handler throws a runtime
RejectedExecutionException upon rejection.
In ThreadPoolExecutor.CallerRunsPolicy, the thread that invokes execute itself
runs the task. This provides a simple feedback control mechanism that will slow
down the rate that new tasks are submitted.
In ThreadPoolExecutor.DiscardPolicy, a task that cannot be executed is simply
dropped.
In ThreadPoolExecutor.DiscardOldestPolicy, if the executor is not shut down,
the task at the head of the work queue is dropped, and then execution is
retried (which can fail again, causing this to be repeated.)
It is possible to define and use other kinds of RejectedExecutionHandler
classes. Doing so requires some care especially when policies are designed to
work only under particular capacity or queuing policies.
Hook methods
```
- Slim
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71784/#review218679
-----------------------------------------------------------
On Nov. 19, 2019, 3:43 p.m., Attila Magyar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71784/
> -----------------------------------------------------------
>
> (Updated Nov. 19, 2019, 3:43 p.m.)
>
>
> Review request for hive, Laszlo Bodor, Harish Jaiprakash, Mustafa Iman, and
> Panos Garefalakis.
>
>
> Bugs: HIVE-22514
> https://issues.apache.org/jira/browse/HIVE-22514
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HiveProtoLoggingHook uses a ScheduledThreadPoolExecutor to submit writer
> tasks and to periodically handle rollover. The builtin
> ScheduledThreadPoolExecutor uses a unbounded queue which cannot be replaced
> from the outside. If log events are generated at a very fast rate this queue
> can grow large.
>
> Since ScheduledThreadPoolExecutor does not support changing the default
> unbounded queue to a bounded one, the queue capacity is checked manually by
> the patch.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a7687d59004
> ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java
> 8eab54859bf
> ql/src/test/org/apache/hadoop/hive/ql/hooks/TestHiveProtoLoggingHook.java
> 450a0b544d6
>
>
> Diff: https://reviews.apache.org/r/71784/diff/1/
>
>
> Testing
> -------
>
> unittest
>
>
> Thanks,
>
> Attila Magyar
>
>
