[
https://issues.apache.org/jira/browse/OOZIE-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16696595#comment-16696595
]
Andras Salamon commented on OOZIE-3382:
---------------------------------------
[~kmarton] Thanks for checking.
I prefer this approach because it does not require the introduction of a new
parameter:
* It would be possible to misconfigure the new parameter. For instance
specifying a large value for {{oozie.servlet.CallbackServlet.max.data.len}} and
low value for the new parameter. Or specifying large value for the new
parameter.
* I don't find the new parameter intuitive. It requires to understand the inner
details of the Ssh action's buffer draining.
I've opened a JIRA fo the {{TestSSh*}} tests: OOZIE-3391
I'll check the flakiness of the test.
> Optimize SshActionExecutor's drainBuffers method
> ------------------------------------------------
>
> Key: OOZIE-3382
> URL: https://issues.apache.org/jira/browse/OOZIE-3382
> Project: Oozie
> Issue Type: Improvement
> Reporter: Andras Salamon
> Assignee: Andras Salamon
> Priority: Major
> Attachments: OOZIE-3382-01.patch, OOZIE-3382-02.patch
>
>
> OOZIE-3354 improved {{SshActionExecutor}} to avoid {{Process#waitFor()}}
> blocks and modified the {{drainBuffers}} method to keep draining the standard
> output (and standard error) continuously.
> Right now the speed of the drain is hardwired. As long as the process is
> running the method only reads 1024 bytes in each cycle (half a second) which
> can take very long time if we want to drain several megabytes (for instance
> {{oozie.servlet.CallbackServlet.max.data.len}} is increased).
> Let's optimize the draining.
> We can either read 1024 bytes multiple times in each cycle (as long as there
> are data in the buffer), or we can increase the value of the buffer size
> (1024).
> In the latter case the default of the buffer size could be half of the
> {{oozie.servlet.CallbackServlet.max.data.len}} value, but we also need an
> additional property to specify the buffer size (to avoid memory problems
> because of using a very big buffer). We can keep 1024 as a minimum buffer
> size.
> It would be also useful to refactor the code and put the buffer draining into
> a separate class and create unit tests for the class. Using this class in
> {{ShellMain}} to avoid code duplication would also be very useful, but we
> have to fix OOZIE-3359 first.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
