[
https://issues.apache.org/jira/browse/HADOOP-4346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raghu Angadi updated HADOOP-4346:
---------------------------------
Attachment: HADOOP-4346.patch
Thanks Nicholas.
Updated patch is attached.
> In case SelectionKey.OP_WRITE, "read" should be "write"
good catch.
> I suggest to simplify the codes for timing as below
I modified the loop a bit like the way you suggested, but not exactly same,
since it requires two calls to currentTime() instead of one for each loop..
even in the common case of no timeouts.
> Hadoop triggers a "soft" fd leak.
> ----------------------------------
>
> Key: HADOOP-4346
> URL: https://issues.apache.org/jira/browse/HADOOP-4346
> Project: Hadoop Core
> Issue Type: Bug
> Components: io
> Affects Versions: 0.17.0
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Attachments: HADOOP-4346-branch-18.patch, HADOOP-4346.patch,
> HADOOP-4346.patch, HADOOP-4346.patch, HADOOP-4346.patch
>
>
> Starting with Hadoop-0.17, most of the network I/O uses non-blocking NIO
> channels. Normal blocking reads and writes are handled by Hadoop and use our
> own cache of selectors. This cache suites well for Hadoop where I/O often
> occurs on many short lived threads. Number of fds consumed is proportional to
> number of threads currently blocked.
> If blocking I/O is done using java.*, Sun's implementation uses internal
> per-thread selectors. These selectors are closed using {{sun.misc.Cleaner}}.
> Looks like this cleaning is kind of like finalizers and tied to GC. This is
> pretty ill suited if we have many threads that are short lived. Until GC
> happens, number of these selectors keeps growing. Each selector consumes 3
> fds.
> Though blocking read and write are handled by Hadoop, {{connect()}} is still
> the default implementation that uses per-thread selector.
> Koji helped a lot in tracking this. Some sections from 'jmap' output and
> other info Koji collected led to this suspicion and will include that in the
> next comment.
> One solution might be to handle connect() also in Hadoop using our selectors.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
