[
https://issues.apache.org/jira/browse/HDFS-941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12866456#action_12866456
]
sam rash commented on HDFS-941:
-------------------------------
+1 for the idea of caching sockets, but I have some questions/concerns about
the implementation.
some comments:
1. avoid making the cache implementation tied to the class ReaderSocketCache.
Don't make the cache a static member of the same class. Let the cache be an
instantiable
object. Let DFSClient store the cache either as an instance or static var
(don't force everything to use the same cache instance--better for testing and
stubbing out as well)
2. a lot of the logic around re-using is complicated--I think this could be
simplified
a. not clear why sockets are always in the cache even if not usable: i
would think adding only when usable and removing when used would be cleaner?
b. if we can keep the cache clean, no need for lazy removal of unusable
sockets
3. shouldn't there be a cap on the # of sockets there can be in the cache?
-again, should only be usable ones, but a max # put into the cache
makes sense. If we have a flurry of reads using tons of sockets to several
DNs, no need to keep 100s or more sockets in a cache
4. general concern about potential socket leaks;
5. seems like this needs more thought into the effects of synchronization: the
freemap has to be traversed every time to get a socket in a sync block. see
above if we can
avoid lazy removal by not putting unusable sockets in the cache (unsuable
either since they are in use or not usable at all)
6. do we have real performance benchmarks from actual clusters that show a
significant benefit? as noted above, the change is fairly complex (caching is
in fact hard :)
and if we don't see a substantial performance improvement, the risk of bugs may
outweigh the benefit
that's my 2c anyway
-sr
> Datanode xceiver protocol should allow reuse of a connection
> ------------------------------------------------------------
>
> Key: HDFS-941
> URL: https://issues.apache.org/jira/browse/HDFS-941
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, hdfs client
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: bc Wong
> Attachments: HDFS-941-1.patch, HDFS-941-2.patch, HDFS-941-3.patch,
> HDFS-941-3.patch, HDFS-941-4.patch
>
>
> Right now each connection into the datanode xceiver only processes one
> operation.
> In the case that an operation leaves the stream in a well-defined state (eg a
> client reads to the end of a block successfully) the same connection could be
> reused for a second operation. This should improve random read performance
> significantly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
