[
https://issues.apache.org/jira/browse/HDFS-8940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14726597#comment-14726597
]
Ming Ma commented on HDFS-8940:
-------------------------------
Thanks [~cmccabe]!
bq. It seems like the ZK-based design is only useful for the case where you
have a small number of items that match the filter.
That is right. We want to explore if it can address specific use cases where
large number of clients waiting for few files that don't change often. It won't
address all use cases.
bq. It seems like we should be able to configure a local edit log file, and get
inotify to read from that.
ANN can optionally write to local edit logs in addition to QJM. So it will work
if inotify is pulled from ANN. Any ideas how to have SBN read from its local
edit log? IIUC for the scenario where it is configured to have 10 SBNs and
there are only 5 JounalNodes, some SBNs will read from their local JournalNode,
some will read from other remote JournalNodes.
bq. even if the ZK-based solution doesn't solve every use-case, perhaps parts
of it can be re-used for a different solution later on. This might be a good
argument for moving the path-based-filtering and user-based-filtering into the
NN, since then it could be shared by multiple inotify scalability solutions.
Exactly. Having filtering definition in NN offers us the flexibility to try out
different mechanisms.
bq. I think the standby NN solution could be very interesting.
Yep it is a good idea. It is simpler and has less dependencies. I am planning
to try out this approach first.
> Support for large-scale multi-tenant inotify service
> ----------------------------------------------------
>
> Key: HDFS-8940
> URL: https://issues.apache.org/jira/browse/HDFS-8940
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Ming Ma
> Attachments: Large-Scale-Multi-Tenant-Inotify-Service.pdf
>
>
> HDFS-6634 provides the core inotify functionality. We would like to extend
> that to provide a large-scale service that ten of thousands of clients can
> subscribe to.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
