[jira] [Commented] (HDFS-6939) Support path-based filtering of inotify events
[ https://issues.apache.org/jira/browse/HDFS-6939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117482#comment-16117482 ] Ming Ma commented on HDFS-6939: --- Yeah we can include this feature if it provides values. Couple questions: * Each RPC getEditsFromTxid call ends up sending the filter over the wire; so filter with lots of paths has perf impact. Do we need to support large number of paths per call? * In the future there could be other type of filters, e.g. a) based on FsEditLogOp type; b) support different logical operators OR, AND, etc. To make it extensible, perhaps we can define an interface with signature shouldNotify(FsEditLogOp) and provide the path-based PathBasedInotifyFilter for now. Then InotifyFSEditLogOpTranslator will be simpler by checking shouldNotify upfront; if we need to add path-and-editop-based filtering, we can just add PathAndOpBasedInotifyFilter without changing InotifyFSEditLogOpTranslator. * DFSClient's existing getInotifyEventStream methods are only used by DistributedFileSystem. So you don't need to keep these old methods on DFSClient; instead have DistributedFileSystem's old getInotifyEventStream methods call DFSClient's new methods. Also maybe we can consider depreciate DistributedFileSystem's old getInotifyEventStream methods. > Support path-based filtering of inotify events > -- > > Key: HDFS-6939 > URL: https://issues.apache.org/jira/browse/HDFS-6939 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client, namenode, qjm >Reporter: James Thomas >Assignee: Surendra Singh Lilhore > Attachments: HDFS-6939-001.patch > > > Users should be able to specify that they only want events involving > particular paths. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6939) Support path-based filtering of inotify events
[
https://issues.apache.org/jira/browse/HDFS-6939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16102635#comment-16102635
]
Hadoop QA commented on HDFS-6939:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HDFS-6939 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-6939 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12751697/HDFS-6939-001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20427/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Support path-based filtering of inotify events
> --
>
> Key: HDFS-6939
> URL: https://issues.apache.org/jira/browse/HDFS-6939
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client, namenode, qjm
>Reporter: James Thomas
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-6939-001.patch
>
>
> Users should be able to specify that they only want events involving
> particular paths.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6939) Support path-based filtering of inotify events
[ https://issues.apache.org/jira/browse/HDFS-6939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16102626#comment-16102626 ] Surendra Singh Lilhore commented on HDFS-6939: -- Hi [~mingma], Can we push this jira and HDFS-8933 in current version?, I think this feature is required by many people. http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/201704.mbox/%3CCADm6LJdY6P5p7EyCopujUY_dg0XhKHtX+tn7BP=dykf4ctp...@mail.gmail.com%3E HDFS-8940 still under discussion and I think it will take time. > Support path-based filtering of inotify events > -- > > Key: HDFS-6939 > URL: https://issues.apache.org/jira/browse/HDFS-6939 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client, namenode, qjm >Reporter: James Thomas >Assignee: Surendra Singh Lilhore > Attachments: HDFS-6939-001.patch > > > Users should be able to specify that they only want events involving > particular paths. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6939) Support path-based filtering of inotify events
[
https://issues.apache.org/jira/browse/HDFS-6939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15805961#comment-15805961
]
Hadoop QA commented on HDFS-6939:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HDFS-6939 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-6939 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12751697/HDFS-6939-001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18070/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Support path-based filtering of inotify events
> --
>
> Key: HDFS-6939
> URL: https://issues.apache.org/jira/browse/HDFS-6939
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client, namenode, qjm
>Reporter: James Thomas
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-6939-001.patch
>
>
> Users should be able to specify that they only want events involving
> particular paths.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6939) Support path-based filtering of inotify events
[ https://issues.apache.org/jira/browse/HDFS-6939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14709649#comment-14709649 ] Surendra Singh Lilhore commented on HDFS-6939: -- Thanks [~mingma] for comments I will go through the design document.. Support path-based filtering of inotify events -- Key: HDFS-6939 URL: https://issues.apache.org/jira/browse/HDFS-6939 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client, namenode, qjm Reporter: James Thomas Assignee: Surendra Singh Lilhore Attachments: HDFS-6939-001.patch Users should be able to specify that they only want events involving particular paths. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6939) Support path-based filtering of inotify events
[ https://issues.apache.org/jira/browse/HDFS-6939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14707828#comment-14707828 ] Ming Ma commented on HDFS-6939: --- Thanks [~surendrasingh] for working on this. I had some discussion with [~cmccabe], [~eddyxu], [~zhz] couple weeks about inotify functionality and how to make it useful for large-scale multi-tenant scenarios. I just uploaded the draft design document in HDFS-8940. Appreciate if you have any input. For this specific work item, we might want to investigate it together with other issues and understand how it can eventually enable more applications. Support path-based filtering of inotify events -- Key: HDFS-6939 URL: https://issues.apache.org/jira/browse/HDFS-6939 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client, namenode, qjm Reporter: James Thomas Assignee: Surendra Singh Lilhore Attachments: HDFS-6939-001.patch Users should be able to specify that they only want events involving particular paths. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6939) Support path-based filtering of inotify events
[
https://issues.apache.org/jira/browse/HDFS-6939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14707259#comment-14707259
]
Hadoop QA commented on HDFS-6939:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 19m 15s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 52s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 54s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 15s | The applied patch generated
1 release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 28s | The applied patch generated 1
new checkstyle issues (total was 3, now 4). |
| {color:red}-1{color} | whitespace | 0m 5s | The patch has 13 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 37s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 37s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 10s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 39m 59s | Tests failed in hadoop-hdfs. |
| {color:green}+1{color} | hdfs tests | 0m 27s | Tests passed in
hadoop-hdfs-client. |
| | | 90m 17s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade |
| | hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality |
| | hadoop.hdfs.TestHttpPolicy |
| | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
| | hadoop.hdfs.server.datanode.TestStorageReport |
| | hadoop.hdfs.server.namenode.TestGenericJournalConf |
| | hadoop.cli.TestCryptoAdminCLI |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles |
| | hadoop.hdfs.qjournal.client.TestQuorumJournalManager |
| | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
| | hadoop.hdfs.TestSetrepIncreasing |
| | hadoop.hdfs.server.namenode.TestSecondaryWebUi |
| | hadoop.hdfs.TestFetchImage |
| | hadoop.hdfs.server.namenode.ha.TestHAStateTransitions |
| | hadoop.hdfs.server.namenode.TestGetBlockLocations |
| | hadoop.hdfs.server.datanode.TestReadOnlySharedStorage |
| | hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots |
| | hadoop.hdfs.server.namenode.TestMetadataVersionOutput |
| |
hadoop.hdfs.server.namenode.snapshot.TestSnapshotNameWithInvalidCharacters |
| | hadoop.hdfs.server.namenode.TestNameNodeResourceChecker |
| | hadoop.hdfs.TestMiniDFSCluster |
| | hadoop.hdfs.TestDFSClientRetries |
| | hadoop.hdfs.TestEncryptionZonesWithKMS |
| | hadoop.hdfs.TestReplication |
| | hadoop.hdfs.server.namenode.TestCheckpoint |
| | hadoop.hdfs.TestDFSOutputStream |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica |
| | hadoop.hdfs.TestDFSConfigKeys |
| | hadoop.hdfs.TestClientBlockVerification |
| | hadoop.hdfs.TestDFSRollback |
| | hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages |
| | hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot |
| | hadoop.hdfs.server.namenode.TestCreateEditsLog |
| | hadoop.tools.TestJMXGet |
| | hadoop.hdfs.server.namenode.ha.TestGetGroupsWithHA |
| | hadoop.hdfs.TestParallelRead |
| | hadoop.hdfs.TestHDFSTrash |
| | hadoop.hdfs.tools.TestDelegationTokenFetcher |
| | hadoop.hdfs.server.blockmanagement.TestBlockReportRateLimiting |
| | hadoop.hdfs.server.namenode.TestQuotaByStorageType |
| | hadoop.hdfs.server.namenode.TestBlockUnderConstruction |
| | hadoop.hdfs.server.datanode.TestIncrementalBlockReports |
| | hadoop.hdfs.TestWriteRead |
| | hadoop.hdfs.TestFileAppend4 |
| | hadoop.hdfs.TestSnapshotCommands |
| | hadoop.hdfs.TestBlocksScheduledCounter |
| | hadoop.hdfs.TestDFSUtil |
| | hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.crypto.TestHdfsCryptoStreams |
| | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.TestFSOutputSummer |
| | hadoop.hdfs.server.namenode.TestDeleteRace |
| | hadoop.hdfs.server.namenode.TestAddBlockRetry |
| | hadoop.hdfs.server.namenode.TestCheckPointForSecurityTokens |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength |
| | hadoop.hdfs.server.namenode.TestListCorruptFileBlocks |
| | hadoop.hdfs.server.datanode.TestDnRespectsBlockReportSplitThreshold |
| | hadoop.hdfs.server.namenode.ha.TestStandbyIsHot |
| |
[jira] [Commented] (HDFS-6939) Support path-based filtering of inotify events
[
https://issues.apache.org/jira/browse/HDFS-6939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706503#comment-14706503
]
Surendra Singh Lilhore commented on HDFS-6939:
--
From inotify design doc.
bq. 1. Allow clients to watch events from specific directories rather than from
the entirety of HDFS. We will need to decide whether undesired events should be
filtered out on the NameNode or client side.
I think its good to add filtering on the Namenode side. Client side user only
can do filtering in his application.
*Patch 001:*
1. Added *Regex* based filterting, User can pass regular expression for
filtering.
2. User can watch events for perticular file, by passing file path.
3. Added two new API in *DistributedFileSystem*
{noformat}
public DFSInotifyEventInputStream
getInotifyEventStream(InotifyFilter filter);
public DFSInotifyEventInputStream getInotifyEventStream(long
lastReadTxid, InotifyFilter filter)
{noformat}
4. In feature we can add more filter option in *InotifyFilter* object
without changing API, for example event based filtering HDFS-8933
Please review and give suggestions .
Support path-based filtering of inotify events
--
Key: HDFS-6939
URL: https://issues.apache.org/jira/browse/HDFS-6939
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs-client, namenode, qjm
Reporter: James Thomas
Assignee: Surendra Singh Lilhore
Attachments: HDFS-6939-001.patch
Users should be able to specify that they only want events involving
particular paths.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
