[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181634#comment-14181634 ] stack commented on HBASE-9864: -- Removed from branch-1 because no assignee and not being worked on. Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack Priority: Blocker In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181636#comment-14181636 ] stack commented on HBASE-9864: -- bq not being worked on. ... in time for 1.0 release (to my knowledge -- correct me if I am wrong) Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack Priority: Blocker In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[
https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13964275#comment-13964275
]
Andrew Purtell commented on HBASE-9864:
---
{quote}
bq. I'm a lot more concerned about a revoke user that doesn't get applied for
hours because someone was napping.
What you thinking Andrew? Chatting w/ Matteo, if a RS fails to update its
local cache, it should abort as we would not being able to update our WAL. No
napping allowed.
{quote}
I'm worried about missed notifications, and more generally about proving at the
time the update is done that it was applied everywhere, because security guys
like proof.
bq. You need something in the 0.98 timeframe too boss?
Thinking about it yes. We have other security fixes and refinements on deck.
Thinking is 0.98 is the security proving ground ahead of 1.0+.
Notifications bus for use by cluster members keeping up-to-date on changes
--
Key: HBASE-9864
URL: https://issues.apache.org/jira/browse/HBASE-9864
Project: HBase
Issue Type: Brainstorming
Reporter: stack
Priority: Blocker
Fix For: 1.0.0
In namespaces and acls, zk callbacks are used so all participating servers
are notified when there is a change in acls/namespaces list.
The new visibility tags feature coming in copies the same model of using zk
with listeners for the features' particular notifications.
Three systems each w/ their own implementation of the notifications all using
zk w/ their own feature-specific watchers.
Should probably unify.
Do we have to go via zk? Seems like all want to be notified when an hbase
table is updated. Could we tell servers directly rather than go via zk?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[
https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13964280#comment-13964280
]
Matteo Bertozzi commented on HBASE-9864:
{code}I'm worried about missed notifications, and more generally about proving
at the time the update is done that it was applied everywhere, because security
guys like proof.{code}
ok. so, do you prefer to fail an operation if a machine is slow?
e.g. the user type 'revoke' and if within the timeout the operation is not
applied to every RS (cache update), the user will get a failure?
Notifications bus for use by cluster members keeping up-to-date on changes
--
Key: HBASE-9864
URL: https://issues.apache.org/jira/browse/HBASE-9864
Project: HBase
Issue Type: Brainstorming
Reporter: stack
Priority: Blocker
Fix For: 1.0.0
In namespaces and acls, zk callbacks are used so all participating servers
are notified when there is a change in acls/namespaces list.
The new visibility tags feature coming in copies the same model of using zk
with listeners for the features' particular notifications.
Three systems each w/ their own implementation of the notifications all using
zk w/ their own feature-specific watchers.
Should probably unify.
Do we have to go via zk? Seems like all want to be notified when an hbase
table is updated. Could we tell servers directly rather than go via zk?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13964297#comment-13964297 ] Andrew Purtell commented on HBASE-9864: --- bq. ok. so, do you prefer to fail an operation if a machine is slow? Yes, the grant or revoke, or label definition, or setauths, etc. should be retried since it failed to be consistently applied. (In addition, as part of cleanup all cache updates elsewhere should be rolled back.) If somehow onerous elsewhere we could build one thing shared by the the security coprocessors and namespace code and do something else for other kinds of distributed cache / state updates. Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack Priority: Blocker Fix For: 1.0.0 In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[
https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13964311#comment-13964311
]
Matteo Bertozzi commented on HBASE-9864:
{quote}
Yes, the grant or revoke, or label definition, or setauths, etc. should be
retried since it failed to be consistently applied.
(In addition, as part of cleanup any cache updates applied should be rolled
back.)
{quote}
ok, we were trying to unify all the cache updates under a single mechanism but
I guess is fine having a relaxed update and a non-relaxed one for security like
features. In this case we can just use the Procedure that we have today for the
non-relaxed notification, but this means to change operations as something like
this:
* RPC revoke
** new Procedure(revoke)
** acquire: fetch new updates
** reached: Add the revoke to table + apply the updates
** abort: discard the updates
Notifications bus for use by cluster members keeping up-to-date on changes
--
Key: HBASE-9864
URL: https://issues.apache.org/jira/browse/HBASE-9864
Project: HBase
Issue Type: Brainstorming
Reporter: stack
Priority: Blocker
Fix For: 1.0.0
In namespaces and acls, zk callbacks are used so all participating servers
are notified when there is a change in acls/namespaces list.
The new visibility tags feature coming in copies the same model of using zk
with listeners for the features' particular notifications.
Three systems each w/ their own implementation of the notifications all using
zk w/ their own feature-specific watchers.
Should probably unify.
Do we have to go via zk? Seems like all want to be notified when an hbase
table is updated. Could we tell servers directly rather than go via zk?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13964322#comment-13964322 ] Andrew Purtell commented on HBASE-9864: --- Above security related discussion relates to HBASE-10544 Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack Priority: Blocker Fix For: 1.0.0 In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13964317#comment-13964317 ] Andrew Purtell commented on HBASE-9864: --- bq. In this case we can just use the Procedure that we have today for the non-relaxed notification, but this means to change operations as something like this: Agreed Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack Priority: Blocker Fix For: 1.0.0 In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963078#comment-13963078 ] Andrew Purtell commented on HBASE-9864: --- bq. When any of these tables are changed, either the editor pokes the Master or the Master 'notices' the change because it is proactively scanning the tables. This part I'm fuzzy about because currently the coprocessors running on the meta table notice the change immediately and trigger a notification by changing znode data. How would this work in the alternative? The reason I suggest a Procedure based update (or equivalent) is so we can guarantee that the change has committed immediately in every local cache, with the additional benefit of knowing if not. (We don't have this last part today because we use ZK watchers, but we really should.) Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack Priority: Blocker Fix For: 1.0.0 In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963407#comment-13963407 ] stack commented on HBASE-9864: -- The editor will not need to poke the master or the master will not need to do proactive scans looking for updates if the argument that master and meta are tied prevails over in HBASE-10569. So, allowing HBASE-10569, system table edits will be noticed 'immediately'. In the proposal, we could pass the edit on the back of the heartbeat or easier, just prompt the heartbeater go do the lookup on the table itself. In the first case, it may be a full heartbeat period before changes are noticed. In the second case, it would be the heartbeat period + time to scan the table. This could be seconds. Is this 'soon-as-possible' too much for say, an ACL update? On plus side, this is easy-to-do and undoes our reliance on a 3rd system (zk) for notification. Later we could do a new Procedure mechanism if needed that writes over a new x-cluster direct channel between Master and RS pushing out the change, waiting on all to acknowledge and taking evasive action if any fails. This would be more work but probably closer to 'immediate'. Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack Priority: Blocker Fix For: 1.0.0 In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963437#comment-13963437 ] Andrew Purtell commented on HBASE-9864: --- bq. On plus side, this is easy-to-do and undoes our reliance on a 3rd system (zk) for notification. I thought other issues are undoing our reliance on ZK? Presumably distributed barriers would be ported to it so therefore no reliance on ZK then? bq. Later we could do a new Procedure mechanism I think it needs to be now. We have anecdotal evidence that ZK watches work but won't have such evidence for something else. We should not be comfortable with security subsystems that need consistent state everywhere running on something that doesn't guarantee that. Every current component that has a cache like this needs this guarantee (ACLs, visibility, namespaces) right? Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack Priority: Blocker Fix For: 1.0.0 In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963440#comment-13963440 ] Andrew Purtell commented on HBASE-9864: --- bq. Is this 'soon-as-possible' too much for say, an ACL update? I'm not concerned with time, it's the guarantee that the update has been applied at every live location (or not) Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack Priority: Blocker Fix For: 1.0.0 In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[
https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963445#comment-13963445
]
Matteo Bertozzi commented on HBASE-9864:
{quote}I'm not concerned with time, it's the guarantee that the update has been
applied at every live location (or not){quote}
If a cache update cannot be applied, is probably because that RS is in a bad
state. and it should be aborted.
if it is just slow (e.g. GC) no one can probably query it, so in that case is
fine if the cache update it will be applied later.
I don't think the procedure is a good thing e.g. for ACL
let say that one node is in GC or has some network problem.
You try to do grant user and it fails because one node is not responding
within N seconds.
by using the heartbeat everyone will be updated as soon as they can
Notifications bus for use by cluster members keeping up-to-date on changes
--
Key: HBASE-9864
URL: https://issues.apache.org/jira/browse/HBASE-9864
Project: HBase
Issue Type: Brainstorming
Reporter: stack
Priority: Blocker
Fix For: 1.0.0
In namespaces and acls, zk callbacks are used so all participating servers
are notified when there is a change in acls/namespaces list.
The new visibility tags feature coming in copies the same model of using zk
with listeners for the features' particular notifications.
Three systems each w/ their own implementation of the notifications all using
zk w/ their own feature-specific watchers.
Should probably unify.
Do we have to go via zk? Seems like all want to be notified when an hbase
table is updated. Could we tell servers directly rather than go via zk?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963447#comment-13963447 ] Andrew Purtell commented on HBASE-9864: --- Part of where I am coming from is HBASE-10569 won't be backported to 0.98 or 0.96. My above comments should not be viewed as against solving the concerns regarding cache updates in a different way in trunk versus 0.98 or earlier. Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack Priority: Blocker Fix For: 1.0.0 In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963449#comment-13963449 ] Andrew Purtell commented on HBASE-9864: --- bq. You try to do grant user and it fails because one node is not responding within N seconds. by using the heartbeat everyone will be updated as soon as they can I'm a lot more concerned about a revoke user that doesn't get applied for hours because someone was napping. Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack Priority: Blocker Fix For: 1.0.0 In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963450#comment-13963450 ] Andrew Purtell commented on HBASE-9864: --- Anyway, like I said above I'm a bit fuzzy on how this would work for 0.98 because of assumptions based on code not available in any release yet. Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack Priority: Blocker Fix For: 1.0.0 In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963767#comment-13963767 ] stack commented on HBASE-9864: -- bq. I'm a lot more concerned about a revoke user that doesn't get applied for hours because someone was napping. What you thinking Andrew? Chatting w/ Matteo, if a RS fails to update its local cache, it should abort as we would not being able to update our WAL. No napping allowed. You need something in the 0.98 timeframe too boss? bq. I thought other issues are undoing our reliance on ZK? Presumably distributed barriers would be ported to it so therefore no reliance on ZK then? There are others to undo our zk heavy-reliance, yes. The suggestion here is that in genericizing a notification system, if possible, avoid zk if we can (Yeah, current zk 'works' but if we could avoid having to go to another system...). Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack Priority: Blocker Fix For: 1.0.0 In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13961659#comment-13961659 ] Mikhail Antonov commented on HBASE-9864: [~apurtell] when you're talking about propagated in the background with best effort and internal distributed non-persistent store, and that it doesn't have to be coupled to ZK, you mean that this store would be kind of option for the consensus library (referred via HBASE-10909), and that it would have 2 modes of replication - one for guaranteed propagation of distributed state (like part of distributed state machine and one for best effort propagation? Do I understand that correct? I.e. when you say best effort, what kind of guarantees you imply? Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack Priority: Blocker Fix For: 1.0.0 In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13961699#comment-13961699 ] Andrew Purtell commented on HBASE-9864: --- Not a store for the consensus library, a distributed store/cache for internal use by components like the security coprocessors and namespace management (all of which currently do their own thing). By best effort I proposed above epidemic propagation. We could tune by interval and fanout. No guarantees besides a likelihood of convergence that can be derived from those parameters. Yes, another mode that guarantees propagation to all RegionServers or returns failure. We could add a simple gossip protocol for the first and use the pluggable distributed barrier facility for the second. The consensus package could handle the second also. Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack Priority: Blocker Fix For: 1.0.0 In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13961705#comment-13961705 ] Mikhail Antonov commented on HBASE-9864: So the emphasis in that case would be on performance over the strong consistency, right? So that if some piece of info hasn't been timely replicated to a particular node, it's fine - the request will look for them using remote call or..? Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack Priority: Blocker Fix For: 1.0.0 In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13961746#comment-13961746 ] Andrew Purtell commented on HBASE-9864: --- In this particular case we would be propagating updates for a distributed cache, where we want to loosly synchronize a cache kept on all RegionServers. If an update does arrive in time before we go to query the external system, then we have saved some work. Otherwise still no problem. So performance, yes, but lightweight more so. Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack Priority: Blocker Fix For: 1.0.0 In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13961751#comment-13961751 ] Mikhail Antonov commented on HBASE-9864: thanks for clarification! now better understand the usecase Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack Priority: Blocker Fix For: 1.0.0 In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13961759#comment-13961759 ] Anoop Sam John commented on HBASE-9864: --- ACL data, Visibility labels and NS details, these are the current items which can use this Notification bus. correct? Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack Priority: Blocker Fix For: 1.0.0 In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13961797#comment-13961797 ] Andrew Purtell commented on HBASE-9864: --- bq. ACL data, visibility labels, and NS details Yes and these would need/use the consistent propagation option. Also a new security related use case for LDAP attribute query cache that would only need lightweight propagation. Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack Priority: Blocker Fix For: 1.0.0 In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962414#comment-13962414 ] stack commented on HBASE-9864: -- Chatting w/ Matteo, would the following do? + For namespaces, tags, and ACL, each RS needs to host an up-to-date copy of the table in its memory. + When any of these tables are changed, either the editor pokes the Master or the Master 'notices' the change because it is proactively scanning the tables. + On change, the master ups an internal, in-memory sequence id. + When the regionserver heartbeats, currently the response is empty. Change the Master so its response is a Set of table names X seqid. + When the regionserver gets the heartbeat reply, it checks the seqid. If any seqids fail to match, scan the src table and then update the regionserver's seqid to match that of the master. + If the Master crashes, it will reset its seqids. They won't match the regionservers. Regionservers will all rescan (redundantly). No zk and piggybacking on system we already have in place? Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack Priority: Blocker Fix For: 1.0.0 In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962420#comment-13962420 ] Mikhail Antonov commented on HBASE-9864: Would it make sense to make backup masters host the copy of all seqids, to avoid global rescan? Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack Priority: Blocker Fix For: 1.0.0 In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962429#comment-13962429 ] Matteo Bertozzi commented on HBASE-9864: The idea, was that a master going down has the same consequence of an update to e.g. ACL table. keeping the copy of the seqid on the backup master is just an optimization and lot of extra work, and I'm not sure if it is really necessary. Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack Priority: Blocker Fix For: 1.0.0 In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962576#comment-13962576 ] ramkrishna.s.vasudevan commented on HBASE-9864: --- bq.When any of these tables are changed, either the editor pokes the Master or the Master 'notices' the change because it is proactively scanning the tables. This editor is now on the master side? Currently master does not get involved in these updations on the ACL/visibility tables. Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack Priority: Blocker Fix For: 1.0.0 In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962587#comment-13962587 ] stack commented on HBASE-9864: -- bq. This editor is now on the master side? It doesn't have to be but over in https://issues.apache.org/jira/browse/HBASE-10295 it is suggested that the master edit system tables only. Would it be hard to do this? Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack Priority: Blocker Fix For: 1.0.0 In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13961461#comment-13961461 ] Andrew Purtell commented on HBASE-9864: --- See HBASE-10919. It could be useful if the envisioned distributed cache there could be propagated among RegionServers in the background with best effort. Update conflicts could be resolved with a latest update wins policy and cache misses during propagation would be fine. Consider a trivial internal distributed non-persistent store where we reserve a small proportion of heap for in-memory storage and propagate updates from RegionServer to RegionServer using an epidemic gossip protocol over UDP. We could use this for the AccessController's table/CF ACL cache if in addition we support pinning of specific items to prevent eviction and provide an alternative update propagation option using Procedures to guarantee the global consistency of the update. Then we would have for distributing internal state a lightweight option for lazy propagation when only that is needed (idempotent updates, eventual consistency) and a heavyweight option for assuring change notification or state updates are delivered to all RegionServers. Neither need be tightly coupled to ZooKeeper. The gossip protocol would have no relationship to ZooKeeper. The Procedure based update should be abstracted from ZK after HBASE-10909. Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack Priority: Blocker Fix For: 1.0.0 In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13942168#comment-13942168 ] Andrew Purtell commented on HBASE-9864: --- The namespace auditor is HBASE-8410 Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack Priority: Blocker Fix For: 1.0.0 In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809985#comment-13809985 ] ramkrishna.s.vasudevan commented on HBASE-9864: --- Yes. I noted the same point in RB. I tried using the Procedure when we started the Visibility implementation and found that is involving master also. So that would not suit these ACLs, Visibility etc. May be make the procedure framework itself more flexible can be checked (not sure). +1 for the idea. Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13810307#comment-13810307 ] Andrew Purtell commented on HBASE-9864: --- Granting or revoking table or CF permissions or clearauths/setauths or add_label is just like a snapshot prepare or schema change action, something that needs to be globally applied and Procedure provides a way to do that with tracking of success or failure in that regard. That seems like a good thing to me. What we don't have with ZK watches today is any guarantee or confirmation that all clients with a watch received notification. In practice it works but ZK makes no guarantee, any given quorum peer can be behind the leader by some delta. The guarantee that ZK provides is that any znode change that a client sees was agreed upon by the majority at the time the proposal was committed. The client can call sync() on a znode to ask the quorum peer to sync with the leader first but to my knowledge there is no syncWatchers(). Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809595#comment-13809595 ] Nicolas Liochon commented on HBASE-9864: We have the cluster status, sent w/ a multicast message. It scales, and the server does not need to know about the client. ZK is interesting as it now has cheap local session (ZOOKEEPER-1147). It depends as well on the reliability we need, and if we need to see all states or not. Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809599#comment-13809599 ] Andrew Purtell commented on HBASE-9864: --- See HBASE-7254 Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9864) Notifications bus for use by cluster members keeping up-to-date on changes
[ https://issues.apache.org/jira/browse/HBASE-9864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809943#comment-13809943 ] Anoop Sam John commented on HBASE-9864: --- One item came up was using Procedure way. But this would involve Master also into these ops. Take case of ACL where the ACL region is associated with an HRS, the ACL admin operations are performed via Endpoints now. So client need to talks with one HRS only. A procedure for ACL admin operation means HM involvement and APIs in HM (?) This was my worry point. bq.Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? Such a generic framework will be good.. Notifications bus for use by cluster members keeping up-to-date on changes -- Key: HBASE-9864 URL: https://issues.apache.org/jira/browse/HBASE-9864 Project: HBase Issue Type: Brainstorming Reporter: stack In namespaces and acls, zk callbacks are used so all participating servers are notified when there is a change in acls/namespaces list. The new visibility tags feature coming in copies the same model of using zk with listeners for the features' particular notifications. Three systems each w/ their own implementation of the notifications all using zk w/ their own feature-specific watchers. Should probably unify. Do we have to go via zk? Seems like all want to be notified when an hbase table is updated. Could we tell servers directly rather than go via zk? -- This message was sent by Atlassian JIRA (v6.1#6144)
