Hi Moshe,
Dynamic configuration, with parameters stored in Zookeeper, is actually quite
complex if you follow it from end to end. Here’s a brief description:
• The Metron configuration parameter mechanism uses Apache Curator to subscribe
to ZK nodes of interest, and maintain a TreeCache with background,
asynchronous, atomic updates. Whenever someone writes new parameter values to
ZK, ZK notifies the Curator client, which copies the new values into the
TreeCache. That’s the bit of code you referenced below. The pub/sub model
supported by ZK and used by Curator, means that this is very low cost, without
active polling.
• For parameters that may be changed without requiring a topology restart,
Metron always reads them from the parameter mechanism, rather than storing the
values in local variables. This sentence needs some commentary:
o Some parameters DO require a topology restart, in particular if they interact
with Storm settings. Updates to these values are ignored, or only have partial
effect, until restart. The documents are clear about which settings can be
updated dynamically.
o Reading from the TreeCache is not significantly more expensive than reading
from an ordinary Map object. So going back to the parameter mechanism instead
of storing the values in local variables is a small cost.
o The TreeCache uses Concurrent data structures where appropriate to assure
that edits happen atomically and at low cost, so you don’t need to worry about
reading inconsistent values from the parameter mechanism.
• The net result is a fully dynamic configuration capability for a broad set of
independently updatable parameters.
Hope this helps,
--Matt
On 3/31/17, 5:53 AM, "moshe jarusalem" <tuu...@gmail.com> wrote:
Hi All,
I have been looking the codes for ConfiguredBolt and its derivatives. I
realized that updateConfig is actually not doing much?
Would you describe how you manage configuration changes might be needed
after bolts are initialized and running?
for convenience, I copied the code here
public void updateConfig(String path, byte[] data) throws IOException {
if (data.length != 0) {
String name = path.substring(path.lastIndexOf("/") + 1);
if (path.startsWith(ConfigurationType.ENRICHMENT.getZookeeperRoot())) {
getConfigurations().updateSensorEnrichmentConfig(name, data);
reloadCallback(name, ConfigurationType.ENRICHMENT);
} else if (ConfigurationType.GLOBAL.getZookeeperRoot().equals(path)) {
getConfigurations().updateGlobalConfig(data);
reloadCallback(name, ConfigurationType.GLOBAL);
}
}
Thanks,
