[
https://issues.apache.org/jira/browse/STORM-1342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Weathers updated STORM-1342:
---------------------------------
Description:
h3. Storm-on-Mesos Worker Logs are in varying directories
When using [storm-on-mesos|https://github.com/mesos/storm] with cgroups, each
topology's workers are isolated into separate containers. By default the
worker logs will be saved into container-specific sandbox directories. These
directories are also topology-specific by definition, because, as just stated,
the containers are specific to each topology.
h3. Problem: Storm supports 1-and-only-1 Logviewer per Worker Host
A challenge with this different way of running Storm is that the [Storm
logviewer|https://github.com/apache/storm/blob/768a85926373355c15cc139fd86268916abc6850/docs/_posts/2013-12-08-storm090-released.md#log-viewer-ui]
runs as a single instance on each worker host. This doesn't play well with
having the topology worker logs in separate per-topology containers. The one
logviewer doesn't know about the various sandbox directories that the Storm
Workers are writing to. And if we just spawned new logviewers for each
container, the problem is that the Storm UI only knows about 1 global port the
logviewer, so you cannot just direct.
These problems are documented (or linked to) from [Issue #6 in the
storm-on-mesos project|https://github.com/mesos/storm/issues/6]
h3. Possible Solutions I can envision
# configure the Storm workers to write to log directories that exist on the raw
host outside of the container sandbox, and run a single logviewer on a host,
which serves up the contents of that directory.
#* violates one of the basic reasons for using containers: isolation.
#* also prevents a standard use case for Mesos: running more than 1 instance of
a Mesos Framework (e.g., "Storm Cluster") at once on same Mesos Cluster. e.g.,
for Blue-Green deployments.
#* a variation on this proposal is to somehow expose the sandbox dirs of all
storm containers to this singleton logviewer process (still has above problems)
# launch a separate logviewer in each container, and somehow register those
logviewers with Storm such that Storm knows for a given host which logviewer
port is assigned to a given topology.
#* this is the proposed solution
# launch a separate logviewer in a separate container, somehow exposing the
logs from these other containers to the host-global logviewer.
h3. Storm Changes for the Proposed Solution
Nimbus or ZooKeeper could serve as a registrar, recording the association
between a slot (host + worker port) and the logviewer port that is serving the
workers logs. And the Storm-on-Mesos framework could update this registry when
launching a new worker. (This proposal definitely calls for thorough vetting
and thinking.)
h3. Storm-on-Mesos Framework Changes for the Proposed Solution
Along with the interaction with the "registrar" proposed above, the
storm-on-mesos framework can be enhanced to launch multiple logviewers on a
given worker host, where each logviewer is dedicated to serving the worker logs
from a specific topology's container/sandbox directory. This would be done by
launching a logviewer process within the topology's container, and assigning it
an arbitrary listening port that has been determined dynamically through mesos
(which treats ports as one of the schedulable resource primitives of a worker
host). [Code implementing this logviewer-port-allocation logic already
exists|https://github.com/mesos/storm/commit/af8c49beac04b530c33c1401c829caaa8e368a35],
but [that specific portion of the code was
reverted|https://github.com/mesos/storm/commit/dc3eee0f0e9c06f6da7b2fe697a8e4fc05b5227e]
because of the issues that inspired this ticket.
was:
h3. Storm-on-Mesos Worker Logs are in varying directories
When using [storm-on-mesos|https://github.com/mesos/storm] with cgroups, each
topology's workers are isolated into separate containers. By default the
worker logs will be saved into container-specific sandbox directories. These
directories are also topology-specific by definition, because, as just stated,
the containers are specific to each topology.
h3. Problem: Storm supports 1-and-only-1 Logviewer per Worker Host
A challenge with this different way of running Storm is that the [Storm
logviewer|https://github.com/apache/storm/blob/768a85926373355c15cc139fd86268916abc6850/docs/_posts/2013-12-08-storm090-released.md#log-viewer-ui]
runs as a single instance on each worker host. This doesn't play well with
having the topology worker logs in separate per-topology containers. The one
logviewer doesn't know about the various sandbox directories that the Storm
Workers are writing to. And if we just spawned new logviewers for each
container, the problem is that the Storm UI only knows about 1 global port the
logviewer, so you cannot just direct.
These problems are documented (or linked to) from [Issue #6 in the
storm-on-mesos project|https://github.com/mesos/storm/issues/6]
h3. Possible Solutions I can envision
# configure the Storm workers to write to log directories that exist on the raw
host outside of the container sandbox, and run a single logviewer on a host,
which serves up the contents of that directory.
#* violates one of the basic reasons for using containers: isolation.
#* also prevents a standard use case for Mesos: running more than 1 instance of
a Mesos Framework (e.g., "Storm Cluster") at once on same Mesos Cluster. e.g.,
for Blue-Green deployments.
#* a variation on this proposal is to somehow expose the sandbox dirs of all
storm containers to this singleton logviewer process (still has above problems)
# launch a separate logviewer in each container, and somehow register those
logviewers with Storm such that Storm knows for a given host which logviewer
port is assigned to a given topology.
#* this is the proposed solution
h3. Storm Changes for the Proposed Solution
Nimbus or ZooKeeper could serve as a registrar, recording the association
between a slot (host + worker port) and the logviewer port that is serving the
workers logs. And the Storm-on-Mesos framework could update this registry when
launching a new worker. (This proposal definitely calls for thorough vetting
and thinking.)
h3. Storm-on-Mesos Framework Changes for the Proposed Solution
Along with the interaction with the "registrar" proposed above, the
storm-on-mesos framework can be enhanced to launch multiple logviewers on a
given worker host, where each logviewer is dedicated to serving the worker logs
from a specific topology's container/sandbox directory. This would be done by
launching a logviewer process within the topology's container, and assigning it
an arbitrary listening port that has been determined dynamically through mesos
(which treats ports as one of the schedulable resource primitives of a worker
host). [Code implementing this logviewer-port-allocation logic already
exists|https://github.com/mesos/storm/commit/af8c49beac04b530c33c1401c829caaa8e368a35],
but [that specific portion of the code was
reverted|https://github.com/mesos/storm/commit/dc3eee0f0e9c06f6da7b2fe697a8e4fc05b5227e]
because of the issues that inspired this ticket.
> support multiple logviewers per host for container-isolated worker logs
> -----------------------------------------------------------------------
>
> Key: STORM-1342
> URL: https://issues.apache.org/jira/browse/STORM-1342
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-core
> Reporter: Erik Weathers
> Priority: Minor
>
> h3. Storm-on-Mesos Worker Logs are in varying directories
> When using [storm-on-mesos|https://github.com/mesos/storm] with cgroups, each
> topology's workers are isolated into separate containers. By default the
> worker logs will be saved into container-specific sandbox directories. These
> directories are also topology-specific by definition, because, as just
> stated, the containers are specific to each topology.
> h3. Problem: Storm supports 1-and-only-1 Logviewer per Worker Host
> A challenge with this different way of running Storm is that the [Storm
> logviewer|https://github.com/apache/storm/blob/768a85926373355c15cc139fd86268916abc6850/docs/_posts/2013-12-08-storm090-released.md#log-viewer-ui]
> runs as a single instance on each worker host. This doesn't play well with
> having the topology worker logs in separate per-topology containers. The one
> logviewer doesn't know about the various sandbox directories that the Storm
> Workers are writing to. And if we just spawned new logviewers for each
> container, the problem is that the Storm UI only knows about 1 global port
> the logviewer, so you cannot just direct.
> These problems are documented (or linked to) from [Issue #6 in the
> storm-on-mesos project|https://github.com/mesos/storm/issues/6]
> h3. Possible Solutions I can envision
> # configure the Storm workers to write to log directories that exist on the
> raw host outside of the container sandbox, and run a single logviewer on a
> host, which serves up the contents of that directory.
> #* violates one of the basic reasons for using containers: isolation.
> #* also prevents a standard use case for Mesos: running more than 1 instance
> of a Mesos Framework (e.g., "Storm Cluster") at once on same Mesos Cluster.
> e.g., for Blue-Green deployments.
> #* a variation on this proposal is to somehow expose the sandbox dirs of all
> storm containers to this singleton logviewer process (still has above
> problems)
> # launch a separate logviewer in each container, and somehow register those
> logviewers with Storm such that Storm knows for a given host which logviewer
> port is assigned to a given topology.
> #* this is the proposed solution
> # launch a separate logviewer in a separate container, somehow exposing the
> logs from these other containers to the host-global logviewer.
> h3. Storm Changes for the Proposed Solution
> Nimbus or ZooKeeper could serve as a registrar, recording the association
> between a slot (host + worker port) and the logviewer port that is serving
> the workers logs. And the Storm-on-Mesos framework could update this registry
> when launching a new worker. (This proposal definitely calls for thorough
> vetting and thinking.)
> h3. Storm-on-Mesos Framework Changes for the Proposed Solution
> Along with the interaction with the "registrar" proposed above, the
> storm-on-mesos framework can be enhanced to launch multiple logviewers on a
> given worker host, where each logviewer is dedicated to serving the worker
> logs from a specific topology's container/sandbox directory. This would be
> done by launching a logviewer process within the topology's container, and
> assigning it an arbitrary listening port that has been determined dynamically
> through mesos (which treats ports as one of the schedulable resource
> primitives of a worker host). [Code implementing this
> logviewer-port-allocation logic already
> exists|https://github.com/mesos/storm/commit/af8c49beac04b530c33c1401c829caaa8e368a35],
> but [that specific portion of the code was
> reverted|https://github.com/mesos/storm/commit/dc3eee0f0e9c06f6da7b2fe697a8e4fc05b5227e]
> because of the issues that inspired this ticket.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
