[
https://issues.apache.org/jira/browse/SAMZA-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15746321#comment-15746321
]
Jake Maes commented on SAMZA-1041:
----------------------------------
Discussion 1 (D1):
Some general comments for the community.
1. Do you have multiple jobs which you would like to deploy and run together?
Do these jobs follow specific ordering to run?
2. Are these jobs connected with Kafka topic? This new feature will manage the
topic for you with autocreation/deletion. We are also thinking about inferring
the topic name and partition count based on the job graph. Any of these are of
specific interests to you?
3. By moving to this new feature, we will need to have the configs of all the
processors in a graph in one place. For each processor, it can have its own
configs as well as shared configs. For example there will likely be a new
"processors.default.*" config prefix to specify default values for processors
and a "processors.<pName>.*" to specify values specific to a processor. This
should make the config less verbose without sacrificing flexibility. Does this
satisfy your requirements and easier to manage?
4. With a full specification of the pipeline, we can perform a number of
validations to ensure everything is configured properly. The first that come to
mind are common pitfalls at LinkedIn:
a. Verify that all processor inputs and outputs are configured properly.
e.g. every processor has at least 1 input and intermediate topics
b. Verify that all streams are configured as expected. e.g. correct number
of partitions
Are there any other validations that you think should be added?
> Multi-stage feature for Samza
> -----------------------------
>
> Key: SAMZA-1041
> URL: https://issues.apache.org/jira/browse/SAMZA-1041
> Project: Samza
> Issue Type: New Feature
> Affects Versions: 0.12.0
> Reporter: Jake Maes
> Assignee: Jake Maes
>
> Samza provides a powerful framework for users to implement and deploy stream
> processors. One of the core concepts in Samza is a processor, which is
> deployed individually as a job. While a single job is sufficient to perform
> some basic stream processing, we have seen users apply Samza to more complex
> problems involving multi-stage jobs or pipelines. These range from jobs that
> require a separate repartitioner to co-partition streams for a join, to more
> complicated pipelines in which the stages are purposely decoupled like
> microservices. Historically, users would create a separate job for each stage
> of the pipeline and deploy them manually. A critical part of this manual
> deployment is creating and modifying the intermediate streams to have the
> appropriate configuration, in particular; partition count. This deployment
> model has proven to be tedious and error-prone because:
> 1. Stream creation is a manual process. If the streams are not pre-created
> with the appropriate configurations, it can lead to unexpected behavior in
> the pipeline. For example failure to join because keys are not being routed
> to a common Task.
> 2. Job deployment is a manual process. Each stage needs to be deployed
> separately, even though they are often deployed in the same cadence.
> 3. Configuration is associated with a processor, which makes it more
> difficult to reuse the processor. Configurations like task inputs and
> container count can vary for the same processor depending on the context
> (pipeline) in which it is executed.
> 4. There is no early validation to detect a misconfigured pipeline. Instead
> users tend to notice that something is wrong long after the initial
> deployment by looking at metrics.
> 5. There are common preconditions for processors (e.g. co-partitioned, or
> deduplicated inputs) that could be handled automatically in a system that has
> the “whole picture” of streams and processors.
> Our goal is to alleviate these issues by introducing a
> yet-to-be-properly-named feature which we will call “pipelines” for now. The
> pipeline feature will allow users to easily compose a collection of
> processors and streams into a directed acyclic graph (DAG) and manage them as
> one unit. A key part of this feature is automatic runtime creation of
> intermediate and output streams. It also enables richer validation,
> simplified deployment, and a foundation for many performance and ease-of-use
> features. For example, repartitioners could be automatically injected where
> needed and processors could be colocated on the same container for
> performance.
> Note that this feature is not the same as SAMZA-914, mostly in terms of scope
> and simplicity. SAMZA-914 focuses on composing operators into a logical flow
> that is executed as one processor. That entire flow is scaled out uniformly
> by adding containers. By contrast, this pipeline feature provides isolation
> and independent scaling of the processors.
> Many of the details have yet to be worked out, but a design doc will be
> posted here soon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
