On Wed, Jan 20, 2016 at 10:32 AM, James Malone <
jamesmal...@google.com.invalid> wrote:
Hello everyone,
Attached to this message is a proposed new project - Apache Dataflow,
a
unified programming model for data processing and integration.
The text of the proposal is included below. Additionally, the
proposal is
in draft form on the wiki where we will make any required changes:
https://wiki.apache.org/incubator/DataflowProposal
We look forward to your feedback and input.
Best,
James
----
= Apache Dataflow =
== Abstract ==
Dataflow is an open source, unified model and set of language-specific
SDKs
for defining and executing data processing workflows, and also data
ingestion and integration flows, supporting Enterprise Integration
Patterns
(EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines
simplify
the mechanics of large-scale batch and streaming data processing and
can
run on a number of runtimes like Apache Flink, Apache Spark, and
Google
Cloud Dataflow (a cloud service). Dataflow also brings DSL in
different
languages, allowing users to easily implement their data integration
processes.
== Proposal ==
Dataflow is a simple, flexible, and powerful system for distributed
data
processing at any scale. Dataflow provides a unified programming
model, a
software development kit to define and construct data processing
pipelines,
and runners to execute Dataflow pipelines in several runtime engines,
like
Apache Spark, Apache Flink, or Google Cloud Dataflow. Dataflow can be
used
for a variety of streaming or batch data processing goals including
ETL,
stream analysis, and aggregate computation. The underlying programming
model for Dataflow provides MapReduce-like parallelism, combined with
support for powerful data windowing, and fine-grained correctness
control.
== Background ==
Dataflow started as a set of Google projects focused on making data
processing easier, faster, and less costly. The Dataflow model is a
successor to MapReduce, FlumeJava, and Millwheel inside Google and is
focused on providing a unified solution for batch and stream
processing.
These projects on which Dataflow is based have been published in
several
papers made available to the public:
* MapReduce - http://research.google.com/archive/mapreduce.html
* Dataflow model - http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf
* FlumeJava - http://notes.stephenholiday.com/FlumeJava.pdf
* MillWheel - http://research.google.com/pubs/pub41378.html
Dataflow was designed from the start to provide a portable programming
layer. When you define a data processing pipeline with the Dataflow
model,
you are creating a job which is capable of being processed by any
number
of
Dataflow processing engines. Several engines have been developed to
run
Dataflow pipelines in other open source runtimes, including a Dataflow
runner for Apache Flink and Apache Spark. There is also a ³direct
runner²,
for execution on the developer machine (mainly for dev/debug
purposes).
Another runner allows a Dataflow program to run on a managed service,
Google Cloud Dataflow, in Google Cloud Platform. The Dataflow Java
SDK is
already available on GitHub, and independent from the Google Cloud
Dataflow
service. Another Python SDK is currently in active development.
In this proposal, the Dataflow SDKs, model, and a set of runners will
be
submitted as an OSS project under the ASF. The runners which are a
part
of
this proposal include those for Spark (from Cloudera), Flink (from
data
Artisans), and local development (from Google); the Google Cloud
Dataflow
service runner is not included in this proposal. Further references to
Dataflow will refer to the Dataflow model, SDKs, and runners which
are a
part of this proposal (Apache Dataflow) only. The initial submission
will
contain the already-released Java SDK; Google intends to submit the
Python
SDK later in the incubation process. The Google Cloud Dataflow service
will
continue to be one of many runners for Dataflow, built on Google Cloud
Platform, to run Dataflow pipelines. Necessarily, Cloud Dataflow will
develop against the Apache project additions, updates, and changes.
Google
Cloud Dataflow will become one user of Apache Dataflow and will
participate
in the project openly and publicly.
The Dataflow programming model has been designed with simplicity,
scalability, and speed as key tenants. In the Dataflow model, you only
need
to think about four top-level concepts when constructing your data
processing job:
* Pipelines - The data processing job made of a series of computations
including input, processing, and output
* PCollections - Bounded (or unbounded) datasets which represent the
input,
intermediate and output data in pipelines
* PTransforms - A data processing step in a pipeline in which one or
more
PCollections are an input and output
* I/O Sources and Sinks - APIs for reading and writing data which are
the
roots and endpoints of the pipeline
== Rationale ==
With Dataflow, Google intended to develop a framework which allowed
developers to be maximally productive in defining the processing, and
then
be able to execute the program at various levels of
latency/cost/completeness without re-architecting or re-writing it.
This
goal was informed by Google¹s past experience developing several
models,
frameworks, and tools useful for large-scale and distributed data
processing. While Google has previously published papers describing
some
of
its technologies, Google decided to take a different approach with
Dataflow. Google open-sourced the SDK and model alongside
commercialization
of the idea and ahead of publishing papers on the topic. As a result,
a
number of open source runtimes exist for Dataflow, such as the Apache
Flink
and Apache Spark runners.
We believe that submitting Dataflow as an Apache project will provide
an
immediate, worthwhile, and substantial contribution to the open source
community. As an incubating project, we believe Dataflow will have a
better
opportunity to provide a meaningful contribution to OSS and also
integrate
with other Apache projects.
In the long term, we believe Dataflow can be a powerful abstraction
layer
for data processing. By providing an abstraction layer for data
pipelines
and processing, data workflows can be increasingly portable,
resilient to
breaking changes in tooling, and compatible across many execution
engines,
runtimes, and open source projects.
== Initial Goals ==
We are breaking our initial goals into immediate (< 2 months),
short-term
(2-4 months), and intermediate-term (> 4 months).
Our immediate goals include the following:
* Plan for reconciling the Dataflow Java SDK and various runners into
one
project
* Plan for refactoring the existing Java SDK for better extensibility
by
SDK and runner writers
* Validating all dependencies are ASL 2.0 or compatible
* Understanding and adapting to the Apache development process
Our short-term goals include:
* Moving the newly-merged lists, and build utilities to Apache
* Start refactoring codebase and move code to Apache Git repo
* Continue development of new features, functions, and fixes in the
Dataflow Java SDK, and Dataflow runners
* Cleaning up the Dataflow SDK sources and crafting a roadmap and plan
for
how to include new major ideas, modules, and runtimes
* Establishment of easy and clear build/test framework for Dataflow
and
associated runtimes; creation of testing, rollback, and validation
policy
* Analysis and design for work needed to make Dataflow a better data
processing abstraction layer for multiple open source frameworks and
environments
Finally, we have a number of intermediate-term goals:
* Roadmapping, planning, and execution of integrations with other OSS
and
non-OSS projects/products
* Inclusion of additional SDK for Python, which is under active
development
== Current Status ==
=== Meritocracy ===
Dataflow was initially developed based on ideas from many employees
within
Google. As an ASL OSS project on GitHub, the Dataflow SDK has received
contributions from data Artisans, Cloudera Labs, and other individual
developers. As a project under incubation, we are committed to
expanding
our effort to build an environment which supports a meritocracy. We
are
focused on engaging the community and other related projects for
support
and contributions. Moreover, we are committed to ensure contributors
and
committers to Dataflow come from a broad mix of organizations through
a
merit-based decision process during incubation. We believe strongly in
the
Dataflow model and are committed to growing an inclusive community of
Dataflow contributors.
=== Community ===
The core of the Dataflow Java SDK has been developed by Google for use
with
Google Cloud Dataflow. Google has active community engagement in the
SDK
GitHub repository (
https://github.com/GoogleCloudPlatform/DataflowJavaSDK
),
on Stack Overflow (
http://stackoverflow.com/questions/tagged/google-cloud-dataflow) and
has
had contributions from a number of organizations and indivuduals.
Everyday, Cloud Dataflow is actively used by a number of organizations
and
institutions for batch and stream processing of data. We believe
acceptance
will allow us to consolidate existing Dataflow-related work, grow the
Dataflow community, and deepen connections between Dataflow and other
open
source projects.
=== Core Developers ===
The core developers for Dataflow and the Dataflow runners are:
* Frances Perry
* Tyler Akidau
* Davor Bonaci
* Luke Cwik
* Ben Chambers
* Kenn Knowles
* Dan Halperin
* Daniel Mills
* Mark Shields
* Craig Chambers
* Maximilian Michels
* Tom White
* Josh Wills
=== Alignment ===
The Dataflow SDK can be used to create Dataflow pipelines which can be
executed on Apache Spark or Apache Flink. Dataflow is also related to
other
Apache projects, such as Apache Crunch. We plan on expanding
functionality
for Dataflow runners, support for additional domain specific
languages,
and
increased portability so Dataflow is a powerful abstraction layer for
data
processing.
== Known Risks ==
=== Orphaned Products ===
The Dataflow SDK is presently used by several organizations, from
small
startups to Fortune 100 companies, to construct production pipelines
which
are executed in Google Cloud Dataflow. Google has a long-term
commitment
to
advance the Dataflow SDK; moreover, Dataflow is seeing increasing
interest,
development, and adoption from organizations outside of Google.
=== Inexperience with Open Source ===
Google believes strongly in open source and the exchange of
information
to
advance new ideas and work. Examples of this commitment are active OSS
projects such as Chromium (https://www.chromium.org) and Kubernetes (
http://kubernetes.io/). With Dataflow, we have tried to be
increasingly
open and forward-looking; we have published a paper in the VLDB
conference
describing the Dataflow model (
http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf) and were quick to
release
the Dataflow SDK as open source software with the launch of Cloud
Dataflow.
Our submission to the Apache Software Foundation is a logical
extension
of
our commitment to open source software.
=== Homogeneous Developers ===
The majority of committers in this proposal belong to Google due to
the
fact that Dataflow has emerged from several internal Google projects.
This
proposal also includes committers outside of Google who are actively
involved with other Apache projects, such as Hadoop, Flink, and Spark.
We
expect our entry into incubation will allow us to expand the number of
individuals and organizations participating in Dataflow development.
Additionally, separation of the Dataflow SDK from Google Cloud
Dataflow
allows us to focus on the open source SDK and model and do what is
best
for
this project.
=== Reliance on Salaried Developers ===
The Dataflow SDK and Dataflow runners have been developed primarily by
salaried developers supporting the Google Cloud Dataflow project.
While
the
Dataflow SDK and Cloud Dataflow have been developed by different teams
(and
this proposal would reinforce that separation) we expect our initial
set
of
developers will still primarily be salaried. Contribution has not been
exclusively from salaried developers, however. For example, the
contrib
directory of the Dataflow SDK (
https://github.com/GoogleCloudPlatform/DataflowJavaSDK/tree/master/contri
b
)
contains items from free-time contributors. Moreover, seperate
projects,
such as ScalaFlow (https://github.com/darkjh/scalaflow) have been
created
around the Dataflow model and SDK. We expect our reliance on salaried
developers will decrease over time during incubation.
=== Relationship with other Apache products ===
Dataflow directly interoperates with or utilizes several existing
Apache
projects.
* Build
** Apache Maven
* Data I/O, Libraries
** Apache Avro
** Apache Commons
* Dataflow runners
** Apache Flink
** Apache Spark
Dataflow when used in batch mode shares similarities with Apache
Crunch;
however, Dataflow is focused on a model, SDK, and abstraction layer
beyond
Spark and Hadoop (MapReduce.) One key goal of Dataflow is to provide
an
intermediate abstraction layer which can easily be implemented and
utilized
across several different processing frameworks.
=== An excessive fascination with the Apache brand ===
With this proposal we are not seeking attention or publicity. Rather,
we
firmly believe in the Dataflow model, SDK, and the ability to make
Dataflow
a powerful yet simple framework for data processing. While the
Dataflow
SDK
and model have been open source, we believe putting code on GitHub can
only
go so far. We see the Apache community, processes, and mission as
critical
for ensuring the Dataflow SDK and model are truly community-driven,
positively impactful, and innovative open source software. While
Google
has
taken a number of steps to advance its various open source projects,
we
believe Dataflow is a great fit for the Apache Software Foundation
due to
its focus on data processing and its relationships to existing ASF
projects.
== Documentation ==
The following documentation is relevant to this proposal. Relevant
portion
of the documentation will be contributed to the Apache Dataflow
project.
* Dataflow website: https://cloud.google.com/dataflow
* Dataflow programming model:
https://cloud.google.com/dataflow/model/programming-model
* Codebases
** Dataflow Java SDK:
https://github.com/GoogleCloudPlatform/DataflowJavaSDK
** Flink Dataflow runner:
https://github.com/dataArtisans/flink-dataflow
** Spark Dataflow runner: https://github.com/cloudera/spark-dataflow
* Dataflow Java SDK issue tracker:
https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues
* google-cloud-dataflow tag on Stack Overflow:
http://stackoverflow.com/questions/tagged/google-cloud-dataflow
== Initial Source ==
The initial source for Dataflow which we will submit to the Apache
Foundation will include several related projects which are currently
hosted
on the GitHub repositories:
* Dataflow Java SDK (
https://github.com/GoogleCloudPlatform/DataflowJavaSDK)
* Flink Dataflow runner
(https://github.com/dataArtisans/flink-dataflow)
* Spark Dataflow runner (https://github.com/cloudera/spark-dataflow)
These projects have always been Apache 2.0 licensed. We intend to
bundle
all of these repositories since they are all complimentary and should
be
maintained in one project. Prior to our submission, we will combine
all
of
these projects into a new git repository.
== Source and Intellectual Property Submission Plan ==
The source for the Dataflow SDK and the three runners (Spark, Flink,
Google
Cloud Dataflow) are already licensed under an Apache 2 license.
* Dataflow SDK -
https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/LICENS
E
* Flink runner -
https://github.com/dataArtisans/flink-dataflow/blob/master/LICENSE
* Spark runner -
https://github.com/cloudera/spark-dataflow/blob/master/LICENSE
Contributors to the Dataflow SDK have also signed the Google
Individual
Contributor License Agreement (
https://cla.developers.google.com/about/google-individual) in order to
contribute to the project.
With respect to trademark rights, Google does not hold a trademark on
the
phrase ³Dataflow.² Based on feedback and guidance we receive during
the
incubation process, we are open to renaming the project if necessary
for
trademark or other concerns.
== External Dependencies ==
All external dependencies are licensed under an Apache 2.0 or
Apache-compatible license. As we grow the Dataflow community we will
configure our build process to require and validate all contributions
and
dependencies are licensed under the Apache 2.0 license or are under an
Apache-compatible license.
== Required Resources ==
=== Mailing Lists ===
We currently use a mix of mailing lists. We will migrate our existing
mailing lists to the following:
* d...@dataflow.incubator.apache.org
* u...@dataflow.incubator.apache.org
* priv...@dataflow.incubator.apache.org
* comm...@dataflow.incubator.apache.org
=== Source Control ===
The Dataflow team currently uses Git and would like to continue to do
so.
We request a Git repository for Dataflow with mirroring to GitHub
enabled.
=== Issue Tracking ===
We request the creation of an Apache-hosted JIRA. The Dataflow
project is
currently using both a public GitHub issue tracker and internal Google
issue tracking. We will migrate and combine from these two sources to
the
Apache JIRA.
== Initial Committers ==
* Aljoscha Krettek [aljos...@apache.org]
* Amit Sela [amitsel...@gmail.com]
* Ben Chambers [bchamb...@google.com]
* Craig Chambers [chamb...@google.com]
* Dan Halperin [dhalp...@google.com]
* Davor Bonaci [da...@google.com]
* Frances Perry [f...@google.com]
* James Malone [jamesmal...@google.com]
* Jean-Baptiste Onofré [jbono...@apache.org]
* Josh Wills [jwi...@apache.org]
* Kostas Tzoumas [kos...@data-artisans.com]
* Kenneth Knowles [k...@google.com]
* Luke Cwik [lc...@google.com]
* Maximilian Michels [m...@apache.org]
* Stephan Ewen [step...@data-artisans.com]
* Tom White [t...@cloudera.com]
* Tyler Akidau [taki...@google.com]
== Affiliations ==
The initial committers are from six organizations. Google developed
Dataflow and the Dataflow SDK, data Artisans developed the Flink
runner,
and Cloudera (Labs) developed the Spark runner.
* Cloudera
** Tom White
* Data Artisans
** Aljoscha Krettek
** Kostas Tzoumas
** Maximilian Michels
** Stephan Ewen
* Google
** Ben Chambers
** Dan Halperin
** Davor Bonaci
** Frances Perry
** James Malone
** Kenneth Knowles
** Luke Cwik
** Tyler Akidau
* PayPal
** Amit Sela
* Slack
** Josh Wills
* Talend
** Jean-Baptiste Onofré
== Sponsors ==
=== Champion ===
* Jean-Baptiste Onofre [jbono...@apache.org]
=== Nominated Mentors ===
* Jim Jagielski [j...@apache.org]
* Venkatesh Seetharam [venkat...@apache.org]
* Bertrand Delacretaz [bdelacre...@apache.org]
* Ted Dunning [tdunn...@apache.org]
=== Sponsoring Entity ===
The Apache Incubator
--
Sean
