Dong Lin created FLINK-22915:
--------------------------------
Summary: Extend Flink ML API to support Estimator/Transformer DAG
Key: FLINK-22915
URL: https://issues.apache.org/jira/browse/FLINK-22915
Project: Flink
Issue Type: Improvement
Reporter: Dong Lin
Currently Flink ML API allows users to compose an Estimator/Transformer from a
pipeline (i.e. linear sequence) of Estimator/Transformer. We propose to extend
the Flink ML API so that users can compose an Estimator/Transformer from a
directed-acyclic-graph (i.e. DAG) of Estimator/Transformer.
This feature is useful for the following use-cases:
1) The preprocessing workflow (shared between training and inference workflows)
may involve the join of multiple tables, where the join of two tables can be
expressed as a Transformer of 2 inputs and 1 output. And the preprocessing
workflow could also involve the spilt operation, where the split operation has
1 input (e.g. the original table) and 2 outputs (e.g. the split of the original
table).
The expression of preprocessing workflow involving the join/split operation
needs to be expressed as a DAG of Transformer.
2) The graph-embedding algorithm can be expressed as an Estimator, where the
Estimator takes as input two tables (e.g. a node table and an edge table). The
corresponding Transformer has 1 input (i.e. the node) and 1 output (i.e. the
node after embedding)
The expression of training workflow involving the graph-embedding Estimator
needs to be expressed as a DAG of Transformer/Estimator.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
