[
https://issues.apache.org/jira/browse/FLINK-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123522#comment-16123522
]
Aljoscha Krettek commented on FLINK-3133:
-----------------------------------------
I would be in favour of closing this issue. The DataStream API is already quite
big as it is and adding more API surface increases the maintenance burden and
also the number of things that can go wrong. Also, the communication between
TaskManagers is still a problem and is made even harder by recent changes to
how people want to deploy Flink, i.e. deployments in Kubernetes, where you have
overlay networks. We would therefore have so solve similar problems as
Queryable State and other efforts are currently solving. (There is work to make
QS use REST to communicate with JobManager/TaskManager and make it work well
with overlay networks).
That being said, I would like to have a good testing framework, similar to
flink-spector (or flink-spector) as part of Flink. I think a generic framework
that does something like {{SavepointMigrationTestBase}} would be very good. In
there, I'm using Accumulators to listen for certain conditions being met and
then cancel the job. This avoids the dreaded {{SuccessException}} that we have
in so many tests. We could also use queryable state to do a similar thing.
However, this is still blocked on having asynchronous job submission, which I
think will be easier to do once a bit more FLIP-6 related work lands, among
other things a new REST-based JobClient (see FLINK-7071, FLINK-7072, and
FLINK-7073).
For reference, {{PAssert}} from Beam is very nice API for testing.
> Introduce collect()/count()/print() methods in DataStream API
> -------------------------------------------------------------
>
> Key: FLINK-3133
> URL: https://issues.apache.org/jira/browse/FLINK-3133
> Project: Flink
> Issue Type: Improvement
> Components: DataStream API
> Affects Versions: 0.10.0, 1.0.0, 0.10.1
> Reporter: Maximilian Michels
> Assignee: Evgeny Kincharov
> Fix For: 1.0.0
>
>
> The DataSet API's methods {{collect()}}, {{count()}}, and {{print()}} should
> be mirrored to the DataStream API.
> The semantics of the calls are different. We need to be able to sample parts
> of a stream, e.g. by supplying a time period in the arguments to the methods.
> Users should use the {{JobClient}} to retrieve the results.
> {code:java}
> StreamExecutionEnvironment env =
> StramEnvironment.getStreamExecutionEnvironment();
> DataStream<DataType> streamData = env.addSource(..).map(..);
> JobClient jobClient = env.executeWithControl();
> Iterable<DataType> sampled = jobClient.sampleStream(streamData,
> Time.seconds(5));
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
