[
https://issues.apache.org/jira/browse/FLINK-8971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16465920#comment-16465920
]
ASF GitHub Bot commented on FLINK-8971:
---------------------------------------
Github user StefanRRichter commented on the issue:
https://github.com/apache/flink/pull/5941
As discussed separately, I think that this could be simplified quiet a bit
and that we do not require connected streams. The operator state does not have
to depend on the input events. For example, it could just be something that is
correlated with the subtask index, and you can reason about repartitioning by
that.
> Create general purpose testing job
> ----------------------------------
>
> Key: FLINK-8971
> URL: https://issues.apache.org/jira/browse/FLINK-8971
> Project: Flink
> Issue Type: Task
> Components: Tests
> Affects Versions: 1.5.0
> Reporter: Till Rohrmann
> Priority: Critical
> Fix For: 1.5.0
>
>
> In order to write better end-to-end tests we need a general purpose testing
> job which comprises as many Flink aspects as possible. These include
> different types for records and state, user defined components, state types
> and operators.
> The job should allow to activate a certain misbehavior, such as slowing
> certain paths down or throwing exceptions to simulate failures.
> The job should come with a data generator which generates input data such
> that the job can verify it's own behavior. This includes the state as well as
> the input/output records.
> We already have the [heavily misbehaved
> job|https://github.com/rmetzger/flink-testing-jobs/blob/flink-1.4/basic-jobs/src/main/java/com/dataartisans/heavymisbehaved/HeavyMisbehavedJob.java]
> which simulates some misbehavior. There is also the [state machine
> job|https://github.com/dataArtisans/flink-testing-jobs/tree/master/streaming-state-machine]
> which can verify itself for invalid state changes which indicate data loss.
> We should incorporate their characteristics into the new general purpose job.
> Additionally, the general purpose job should contain the following aspects:
> * Job containing a sliding window aggregation
> * At least one generic Kryo type
> * At least one generic Avro type
> * At least one Avro specific record type
> * At least one input type for which we register a Kryo serializer
> * At least one input type for which we provide a user defined serializer
> * At least one state type for which we provide a user defined serializer
> * At least one state type which uses the AvroSerializer
> * Include an operator with ValueState
> * Value state changes should be verified (e.g. predictable series of values)
> * Include an operator with operator state
> * Include an operator with broadcast state
> * Broadcast state changes should be verified (e.g. predictable series of
> values)
> * Include union state
> * User defined watermark assigner
> The job should be made available in the {{flink-end-to-end-tests}} module.
> This issue is intended to serve as an umbrella issue for developing and
> extending this job.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
