[jira] [Assigned] (FLINK-16784) Support KeyedBroadcastProcessFunction state bootstrapping.
[ https://issues.apache.org/jira/browse/FLINK-16784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov reassigned FLINK-16784: - Assignee: Or Keren > Support KeyedBroadcastProcessFunction state bootstrapping. > --- > > Key: FLINK-16784 > URL: https://issues.apache.org/jira/browse/FLINK-16784 > Project: Flink > Issue Type: Improvement > Components: API / State Processor >Reporter: Seth Wiesman >Assignee: Or Keren >Priority: Not a Priority > Labels: auto-deprioritized-major, auto-deprioritized-minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35240) Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record
[
https://issues.apache.org/jira/browse/FLINK-35240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841271#comment-17841271
]
Alexander Fedulov commented on FLINK-35240:
---
[~gongzhongqiang] and [~robyoung] you are right!
Unfortunately the docs of the {{FLUSH_PASSED_TO_STREAM}} are lying about it
only being relevant for the {{flush}} method, whereas it indeed controls
individual flushes from the {{close()}} method
{code:java}
/**
* Feature that specifies that calls to {@link #flush} will cause
* matching flush() to underlying {@link OutputStream}
* or {@link Writer}; if disabled this will not be done.
* Main reason to disable this feature is to prevent flushing at
* generator level, if it is not possible to prevent method being
* called by other code (like ObjectMapper or third
* party libraries).
*
* Feature is enabled by default.
*/
FLUSH_PASSED_TO_STREAM(true){code}
> Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record
> -
>
> Key: FLINK-35240
> URL: https://issues.apache.org/jira/browse/FLINK-35240
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem
>Reporter: Zhongqiang Gong
>Priority: Minor
> Attachments: image-2024-04-26-00-23-29-975.png,
> image-2024-04-26-17-16-07-925.png, image-2024-04-26-17-16-20-647.png,
> image-2024-04-26-17-16-30-293.png, screenshot-1.png
>
>
> *Reproduce：*
> * According to user email:
> https://lists.apache.org/thread/9j5z8hv4vjkd54dkzqy1ryyvm0l5rxhc
> * !image-2024-04-26-00-23-29-975.png!
> *Analysis：*
> * `org.apache.flink.formats.csv.CsvBulkWriter#addElement` will flush per
> record.
> *Solution：*
> * I think maybe we can disable `FLUSH_AFTER_WRITE_VALUE` to avoid flush when
> a record added.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35240) Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record
[ https://issues.apache.org/jira/browse/FLINK-35240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-35240: -- Attachment: image-2024-04-26-17-16-20-647.png > Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record > - > > Key: FLINK-35240 > URL: https://issues.apache.org/jira/browse/FLINK-35240 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Reporter: Zhongqiang Gong >Priority: Minor > Attachments: image-2024-04-26-00-23-29-975.png, > image-2024-04-26-17-16-07-925.png, image-2024-04-26-17-16-20-647.png, > image-2024-04-26-17-16-30-293.png, screenshot-1.png > > > *Reproduce：* > * According to user email: > https://lists.apache.org/thread/9j5z8hv4vjkd54dkzqy1ryyvm0l5rxhc > * !image-2024-04-26-00-23-29-975.png! > *Analysis：* > * `org.apache.flink.formats.csv.CsvBulkWriter#addElement` will flush per > record. > *Solution：* > * I think maybe we can disable `FLUSH_AFTER_WRITE_VALUE` to avoid flush when > a record added. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35240) Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record
[ https://issues.apache.org/jira/browse/FLINK-35240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-35240: -- Attachment: image-2024-04-26-17-16-30-293.png > Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record > - > > Key: FLINK-35240 > URL: https://issues.apache.org/jira/browse/FLINK-35240 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Reporter: Zhongqiang Gong >Priority: Minor > Attachments: image-2024-04-26-00-23-29-975.png, > image-2024-04-26-17-16-07-925.png, image-2024-04-26-17-16-20-647.png, > image-2024-04-26-17-16-30-293.png, screenshot-1.png > > > *Reproduce：* > * According to user email: > https://lists.apache.org/thread/9j5z8hv4vjkd54dkzqy1ryyvm0l5rxhc > * !image-2024-04-26-00-23-29-975.png! > *Analysis：* > * `org.apache.flink.formats.csv.CsvBulkWriter#addElement` will flush per > record. > *Solution：* > * I think maybe we can disable `FLUSH_AFTER_WRITE_VALUE` to avoid flush when > a record added. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35240) Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record
[ https://issues.apache.org/jira/browse/FLINK-35240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-35240: -- Attachment: image-2024-04-26-17-16-07-925.png > Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record > - > > Key: FLINK-35240 > URL: https://issues.apache.org/jira/browse/FLINK-35240 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Reporter: Zhongqiang Gong >Priority: Minor > Attachments: image-2024-04-26-00-23-29-975.png, > image-2024-04-26-17-16-07-925.png, image-2024-04-26-17-16-20-647.png, > image-2024-04-26-17-16-30-293.png, screenshot-1.png > > > *Reproduce：* > * According to user email: > https://lists.apache.org/thread/9j5z8hv4vjkd54dkzqy1ryyvm0l5rxhc > * !image-2024-04-26-00-23-29-975.png! > *Analysis：* > * `org.apache.flink.formats.csv.CsvBulkWriter#addElement` will flush per > record. > *Solution：* > * I think maybe we can disable `FLUSH_AFTER_WRITE_VALUE` to avoid flush when > a record added. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35240) Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record
[ https://issues.apache.org/jira/browse/FLINK-35240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841117#comment-17841117 ] Alexander Fedulov commented on FLINK-35240: --- I don't think touching any of the flush-specific properties should be necessary. You can see in the FlameGraph that flush calls are due to close being called and, as [~robyoung] mentioned, this is what JsonGenerator.Feature#AUTO_CLOSE_TARGET is there for. > Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record > - > > Key: FLINK-35240 > URL: https://issues.apache.org/jira/browse/FLINK-35240 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Reporter: Zhongqiang Gong >Priority: Minor > Attachments: image-2024-04-26-00-23-29-975.png, screenshot-1.png > > > *Reproduce：* > * According to user email: > https://lists.apache.org/thread/9j5z8hv4vjkd54dkzqy1ryyvm0l5rxhc > * !image-2024-04-26-00-23-29-975.png! > *Analysis：* > * `org.apache.flink.formats.csv.CsvBulkWriter#addElement` will flush per > record. > *Solution：* > * I think maybe we can disable `FLUSH_AFTER_WRITE_VALUE` to avoid flush when > a record added. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35240) Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record
[
https://issues.apache.org/jira/browse/FLINK-35240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840935#comment-17840935
]
Alexander Fedulov commented on FLINK-35240:
---
[~gongzhongqiang] this is pretty strange, because I remember that I ran into
this issue during development and this is exactly why this line is there:
[https://github.com/apache/flink/blob/c0bf0ac3fb1fe4814bff09807ed2040bb13da052/flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvBulkWriter.java#L60]
Maybe Jackson version got bumped and something in the internal initialization
of the _generatorFactory and the state of the "parent" mapper has changed. The
pointer to it is derived from the parent mapper, so it should, in theory, not
make a difference whether the setting is applied before or after the
{{.writer()}} call in the CsvBulkWriter constructor:
[ObjectWriter.java#L105|https://github.com/FasterXML/jackson-databind/blob/jackson-databind-2.14.2/src/main/java/com/fasterxml/jackson/databind/ObjectWriter.java#L105]
Try to move the {{JsonGenerator.Feature#AUTO_CLOSE_TARGET}} configuration above
the .writer() call.
It seems the better way could also be to configure it on the mapper itself, not
the generator, i.e.:
{{mapper.configure(JsonGenerator.Feature.AUTO_CLOSE_TARGET, false);}}
I am pretty sure
*com.fasterxml.jackson.core.JsonGenerator.Feature#AUTO_CLOSE_TARGET* is the
right property that is supposed to control this behavior:
[JsonFactory.java#L1474-L1482|https://github.com/FasterXML/jackson-core/blob/2.18/src/main/java/com/fasterxml/jackson/core/JsonFactory.java#L1474-L1482]
> Disable FLUSH_AFTER_WRITE_VALUE to avoid flush per record
> -
>
> Key: FLINK-35240
> URL: https://issues.apache.org/jira/browse/FLINK-35240
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem
>Reporter: Zhongqiang Gong
>Priority: Minor
> Attachments: image-2024-04-26-00-23-29-975.png
>
>
> *Reproduce：*
> * According to user email:
> https://lists.apache.org/thread/9j5z8hv4vjkd54dkzqy1ryyvm0l5rxhc
> * !image-2024-04-26-00-23-29-975.png!
> *Analysis：*
> * `org.apache.flink.formats.csv.CsvBulkWriter#addElement` will flush per
> record.
> *Solution：*
> * I think maybe we can disable `FLUSH_AFTER_WRITE_VALUE` to avoid flush when
> a record added.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-35052) Webhook validator should reject unsupported Flink versions
Alexander Fedulov created FLINK-35052: - Summary: Webhook validator should reject unsupported Flink versions Key: FLINK-35052 URL: https://issues.apache.org/jira/browse/FLINK-35052 Project: Flink Issue Type: Improvement Components: Kubernetes Operator Reporter: Alexander Fedulov Assignee: Alexander Fedulov The admission webhook currently does not verify if FlinkDeployment CR utilizes Flink versions that are not supported by the Operator. This causes the CR to be accepted and the failure to be postponed until the reconciliation phase. We should instead fail fast and provide users direct feedback. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34117) CompactCoordinator for table file sink loses data upon job termination
[ https://issues.apache.org/jira/browse/FLINK-34117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17810848#comment-17810848 ] Alexander Fedulov commented on FLINK-34117: --- [~leonard] thanks! Here is the backport PR https://github.com/apache/flink/pull/24195 > CompactCoordinator for table file sink loses data upon job termination > -- > > Key: FLINK-34117 > URL: https://issues.apache.org/jira/browse/FLINK-34117 > Project: Flink > Issue Type: Bug > Components: Connectors / FileSystem >Affects Versions: 1.18.1 >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > CompactCoordinator accumulates data in currentInputFiles and only rolls them > into inputFiles in snapshotState(). At the same time it relies on separately > receiving checkpoint indications from the upstream operator via > processElement() (EndCheckpoint). If the job terminates, the final > EndCheckpoint can arrive before the snapshotState() gets called. This leads > to data loss (all events in currentInputFiles get discarded). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33694) GCS filesystem does not respect gs.storage.root.url config option
[
https://issues.apache.org/jira/browse/FLINK-33694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Fedulov updated FLINK-33694:
--
Release Note: This fix resolves the issue where the `gs.storage.root.url`
setting in the Hadoop configuration was not being acknowledged by the Sink.
Warning: If you have been using this property to configure the GCS Source,
please ensure that your tests or pipelines are not adversely affected by the
GCS Sink now also correctly adhering to this configuration.
> GCS filesystem does not respect gs.storage.root.url config option
> -
>
> Key: FLINK-33694
> URL: https://issues.apache.org/jira/browse/FLINK-33694
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
>Affects Versions: 1.18.0, 1.17.2
>Reporter: Patrick Lucas
>Assignee: Patrick Lucas
>Priority: Major
> Labels: gcs, pull-request-available
> Fix For: 1.19.0, 1.17.3, 1.18.2
>
>
> The GCS FileSystem's RecoverableWriter implementation uses the GCS SDK
> directly rather than going through Hadoop. While support has been added to
> configure credentials correctly based on the standard Hadoop implementation
> configuration, no other options are passed through to the underlying client.
> Because this only affects the RecoverableWriter-related codepaths, it can
> result in very surprising differing behavior whether the FileSystem is being
> used as a source or a sink—while a {{{}gs://{}}}-URI FileSource may work
> fine, a {{{}gs://{}}}-URI FileSink may not work at all.
> We use [fake-gcs-server|https://github.com/fsouza/fake-gcs-server] in
> testing, and so we override the Hadoop GCS FileSystem config option
> {{{}gs.storage.root.url{}}}. However, because this option is not considered
> when creating the GCS client for the RecoverableWriter codepath, in a
> FileSink the GCS FileSystem attempts to write to the real GCS service rather
> than fake-gcs-server. At the same time, a FileSource works as expected,
> reading from fake-gcs-server.
> The fix should be fairly straightforward, reading the {{gs.storage.root.url}}
> config option from the Hadoop FileSystem config in
> [{{GSFileSystemOptions}}|https://github.com/apache/flink/blob/release-1.18.0/flink-filesystems/flink-gs-fs-hadoop/src/main/java/org/apache/flink/fs/gs/GSFileSystemOptions.java#L30]
> and, if set, passing it to {{storageOptionsBuilder}} in
> [{{GSFileSystemFactory}}|https://github.com/apache/flink/blob/release-1.18.0/flink-filesystems/flink-gs-fs-hadoop/src/main/java/org/apache/flink/fs/gs/GSFileSystemFactory.java].
> The only workaround for this is to build a custom flink-gs-fs-hadoop JAR with
> a patch and use it as a plugin.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34117) CompactCoordinator for table file sink loses data upon job termination
Alexander Fedulov created FLINK-34117: - Summary: CompactCoordinator for table file sink loses data upon job termination Key: FLINK-34117 URL: https://issues.apache.org/jira/browse/FLINK-34117 Project: Flink Issue Type: Bug Reporter: Alexander Fedulov Assignee: Alexander Fedulov CompactCoordinator accumulates data in currentInputFiles and only rolls them into inputFiles in snapshotState(). At the same time it relies on separately receiving checkpoint indications from the upstream operator via processElement() (EndCheckpoint). If the job terminates, the final EndCheckpoint can arrive before the snapshotState() gets called. This leads to data loss (all events in currentInputFiles get discarded). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33789) Expose restart time as a metric
[ https://issues.apache.org/jira/browse/FLINK-33789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-33789: -- Summary: Expose restart time as a metric (was: Expose restart time as metric) > Expose restart time as a metric > --- > > Key: FLINK-33789 > URL: https://issues.apache.org/jira/browse/FLINK-33789 > Project: Flink > Issue Type: New Feature > Components: Autoscaler, Kubernetes Operator >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > Labels: pull-request-available > Fix For: kubernetes-operator-1.8.0 > > > FLINK-30593 added restart time tracking. It would be convenient to also > report is as a metric. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33789) Expose restart time as metric
[ https://issues.apache.org/jira/browse/FLINK-33789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-33789: -- Description: FLINK-30593 added restart time tracking. It would be convenient to also report is as a metric. (was: FLINK-30593 added restart time tracking. It would be convenient to also report is as metric.) > Expose restart time as metric > - > > Key: FLINK-33789 > URL: https://issues.apache.org/jira/browse/FLINK-33789 > Project: Flink > Issue Type: New Feature > Components: Autoscaler, Kubernetes Operator >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > Labels: pull-request-available > Fix For: kubernetes-operator-1.8.0 > > > FLINK-30593 added restart time tracking. It would be convenient to also > report is as a metric. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33789) Expose restart time as metric
[ https://issues.apache.org/jira/browse/FLINK-33789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-33789: -- Description: FLINK-30593 added restart time tracking. It would be convenient to also report is as metric. (was: Currently the autoscaler uses a preconfigured restart time for the job. We should dynamically adjust this on the observered restart times for scale operations.) > Expose restart time as metric > - > > Key: FLINK-33789 > URL: https://issues.apache.org/jira/browse/FLINK-33789 > Project: Flink > Issue Type: New Feature > Components: Autoscaler, Kubernetes Operator >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > Labels: pull-request-available > Fix For: kubernetes-operator-1.8.0 > > > FLINK-30593 added restart time tracking. It would be convenient to also > report is as metric. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33789) Expose restart time as metric
Alexander Fedulov created FLINK-33789: - Summary: Expose restart time as metric Key: FLINK-33789 URL: https://issues.apache.org/jira/browse/FLINK-33789 Project: Flink Issue Type: New Feature Components: Autoscaler, Kubernetes Operator Reporter: Alexander Fedulov Assignee: Alexander Fedulov Fix For: kubernetes-operator-1.8.0 Currently the autoscaler uses a preconfigured restart time for the job. We should dynamically adjust this on the observered restart times for scale operations. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28229) Introduce Source API alternatives for StreamExecutionEnvironment#fromCollection() methods
[ https://issues.apache.org/jira/browse/FLINK-28229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-28229: -- Labels: pull-request-available (was: pull-request-available stale-assigned) > Introduce Source API alternatives for > StreamExecutionEnvironment#fromCollection() methods > - > > Key: FLINK-28229 > URL: https://issues.apache.org/jira/browse/FLINK-28229 > Project: Flink > Issue Type: Sub-task >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > Labels: pull-request-available > > * FromElementsFunction > * FromIteratorFunction > are based on SourceFunction API -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-28051) Promote ExternallyInducedSourceReader to non-experimental @Public
[ https://issues.apache.org/jira/browse/FLINK-28051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17790034#comment-17790034 ] Alexander Fedulov commented on FLINK-28051: --- [~Brian Zhou] can I assign this ticket to you or one of your colleagues? > Promote ExternallyInducedSourceReader to non-experimental @Public > - > > Key: FLINK-28051 > URL: https://issues.apache.org/jira/browse/FLINK-28051 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common, Tests >Reporter: Alexander Fedulov >Priority: Major > > It needs to be evaluated if ExternallyInducedSourceReader can be promoted. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-28051) Promote ExternallyInducedSourceReader to non-experimental @Public
[ https://issues.apache.org/jira/browse/FLINK-28051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17790032#comment-17790032 ] Alexander Fedulov commented on FLINK-28051: --- Hi [~Brian Zhou], thanks a lot for the clarification. Looking forward to contributions from the Pravega community. > Promote ExternallyInducedSourceReader to non-experimental @Public > - > > Key: FLINK-28051 > URL: https://issues.apache.org/jira/browse/FLINK-28051 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common, Tests >Reporter: Alexander Fedulov >Priority: Major > > It needs to be evaluated if ExternallyInducedSourceReader can be promoted. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28051) Promote ExternallyInducedSourceReader to non-experimental @Public
[ https://issues.apache.org/jira/browse/FLINK-28051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-28051: -- Summary: Promote ExternallyInducedSourceReader to non-experimental @Public (was: Introduce Source API alternative to ExternallyInducedSource) > Promote ExternallyInducedSourceReader to non-experimental @Public > - > > Key: FLINK-28051 > URL: https://issues.apache.org/jira/browse/FLINK-28051 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common, Tests >Reporter: Alexander Fedulov >Priority: Major > > It needs to be evaluated if ExternallyInducedSourceReader can be promoted. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33525) Migrate LoadSimulationPipeline in Flink K8S Operator to the new Source API
Alexander Fedulov created FLINK-33525: - Summary: Migrate LoadSimulationPipeline in Flink K8S Operator to the new Source API Key: FLINK-33525 URL: https://issues.apache.org/jira/browse/FLINK-33525 Project: Flink Issue Type: Sub-task Reporter: Alexander Fedulov https://github.com/apache/flink-kubernetes-operator/blob/main/examples/autoscaling/src/main/java/autoscaling/LoadSimulationPipeline.java#L100C51-L100C65 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-33525) Migrate LoadSimulationPipeline in Flink K8S Operator to the new Source API
[ https://issues.apache.org/jira/browse/FLINK-33525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov reassigned FLINK-33525: - Assignee: (was: Alexander Fedulov) > Migrate LoadSimulationPipeline in Flink K8S Operator to the new Source API > -- > > Key: FLINK-33525 > URL: https://issues.apache.org/jira/browse/FLINK-33525 > Project: Flink > Issue Type: Sub-task >Reporter: Alexander Fedulov >Priority: Major > > https://github.com/apache/flink-kubernetes-operator/blob/main/examples/autoscaling/src/main/java/autoscaling/LoadSimulationPipeline.java#L100C51-L100C65 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-33525) Migrate LoadSimulationPipeline in Flink K8S Operator to the new Source API
[ https://issues.apache.org/jira/browse/FLINK-33525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov reassigned FLINK-33525: - Assignee: Alexander Fedulov > Migrate LoadSimulationPipeline in Flink K8S Operator to the new Source API > -- > > Key: FLINK-33525 > URL: https://issues.apache.org/jira/browse/FLINK-33525 > Project: Flink > Issue Type: Sub-task >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > https://github.com/apache/flink-kubernetes-operator/blob/main/examples/autoscaling/src/main/java/autoscaling/LoadSimulationPipeline.java#L100C51-L100C65 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-30593) Determine restart time on the fly for Autoscaler
[ https://issues.apache.org/jira/browse/FLINK-30593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov reassigned FLINK-30593: - Assignee: Alexander Fedulov > Determine restart time on the fly for Autoscaler > > > Key: FLINK-30593 > URL: https://issues.apache.org/jira/browse/FLINK-30593 > Project: Flink > Issue Type: New Feature > Components: Autoscaler, Kubernetes Operator >Reporter: Gyula Fora >Assignee: Alexander Fedulov >Priority: Major > > Currently the autoscaler uses a preconfigured restart time for the job. We > should dynamically adjust this on the observered restart times for scale > operations. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32694) Migrate ParallelSourceFunction classes to Source V2 API or prepare for removal
[ https://issues.apache.org/jira/browse/FLINK-32694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-32694: -- Description: * SimpleEndlessSourceWithBloatedState in HeavyDeploymentStressTestProgram (org.apache.flink.deployment) * -StatefulSequenceSource (org.apache.flink.streaming.api.functions.source)- Migration not needed: https://github.com/apache/flink/pull/23611 * ArrowSourceFunction (org.apache.flink.table.runtime.arrow.sources) * InputFormatSourceFunction (org.apache.flink.streaming.api.functions.source) * -EventsGeneratorSource (org.apache.flink.streaming.examples.statemachine.generator)- resolved in FLINK-32670 * FromSplittableIteratorFunction (org.apache.flink.streaming.api.functions.source) * DataGeneratorSource (org.apache.flink.streaming.api.functions.source.datagen) * – Tests: * MySource in StatefulStreamingJob (org.apache.flink.test) * RandomLongSource in StickyAllocationAndLocalRecoveryTestJob (org.apache.flink.streaming.tests) * SequenceGeneratorSource (org.apache.flink.streaming.tests) * TtlStateUpdateSource (org.apache.flink.streaming.tests) * StringSourceFunction in NettyShuffleMemoryControlTestProgram (org.apache.flink.streaming.tests) was: * SimpleEndlessSourceWithBloatedState in HeavyDeploymentStressTestProgram (org.apache.flink.deployment) * StatefulSequenceSource (org.apache.flink.streaming.api.functions.source) * ArrowSourceFunction (org.apache.flink.table.runtime.arrow.sources) * InputFormatSourceFunction (org.apache.flink.streaming.api.functions.source) * -EventsGeneratorSource (org.apache.flink.streaming.examples.statemachine.generator)- resolved in FLINK-32670 * FromSplittableIteratorFunction (org.apache.flink.streaming.api.functions.source) * DataGeneratorSource (org.apache.flink.streaming.api.functions.source.datagen) * – Tests: * MySource in StatefulStreamingJob (org.apache.flink.test) * RandomLongSource in StickyAllocationAndLocalRecoveryTestJob (org.apache.flink.streaming.tests) * SequenceGeneratorSource (org.apache.flink.streaming.tests) * TtlStateUpdateSource (org.apache.flink.streaming.tests) * StringSourceFunction in NettyShuffleMemoryControlTestProgram (org.apache.flink.streaming.tests) > Migrate ParallelSourceFunction classes to Source V2 API or prepare for removal > -- > > Key: FLINK-32694 > URL: https://issues.apache.org/jira/browse/FLINK-32694 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > > * SimpleEndlessSourceWithBloatedState in HeavyDeploymentStressTestProgram > (org.apache.flink.deployment) > * -StatefulSequenceSource (org.apache.flink.streaming.api.functions.source)- > Migration not needed: https://github.com/apache/flink/pull/23611 > * ArrowSourceFunction (org.apache.flink.table.runtime.arrow.sources) > * InputFormatSourceFunction (org.apache.flink.streaming.api.functions.source) > * -EventsGeneratorSource > (org.apache.flink.streaming.examples.statemachine.generator)- resolved in > FLINK-32670 > * FromSplittableIteratorFunction > (org.apache.flink.streaming.api.functions.source) > * DataGeneratorSource > (org.apache.flink.streaming.api.functions.source.datagen) > * – Tests: > * MySource in StatefulStreamingJob (org.apache.flink.test) > * RandomLongSource in StickyAllocationAndLocalRecoveryTestJob > (org.apache.flink.streaming.tests) > * SequenceGeneratorSource (org.apache.flink.streaming.tests) > * TtlStateUpdateSource (org.apache.flink.streaming.tests) > * StringSourceFunction in NettyShuffleMemoryControlTestProgram > (org.apache.flink.streaming.tests) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32694) Migrate ParallelSourceFunction classes to Source V2 API or prepare for removal
[ https://issues.apache.org/jira/browse/FLINK-32694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-32694: -- Summary: Migrate ParallelSourceFunction classes to Source V2 API or prepare for removal (was: Migrate classes that implement ParallelSourceFunction to Source V2 API) > Migrate ParallelSourceFunction classes to Source V2 API or prepare for removal > -- > > Key: FLINK-32694 > URL: https://issues.apache.org/jira/browse/FLINK-32694 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > > * SimpleEndlessSourceWithBloatedState in HeavyDeploymentStressTestProgram > (org.apache.flink.deployment) > * StatefulSequenceSource (org.apache.flink.streaming.api.functions.source) > * ArrowSourceFunction (org.apache.flink.table.runtime.arrow.sources) > * InputFormatSourceFunction (org.apache.flink.streaming.api.functions.source) > * -EventsGeneratorSource > (org.apache.flink.streaming.examples.statemachine.generator)- resolved in > FLINK-32670 > * FromSplittableIteratorFunction > (org.apache.flink.streaming.api.functions.source) > * DataGeneratorSource > (org.apache.flink.streaming.api.functions.source.datagen) > * – Tests: > * MySource in StatefulStreamingJob (org.apache.flink.test) > * RandomLongSource in StickyAllocationAndLocalRecoveryTestJob > (org.apache.flink.streaming.tests) > * SequenceGeneratorSource (org.apache.flink.streaming.tests) > * TtlStateUpdateSource (org.apache.flink.streaming.tests) > * StringSourceFunction in NettyShuffleMemoryControlTestProgram > (org.apache.flink.streaming.tests) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-33367) Invalid Check in DefaultFileFilter
[
https://issues.apache.org/jira/browse/FLINK-33367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Fedulov closed FLINK-33367.
-
Release Note: Not an actual issue.
Resolution: Won't Fix
I am closing this as there is no description of the actual issue.
> Invalid Check in DefaultFileFilter
> --
>
> Key: FLINK-33367
> URL: https://issues.apache.org/jira/browse/FLINK-33367
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Affects Versions: 1.16.2
>Reporter: Chirag Dewan
>Priority: Minor
>
> There is a null check in DefaultFileFilter:
>
> if (fileName == null || fileName.length() == 0)
> { return true; }
>
> So 2 questions here -
> 1) Can a file name ever be null?
> 2) What will be the behavior with return true? Should it be return false
> rather?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33335) Reactivate missing e2e tests
[ https://issues.apache.org/jira/browse/FLINK-5?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-5: -- Summary: Reactivate missing e2e tests (was: Remove unused e2e tests) > Reactivate missing e2e tests > > > Key: FLINK-5 > URL: https://issues.apache.org/jira/browse/FLINK-5 > Project: Flink > Issue Type: Improvement >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > FLINK-17375 removed _run-pre-commit-tests.sh_ in Flink 1.12 [1]. Since then > the following tests are not executed anymore: > _test_state_migration.sh_ > _test_state_evolution.sh_ > _test_streaming_kinesis.sh_ > _test_streaming_classloader.sh_ > _test_streaming_distributed_cache_via_blob.sh_ > [1] > https://github.com/apache/flink/pull/12268/files#diff-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33335) Remove unused e2e tests
[ https://issues.apache.org/jira/browse/FLINK-5?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17779442#comment-17779442 ] Alexander Fedulov commented on FLINK-5: --- [~rmetzger] thanks for the clarification! I am changing the name of the ticket Remove unused e2e tests -> Reactivate missing e2e tests :) > Remove unused e2e tests > --- > > Key: FLINK-5 > URL: https://issues.apache.org/jira/browse/FLINK-5 > Project: Flink > Issue Type: Improvement >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > FLINK-17375 removed _run-pre-commit-tests.sh_ in Flink 1.12 [1]. Since then > the following tests are not executed anymore: > _test_state_migration.sh_ > _test_state_evolution.sh_ > _test_streaming_kinesis.sh_ > _test_streaming_classloader.sh_ > _test_streaming_distributed_cache_via_blob.sh_ > [1] > https://github.com/apache/flink/pull/12268/files#diff-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-33058) Support for JSON-encoded Avro
[ https://issues.apache.org/jira/browse/FLINK-33058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov reassigned FLINK-33058: - Assignee: Dale Lane > Support for JSON-encoded Avro > - > > Key: FLINK-33058 > URL: https://issues.apache.org/jira/browse/FLINK-33058 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Reporter: Dale Lane >Assignee: Dale Lane >Priority: Minor > Labels: avro, flink, flink-formats, pull-request-available > > Avro supports two serialization encoding methods: binary and JSON > cf. [https://avro.apache.org/docs/1.11.1/specification/#encodings] > flink-avro currently has a hard-coded assumption that Avro data is > binary-encoded (and cannot process Avro data that has been JSON-encoded). > I propose adding a new optional format option to flink-avro: *avro.encoding* > It will support two options: 'binary' and 'json'. > It unset, it will default to 'binary' to maintain compatibility/consistency > with current behaviour. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33335) Remove unused e2e tests
[ https://issues.apache.org/jira/browse/FLINK-5?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-5: -- Description: FLINK-17375 removed _run-pre-commit-tests.sh_ in Flink 1.12 [1]. Since then the following tests are not executed anymore: _test_state_migration.sh_ _test_state_evolution.sh_ _test_streaming_kinesis.sh_ _test_streaming_classloader.sh_ _test_streaming_distributed_cache_via_blob.sh_ [1] https://github.com/apache/flink/pull/12268/files#diff-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e was: FLINK-17375 removed _run-pre-commit-tests.sh_ in Flink 1.12 [1]. Since then the following tests are not executed anymore: _test_state_migration.sh_ _test_state_evolution.sh_ _test_streaming_kinesis.sh_ _test_streaming_classloader.sh_ _test_streaming_distributed_cache_via_blob.sh_ [1]https://github.com/apache/flink/pull/12268/files#diff-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e > Remove unused e2e tests > --- > > Key: FLINK-5 > URL: https://issues.apache.org/jira/browse/FLINK-5 > Project: Flink > Issue Type: Improvement >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > FLINK-17375 removed _run-pre-commit-tests.sh_ in Flink 1.12 [1]. Since then > the following tests are not executed anymore: > _test_state_migration.sh_ > _test_state_evolution.sh_ > _test_streaming_kinesis.sh_ > _test_streaming_classloader.sh_ > _test_streaming_distributed_cache_via_blob.sh_ > [1] > https://github.com/apache/flink/pull/12268/files#diff-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33335) Remove unused e2e tests
[ https://issues.apache.org/jira/browse/FLINK-5?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778126#comment-17778126 ] Alexander Fedulov edited comment on FLINK-5 at 10/22/23 10:07 AM: -- [~rmetzger] [~chesnay] Could you please confirm that disabling of the aforementioned tests was intentional? I would like to understand if we can simply drop _org.apache.flink.test.StatefulStreamingJob_ without moving it to a FLIP-27 source. was (Author: afedulov): [~rmetzger] [~chesnay] Could you please confirm that disabling of the aforementioned tests was intentional? I would like to understand if we can simply drop `org.apache.flink.test.StatefulStreamingJob` without moving it to a FLIP-27 source. > Remove unused e2e tests > --- > > Key: FLINK-5 > URL: https://issues.apache.org/jira/browse/FLINK-5 > Project: Flink > Issue Type: Improvement >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > FLINK-17375 removed _run-pre-commit-tests.sh_ in Flink 1.12 [1]. Since then > the following tests are not executed anymore: > _test_state_migration.sh_ > _test_state_evolution.sh_ > _test_streaming_kinesis.sh_ > _test_streaming_classloader.sh_ > _test_streaming_distributed_cache_via_blob.sh_ > [1]https://github.com/apache/flink/pull/12268/files#diff-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33335) Remove unused e2e tests
[ https://issues.apache.org/jira/browse/FLINK-5?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-5: -- Description: FLINK-17375 removed run-pre-commit-tests.sh in Flink 1.12 [1]. Since then the following tests are not executed anymore: test_state_migration.sh test_state_evolution.sh test_streaming_kinesis.sh test_streaming_classloader.sh test_streaming_distributed_cache_via_blob.sh [1]https://github.com/apache/flink/pull/12268/files#diff-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e was: FLINK-17375 removed run-pre-commit-tests.sh in Flink 1.12 [1]. Since then the following tests are not executed anymore: test_state_migration.sh test_state_evolution.sh test_streaming_kinesis.sh test_streaming_classloader.sh test_streaming_distributed_cache_via_blob.sh [1]https://github.com/apache/flink/pull/12268/files#diff-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e > Remove unused e2e tests > --- > > Key: FLINK-5 > URL: https://issues.apache.org/jira/browse/FLINK-5 > Project: Flink > Issue Type: Improvement >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > FLINK-17375 removed run-pre-commit-tests.sh in Flink 1.12 [1]. Since then the > following tests are not executed anymore: > test_state_migration.sh > test_state_evolution.sh > test_streaming_kinesis.sh > test_streaming_classloader.sh > test_streaming_distributed_cache_via_blob.sh > [1]https://github.com/apache/flink/pull/12268/files#diff-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33335) Remove unused e2e tests
[ https://issues.apache.org/jira/browse/FLINK-5?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-5: -- Description: FLINK-17375 removed run-pre-commit-tests.sh in Flink 1.12 [1]. Since then the following tests are not executed anymore: test_state_migration.sh test_state_evolution.sh test_streaming_kinesis.sh test_streaming_classloader.sh test_streaming_distributed_cache_via_blob.sh [1]https://github.com/apache/flink/pull/12268/files#diff-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e was: FLINK-17375 [removed|#diff-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e] run-pre-commit-tests.sh in Flink 1.12. Since then the following tests are not executed anymore: test_state_migration.sh test_state_evolution.sh test_streaming_kinesis.sh test_streaming_classloader.sh test_streaming_distributed_cache_via_blob.sh > Remove unused e2e tests > --- > > Key: FLINK-5 > URL: https://issues.apache.org/jira/browse/FLINK-5 > Project: Flink > Issue Type: Improvement >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > FLINK-17375 removed run-pre-commit-tests.sh in Flink 1.12 [1]. Since then the > following tests are not executed anymore: > test_state_migration.sh > test_state_evolution.sh > test_streaming_kinesis.sh > test_streaming_classloader.sh > test_streaming_distributed_cache_via_blob.sh > > [1]https://github.com/apache/flink/pull/12268/files#diff-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33335) Remove unused e2e tests
[ https://issues.apache.org/jira/browse/FLINK-5?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-5: -- Description: FLINK-17375 removed _run-pre-commit-tests.sh_ in Flink 1.12 [1]. Since then the following tests are not executed anymore: _test_state_migration.sh_ _test_state_evolution.sh_ _test_streaming_kinesis.sh_ _test_streaming_classloader.sh_ _test_streaming_distributed_cache_via_blob.sh_ [1]https://github.com/apache/flink/pull/12268/files#diff-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e was: FLINK-17375 removed run-pre-commit-tests.sh in Flink 1.12 [1]. Since then the following tests are not executed anymore: test_state_migration.sh test_state_evolution.sh test_streaming_kinesis.sh test_streaming_classloader.sh test_streaming_distributed_cache_via_blob.sh [1]https://github.com/apache/flink/pull/12268/files#diff-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e > Remove unused e2e tests > --- > > Key: FLINK-5 > URL: https://issues.apache.org/jira/browse/FLINK-5 > Project: Flink > Issue Type: Improvement >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > FLINK-17375 removed _run-pre-commit-tests.sh_ in Flink 1.12 [1]. Since then > the following tests are not executed anymore: > _test_state_migration.sh_ > _test_state_evolution.sh_ > _test_streaming_kinesis.sh_ > _test_streaming_classloader.sh_ > _test_streaming_distributed_cache_via_blob.sh_ > [1]https://github.com/apache/flink/pull/12268/files#diff-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33335) Remove unused e2e tests
[ https://issues.apache.org/jira/browse/FLINK-5?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-5: -- Description: FLINK-17375 [removed|#diff-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e]] run-pre-commit-tests.sh in Flink 1.12. Since then the following tests are not executed anymore: test_state_migration.sh test_state_evolution.sh test_streaming_kinesis.sh test_streaming_classloader.sh test_streaming_distributed_cache_via_blob.sh was: FLINK-17375 [removed|[https://github.com/apache/flink/pull/12268/files#diff-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e]] run-pre-commit-tests.sh in Flink 1.12. Since then the following tests are not executed anymore: test_state_migration.sh test_state_evolution.sh test_streaming_kinesis.sh test_streaming_classloader.sh test_streaming_distributed_cache_via_blob.sh > Remove unused e2e tests > --- > > Key: FLINK-5 > URL: https://issues.apache.org/jira/browse/FLINK-5 > Project: Flink > Issue Type: Improvement >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > FLINK-17375 > [removed|#diff-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e]] > run-pre-commit-tests.sh in Flink 1.12. Since then the following tests are > not executed anymore: > test_state_migration.sh > test_state_evolution.sh > test_streaming_kinesis.sh > test_streaming_classloader.sh > test_streaming_distributed_cache_via_blob.sh -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33335) Remove unused e2e tests
[ https://issues.apache.org/jira/browse/FLINK-5?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-5: -- Description: FLINK-17375 [removed|#diff-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e] run-pre-commit-tests.sh in Flink 1.12. Since then the following tests are not executed anymore: test_state_migration.sh test_state_evolution.sh test_streaming_kinesis.sh test_streaming_classloader.sh test_streaming_distributed_cache_via_blob.sh was: FLINK-17375 [removed|#diff-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e]] run-pre-commit-tests.sh in Flink 1.12. Since then the following tests are not executed anymore: test_state_migration.sh test_state_evolution.sh test_streaming_kinesis.sh test_streaming_classloader.sh test_streaming_distributed_cache_via_blob.sh > Remove unused e2e tests > --- > > Key: FLINK-5 > URL: https://issues.apache.org/jira/browse/FLINK-5 > Project: Flink > Issue Type: Improvement >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > FLINK-17375 > [removed|#diff-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e] > run-pre-commit-tests.sh in Flink 1.12. Since then the following tests are > not executed anymore: > test_state_migration.sh > test_state_evolution.sh > test_streaming_kinesis.sh > test_streaming_classloader.sh > test_streaming_distributed_cache_via_blob.sh -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-33335) Remove unused e2e tests
[ https://issues.apache.org/jira/browse/FLINK-5?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov reassigned FLINK-5: - Assignee: Alexander Fedulov > Remove unused e2e tests > --- > > Key: FLINK-5 > URL: https://issues.apache.org/jira/browse/FLINK-5 > Project: Flink > Issue Type: Improvement >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > FLINK\-17375 removed run\-pre\-commit\-tests.sh in Flink 1.12. Since then the > following tests are not executed anymore: > test\_state\_migration.sh > test\_state\_evolution.sh > test\_streaming\_kinesis.sh > test\_streaming\_classloader.sh > test\_streaming\_distributed\_cache\_via\_blob.sh > [https://github.com/apache/flink/pull/12268/files#diff\-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e|https://github.com/apache/flink/pull/12268/files#diff-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33335) Remove unused e2e tests
[ https://issues.apache.org/jira/browse/FLINK-5?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778126#comment-17778126 ] Alexander Fedulov commented on FLINK-5: --- [~rmetzger] [~chesnay] Could you please confirm that disabling of the aforementioned tests was intentional? I would like to understand if we can simply drop `org.apache.flink.test.StatefulStreamingJob` without moving it to a FLIP-27 source. > Remove unused e2e tests > --- > > Key: FLINK-5 > URL: https://issues.apache.org/jira/browse/FLINK-5 > Project: Flink > Issue Type: Improvement >Reporter: Alexander Fedulov >Priority: Major > > FLINK\-17375 removed run\-pre\-commit\-tests.sh in Flink 1.12. Since then the > following tests are not executed anymore: > test\_state\_migration.sh > test\_state\_evolution.sh > test\_streaming\_kinesis.sh > test\_streaming\_classloader.sh > test\_streaming\_distributed\_cache\_via\_blob.sh > [https://github.com/apache/flink/pull/12268/files#diff\-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e|https://github.com/apache/flink/pull/12268/files#diff-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33335) Remove unused e2e tests
[ https://issues.apache.org/jira/browse/FLINK-5?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-5: -- Description: FLINK-17375 [removed|[https://github.com/apache/flink/pull/12268/files#diff-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e]] run-pre-commit-tests.sh in Flink 1.12. Since then the following tests are not executed anymore: test_state_migration.sh test_state_evolution.sh test_streaming_kinesis.sh test_streaming_classloader.sh test_streaming_distributed_cache_via_blob.sh was: FLINK\-17375 removed run\-pre\-commit\-tests.sh in Flink 1.12. Since then the following tests are not executed anymore: test\_state\_migration.sh test\_state\_evolution.sh test\_streaming\_kinesis.sh test\_streaming\_classloader.sh test\_streaming\_distributed\_cache\_via\_blob.sh [https://github.com/apache/flink/pull/12268/files#diff\-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e|https://github.com/apache/flink/pull/12268/files#diff-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e] > Remove unused e2e tests > --- > > Key: FLINK-5 > URL: https://issues.apache.org/jira/browse/FLINK-5 > Project: Flink > Issue Type: Improvement >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > FLINK-17375 > [removed|[https://github.com/apache/flink/pull/12268/files#diff-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e]] > run-pre-commit-tests.sh in Flink 1.12. Since then the following tests are > not executed anymore: > test_state_migration.sh > test_state_evolution.sh > test_streaming_kinesis.sh > test_streaming_classloader.sh > test_streaming_distributed_cache_via_blob.sh -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33335) Remove unused e2e tests
Alexander Fedulov created FLINK-5: - Summary: Remove unused e2e tests Key: FLINK-5 URL: https://issues.apache.org/jira/browse/FLINK-5 Project: Flink Issue Type: Improvement Reporter: Alexander Fedulov FLINK\-17375 removed run\-pre\-commit\-tests.sh in Flink 1.12. Since then the following tests are not executed anymore: test\_state\_migration.sh test\_state\_evolution.sh test\_streaming\_kinesis.sh test\_streaming\_classloader.sh test\_streaming\_distributed\_cache\_via\_blob.sh [https://github.com/apache/flink/pull/12268/files#diff\-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e|https://github.com/apache/flink/pull/12268/files#diff-39f0aea40d2dd3f026544bb4c2502b2e9eab4c825df5f2b68c6d4ca8c39d7b5e] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32694) Migrate classes that implement ParallelSourceFunction to Source V2 API
[ https://issues.apache.org/jira/browse/FLINK-32694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-32694: -- Description: * SimpleEndlessSourceWithBloatedState in HeavyDeploymentStressTestProgram (org.apache.flink.deployment) * StatefulSequenceSource (org.apache.flink.streaming.api.functions.source) * ArrowSourceFunction (org.apache.flink.table.runtime.arrow.sources) * InputFormatSourceFunction (org.apache.flink.streaming.api.functions.source) * -EventsGeneratorSource (org.apache.flink.streaming.examples.statemachine.generator)- resolved in FLINK-32670 * FromSplittableIteratorFunction (org.apache.flink.streaming.api.functions.source) * DataGeneratorSource (org.apache.flink.streaming.api.functions.source.datagen) * – Tests: * MySource in StatefulStreamingJob (org.apache.flink.test) * RandomLongSource in StickyAllocationAndLocalRecoveryTestJob (org.apache.flink.streaming.tests) * SequenceGeneratorSource (org.apache.flink.streaming.tests) * TtlStateUpdateSource (org.apache.flink.streaming.tests) * StringSourceFunction in NettyShuffleMemoryControlTestProgram (org.apache.flink.streaming.tests) was: * SimpleEndlessSourceWithBloatedState in HeavyDeploymentStressTestProgram (org.apache.flink.deployment) * StatefulSequenceSource (org.apache.flink.streaming.api.functions.source) * ArrowSourceFunction (org.apache.flink.table.runtime.arrow.sources) * InputFormatSourceFunction (org.apache.flink.streaming.api.functions.source) * EventsGeneratorSource (org.apache.flink.streaming.examples.statemachine.generator) * FromSplittableIteratorFunction (org.apache.flink.streaming.api.functions.source) * DataGeneratorSource (org.apache.flink.streaming.api.functions.source.datagen) * -- Tests: * MySource in StatefulStreamingJob (org.apache.flink.test) * RandomLongSource in StickyAllocationAndLocalRecoveryTestJob (org.apache.flink.streaming.tests) * SequenceGeneratorSource (org.apache.flink.streaming.tests) * TtlStateUpdateSource (org.apache.flink.streaming.tests) * StringSourceFunction in NettyShuffleMemoryControlTestProgram (org.apache.flink.streaming.tests) > Migrate classes that implement ParallelSourceFunction to Source V2 API > -- > > Key: FLINK-32694 > URL: https://issues.apache.org/jira/browse/FLINK-32694 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > > * SimpleEndlessSourceWithBloatedState in HeavyDeploymentStressTestProgram > (org.apache.flink.deployment) > * StatefulSequenceSource (org.apache.flink.streaming.api.functions.source) > * ArrowSourceFunction (org.apache.flink.table.runtime.arrow.sources) > * InputFormatSourceFunction (org.apache.flink.streaming.api.functions.source) > * -EventsGeneratorSource > (org.apache.flink.streaming.examples.statemachine.generator)- resolved in > FLINK-32670 > * FromSplittableIteratorFunction > (org.apache.flink.streaming.api.functions.source) > * DataGeneratorSource > (org.apache.flink.streaming.api.functions.source.datagen) > * – Tests: > * MySource in StatefulStreamingJob (org.apache.flink.test) > * RandomLongSource in StickyAllocationAndLocalRecoveryTestJob > (org.apache.flink.streaming.tests) > * SequenceGeneratorSource (org.apache.flink.streaming.tests) > * TtlStateUpdateSource (org.apache.flink.streaming.tests) > * StringSourceFunction in NettyShuffleMemoryControlTestProgram > (org.apache.flink.streaming.tests) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28045) [umbrella] Blockers for SourceFunction API removal (in Flink 2.0)
[
https://issues.apache.org/jira/browse/FLINK-28045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Fedulov updated FLINK-28045:
--
Description:
This ticket should only contain items where SourceFunction is already used in
the existing code base. For new functionality and Source V2 API improvements,
use FLINK-32692.
—
Current state of migration to Source V2 API for officially-supported connectors
(/) FileSystem (Public)
(/) Kafka (PublicEvolving)
(/) Cassandra (PublicEvolving)
(/) Pulsar (PublicEvolving)
(/) HiveSource (PublicEvolving)
(/) HybridSource (PublicEvolving)
(/) MongoDB (PublicEvolving)
(/) Hive (PublicEvolving)
(?) Kinesis (Experimental)
(flagoff) JDBC - not impacted (based on {{{}TableFunction{}}})
(flagoff) HBase - not impacted (based on {{{}TableFunction{}}})
(x) RabbitMQ
(x) Google Pub Sub
Links to externalized connector repos:
https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development
was:
This ticket should only contain items where SourceFunction is already used in
the existing code base. For new functionality and Source V2 API improvements,
use FLINK-32692.
—
Current state of migration to Source V2 API for officially-supported connectors
(/) FileSystem (Public)
(/) Kafka (PublicEvolving)
(/) Cassandra (PublicEvolving)
(/) Pulsar (PublicEvolving)
(/) HiveSource (PublicEvolving)
(/) HybridSource (PublicEvolving)
(/) MongoDB (PublicEvolving)
(/) Hive (PublicEvolving)
(?) Kinesis (Experimental)
(flagoff) JDBC - not impacted (based on {{{}TableFunction{}}})
(flagoff) HBase - not impacted (based on {{{}TableFunction{}}})
(x) RabbitMQ
(x) Google Pub Sub
> [umbrella] Blockers for SourceFunction API removal (in Flink 2.0)
> -
>
> Key: FLINK-28045
> URL: https://issues.apache.org/jira/browse/FLINK-28045
> Project: Flink
> Issue Type: Technical Debt
> Components: Connectors / Common
>Reporter: Alexander Fedulov
>Assignee: Alexander Fedulov
>Priority: Major
>
> This ticket should only contain items where SourceFunction is already used in
> the existing code base. For new functionality and Source V2 API improvements,
> use FLINK-32692.
> —
> Current state of migration to Source V2 API for officially-supported
> connectors
> (/) FileSystem (Public)
> (/) Kafka (PublicEvolving)
> (/) Cassandra (PublicEvolving)
> (/) Pulsar (PublicEvolving)
> (/) HiveSource (PublicEvolving)
> (/) HybridSource (PublicEvolving)
> (/) MongoDB (PublicEvolving)
> (/) Hive (PublicEvolving)
> (?) Kinesis (Experimental)
> (flagoff) JDBC - not impacted (based on {{{}TableFunction{}}})
> (flagoff) HBase - not impacted (based on {{{}TableFunction{}}})
> (x) RabbitMQ
> (x) Google Pub Sub
>
> Links to externalized connector repos:
> https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-33326) Remove usage of FromElementsFunction
Alexander Fedulov created FLINK-33326: - Summary: Remove usage of FromElementsFunction Key: FLINK-33326 URL: https://issues.apache.org/jira/browse/FLINK-33326 Project: Flink Issue Type: Sub-task Components: API / Core Reporter: Alexander Fedulov Assignee: Alexander Fedulov -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33286) DataGeneratorSource should support automatic return type detection
[ https://issues.apache.org/jira/browse/FLINK-33286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-33286: -- Description: Currently, DataGeneratorSource requires both GeneratorFunction and TypeInformation to be passed during its construction. Given that the generator function has a fixed API, it should be possible to reliably extract the OUT type automatically for both generator functions passed both as lambda and as objects. was: Currently, DataGeneratorSource requires both GeneratorFunction and TypeInformation to be passed during its construction. Given that the generator function has a fixed API, it should be possible to reliably extract the OUT type automatically for both lambda generator functions and for objects. > DataGeneratorSource should support automatic return type detection > -- > > Key: FLINK-33286 > URL: https://issues.apache.org/jira/browse/FLINK-33286 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.17.1 >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > Currently, DataGeneratorSource requires both GeneratorFunction and > TypeInformation to be passed during its construction. Given that the > generator function has a fixed API, it should be possible to reliably extract > the OUT type automatically for both generator functions passed both as lambda > and as objects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-33286) DataGeneratorSource should support automatic return type detection
[ https://issues.apache.org/jira/browse/FLINK-33286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov reassigned FLINK-33286: - Assignee: Alexander Fedulov > DataGeneratorSource should support automatic return type detection > -- > > Key: FLINK-33286 > URL: https://issues.apache.org/jira/browse/FLINK-33286 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.17.1 >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > Currently, DataGeneratorSource requires both GeneratorFunction and > TypeInformation to be passed during its construction. Given that the > generator function has a fixed API, it should be possible to reliably extract > the OUT type automatically for both lambda generator functions and for > objects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33286) DataGeneratorSource should support automatic return type detection
[ https://issues.apache.org/jira/browse/FLINK-33286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17775923#comment-17775923 ] Alexander Fedulov commented on FLINK-33286: --- Connectors / DataGen component is missing. If you read this comment and have the required permissions, please create. > DataGeneratorSource should support automatic return type detection > -- > > Key: FLINK-33286 > URL: https://issues.apache.org/jira/browse/FLINK-33286 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.17.1 >Reporter: Alexander Fedulov >Priority: Major > > Currently, DataGeneratorSource requires both GeneratorFunction and > TypeInformation to be passed during its construction. Given that the > generator function has a fixed API, it should be possible to reliably extract > the OUT type automatically for both lambda generator functions and for > objects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33286) DataGeneratorSource should support automatic return type detection
Alexander Fedulov created FLINK-33286: - Summary: DataGeneratorSource should support automatic return type detection Key: FLINK-33286 URL: https://issues.apache.org/jira/browse/FLINK-33286 Project: Flink Issue Type: Improvement Affects Versions: 1.17.1 Reporter: Alexander Fedulov Currently, DataGeneratorSource requires both GeneratorFunction and TypeInformation to be passed during its construction. Given that the generator function has a fixed API, it should be possible to reliably extract the OUT type automatically for both lambda generator functions and for objects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-33218) First Steps - error when run with zsh
[
https://issues.apache.org/jira/browse/FLINK-33218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Fedulov reassigned FLINK-33218:
-
Assignee: Robin Moffatt
> First Steps - error when run with zsh
> -
>
> Key: FLINK-33218
> URL: https://issues.apache.org/jira/browse/FLINK-33218
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Reporter: Robin Moffatt
>Assignee: Robin Moffatt
>Priority: Not a Priority
> Labels: pull-request-available
>
> If a user of zsh (the default on MacOS) runs the literal command that's given
> under "Browsing the project directory" they get an error:
> {code:java}
> $ cd flink-* && ls -l
> cd: string not in pwd: flink-1.17.1
> {code}
>
> This is because the behaviour of `cd` is different under zsh than bash and
> the glob triggers this. I've written up [an
> explanation|https://rmoff.net/2023/10/04/cd-string-not-in-pwd/] for those
> interested.
> IMO the fix is to hardcode the version in the instructions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31278) exit code 137 (i.e. OutOfMemoryError) in core module
[
https://issues.apache.org/jira/browse/FLINK-31278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17772852#comment-17772852
]
Alexander Fedulov commented on FLINK-31278:
---
https://dev.azure.com/alexanderfedulov/Flink/_build/results?buildId=557=logs=7f652c99-c3cd-5aee-11e2-f8e88140dbea=1fe0f51b-b0ac-5dfd-7645-a1ae7486f7da=8363
> exit code 137 (i.e. OutOfMemoryError) in core module
>
>
> Key: FLINK-31278
> URL: https://issues.apache.org/jira/browse/FLINK-31278
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.17.0, 1.19.0
>Reporter: Matthias Pohl
>Priority: Major
> Labels: auto-deprioritized-critical, pull-request-available,
> test-stability
>
> The following build failed due to a 137 exit code indicating an
> OutOfMemoryError:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46643=logs=77a9d8e1-d610-59b3-fc2a-4766541e0e33=125e07e7-8de0-5c6c-a541-a567415af3ef=7847
> {code}
> [...]
> Mar 01 05:29:06 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time
> elapsed: 0.65 s - in
> org.apache.flink.runtime.io.compression.BlockCompressionTest
> Mar 01 05:29:06 [INFO] Running
> org.apache.flink.runtime.dispatcher.DispatcherCachedOperationsHandlerTest
> Mar 01 05:29:07 [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time
> elapsed: 1.142 s - in
> org.apache.flink.runtime.dispatcher.DispatcherCachedOperationsHandlerTest
> Mar 01 05:29:08 [INFO] Running
> org.apache.flink.runtime.dispatcher.MemoryExecutionGraphInfoStoreTest
> ##[error]Exit code 137 returned from process: file name '/usr/bin/docker',
> arguments 'exec -i -u 1001 -w /home/vsts_azpcontainer
> 5953b171e8ed4caba7af2b326533e249211ed4dcc48640edb3c1b0cbbcdf1a21
> /__a/externals/node/bin/node /__w/_temp/containerHandlerInvoker.js'.
> Finishing: Test - core
> {code}
> This build ran on an Azure pipeline machine (Azure Pipelines 9) and,
> therefore, cannot be caused by FLINK-18356. That said, there was a concurrent
> 137 exit code build failure happening on agent "Azure Pipelines 21" (see
> [20230301.3|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46643=logs=77a9d8e1-d610-59b3-fc2a-4766541e0e33=125e07e7-8de0-5c6c-a541-a567415af3ef=7847])
> ~10mins later
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31278) exit code 137 (i.e. OutOfMemoryError) in core module
[
https://issues.apache.org/jira/browse/FLINK-31278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17772849#comment-17772849
]
Alexander Fedulov commented on FLINK-31278:
---
https://dev.azure.com/alexanderfedulov/Flink/_build/results?buildId=557=logs=9dc1b5dc-bcfa-5f83-eaa7-0cb181ddc267=511d2595-ec54-5ab7-86ce-92f328796f20=8376
> exit code 137 (i.e. OutOfMemoryError) in core module
>
>
> Key: FLINK-31278
> URL: https://issues.apache.org/jira/browse/FLINK-31278
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.17.0, 1.19.0
>Reporter: Matthias Pohl
>Priority: Major
> Labels: auto-deprioritized-critical, pull-request-available,
> test-stability
>
> The following build failed due to a 137 exit code indicating an
> OutOfMemoryError:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46643=logs=77a9d8e1-d610-59b3-fc2a-4766541e0e33=125e07e7-8de0-5c6c-a541-a567415af3ef=7847
> {code}
> [...]
> Mar 01 05:29:06 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time
> elapsed: 0.65 s - in
> org.apache.flink.runtime.io.compression.BlockCompressionTest
> Mar 01 05:29:06 [INFO] Running
> org.apache.flink.runtime.dispatcher.DispatcherCachedOperationsHandlerTest
> Mar 01 05:29:07 [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time
> elapsed: 1.142 s - in
> org.apache.flink.runtime.dispatcher.DispatcherCachedOperationsHandlerTest
> Mar 01 05:29:08 [INFO] Running
> org.apache.flink.runtime.dispatcher.MemoryExecutionGraphInfoStoreTest
> ##[error]Exit code 137 returned from process: file name '/usr/bin/docker',
> arguments 'exec -i -u 1001 -w /home/vsts_azpcontainer
> 5953b171e8ed4caba7af2b326533e249211ed4dcc48640edb3c1b0cbbcdf1a21
> /__a/externals/node/bin/node /__w/_temp/containerHandlerInvoker.js'.
> Finishing: Test - core
> {code}
> This build ran on an Azure pipeline machine (Azure Pipelines 9) and,
> therefore, cannot be caused by FLINK-18356. That said, there was a concurrent
> 137 exit code build failure happening on agent "Azure Pipelines 21" (see
> [20230301.3|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46643=logs=77a9d8e1-d610-59b3-fc2a-4766541e0e33=125e07e7-8de0-5c6c-a541-a567415af3ef=7847])
> ~10mins later
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33199) ArchitectureTests should test for fully qualified class names instead of Class objects
[ https://issues.apache.org/jira/browse/FLINK-33199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-33199: -- Summary: ArchitectureTests should test for fully qualified class names instead of Class objects (was: ArchitectureTests should test for canonical class names instead of Class objects) > ArchitectureTests should test for fully qualified class names instead of > Class objects > -- > > Key: FLINK-33199 > URL: https://issues.apache.org/jira/browse/FLINK-33199 > Project: Flink > Issue Type: Improvement > Components: Tests >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > Currently architecture tests rely on importing such classes as > MiniClusterExtension. This introduces a production scope dependency on > flink-test-utils which in turn depends on flink-streaming-java. This is > problematic because adding architecture tests to any direct or transitive > dependency of flink-streaming-java creates a dependency cycle. > Example: https://github.com/apache/flink/pull/22850#discussion_r1243343382 > In general, since architecture tests are supposed to be used freely in any > submodule, it is desirable to reduce its dependency surface as much as > possible to prevent such cycles. > This can be achieved by moving away from using Class objects and employing > fully qualified type names checks instead. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-33199) ArchitectureTests should test for canonical class names instead of Class objects
[ https://issues.apache.org/jira/browse/FLINK-33199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov reassigned FLINK-33199: - Assignee: Alexander Fedulov > ArchitectureTests should test for canonical class names instead of Class > objects > > > Key: FLINK-33199 > URL: https://issues.apache.org/jira/browse/FLINK-33199 > Project: Flink > Issue Type: Improvement > Components: Tests >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > Currently architecture tests rely on importing such classes as > MiniClusterExtension. This introduces a production scope dependency on > flink-test-utils which in turn depends on flink-streaming-java. This is > problematic because adding architecture tests to any direct or transitive > dependency of flink-streaming-java creates a dependency cycle. > Example: https://github.com/apache/flink/pull/22850#discussion_r1243343382 > In general, since architecture tests are supposed to be used freely in any > submodule, it is desirable to reduce its dependency surface as much as > possible to prevent such cycles. > This can be achieved by moving away from using Class objects and employing > fully qualified type names checks instead. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33199) ArchitectureTests should test for canonical class names instead of Class objects
Alexander Fedulov created FLINK-33199: - Summary: ArchitectureTests should test for canonical class names instead of Class objects Key: FLINK-33199 URL: https://issues.apache.org/jira/browse/FLINK-33199 Project: Flink Issue Type: Improvement Components: Tests Reporter: Alexander Fedulov Currently architecture tests rely on importing such classes as MiniClusterExtension. This introduces a production scope dependency on flink-test-utils which in turn depends on flink-streaming-java. This is problematic because adding architecture tests to any direct or transitive dependency of flink-streaming-java creates a dependency cycle. Example: https://github.com/apache/flink/pull/22850#discussion_r1243343382 In general, since architecture tests are supposed to be used freely in any submodule, it is desirable to reduce its dependency surface as much as possible to prevent such cycles. This can be achieved by moving away from using Class objects and employing fully qualified type names checks instead. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32904) Support cron expressions for periodic snapshots triggering
[ https://issues.apache.org/jira/browse/FLINK-32904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-32904: -- Description: Current functionality to trigger savepoints and checkpoints is based on specifying a fixed interval. This approach gives little control to prevent potentially overlapping snapshots running simultaneously in adjacent deployments and can cause issues for high scale, large state jobs. In order to have better control over the exact schedule, triggering based on cron expressions should be added. (was: Similar to the support for periodic savepoints, the operator should support triggering periodic checkpoints to break the incremental checkpoint chain. Support for external triggering will come with 1.17: https://issues.apache.org/jira/browse/FLINK-27101 ) > Support cron expressions for periodic snapshots triggering > -- > > Key: FLINK-32904 > URL: https://issues.apache.org/jira/browse/FLINK-32904 > Project: Flink > Issue Type: New Feature > Components: Kubernetes Operator >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > Labels: pull-request-available > Fix For: kubernetes-operator-1.7.0 > > > Current functionality to trigger savepoints and checkpoints is based on > specifying a fixed interval. This approach gives little control to prevent > potentially overlapping snapshots running simultaneously in adjacent > deployments and can cause issues for high scale, large state jobs. In order > to have better control over the exact schedule, triggering based on cron > expressions should be added. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-32904) Support cron expressions for periodic snapshots triggering
Alexander Fedulov created FLINK-32904: - Summary: Support cron expressions for periodic snapshots triggering Key: FLINK-32904 URL: https://issues.apache.org/jira/browse/FLINK-32904 Project: Flink Issue Type: New Feature Components: Kubernetes Operator Reporter: Alexander Fedulov Assignee: Alexander Fedulov Fix For: kubernetes-operator-1.7.0 Similar to the support for periodic savepoints, the operator should support triggering periodic checkpoints to break the incremental checkpoint chain. Support for external triggering will come with 1.17: https://issues.apache.org/jira/browse/FLINK-27101 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-32821) Streaming examples failed to execute due to error in packaging
[ https://issues.apache.org/jira/browse/FLINK-32821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17756913#comment-17756913 ] Alexander Fedulov commented on FLINK-32821: --- [~huweihua] Oh, I see, good catch! Ideally we should have some tests to get this covered. > Streaming examples failed to execute due to error in packaging > -- > > Key: FLINK-32821 > URL: https://issues.apache.org/jira/browse/FLINK-32821 > Project: Flink > Issue Type: Bug > Components: Examples >Affects Versions: 1.18.0 >Reporter: Zhanghao Chen >Assignee: Zhanghao Chen >Priority: Major > Labels: pull-request-available > > 7 out of the 8 streaming examples failed to run: > * Iteration & SessionWindowing & SocketWindowWordCount & WindowJoin failed > to run due to java.lang.NoClassDefFoundError: > org/apache/flink/streaming/examples/utils/ParameterTool > * MatrixVectorMul & TopSpeedWindowing & StateMachineExample failed to run > due to: Caused by: java.lang.ClassNotFoundException: > org.apache.flink.connector.datagen.source.GeneratorFunction > The NoClassDefFoundError with ParameterTool is introduced by FLINK-32558 > Properly deprecate DataSet API - ASF JIRA (apache.org), and we'd better > resolve FLINK-32820 ParameterTool is mistakenly marked as deprecated - ASF > JIRA (apache.org) first before we come to a fix for this problem. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-32821) Streaming examples failed to execute due to error in packaging
[ https://issues.apache.org/jira/browse/FLINK-32821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755534#comment-17755534 ] Alexander Fedulov commented on FLINK-32821: --- [~Zhanghao Chen] I am still not sure why you faced this issue. Bundling datagen was part of the original PR: [https://github.com/apache/flink/pull/23079/files#diff-c7da2a9d0258717dbd97328195d580aedc3fe51240aeb4fc0f292d946dcdb4af] Maybe there was some unlucky timing because Leonard might have merged the commits separately and maybe you pulled right before this one got integrated [https://github.com/apache/flink/commit/dfb9cb851dc1f0908ea6c3ce1230dd8ca2b48733] Anyhow, I believe we can close this issue. > Streaming examples failed to execute due to error in packaging > -- > > Key: FLINK-32821 > URL: https://issues.apache.org/jira/browse/FLINK-32821 > Project: Flink > Issue Type: Bug > Components: Examples >Affects Versions: 1.18.0 >Reporter: Zhanghao Chen >Assignee: Zhanghao Chen >Priority: Major > Labels: pull-request-available > > 7 out of the 8 streaming examples failed to run: > * Iteration & SessionWindowing & SocketWindowWordCount & WindowJoin failed > to run due to java.lang.NoClassDefFoundError: > org/apache/flink/streaming/examples/utils/ParameterTool > * MatrixVectorMul & TopSpeedWindowing & StateMachineExample failed to run > due to: Caused by: java.lang.ClassNotFoundException: > org.apache.flink.connector.datagen.source.GeneratorFunction > The NoClassDefFoundError with ParameterTool is introduced by FLINK-32558 > Properly deprecate DataSet API - ASF JIRA (apache.org), and we'd better > resolve FLINK-32820 ParameterTool is mistakenly marked as deprecated - ASF > JIRA (apache.org) first before we come to a fix for this problem. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-32821) Streaming examples failed to execute due to error in packaging
[ https://issues.apache.org/jira/browse/FLINK-32821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752918#comment-17752918 ] Alexander Fedulov commented on FLINK-32821: --- [~Zhanghao Chen] I cannot reproduce the listed issues. How do you execute the examples? Do you face the issue also when running in IDE? > Streaming examples failed to execute due to error in packaging > -- > > Key: FLINK-32821 > URL: https://issues.apache.org/jira/browse/FLINK-32821 > Project: Flink > Issue Type: Bug > Components: Examples >Affects Versions: 1.18.0 >Reporter: Zhanghao Chen >Assignee: Zhanghao Chen >Priority: Major > > 5 out of the 7 streaming examples failed to run: > * Iteration & SessionWindowing & SocketWindowWordCount & WindowJoin failed > to run due to java.lang.NoClassDefFoundError: > org/apache/flink/streaming/examples/utils/ParameterTool > * TopSpeedWindowing failed to run due to: Caused by: > java.lang.ClassNotFoundException: > org.apache.flink.connector.datagen.source.GeneratorFunction > The NoClassDefFoundError with ParameterTool is introduced by [FLINK-32558] > Properly deprecate DataSet API - ASF JIRA (apache.org), and we'd better > resolve [FLINK-32820] ParameterTool is mistakenly marked as deprecated - ASF > JIRA (apache.org) first before we come to a fix for this problem. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-32195) Add SQL Gateway custom headers support
[ https://issues.apache.org/jira/browse/FLINK-32195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov closed FLINK-32195. - Release Note: Duplicate of FLINK-32373 Resolution: Duplicate > Add SQL Gateway custom headers support > -- > > Key: FLINK-32195 > URL: https://issues.apache.org/jira/browse/FLINK-32195 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Gateway >Affects Versions: 1.17.0 >Reporter: Elkhan Dadashov >Priority: Major > Labels: features > > For some use cases, it might be needed setting a few extra HTTP headers with > a request to FlinkSQL Gateway, for example, a cookie for Auth/session. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32693) Cascade deprecation to classes that directly implement SourceFunction
[ https://issues.apache.org/jira/browse/FLINK-32693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-32693: -- Parent: (was: FLINK-28045) Issue Type: Technical Debt (was: Sub-task) > Cascade deprecation to classes that directly implement SourceFunction > - > > Key: FLINK-32693 > URL: https://issues.apache.org/jira/browse/FLINK-32693 > Project: Flink > Issue Type: Technical Debt > Components: Connectors / Common >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > * SocketTextStreamFunction (org.apache.flink.streaming.api.functions.source) > * FiniteTestSource (org.apache.flink.streaming.util) > * FromIteratorFunction (org.apache.flink.streaming.api.functions.source) > * Generator in FileSinkProgram (org.apache.flink.connector.file.sink) > * RichSourceFunction (org.apache.flink.streaming.api.functions.source) > * FileMonitoringFunction (org.apache.flink.streaming.api.functions.source) > * FromElementsFunction (org.apache.flink.streaming.api.functions.source) > * — Tests > * PeriodicSourceGenerator in PeriodicStreamingJob > (org.apache.flink.streaming.tests) > * Generator in StreamSQLTestProgram (org.apache.flink.sql.tests) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-32693) Cascade deprecation to classes that directly implement SourceFunction
[ https://issues.apache.org/jira/browse/FLINK-32693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov closed FLINK-32693. - Release Note: Subsumed by FLINK-32670 Resolution: Duplicate > Cascade deprecation to classes that directly implement SourceFunction > - > > Key: FLINK-32693 > URL: https://issues.apache.org/jira/browse/FLINK-32693 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > * SocketTextStreamFunction (org.apache.flink.streaming.api.functions.source) > * FiniteTestSource (org.apache.flink.streaming.util) > * FromIteratorFunction (org.apache.flink.streaming.api.functions.source) > * Generator in FileSinkProgram (org.apache.flink.connector.file.sink) > * RichSourceFunction (org.apache.flink.streaming.api.functions.source) > * FileMonitoringFunction (org.apache.flink.streaming.api.functions.source) > * FromElementsFunction (org.apache.flink.streaming.api.functions.source) > * — Tests > * PeriodicSourceGenerator in PeriodicStreamingJob > (org.apache.flink.streaming.tests) > * Generator in StreamSQLTestProgram (org.apache.flink.sql.tests) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-32719) Block upgrades during checkpoints when LATEST_STATE recovery is configured
Alexander Fedulov created FLINK-32719: - Summary: Block upgrades during checkpoints when LATEST_STATE recovery is configured Key: FLINK-32719 URL: https://issues.apache.org/jira/browse/FLINK-32719 Project: Flink Issue Type: Bug Components: Deployment / Kubernetes Reporter: Alexander Fedulov Savepoints are allowed to finish if LATEST_STATE recovery is configured. The same optimization should be added with regards to checkpoints (introduced in FLINK-29634) Can be tested [similar to savepoints.|https://github.com/apache/flink-kubernetes-operator/blob/8d307086122bdf4f74aaa9a69b5b8cc9025afdc8/flink-kubernetes-operator/src/test/java/org/apache/flink/kubernetes/operator/reconciler/sessionjob/SessionJobReconcilerTest.java#L388] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-32713) Cascade deprecation to non-private methods that reference SourceFunction
[ https://issues.apache.org/jira/browse/FLINK-32713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov reassigned FLINK-32713: - Assignee: Alexander Fedulov > Cascade deprecation to non-private methods that reference SourceFunction > > > Key: FLINK-32713 > URL: https://issues.apache.org/jira/browse/FLINK-32713 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-32713) Cascade deprecation to non-private methods that reference SourceFunction
Alexander Fedulov created FLINK-32713: - Summary: Cascade deprecation to non-private methods that reference SourceFunction Key: FLINK-32713 URL: https://issues.apache.org/jira/browse/FLINK-32713 Project: Flink Issue Type: Sub-task Components: Connectors / Common Reporter: Alexander Fedulov -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32670) Cascade deprecation to classes that implement SourceFunction
[ https://issues.apache.org/jira/browse/FLINK-32670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-32670: -- Description: (was: ParallelSourceFunction, RichParallelSourceFunction, ExternallyInducedSource) > Cascade deprecation to classes that implement SourceFunction > > > Key: FLINK-32670 > URL: https://issues.apache.org/jira/browse/FLINK-32670 > Project: Flink > Issue Type: Sub-task >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32670) Cascade deprecation to classes that implement SourceFunction
[ https://issues.apache.org/jira/browse/FLINK-32670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-32670: -- Summary: Cascade deprecation to classes that implement SourceFunction (was: Annotate interfaces that inherit from SourceFunction as deprecated ) > Cascade deprecation to classes that implement SourceFunction > > > Key: FLINK-32670 > URL: https://issues.apache.org/jira/browse/FLINK-32670 > Project: Flink > Issue Type: Sub-task >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > Labels: pull-request-available > > ParallelSourceFunction, RichParallelSourceFunction, ExternallyInducedSource -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32694) Migrate classes that implement ParallelSourceFunction to Source V2 API
[ https://issues.apache.org/jira/browse/FLINK-32694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-32694: -- Summary: Migrate classes that implement ParallelSourceFunction to Source V2 API (was: Cascade deprecation to classes that implement ParallelSourceFunction) > Migrate classes that implement ParallelSourceFunction to Source V2 API > -- > > Key: FLINK-32694 > URL: https://issues.apache.org/jira/browse/FLINK-32694 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > > * SimpleEndlessSourceWithBloatedState in HeavyDeploymentStressTestProgram > (org.apache.flink.deployment) > * StatefulSequenceSource (org.apache.flink.streaming.api.functions.source) > * ArrowSourceFunction (org.apache.flink.table.runtime.arrow.sources) > * InputFormatSourceFunction (org.apache.flink.streaming.api.functions.source) > * EventsGeneratorSource > (org.apache.flink.streaming.examples.statemachine.generator) > * FromSplittableIteratorFunction > (org.apache.flink.streaming.api.functions.source) > * DataGeneratorSource > (org.apache.flink.streaming.api.functions.source.datagen) > * -- Tests: > * MySource in StatefulStreamingJob (org.apache.flink.test) > * RandomLongSource in StickyAllocationAndLocalRecoveryTestJob > (org.apache.flink.streaming.tests) > * SequenceGeneratorSource (org.apache.flink.streaming.tests) > * TtlStateUpdateSource (org.apache.flink.streaming.tests) > * StringSourceFunction in NettyShuffleMemoryControlTestProgram > (org.apache.flink.streaming.tests) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-32694) Cascade deprecation to classes that implement ParallelSourceFunction
[ https://issues.apache.org/jira/browse/FLINK-32694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov reassigned FLINK-32694: - Assignee: Alexander Fedulov > Cascade deprecation to classes that implement ParallelSourceFunction > > > Key: FLINK-32694 > URL: https://issues.apache.org/jira/browse/FLINK-32694 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > > * SimpleEndlessSourceWithBloatedState in HeavyDeploymentStressTestProgram > (org.apache.flink.deployment) > * StatefulSequenceSource (org.apache.flink.streaming.api.functions.source) > * ArrowSourceFunction (org.apache.flink.table.runtime.arrow.sources) > * InputFormatSourceFunction (org.apache.flink.streaming.api.functions.source) > * EventsGeneratorSource > (org.apache.flink.streaming.examples.statemachine.generator) > * FromSplittableIteratorFunction > (org.apache.flink.streaming.api.functions.source) > * DataGeneratorSource > (org.apache.flink.streaming.api.functions.source.datagen) > * -- Tests: > * MySource in StatefulStreamingJob (org.apache.flink.test) > * RandomLongSource in StickyAllocationAndLocalRecoveryTestJob > (org.apache.flink.streaming.tests) > * SequenceGeneratorSource (org.apache.flink.streaming.tests) > * TtlStateUpdateSource (org.apache.flink.streaming.tests) > * StringSourceFunction in NettyShuffleMemoryControlTestProgram > (org.apache.flink.streaming.tests) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32693) Cascade deprecation to classes that directly implement SourceFunction
[ https://issues.apache.org/jira/browse/FLINK-32693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-32693: -- Description: * SocketTextStreamFunction (org.apache.flink.streaming.api.functions.source) * FiniteTestSource (org.apache.flink.streaming.util) * FromIteratorFunction (org.apache.flink.streaming.api.functions.source) * Generator in FileSinkProgram (org.apache.flink.connector.file.sink) * RichSourceFunction (org.apache.flink.streaming.api.functions.source) * FileMonitoringFunction (org.apache.flink.streaming.api.functions.source) * FromElementsFunction (org.apache.flink.streaming.api.functions.source) * — Tests * PeriodicSourceGenerator in PeriodicStreamingJob (org.apache.flink.streaming.tests) * Generator in StreamSQLTestProgram (org.apache.flink.sql.tests) was: * SocketTextStreamFunction (org.apache.flink.streaming.api.functions.source) * FiniteTestSource (org.apache.flink.streaming.util) * FromIteratorFunction (org.apache.flink.streaming.api.functions.source) * Generator in FileSinkProgram (org.apache.flink.connector.file.sink) * RichSourceFunction (org.apache.flink.streaming.api.functions.source) * FileMonitoringFunction (org.apache.flink.streaming.api.functions.source) * FromElementsFunction (org.apache.flink.streaming.api.functions.source) * PeriodicSourceGenerator in PeriodicStreamingJob (org.apache.flink.streaming.tests) * Generator in StreamSQLTestProgram (org.apache.flink.sql.tests) * ParallelSourceFunction (org.apache.flink.streaming.api.functions.source) * ExternallyInducedSource (org.apache.flink.streaming.api.checkpoint) > Cascade deprecation to classes that directly implement SourceFunction > - > > Key: FLINK-32693 > URL: https://issues.apache.org/jira/browse/FLINK-32693 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > * SocketTextStreamFunction (org.apache.flink.streaming.api.functions.source) > * FiniteTestSource (org.apache.flink.streaming.util) > * FromIteratorFunction (org.apache.flink.streaming.api.functions.source) > * Generator in FileSinkProgram (org.apache.flink.connector.file.sink) > * RichSourceFunction (org.apache.flink.streaming.api.functions.source) > * FileMonitoringFunction (org.apache.flink.streaming.api.functions.source) > * FromElementsFunction (org.apache.flink.streaming.api.functions.source) > * — Tests > * PeriodicSourceGenerator in PeriodicStreamingJob > (org.apache.flink.streaming.tests) > * Generator in StreamSQLTestProgram (org.apache.flink.sql.tests) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32695) Migrate SourceFunction in Tests to Source V2 API
[ https://issues.apache.org/jira/browse/FLINK-32695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-32695: -- Description: * ProcessingTimeServiceSource in StreamSourceOperatorLatencyMetricsTest (org.apache.flink.streaming.runtime.operators) * TestSourceFunction in CommonExecSinkITCase (org.apache.flink.table.planner.plan.nodes.exec.common) * CheckpointingNonParallelSourceWithListState in MigrationTestUtils (org.apache.flink.test.checkpointing.utils) * NonSerializableTupleSource in StreamingOperatorsITCase (org.apache.flink.test.streaming.api) * ImmediatelyFinishingSource in StreamTaskFinalCheckpointsTest (org.apache.flink.streaming.runtime.tasks) * SimpleStringGenerator in CheckpointedStreamingProgram (org.apache.flink.test.classloading.jar) * TupleSource in StreamingOperatorsITCase (org.apache.flink.test.streaming.api) * SystemExitSourceFunction in StreamTaskSystemExitTest (org.apache.flink.streaming.runtime.tasks) * EmptySource in SourceStreamTaskTest (org.apache.flink.streaming.runtime.tasks) * NumberSource in IgnoreInFlightDataITCase (org.apache.flink.test.checkpointing) * MyTimestampSource in TimestampITCase (org.apache.flink.test.streaming.runtime) * InfiniteTestSource in StreamTaskTimerITCase (org.apache.flink.test.streaming.runtime) * SavepointSource in SavepointReaderITTestBase (org.apache.flink.state.api) * Source in TimersSavepointITCase (org.apache.flink.test.checkpointing) * MyTimestampSourceInfinite in TimestampITCase (org.apache.flink.test.streaming.runtime) * TestSource in WithMasterCheckpointHookConfigTest (org.apache.flink.streaming.graph) * MockSourceFunction in StreamTaskTest (org.apache.flink.streaming.runtime.tasks) * MyCustomSourceFunction (org.apache.flink.python.util) * RowSourceFunction in StreamExecutionEnvironmentTest (org.apache.flink.streaming.api) * FromRowDataSourceFunction in TestValuesTableFactory (org.apache.flink.table.planner.factories) * CancelLockingSource in SourceStreamTaskTest (org.apache.flink.streaming.runtime.tasks) * LockStepSourceWithOneWmPerElement in SourceTaskTerminationTest (org.apache.flink.streaming.runtime.tasks) * NonStoppingSource in SourceStreamTaskTest (org.apache.flink.streaming.runtime.tasks) * FromElementSourceFunctionWithWatermark in TestValuesRuntimeFunctions (org.apache.flink.table.planner.factories) * TestSource in InterruptSensitiveRestoreTest (org.apache.flink.streaming.runtime.tasks) * SavepointSource in DataSetSavepointReaderITTestBase (org.apache.flink.state.api) * FailingCollectionSource (org.apache.flink.table.planner.runtime.utils) * SessionEventGeneratorDataSource in SessionWindowITCase (org.apache.flink.test.windowing.sessionwindows) * Source in BuiltInAggregateFunctionTestBase (org.apache.flink.table.planner.functions) * InterruptedSource in SourceStreamTaskTest (org.apache.flink.streaming.runtime.tasks) * MyNonWatermarkingSource in TimestampITCase (org.apache.flink.test.streaming.runtime) * NormalSource in NotifyCheckpointAbortedITCase (org.apache.flink.test.checkpointing) * MaxWatermarkSource (org.apache.flink.state.api.utils) * TestSource in RestoreUpgradedJobITCase (org.apache.flink.test.checkpointing) * InfiniteSource in StreamSourceOperatorWatermarksTest (org.apache.flink.streaming.runtime.operators) * InfiniteLongSourceFunction in JobCancelingITCase (org.apache.flink.test.cancelling) * TestSource in TypeFillTest (org.apache.flink.streaming.api) * ManuallyClosedSourceFunction in CoordinatorEventsToStreamOperatorRecipientExactlyOnceITCase (org.apache.flink.streaming.runtime.tasks) * GenericSourceFunction in YarnTestCacheJob (org.apache.flink.yarn.testjob) * MockSource in SourceStreamTaskTest (org.apache.flink.streaming.runtime.tasks) * DummySource in ReactiveModeITCase (org.apache.flink.test.scheduling) * — * Anonymous in testDisablingBufferTimeout() in BufferTimeoutITCase (org.apache.flink.test.streaming.runtime) * Anonymous in testWatermarkForwarding() in SideOutputITCase (org.apache.flink.test.streaming.runtime) * Anonymous in testCoGroup() in CoGroupJoinITCase (org.apache.flink.test.streaming.runtime) * Anonymous in testCoGroup() in CoGroupJoinITCase (org.apache.flink.test.streaming.runtime) * Anonymous in testJoin() in CoGroupJoinITCase (org.apache.flink.test.streaming.runtime) * Anonymous in testJoin() in CoGroupJoinITCase (org.apache.flink.test.streaming.runtime) * Anonymous in testSelfJoin() in CoGroupJoinITCase (org.apache.flink.test.streaming.runtime) * Anonymous in testTimestampExtractorWithAutoInterval() in TimestampITCase (org.apache.flink.test.streaming.runtime) * Anonymous in testTimestampExtractorWithCustomWatermarkEmit() in TimestampITCase (org.apache.flink.test.streaming.runtime) * Anonymous in testTimestampExtractorWithDecreasingCustomWatermarkEmit() in TimestampITCase (org.apache.flink.test.streaming.runtime) *
[jira] [Updated] (FLINK-32694) Cascade deprecation to classes that implement ParallelSourceFunction
[ https://issues.apache.org/jira/browse/FLINK-32694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-32694: -- Description: * SimpleEndlessSourceWithBloatedState in HeavyDeploymentStressTestProgram (org.apache.flink.deployment) * StatefulSequenceSource (org.apache.flink.streaming.api.functions.source) * ArrowSourceFunction (org.apache.flink.table.runtime.arrow.sources) * InputFormatSourceFunction (org.apache.flink.streaming.api.functions.source) * EventsGeneratorSource (org.apache.flink.streaming.examples.statemachine.generator) * FromSplittableIteratorFunction (org.apache.flink.streaming.api.functions.source) * DataGeneratorSource (org.apache.flink.streaming.api.functions.source.datagen) * -- Tests: * MySource in StatefulStreamingJob (org.apache.flink.test) * RandomLongSource in StickyAllocationAndLocalRecoveryTestJob (org.apache.flink.streaming.tests) * SequenceGeneratorSource (org.apache.flink.streaming.tests) * TtlStateUpdateSource (org.apache.flink.streaming.tests) * StringSourceFunction in NettyShuffleMemoryControlTestProgram (org.apache.flink.streaming.tests) was: * RichParallelSourceFunction (org.apache.flink.streaming.api.functions.source) * MySource in StatefulStreamingJob (org.apache.flink.test) * RandomLongSource in StickyAllocationAndLocalRecoveryTestJob (org.apache.flink.streaming.tests) * SequenceGeneratorSource (org.apache.flink.streaming.tests) * TtlStateUpdateSource (org.apache.flink.streaming.tests) * SimpleEndlessSourceWithBloatedState in HeavyDeploymentStressTestProgram (org.apache.flink.deployment) * StringSourceFunction in NettyShuffleMemoryControlTestProgram (org.apache.flink.streaming.tests) * StatefulSequenceSource (org.apache.flink.streaming.api.functions.source) * ArrowSourceFunction (org.apache.flink.table.runtime.arrow.sources) * InputFormatSourceFunction (org.apache.flink.streaming.api.functions.source) * EventsGeneratorSource (org.apache.flink.streaming.examples.statemachine.generator) * FromSplittableIteratorFunction (org.apache.flink.streaming.api.functions.source) * DataGeneratorSource (org.apache.flink.streaming.api.functions.source.datagen) > Cascade deprecation to classes that implement ParallelSourceFunction > > > Key: FLINK-32694 > URL: https://issues.apache.org/jira/browse/FLINK-32694 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common >Reporter: Alexander Fedulov >Priority: Major > > > * SimpleEndlessSourceWithBloatedState in HeavyDeploymentStressTestProgram > (org.apache.flink.deployment) > * StatefulSequenceSource (org.apache.flink.streaming.api.functions.source) > * ArrowSourceFunction (org.apache.flink.table.runtime.arrow.sources) > * InputFormatSourceFunction (org.apache.flink.streaming.api.functions.source) > * EventsGeneratorSource > (org.apache.flink.streaming.examples.statemachine.generator) > * FromSplittableIteratorFunction > (org.apache.flink.streaming.api.functions.source) > * DataGeneratorSource > (org.apache.flink.streaming.api.functions.source.datagen) > * -- Tests: > * MySource in StatefulStreamingJob (org.apache.flink.test) > * RandomLongSource in StickyAllocationAndLocalRecoveryTestJob > (org.apache.flink.streaming.tests) > * SequenceGeneratorSource (org.apache.flink.streaming.tests) > * TtlStateUpdateSource (org.apache.flink.streaming.tests) > * StringSourceFunction in NettyShuffleMemoryControlTestProgram > (org.apache.flink.streaming.tests) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32695) Migrate SourceFunction in Tests to Source V2 API
[ https://issues.apache.org/jira/browse/FLINK-32695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-32695: -- Description: * ProcessingTimeServiceSource in StreamSourceOperatorLatencyMetricsTest (org.apache.flink.streaming.runtime.operators) * TestSourceFunction in CommonExecSinkITCase (org.apache.flink.table.planner.plan.nodes.exec.common) * CheckpointingNonParallelSourceWithListState in MigrationTestUtils (org.apache.flink.test.checkpointing.utils) * NonSerializableTupleSource in StreamingOperatorsITCase (org.apache.flink.test.streaming.api) * ImmediatelyFinishingSource in StreamTaskFinalCheckpointsTest (org.apache.flink.streaming.runtime.tasks) * SimpleStringGenerator in CheckpointedStreamingProgram (org.apache.flink.test.classloading.jar) * TupleSource in StreamingOperatorsITCase (org.apache.flink.test.streaming.api) * SystemExitSourceFunction in StreamTaskSystemExitTest (org.apache.flink.streaming.runtime.tasks) * EmptySource in SourceStreamTaskTest (org.apache.flink.streaming.runtime.tasks) * NumberSource in IgnoreInFlightDataITCase (org.apache.flink.test.checkpointing) * MyTimestampSource in TimestampITCase (org.apache.flink.test.streaming.runtime) * InfiniteTestSource in StreamTaskTimerITCase (org.apache.flink.test.streaming.runtime) * SavepointSource in SavepointReaderITTestBase (org.apache.flink.state.api) * Source in TimersSavepointITCase (org.apache.flink.test.checkpointing) * MyTimestampSourceInfinite in TimestampITCase (org.apache.flink.test.streaming.runtime) * TestSource in WithMasterCheckpointHookConfigTest (org.apache.flink.streaming.graph) * MockSourceFunction in StreamTaskTest (org.apache.flink.streaming.runtime.tasks) * MyCustomSourceFunction (org.apache.flink.python.util) * RowSourceFunction in StreamExecutionEnvironmentTest (org.apache.flink.streaming.api) * FromRowDataSourceFunction in TestValuesTableFactory (org.apache.flink.table.planner.factories) * CancelLockingSource in SourceStreamTaskTest (org.apache.flink.streaming.runtime.tasks) * LockStepSourceWithOneWmPerElement in SourceTaskTerminationTest (org.apache.flink.streaming.runtime.tasks) * NonStoppingSource in SourceStreamTaskTest (org.apache.flink.streaming.runtime.tasks) * FromElementSourceFunctionWithWatermark in TestValuesRuntimeFunctions (org.apache.flink.table.planner.factories) * TestSource in InterruptSensitiveRestoreTest (org.apache.flink.streaming.runtime.tasks) * SavepointSource in DataSetSavepointReaderITTestBase (org.apache.flink.state.api) * FailingCollectionSource (org.apache.flink.table.planner.runtime.utils) * SessionEventGeneratorDataSource in SessionWindowITCase (org.apache.flink.test.windowing.sessionwindows) * Source in BuiltInAggregateFunctionTestBase (org.apache.flink.table.planner.functions) * InterruptedSource in SourceStreamTaskTest (org.apache.flink.streaming.runtime.tasks) * MyNonWatermarkingSource in TimestampITCase (org.apache.flink.test.streaming.runtime) * NormalSource in NotifyCheckpointAbortedITCase (org.apache.flink.test.checkpointing) * MaxWatermarkSource (org.apache.flink.state.api.utils) * TestSource in RestoreUpgradedJobITCase (org.apache.flink.test.checkpointing) * InfiniteSource in StreamSourceOperatorWatermarksTest (org.apache.flink.streaming.runtime.operators) * InfiniteLongSourceFunction in JobCancelingITCase (org.apache.flink.test.cancelling) * TestSource in TypeFillTest (org.apache.flink.streaming.api) * ManuallyClosedSourceFunction in CoordinatorEventsToStreamOperatorRecipientExactlyOnceITCase (org.apache.flink.streaming.runtime.tasks) * GenericSourceFunction in YarnTestCacheJob (org.apache.flink.yarn.testjob) * MockSource in SourceStreamTaskTest (org.apache.flink.streaming.runtime.tasks) * DummySource in ReactiveModeITCase (org.apache.flink.test.scheduling) * --- * PeriodicSourceGenerator in PeriodicStreamingJob (org.apache.flink.streaming.tests) * Generator in StreamSQLTestProgram (org.apache.flink.sql.tests) * --- * Anonymous in testDisablingBufferTimeout() in BufferTimeoutITCase (org.apache.flink.test.streaming.runtime) * Anonymous in testWatermarkForwarding() in SideOutputITCase (org.apache.flink.test.streaming.runtime) * Anonymous in testCoGroup() in CoGroupJoinITCase (org.apache.flink.test.streaming.runtime) * Anonymous in testCoGroup() in CoGroupJoinITCase (org.apache.flink.test.streaming.runtime) * Anonymous in testJoin() in CoGroupJoinITCase (org.apache.flink.test.streaming.runtime) * Anonymous in testJoin() in CoGroupJoinITCase (org.apache.flink.test.streaming.runtime) * Anonymous in testSelfJoin() in CoGroupJoinITCase (org.apache.flink.test.streaming.runtime) * Anonymous in testTimestampExtractorWithAutoInterval() in TimestampITCase (org.apache.flink.test.streaming.runtime) * Anonymous in testTimestampExtractorWithCustomWatermarkEmit() in TimestampITCase
[jira] [Updated] (FLINK-28045) [umbrella] Blockers for SourceFunction API removal (in Flink 2.0)
[
https://issues.apache.org/jira/browse/FLINK-28045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Fedulov updated FLINK-28045:
--
Description:
This ticket should only contain items where SourceFunction is already used in
the existing code base. For new functionality and Source V2 API improvements,
use FLINK-32692.
—
Current state of migration to Source V2 API for officially-supported connectors
(/) FileSystem (Public)
(/) Kafka (PublicEvolving)
(/) Cassandra (PublicEvolving)
(/) Pulsar (PublicEvolving)
(/) HiveSource (PublicEvolving)
(/) HybridSource (PublicEvolving)
(/) MongoDB (PublicEvolving)
(/) Hive (PublicEvolving)
(?) Kinesis (Experimental)
(flagoff) JDBC - not impacted (based on {{{}TableFunction{}}})
(flagoff) HBase - not impacted (based on {{{}TableFunction{}}})
(x) RabbitMQ
(x) Google Pub Sub
was:
This ticket should only contain items where SourceFunction is already used in
the existing code base. For new functionality and Source API improvements, use
FLINK-32692.
---
Current state of migration to Source V2 API for officially-supported connectors
(/) FileSystem (Public)
(/) Kafka (PublicEvolving)
(/) Cassandra (PublicEvolving)
(/) Pulsar (PublicEvolving)
(/) HiveSource (PublicEvolving)
(/) HybridSource (PublicEvolving)
(/) MongoDB (PublicEvolving)
(/) Hive (PublicEvolving)
(?) Kinesis (Experimental)
(flagoff) JDBC - not impacted (based on {{{}TableFunction{}}})
(flagoff) HBase - not impacted (based on {{{}TableFunction{}}})
(x) RabbitMQ
(x) Google Pub Sub
> [umbrella] Blockers for SourceFunction API removal (in Flink 2.0)
> -
>
> Key: FLINK-28045
> URL: https://issues.apache.org/jira/browse/FLINK-28045
> Project: Flink
> Issue Type: Technical Debt
> Components: Connectors / Common
>Reporter: Alexander Fedulov
>Assignee: Alexander Fedulov
>Priority: Major
>
> This ticket should only contain items where SourceFunction is already used in
> the existing code base. For new functionality and Source V2 API improvements,
> use FLINK-32692.
> —
> Current state of migration to Source V2 API for officially-supported
> connectors
> (/) FileSystem (Public)
> (/) Kafka (PublicEvolving)
> (/) Cassandra (PublicEvolving)
> (/) Pulsar (PublicEvolving)
> (/) HiveSource (PublicEvolving)
> (/) HybridSource (PublicEvolving)
> (/) MongoDB (PublicEvolving)
> (/) Hive (PublicEvolving)
> (?) Kinesis (Experimental)
> (flagoff) JDBC - not impacted (based on {{{}TableFunction{}}})
> (flagoff) HBase - not impacted (based on {{{}TableFunction{}}})
> (x) RabbitMQ
> (x) Google Pub Sub
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-28045) [umbrella] Blockers for SourceFunction API removal (in Flink 2.0)
[
https://issues.apache.org/jira/browse/FLINK-28045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17747894#comment-17747894
]
Alexander Fedulov commented on FLINK-28045:
---
Hi [~dannycranmer], thanks for the confirmation!
> [umbrella] Blockers for SourceFunction API removal (in Flink 2.0)
> -
>
> Key: FLINK-28045
> URL: https://issues.apache.org/jira/browse/FLINK-28045
> Project: Flink
> Issue Type: Technical Debt
> Components: Connectors / Common
>Reporter: Alexander Fedulov
>Assignee: Alexander Fedulov
>Priority: Major
>
> This ticket should only contain items where SourceFunction is already used in
> the existing code base. For new functionality and Source API improvements,
> use FLINK-32692.
> ---
> Current state of migration to Source V2 API for officially-supported
> connectors
> (/) FileSystem (Public)
> (/) Kafka (PublicEvolving)
> (/) Cassandra (PublicEvolving)
> (/) Pulsar (PublicEvolving)
> (/) HiveSource (PublicEvolving)
> (/) HybridSource (PublicEvolving)
> (/) MongoDB (PublicEvolving)
> (/) Hive (PublicEvolving)
> (?) Kinesis (Experimental)
> (flagoff) JDBC - not impacted (based on {{{}TableFunction{}}})
> (flagoff) HBase - not impacted (based on {{{}TableFunction{}}})
> (x) RabbitMQ
> (x) Google Pub Sub
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-28045) [umbrella] Blockers for SourceFunction API removal (in Flink 2.0)
[
https://issues.apache.org/jira/browse/FLINK-28045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17747665#comment-17747665
]
Alexander Fedulov commented on FLINK-28045:
---
[~danny.cranmer] I see that Kinesis Source V2 connector is currently marked as
Experimental. What is its current state? Do you see any obstacles in being able
to promote it to PublicEvolving in the near(ish) future?
> [umbrella] Blockers for SourceFunction API removal (in Flink 2.0)
> -
>
> Key: FLINK-28045
> URL: https://issues.apache.org/jira/browse/FLINK-28045
> Project: Flink
> Issue Type: Technical Debt
> Components: Connectors / Common
>Reporter: Alexander Fedulov
>Assignee: Alexander Fedulov
>Priority: Major
>
> This ticket should only contain items where SourceFunction is already used in
> the existing code base. For new functionality and Source API improvements,
> use FLINK-32692.
> ---
> Current state of migration to Source V2 API for officially-supported
> connectors
> (/) FileSystem (Public)
> (/) Kafka (PublicEvolving)
> (/) Cassandra (PublicEvolving)
> (/) Pulsar (PublicEvolving)
> (/) HiveSource (PublicEvolving)
> (/) HybridSource (PublicEvolving)
> (/) MongoDB (PublicEvolving)
> (/) Hive (PublicEvolving)
> (?) Kinesis (Experimental)
> (flagoff) JDBC - not impacted (based on {{{}TableFunction{}}})
> (flagoff) HBase - not impacted (based on {{{}TableFunction{}}})
> (x) RabbitMQ
> (x) Google Pub Sub
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28045) [umbrella] Blockers for SourceFunction API (in Flink 2.0)
[
https://issues.apache.org/jira/browse/FLINK-28045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Fedulov updated FLINK-28045:
--
Summary: [umbrella] Blockers for SourceFunction API (in Flink 2.0) (was:
[umbrella] Blockers for SourceFunction API removal)
> [umbrella] Blockers for SourceFunction API (in Flink 2.0)
> -
>
> Key: FLINK-28045
> URL: https://issues.apache.org/jira/browse/FLINK-28045
> Project: Flink
> Issue Type: Technical Debt
> Components: Connectors / Common
>Reporter: Alexander Fedulov
>Assignee: Alexander Fedulov
>Priority: Major
>
> This ticket should only contain items where SourceFunction is already used in
> the existing code base. For new functionality and Source API improvements,
> use FLINK-32692.
> ---
> Current state of migration to Source V2 API for officially-supported
> connectors
> (/) FileSystem (Public)
> (/) Kafka (PublicEvolving)
> (/) Cassandra (PublicEvolving)
> (/) Pulsar (PublicEvolving)
> (/) HiveSource (PublicEvolving)
> (/) HybridSource (PublicEvolving)
> (/) MongoDB (PublicEvolving)
> (/) Hive (PublicEvolving)
> (?) Kinesis (Experimental)
> (flagoff) JDBC - not impacted (based on {{{}TableFunction{}}})
> (flagoff) HBase - not impacted (based on {{{}TableFunction{}}})
> (x) RabbitMQ
> (x) Google Pub Sub
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28045) [umbrella] Blockers for SourceFunction API removal (in Flink 2.0)
[
https://issues.apache.org/jira/browse/FLINK-28045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Fedulov updated FLINK-28045:
--
Summary: [umbrella] Blockers for SourceFunction API removal (in Flink 2.0)
(was: [umbrella] Blockers for SourceFunction API (in Flink 2.0))
> [umbrella] Blockers for SourceFunction API removal (in Flink 2.0)
> -
>
> Key: FLINK-28045
> URL: https://issues.apache.org/jira/browse/FLINK-28045
> Project: Flink
> Issue Type: Technical Debt
> Components: Connectors / Common
>Reporter: Alexander Fedulov
>Assignee: Alexander Fedulov
>Priority: Major
>
> This ticket should only contain items where SourceFunction is already used in
> the existing code base. For new functionality and Source API improvements,
> use FLINK-32692.
> ---
> Current state of migration to Source V2 API for officially-supported
> connectors
> (/) FileSystem (Public)
> (/) Kafka (PublicEvolving)
> (/) Cassandra (PublicEvolving)
> (/) Pulsar (PublicEvolving)
> (/) HiveSource (PublicEvolving)
> (/) HybridSource (PublicEvolving)
> (/) MongoDB (PublicEvolving)
> (/) Hive (PublicEvolving)
> (?) Kinesis (Experimental)
> (flagoff) JDBC - not impacted (based on {{{}TableFunction{}}})
> (flagoff) HBase - not impacted (based on {{{}TableFunction{}}})
> (x) RabbitMQ
> (x) Google Pub Sub
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28045) [umbrella] Blockers for SourceFunction API removal
[
https://issues.apache.org/jira/browse/FLINK-28045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Fedulov updated FLINK-28045:
--
Description:
This ticket should only contain items where SourceFunction is already used in
the existing code base. For new functionality and Source API improvements, use
FLINK-32692.
---
Current state of migration to Source V2 API for officially-supported connectors
(/) FileSystem (Public)
(/) Kafka (PublicEvolving)
(/) Cassandra (PublicEvolving)
(/) Pulsar (PublicEvolving)
(/) HiveSource (PublicEvolving)
(/) HybridSource (PublicEvolving)
(/) MongoDB (PublicEvolving)
(/) Hive (PublicEvolving)
(?) Kinesis (Experimental)
(flagoff) JDBC - not impacted (based on {{{}TableFunction{}}})
(flagoff) HBase - not impacted (based on {{{}TableFunction{}}})
(x) RabbitMQ
(x) Google Pub Sub
was:
This ticket should only contain items where SourceFunction is already used in
the existing code base. For new functionality and Source API improvements, use
FLINK-32692
(/) FileSystem (Public)
(/) Kafka (PublicEvolving)
(/) Cassandra (PublicEvolving)
(/) Pulsar (PublicEvolving)
(/) HiveSource (PublicEvolving)
(/) HybridSource (PublicEvolving)
(/) MongoDB (PublicEvolving)
(/) Hive (PublicEvolving)
(?) Kinesis (Experimental)
(flagoff) JDBC - not impacted (based on {{{}TableFunction{}}})
(flagoff) HBase - not impacted (based on {{{}TableFunction{}}})
(x) RabbitMQ
(x)Google Pub Sub
> [umbrella] Blockers for SourceFunction API removal
> --
>
> Key: FLINK-28045
> URL: https://issues.apache.org/jira/browse/FLINK-28045
> Project: Flink
> Issue Type: Technical Debt
> Components: Connectors / Common
>Reporter: Alexander Fedulov
>Assignee: Alexander Fedulov
>Priority: Major
>
> This ticket should only contain items where SourceFunction is already used in
> the existing code base. For new functionality and Source API improvements,
> use FLINK-32692.
> ---
> Current state of migration to Source V2 API for officially-supported
> connectors
> (/) FileSystem (Public)
> (/) Kafka (PublicEvolving)
> (/) Cassandra (PublicEvolving)
> (/) Pulsar (PublicEvolving)
> (/) HiveSource (PublicEvolving)
> (/) HybridSource (PublicEvolving)
> (/) MongoDB (PublicEvolving)
> (/) Hive (PublicEvolving)
> (?) Kinesis (Experimental)
> (flagoff) JDBC - not impacted (based on {{{}TableFunction{}}})
> (flagoff) HBase - not impacted (based on {{{}TableFunction{}}})
> (x) RabbitMQ
> (x) Google Pub Sub
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-32696) Promote Kinesis connector support to PublicEvolving
Alexander Fedulov created FLINK-32696: - Summary: Promote Kinesis connector support to PublicEvolving Key: FLINK-32696 URL: https://issues.apache.org/jira/browse/FLINK-32696 Project: Flink Issue Type: Sub-task Components: Connectors / Kinesis Reporter: Alexander Fedulov -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28045) [umbrella] Blockers for SourceFunction API removal
[
https://issues.apache.org/jira/browse/FLINK-28045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Fedulov updated FLINK-28045:
--
Description:
This ticket should only contain items where SourceFunction is already used in
the existing code base. For new functionality and Source API improvements, use
FLINK-32692
(/) FileSystem (Public)
(/) Kafka (PublicEvolving)
(/) Cassandra (PublicEvolving)
(/) Pulsar (PublicEvolving)
(/) HiveSource (PublicEvolving)
(/) HybridSource (PublicEvolving)
(/) MongoDB (PublicEvolving)
(/) Hive (PublicEvolving)
(?) Kinesis (Experimental)
(flagoff) JDBC - not impacted (based on {{{}TableFunction{}}})
(flagoff) HBase - not impacted (based on {{{}TableFunction{}}})
(x) RabbitMQ
(x)Google Pub Sub
was:This ticket should only contain items where SourceFunction is already
used in the existing code base. For new functionality and Source API
improvements, use FLINK-32692
> [umbrella] Blockers for SourceFunction API removal
> --
>
> Key: FLINK-28045
> URL: https://issues.apache.org/jira/browse/FLINK-28045
> Project: Flink
> Issue Type: Technical Debt
> Components: Connectors / Common
>Reporter: Alexander Fedulov
>Assignee: Alexander Fedulov
>Priority: Major
>
> This ticket should only contain items where SourceFunction is already used in
> the existing code base. For new functionality and Source API improvements,
> use FLINK-32692
> (/) FileSystem (Public)
> (/) Kafka (PublicEvolving)
> (/) Cassandra (PublicEvolving)
> (/) Pulsar (PublicEvolving)
> (/) HiveSource (PublicEvolving)
> (/) HybridSource (PublicEvolving)
> (/) MongoDB (PublicEvolving)
> (/) Hive (PublicEvolving)
> (?) Kinesis (Experimental)
> (flagoff) JDBC - not impacted (based on {{{}TableFunction{}}})
> (flagoff) HBase - not impacted (based on {{{}TableFunction{}}})
> (x) RabbitMQ
> (x)Google Pub Sub
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32695) Migrate SourceFunction in Tests to Source V2 API
[ https://issues.apache.org/jira/browse/FLINK-32695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-32695: -- Summary: Migrate SourceFunction in Tests to Source V2 API (was: Migrate SourceFunction in Tests to Source API) > Migrate SourceFunction in Tests to Source V2 API > > > Key: FLINK-32695 > URL: https://issues.apache.org/jira/browse/FLINK-32695 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > * ProcessingTimeServiceSource in StreamSourceOperatorLatencyMetricsTest > (org.apache.flink.streaming.runtime.operators) > * TestSourceFunction in CommonExecSinkITCase > (org.apache.flink.table.planner.plan.nodes.exec.common) > * CheckpointingNonParallelSourceWithListState in MigrationTestUtils > (org.apache.flink.test.checkpointing.utils) > * NonSerializableTupleSource in StreamingOperatorsITCase > (org.apache.flink.test.streaming.api) > * ImmediatelyFinishingSource in StreamTaskFinalCheckpointsTest > (org.apache.flink.streaming.runtime.tasks) > * SimpleStringGenerator in CheckpointedStreamingProgram > (org.apache.flink.test.classloading.jar) > * TupleSource in StreamingOperatorsITCase > (org.apache.flink.test.streaming.api) > * SystemExitSourceFunction in StreamTaskSystemExitTest > (org.apache.flink.streaming.runtime.tasks) > * EmptySource in SourceStreamTaskTest > (org.apache.flink.streaming.runtime.tasks) > * NumberSource in IgnoreInFlightDataITCase > (org.apache.flink.test.checkpointing) > * MyTimestampSource in TimestampITCase > (org.apache.flink.test.streaming.runtime) > * InfiniteTestSource in StreamTaskTimerITCase > (org.apache.flink.test.streaming.runtime) > * SavepointSource in SavepointReaderITTestBase (org.apache.flink.state.api) > * Source in TimersSavepointITCase (org.apache.flink.test.checkpointing) > * MyTimestampSourceInfinite in TimestampITCase > (org.apache.flink.test.streaming.runtime) > * TestSource in WithMasterCheckpointHookConfigTest > (org.apache.flink.streaming.graph) > * MockSourceFunction in StreamTaskTest > (org.apache.flink.streaming.runtime.tasks) > * MyCustomSourceFunction (org.apache.flink.python.util) > * RowSourceFunction in StreamExecutionEnvironmentTest > (org.apache.flink.streaming.api) > * FromRowDataSourceFunction in TestValuesTableFactory > (org.apache.flink.table.planner.factories) > * CancelLockingSource in SourceStreamTaskTest > (org.apache.flink.streaming.runtime.tasks) > * LockStepSourceWithOneWmPerElement in SourceTaskTerminationTest > (org.apache.flink.streaming.runtime.tasks) > * NonStoppingSource in SourceStreamTaskTest > (org.apache.flink.streaming.runtime.tasks) > * FromElementSourceFunctionWithWatermark in TestValuesRuntimeFunctions > (org.apache.flink.table.planner.factories) > * TestSource in InterruptSensitiveRestoreTest > (org.apache.flink.streaming.runtime.tasks) > * SavepointSource in DataSetSavepointReaderITTestBase > (org.apache.flink.state.api) > * FailingCollectionSource (org.apache.flink.table.planner.runtime.utils) > * SessionEventGeneratorDataSource in SessionWindowITCase > (org.apache.flink.test.windowing.sessionwindows) > * Source in BuiltInAggregateFunctionTestBase > (org.apache.flink.table.planner.functions) > * InterruptedSource in SourceStreamTaskTest > (org.apache.flink.streaming.runtime.tasks) > * MyNonWatermarkingSource in TimestampITCase > (org.apache.flink.test.streaming.runtime) > * NormalSource in NotifyCheckpointAbortedITCase > (org.apache.flink.test.checkpointing) > * MaxWatermarkSource (org.apache.flink.state.api.utils) > * TestSource in RestoreUpgradedJobITCase > (org.apache.flink.test.checkpointing) > * InfiniteSource in StreamSourceOperatorWatermarksTest > (org.apache.flink.streaming.runtime.operators) > * InfiniteLongSourceFunction in JobCancelingITCase > (org.apache.flink.test.cancelling) > * TestSource in TypeFillTest (org.apache.flink.streaming.api) > * ManuallyClosedSourceFunction in > CoordinatorEventsToStreamOperatorRecipientExactlyOnceITCase > (org.apache.flink.streaming.runtime.tasks) > * GenericSourceFunction in YarnTestCacheJob (org.apache.flink.yarn.testjob) > * MockSource in SourceStreamTaskTest > (org.apache.flink.streaming.runtime.tasks) > * DummySource in ReactiveModeITCase (org.apache.flink.test.scheduling) > * Anonymous in testDisablingBufferTimeout() in BufferTimeoutITCase > (org.apache.flink.test.streaming.runtime) > * Anonymous in testWatermarkForwarding() in SideOutputITCase > (org.apache.flink.test.streaming.runtime) > * Anonymous in testCoGroup() in CoGroupJoinITCase > (org.apache.flink.test.streaming.runtime) > * Anonymous in testCoGroup() in CoGroupJoinITCase >
[jira] [Updated] (FLINK-32692) [umbrella] Nice-to-haves for SourceFunction API removal
[ https://issues.apache.org/jira/browse/FLINK-32692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-32692: -- Description: See also FLINK-28045 (was: See also https://issues.apache.org/jira/browse/FLINK-28045) > [umbrella] Nice-to-haves for SourceFunction API removal > --- > > Key: FLINK-32692 > URL: https://issues.apache.org/jira/browse/FLINK-32692 > Project: Flink > Issue Type: Technical Debt >Reporter: Alexander Fedulov >Priority: Major > > See also FLINK-28045 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28045) [umbrella] Blockers for SourceFunction API removal
[ https://issues.apache.org/jira/browse/FLINK-28045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-28045: -- Description: This ticket should only contain items where SourceFunction is already used in the existing code base. For new functionality and Source API improvements, use FLINK-32692 (was: This ticket should only contain items where SourceFunction is already used in the existing code base. For new functionality and Source API improvements, use https://issues.apache.org/jira/browse/FLINK-32692) > [umbrella] Blockers for SourceFunction API removal > -- > > Key: FLINK-28045 > URL: https://issues.apache.org/jira/browse/FLINK-28045 > Project: Flink > Issue Type: Technical Debt > Components: Connectors / Common >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > This ticket should only contain items where SourceFunction is already used in > the existing code base. For new functionality and Source API improvements, > use FLINK-32692 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32692) [umbrella] Nice-to-haves for SourceFunction API removal
[ https://issues.apache.org/jira/browse/FLINK-32692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-32692: -- Description: See also https://issues.apache.org/jira/browse/FLINK-28045 > [umbrella] Nice-to-haves for SourceFunction API removal > --- > > Key: FLINK-32692 > URL: https://issues.apache.org/jira/browse/FLINK-32692 > Project: Flink > Issue Type: Technical Debt >Reporter: Alexander Fedulov >Priority: Major > > See also https://issues.apache.org/jira/browse/FLINK-28045 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28051) Introduce Source API alternative to ExternallyInducedSource
[ https://issues.apache.org/jira/browse/FLINK-28051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-28051: -- Summary: Introduce Source API alternative to ExternallyInducedSource (was: Introduce stable Source API alternative to ExternallyInducedSource) > Introduce Source API alternative to ExternallyInducedSource > --- > > Key: FLINK-28051 > URL: https://issues.apache.org/jira/browse/FLINK-28051 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common, Tests >Reporter: Alexander Fedulov >Priority: Major > > It needs to be evaluated if ExternallyInducedSourceReader can be promoted. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28045) [umbrella] Blockers for SourceFunction API removal
[ https://issues.apache.org/jira/browse/FLINK-28045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-28045: -- Description: This ticket should only contain items where SourceFunction is already used in the existing code base. For new functionality and Source API improvements, use https://issues.apache.org/jira/browse/FLINK-32692 (was: This ticket should only contain items where SourceFunction is used in the existing code base and needs to be removed. For new functionality and Source API improvements, use https://issues.apache.org/jira/browse/FLINK-32692) > [umbrella] Blockers for SourceFunction API removal > -- > > Key: FLINK-28045 > URL: https://issues.apache.org/jira/browse/FLINK-28045 > Project: Flink > Issue Type: Technical Debt > Components: Connectors / Common >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > This ticket should only contain items where SourceFunction is already used in > the existing code base. For new functionality and Source API improvements, > use https://issues.apache.org/jira/browse/FLINK-32692 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28045) [umbrella] Blockers for SourceFunction API removal
[ https://issues.apache.org/jira/browse/FLINK-28045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-28045: -- Description: Should only contain items where SourceFunction is used in the existing code base and needs to be removed. For new functionality and Source API improvements, use https://issues.apache.org/jira/browse/FLINK-32692 > [umbrella] Blockers for SourceFunction API removal > -- > > Key: FLINK-28045 > URL: https://issues.apache.org/jira/browse/FLINK-28045 > Project: Flink > Issue Type: Technical Debt > Components: Connectors / Common >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > Should only contain items where SourceFunction is used in the existing code > base and needs to be removed. For new functionality and Source API > improvements, use https://issues.apache.org/jira/browse/FLINK-32692 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28045) [umbrella] Blockers for SourceFunction API removal
[ https://issues.apache.org/jira/browse/FLINK-28045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-28045: -- Description: This ticket should only contain items where SourceFunction is used in the existing code base and needs to be removed. For new functionality and Source API improvements, use https://issues.apache.org/jira/browse/FLINK-32692 (was: Should only contain items where SourceFunction is used in the existing code base and needs to be removed. For new functionality and Source API improvements, use https://issues.apache.org/jira/browse/FLINK-32692) > [umbrella] Blockers for SourceFunction API removal > -- > > Key: FLINK-28045 > URL: https://issues.apache.org/jira/browse/FLINK-28045 > Project: Flink > Issue Type: Technical Debt > Components: Connectors / Common >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > This ticket should only contain items where SourceFunction is used in the > existing code base and needs to be removed. For new functionality and Source > API improvements, use https://issues.apache.org/jira/browse/FLINK-32692 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-32695) Migrate SourceFunction in Tests to Source API
Alexander Fedulov created FLINK-32695: - Summary: Migrate SourceFunction in Tests to Source API Key: FLINK-32695 URL: https://issues.apache.org/jira/browse/FLINK-32695 Project: Flink Issue Type: Sub-task Components: Connectors / Common Reporter: Alexander Fedulov Assignee: Alexander Fedulov * ProcessingTimeServiceSource in StreamSourceOperatorLatencyMetricsTest (org.apache.flink.streaming.runtime.operators) * TestSourceFunction in CommonExecSinkITCase (org.apache.flink.table.planner.plan.nodes.exec.common) * CheckpointingNonParallelSourceWithListState in MigrationTestUtils (org.apache.flink.test.checkpointing.utils) * NonSerializableTupleSource in StreamingOperatorsITCase (org.apache.flink.test.streaming.api) * ImmediatelyFinishingSource in StreamTaskFinalCheckpointsTest (org.apache.flink.streaming.runtime.tasks) * SimpleStringGenerator in CheckpointedStreamingProgram (org.apache.flink.test.classloading.jar) * TupleSource in StreamingOperatorsITCase (org.apache.flink.test.streaming.api) * SystemExitSourceFunction in StreamTaskSystemExitTest (org.apache.flink.streaming.runtime.tasks) * EmptySource in SourceStreamTaskTest (org.apache.flink.streaming.runtime.tasks) * NumberSource in IgnoreInFlightDataITCase (org.apache.flink.test.checkpointing) * MyTimestampSource in TimestampITCase (org.apache.flink.test.streaming.runtime) * InfiniteTestSource in StreamTaskTimerITCase (org.apache.flink.test.streaming.runtime) * SavepointSource in SavepointReaderITTestBase (org.apache.flink.state.api) * Source in TimersSavepointITCase (org.apache.flink.test.checkpointing) * MyTimestampSourceInfinite in TimestampITCase (org.apache.flink.test.streaming.runtime) * TestSource in WithMasterCheckpointHookConfigTest (org.apache.flink.streaming.graph) * MockSourceFunction in StreamTaskTest (org.apache.flink.streaming.runtime.tasks) * MyCustomSourceFunction (org.apache.flink.python.util) * RowSourceFunction in StreamExecutionEnvironmentTest (org.apache.flink.streaming.api) * FromRowDataSourceFunction in TestValuesTableFactory (org.apache.flink.table.planner.factories) * CancelLockingSource in SourceStreamTaskTest (org.apache.flink.streaming.runtime.tasks) * LockStepSourceWithOneWmPerElement in SourceTaskTerminationTest (org.apache.flink.streaming.runtime.tasks) * NonStoppingSource in SourceStreamTaskTest (org.apache.flink.streaming.runtime.tasks) * FromElementSourceFunctionWithWatermark in TestValuesRuntimeFunctions (org.apache.flink.table.planner.factories) * TestSource in InterruptSensitiveRestoreTest (org.apache.flink.streaming.runtime.tasks) * SavepointSource in DataSetSavepointReaderITTestBase (org.apache.flink.state.api) * FailingCollectionSource (org.apache.flink.table.planner.runtime.utils) * SessionEventGeneratorDataSource in SessionWindowITCase (org.apache.flink.test.windowing.sessionwindows) * Source in BuiltInAggregateFunctionTestBase (org.apache.flink.table.planner.functions) * InterruptedSource in SourceStreamTaskTest (org.apache.flink.streaming.runtime.tasks) * MyNonWatermarkingSource in TimestampITCase (org.apache.flink.test.streaming.runtime) * NormalSource in NotifyCheckpointAbortedITCase (org.apache.flink.test.checkpointing) * MaxWatermarkSource (org.apache.flink.state.api.utils) * TestSource in RestoreUpgradedJobITCase (org.apache.flink.test.checkpointing) * InfiniteSource in StreamSourceOperatorWatermarksTest (org.apache.flink.streaming.runtime.operators) * InfiniteLongSourceFunction in JobCancelingITCase (org.apache.flink.test.cancelling) * TestSource in TypeFillTest (org.apache.flink.streaming.api) * ManuallyClosedSourceFunction in CoordinatorEventsToStreamOperatorRecipientExactlyOnceITCase (org.apache.flink.streaming.runtime.tasks) * GenericSourceFunction in YarnTestCacheJob (org.apache.flink.yarn.testjob) * MockSource in SourceStreamTaskTest (org.apache.flink.streaming.runtime.tasks) * DummySource in ReactiveModeITCase (org.apache.flink.test.scheduling) * Anonymous in testDisablingBufferTimeout() in BufferTimeoutITCase (org.apache.flink.test.streaming.runtime) * Anonymous in testWatermarkForwarding() in SideOutputITCase (org.apache.flink.test.streaming.runtime) * Anonymous in testCoGroup() in CoGroupJoinITCase (org.apache.flink.test.streaming.runtime) * Anonymous in testCoGroup() in CoGroupJoinITCase (org.apache.flink.test.streaming.runtime) * Anonymous in testJoin() in CoGroupJoinITCase (org.apache.flink.test.streaming.runtime) * Anonymous in testJoin() in CoGroupJoinITCase (org.apache.flink.test.streaming.runtime) * Anonymous in testSelfJoin() in CoGroupJoinITCase (org.apache.flink.test.streaming.runtime) * Anonymous in testTimestampExtractorWithAutoInterval() in TimestampITCase (org.apache.flink.test.streaming.runtime) * Anonymous in
[jira] [Updated] (FLINK-28048) Introduce Source API alternative to FiniteTestSource
[ https://issues.apache.org/jira/browse/FLINK-28048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-28048: -- Description: This also has to verify that Iceberg connector tests mentioned in FLINK-28054 also get covered by the solution. > Introduce Source API alternative to FiniteTestSource > > > Key: FLINK-28048 > URL: https://issues.apache.org/jira/browse/FLINK-28048 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common, Tests >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > This also has to verify that Iceberg connector tests mentioned in FLINK-28054 > also get covered by the solution. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-32694) Cascade deprecation to classes that implement ParallelSourceFunction
Alexander Fedulov created FLINK-32694: - Summary: Cascade deprecation to classes that implement ParallelSourceFunction Key: FLINK-32694 URL: https://issues.apache.org/jira/browse/FLINK-32694 Project: Flink Issue Type: Sub-task Components: Connectors / Common Reporter: Alexander Fedulov * RichParallelSourceFunction (org.apache.flink.streaming.api.functions.source) * MySource in StatefulStreamingJob (org.apache.flink.test) * RandomLongSource in StickyAllocationAndLocalRecoveryTestJob (org.apache.flink.streaming.tests) * SequenceGeneratorSource (org.apache.flink.streaming.tests) * TtlStateUpdateSource (org.apache.flink.streaming.tests) * SimpleEndlessSourceWithBloatedState in HeavyDeploymentStressTestProgram (org.apache.flink.deployment) * StringSourceFunction in NettyShuffleMemoryControlTestProgram (org.apache.flink.streaming.tests) * StatefulSequenceSource (org.apache.flink.streaming.api.functions.source) * ArrowSourceFunction (org.apache.flink.table.runtime.arrow.sources) * InputFormatSourceFunction (org.apache.flink.streaming.api.functions.source) * EventsGeneratorSource (org.apache.flink.streaming.examples.statemachine.generator) * FromSplittableIteratorFunction (org.apache.flink.streaming.api.functions.source) * DataGeneratorSource (org.apache.flink.streaming.api.functions.source.datagen) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32693) Cascade deprecation to classes that directly implement SourceFunction
[ https://issues.apache.org/jira/browse/FLINK-32693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-32693: -- Description: * SocketTextStreamFunction (org.apache.flink.streaming.api.functions.source) * FiniteTestSource (org.apache.flink.streaming.util) * FromIteratorFunction (org.apache.flink.streaming.api.functions.source) * Generator in FileSinkProgram (org.apache.flink.connector.file.sink) * RichSourceFunction (org.apache.flink.streaming.api.functions.source) * FileMonitoringFunction (org.apache.flink.streaming.api.functions.source) * FromElementsFunction (org.apache.flink.streaming.api.functions.source) * PeriodicSourceGenerator in PeriodicStreamingJob (org.apache.flink.streaming.tests) * Generator in StreamSQLTestProgram (org.apache.flink.sql.tests) * ParallelSourceFunction (org.apache.flink.streaming.api.functions.source) * ExternallyInducedSource (org.apache.flink.streaming.api.checkpoint) * in addSource() in StreamExecutionEnvironment (org.apache.flink.streaming.api.scala) > Cascade deprecation to classes that directly implement SourceFunction > - > > Key: FLINK-32693 > URL: https://issues.apache.org/jira/browse/FLINK-32693 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common >Reporter: Alexander Fedulov >Priority: Major > > * SocketTextStreamFunction (org.apache.flink.streaming.api.functions.source) > * FiniteTestSource (org.apache.flink.streaming.util) > * FromIteratorFunction (org.apache.flink.streaming.api.functions.source) > * Generator in FileSinkProgram (org.apache.flink.connector.file.sink) > * RichSourceFunction (org.apache.flink.streaming.api.functions.source) > * FileMonitoringFunction (org.apache.flink.streaming.api.functions.source) > * FromElementsFunction (org.apache.flink.streaming.api.functions.source) > * PeriodicSourceGenerator in PeriodicStreamingJob > (org.apache.flink.streaming.tests) > * Generator in StreamSQLTestProgram (org.apache.flink.sql.tests) > * ParallelSourceFunction (org.apache.flink.streaming.api.functions.source) > * ExternallyInducedSource (org.apache.flink.streaming.api.checkpoint) > * in addSource() in StreamExecutionEnvironment > (org.apache.flink.streaming.api.scala) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-32693) Cascade deprecation to classes that directly implement SourceFunction
[ https://issues.apache.org/jira/browse/FLINK-32693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov reassigned FLINK-32693: - Assignee: Alexander Fedulov > Cascade deprecation to classes that directly implement SourceFunction > - > > Key: FLINK-32693 > URL: https://issues.apache.org/jira/browse/FLINK-32693 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > * SocketTextStreamFunction (org.apache.flink.streaming.api.functions.source) > * FiniteTestSource (org.apache.flink.streaming.util) > * FromIteratorFunction (org.apache.flink.streaming.api.functions.source) > * Generator in FileSinkProgram (org.apache.flink.connector.file.sink) > * RichSourceFunction (org.apache.flink.streaming.api.functions.source) > * FileMonitoringFunction (org.apache.flink.streaming.api.functions.source) > * FromElementsFunction (org.apache.flink.streaming.api.functions.source) > * PeriodicSourceGenerator in PeriodicStreamingJob > (org.apache.flink.streaming.tests) > * Generator in StreamSQLTestProgram (org.apache.flink.sql.tests) > * ParallelSourceFunction (org.apache.flink.streaming.api.functions.source) > * ExternallyInducedSource (org.apache.flink.streaming.api.checkpoint) > * in addSource() in StreamExecutionEnvironment > (org.apache.flink.streaming.api.scala) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32693) Cascade deprecation to classes that directly implement SourceFunction
[ https://issues.apache.org/jira/browse/FLINK-32693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-32693: -- Description: * SocketTextStreamFunction (org.apache.flink.streaming.api.functions.source) * FiniteTestSource (org.apache.flink.streaming.util) * FromIteratorFunction (org.apache.flink.streaming.api.functions.source) * Generator in FileSinkProgram (org.apache.flink.connector.file.sink) * RichSourceFunction (org.apache.flink.streaming.api.functions.source) * FileMonitoringFunction (org.apache.flink.streaming.api.functions.source) * FromElementsFunction (org.apache.flink.streaming.api.functions.source) * PeriodicSourceGenerator in PeriodicStreamingJob (org.apache.flink.streaming.tests) * Generator in StreamSQLTestProgram (org.apache.flink.sql.tests) * ParallelSourceFunction (org.apache.flink.streaming.api.functions.source) * ExternallyInducedSource (org.apache.flink.streaming.api.checkpoint) was: * SocketTextStreamFunction (org.apache.flink.streaming.api.functions.source) * FiniteTestSource (org.apache.flink.streaming.util) * FromIteratorFunction (org.apache.flink.streaming.api.functions.source) * Generator in FileSinkProgram (org.apache.flink.connector.file.sink) * RichSourceFunction (org.apache.flink.streaming.api.functions.source) * FileMonitoringFunction (org.apache.flink.streaming.api.functions.source) * FromElementsFunction (org.apache.flink.streaming.api.functions.source) * PeriodicSourceGenerator in PeriodicStreamingJob (org.apache.flink.streaming.tests) * Generator in StreamSQLTestProgram (org.apache.flink.sql.tests) * ParallelSourceFunction (org.apache.flink.streaming.api.functions.source) * ExternallyInducedSource (org.apache.flink.streaming.api.checkpoint) * in addSource() in StreamExecutionEnvironment (org.apache.flink.streaming.api.scala) > Cascade deprecation to classes that directly implement SourceFunction > - > > Key: FLINK-32693 > URL: https://issues.apache.org/jira/browse/FLINK-32693 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > > * SocketTextStreamFunction (org.apache.flink.streaming.api.functions.source) > * FiniteTestSource (org.apache.flink.streaming.util) > * FromIteratorFunction (org.apache.flink.streaming.api.functions.source) > * Generator in FileSinkProgram (org.apache.flink.connector.file.sink) > * RichSourceFunction (org.apache.flink.streaming.api.functions.source) > * FileMonitoringFunction (org.apache.flink.streaming.api.functions.source) > * FromElementsFunction (org.apache.flink.streaming.api.functions.source) > * PeriodicSourceGenerator in PeriodicStreamingJob > (org.apache.flink.streaming.tests) > * Generator in StreamSQLTestProgram (org.apache.flink.sql.tests) > * ParallelSourceFunction (org.apache.flink.streaming.api.functions.source) > * ExternallyInducedSource (org.apache.flink.streaming.api.checkpoint) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-32693) Cascade deprecation to classes that directly implement SourceFunction
Alexander Fedulov created FLINK-32693: - Summary: Cascade deprecation to classes that directly implement SourceFunction Key: FLINK-32693 URL: https://issues.apache.org/jira/browse/FLINK-32693 Project: Flink Issue Type: Sub-task Components: Connectors / Common Reporter: Alexander Fedulov -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28050) Introduce Source API alternative to SourceExecutionContext#fromElements(*) methods
[ https://issues.apache.org/jira/browse/FLINK-28050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-28050: -- Summary: Introduce Source API alternative to SourceExecutionContext#fromElements(*) methods (was: Introduce Source API alternative to SourceExecutionContext#fromCollection(*) methods) > Introduce Source API alternative to SourceExecutionContext#fromElements(*) > methods > -- > > Key: FLINK-28050 > URL: https://issues.apache.org/jira/browse/FLINK-28050 > Project: Flink > Issue Type: Sub-task > Components: API / DataStream >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28054) Introduce Source API utilities to control records emitted per checkpoint
[ https://issues.apache.org/jira/browse/FLINK-28054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-28054: -- Parent: (was: FLINK-28045) Issue Type: Improvement (was: Sub-task) > Introduce Source API utilities to control records emitted per checkpoint > > > Key: FLINK-28054 > URL: https://issues.apache.org/jira/browse/FLINK-28054 > Project: Flink > Issue Type: Improvement >Reporter: Alexander Fedulov >Priority: Major > > For some connectors, it is required to verify the content of a checkpoint > after processing specific records. See [1] for more details. This can be done > using the SourceFunction like demonstrated here [2]. An abstraction to > support this use case with the new Source API is required. > Idea: SourceReader interface has notifyCheckpointComplete() callback, so a > higher level implementation could be provided to emit records in lockstep > with checkpoints. > > [1]https://lists.apache.org/thread/9kfppzbq7r2gk5mhykc7m786sv0fc8j2 > [2]https://github.com/apache/iceberg/blob/master/flink/v1.15/flink/src/test/java/org/apache/iceberg/flink/source/BoundedTestSource.java > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28056) Introduce high level API for cases that do not require specific SourceEnumerator logic.
[ https://issues.apache.org/jira/browse/FLINK-28056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-28056: -- Parent: FLINK-32692 Issue Type: Sub-task (was: Improvement) > Introduce high level API for cases that do not require specific > SourceEnumerator logic. > --- > > Key: FLINK-28056 > URL: https://issues.apache.org/jira/browse/FLINK-28056 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common >Reporter: Alexander Fedulov >Priority: Major > > There are simple cases that do not require any specific SplitEnumerator > logic. It should be as simple to implement them with the Source API as it is > with SourceFunction. > NoOpSplitEnumerator? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28056) Introduce high level API for cases that do not require specific SourceEnumerator logic.
[ https://issues.apache.org/jira/browse/FLINK-28056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-28056: -- Parent: (was: FLINK-28045) Issue Type: Improvement (was: Sub-task) > Introduce high level API for cases that do not require specific > SourceEnumerator logic. > --- > > Key: FLINK-28056 > URL: https://issues.apache.org/jira/browse/FLINK-28056 > Project: Flink > Issue Type: Improvement > Components: Connectors / Common >Reporter: Alexander Fedulov >Priority: Major > > There are simple cases that do not require any specific SplitEnumerator > logic. It should be as simple to implement them with the Source API as it is > with SourceFunction. > NoOpSplitEnumerator? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28049) Introduce FLIP-208 functionality to stop Source based on consumed records
[ https://issues.apache.org/jira/browse/FLINK-28049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-28049: -- Parent: FLINK-32692 Issue Type: Sub-task (was: Improvement) > Introduce FLIP-208 functionality to stop Source based on consumed records > - > > Key: FLINK-28049 > URL: https://issues.apache.org/jira/browse/FLINK-28049 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common >Reporter: Alexander Fedulov >Priority: Major > Labels: pull-request-available > > https://cwiki.apache.org/confluence/x/fZbkCw -- This message was sent by Atlassian Jira (v8.20.10#820010)
