[jira] [Updated] (FLINK-33463) Support the implementation of dynamic source tables based on the new source
[ https://issues.apache.org/jira/browse/FLINK-33463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-33463: --- Labels: pull-request-available (was: ) > Support the implementation of dynamic source tables based on the new source > --- > > Key: FLINK-33463 > URL: https://issues.apache.org/jira/browse/FLINK-33463 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / JDBC >Reporter: RocMarshal >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-33463][Connector/JDBC] Support the implementation of dynamic source tables based on the new source [flink-connector-jdbc]
RocMarshal opened a new pull request, #117: URL: https://github.com/apache/flink-connector-jdbc/pull/117 - Support the implementation of dynamic source tables based on the new source -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33379) Bump CI flink version on flink-connector-elasticsearch
[ https://issues.apache.org/jira/browse/FLINK-33379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839184#comment-17839184 ] Yubin Li commented on FLINK-33379: -- [~pgodowski] sorry for the late reply, it seems that 3.1.0 branch not created yet, you could compile using master branch to support Flink 1.18 > Bump CI flink version on flink-connector-elasticsearch > -- > > Key: FLINK-33379 > URL: https://issues.apache.org/jira/browse/FLINK-33379 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: elasticsearch-3.1.0 >Reporter: Yubin Li >Assignee: Yubin Li >Priority: Major > Labels: pull-request-available > Fix For: elasticsearch-3.1.0 > > Attachments: image-2023-10-27-16-54-04-937.png > > > As Flink 1.18 released, bump the flink version in es connector . -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34918) Introduce comment for Catalog
[ https://issues.apache.org/jira/browse/FLINK-34918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yubin Li updated FLINK-34918: - Summary: Introduce comment for Catalog (was: Introduce comment support for Catalog) > Introduce comment for Catalog > - > > Key: FLINK-34918 > URL: https://issues.apache.org/jira/browse/FLINK-34918 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Affects Versions: 1.20.0 >Reporter: Yubin Li >Assignee: Yubin Li >Priority: Major > > We propose to introduce `getComment()` method in `CatalogDescriptor`, and the > reasons are as follows. > 1. For the sake of design consistency, follow the design of FLIP-295 [1] > which introduced `CatalogStore` component, `CatalogDescriptor` includes names > and attributes, both of which are used to describe the catalog, and `comment` > can be added smoothly. > 2. Extending the existing class rather than add new method to the existing > interface, Especially, the `Catalog` interface, as a core interface, is used > by a series of important components such as `CatalogFactory`, > `CatalogManager` and `FactoryUtil`, and is implemented by a large number of > connectors such as JDBC, Paimon, and Hive. Adding methods to it will greatly > increase the implementation complexity, and more importantly, increase the > cost of iteration, maintenance, and verification. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35177] Fix DataGen Connector documentation [flink]
flinkbot commented on PR #24692: URL: https://github.com/apache/flink/pull/24692#issuecomment-2067479443 ## CI report: * 4d0723dfc64326e5fab53b2b59c495f341a865df UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-35177) Datagen examples in documentation do not compile
[
https://issues.apache.org/jira/browse/FLINK-35177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35177:
---
Labels: pull-request-available (was: )
> Datagen examples in documentation do not compile
>
>
> Key: FLINK-35177
> URL: https://issues.apache.org/jira/browse/FLINK-35177
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.19.0
>Reporter: Sergei Morozov
>Priority: Not a Priority
> Labels: pull-request-available
>
> Currently, the
> [examples|https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/connectors/datastream/datagen/#rate-limiting]
> look like this:
> {code:java}
> GeneratorFunction generatorFunction = index -> index;
> double recordsPerSecond = 100;
> DataGeneratorSource source =
> new DataGeneratorSource<>(
> generatorFunction,
> Long.MAX_VALUE,
> RateLimiterStrategy.perSecond(recordsPerSecond),
> Types.STRING);
> {code}
> The generator function returns Long but the DataGeneratorSource uses String,
> so their types do not match.
> Either the generator function needs to be changed to return a string, or the
> source needs to use Long.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] [FLINK-35177] Fix DataGen Connector documentation [flink]
morozov opened a new pull request, #24692: URL: https://github.com/apache/flink/pull/24692 ## What is the purpose of the change The code [examples](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/connectors/datastream/datagen/#rate-limiting) used in the documentation are incorrect. Currently, they look like this: ```java GeneratorFunction generatorFunction = index -> index; double recordsPerSecond = 100; DataGeneratorSource source = new DataGeneratorSource<>( generatorFunction, Long.MAX_VALUE, RateLimiterStrategy.perSecond(recordsPerSecond), Types.STRING); ``` The generator function returns `Long `but the `DataGeneratorSource` uses `String` and `Types.STRING`, so the types do not match, and the example code cannot be compiled. ## Brief change log The types used by the `DataGeneratorSource` are updated to match the return type of the generator function. ## Verifying this change Please make sure both new and modified tests in this PR follow [the conventions for tests defined in our code quality guide](https://flink.apache.org/how-to-contribute/code-style-and-quality-common/#7-testing). This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-35177) Datagen examples in documentation do not compile
[
https://issues.apache.org/jira/browse/FLINK-35177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergei Morozov updated FLINK-35177:
---
Description:
Currently, the
[examples|https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/connectors/datastream/datagen/#rate-limiting]
look like this:
{code:java}
GeneratorFunction generatorFunction = index -> index;
double recordsPerSecond = 100;
DataGeneratorSource source =
new DataGeneratorSource<>(
generatorFunction,
Long.MAX_VALUE,
RateLimiterStrategy.perSecond(recordsPerSecond),
Types.STRING);
{code}
The generator function returns Long but the DataGeneratorSource uses String, so
their types do not match.
Either the generator function needs to be changed to return a string, or the
source needs to use Long.
was:
Currently, the examples look like this:
{code}
GeneratorFunction generatorFunction = index -> index;
double recordsPerSecond = 100;
DataGeneratorSource source =
new DataGeneratorSource<>(
generatorFunction,
Long.MAX_VALUE,
RateLimiterStrategy.perSecond(recordsPerSecond),
Types.STRING);
{code}
The generator function returns Long but the DataGeneratorSource uses String, so
their types do not match.
Either the generator function needs to be changed to return a string, or the
source needs to use Long.
> Datagen examples in documentation do not compile
>
>
> Key: FLINK-35177
> URL: https://issues.apache.org/jira/browse/FLINK-35177
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.19.0
>Reporter: Sergei Morozov
>Priority: Not a Priority
>
> Currently, the
> [examples|https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/connectors/datastream/datagen/#rate-limiting]
> look like this:
> {code:java}
> GeneratorFunction generatorFunction = index -> index;
> double recordsPerSecond = 100;
> DataGeneratorSource source =
> new DataGeneratorSource<>(
> generatorFunction,
> Long.MAX_VALUE,
> RateLimiterStrategy.perSecond(recordsPerSecond),
> Types.STRING);
> {code}
> The generator function returns Long but the DataGeneratorSource uses String,
> so their types do not match.
> Either the generator function needs to be changed to return a string, or the
> source needs to use Long.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-35177) Datagen examples in documentation do not compile
Sergei Morozov created FLINK-35177:
--
Summary: Datagen examples in documentation do not compile
Key: FLINK-35177
URL: https://issues.apache.org/jira/browse/FLINK-35177
Project: Flink
Issue Type: Bug
Components: Documentation
Affects Versions: 1.19.0
Reporter: Sergei Morozov
Currently, the examples look like this:
{code}
GeneratorFunction generatorFunction = index -> index;
double recordsPerSecond = 100;
DataGeneratorSource source =
new DataGeneratorSource<>(
generatorFunction,
Long.MAX_VALUE,
RateLimiterStrategy.perSecond(recordsPerSecond),
Types.STRING);
{code}
The generator function returns Long but the DataGeneratorSource uses String, so
their types do not match.
Either the generator function needs to be changed to return a string, or the
source needs to use Long.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-35175][filesystem] Avoid reading with ByteBuffer on hadoop version below 3.3.0 [flink]
masteryhx commented on PR #24691: URL: https://github.com/apache/flink/pull/24691#issuecomment-2066998011 @Zakelly @RyanSkraba Could you help to take a look ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-35175) HadoopDataInputStream can't compile with Hadoop 3.2.3
[
https://issues.apache.org/jira/browse/FLINK-35175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839068#comment-17839068
]
Hangxiang Yu commented on FLINK-35175:
--
[~rskraba] Thanks for reporting this!
I think it's better if we could upgrade the hadoop version, and I also saw
before ticket about it: https://issues.apache.org/jira/browse/FLINK-33584
But seems there still are some blockers.
So I'd prefer to prepare a PR to remove the optimization currently and add TODO
here.
I will also try to help to push forward the upgradation.
Thanks again.
> HadoopDataInputStream can't compile with Hadoop 3.2.3
> -
>
> Key: FLINK-35175
> URL: https://issues.apache.org/jira/browse/FLINK-35175
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.20.0
>Reporter: Ryan Skraba
>Priority: Critical
> Labels: pull-request-available
>
> Unfortunately, introduced in FLINK-35045:
> [PREADWRITEBUFFER|https://github.com/apache/flink/commit/a312a3bdd258e0ff7d6f94e979b32e2bc762b82f#diff-3ed57be01895ba0f792110e40f4283427c55528f11a5105b4bf34ebd4e6fef0dR182]
> was added in Hadoop releases
> [3.3.0|https://github.com/apache/hadoop/blob/rel/release-3.3.0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StreamCapabilities.java#L72]
> and
> [2.10.0|https://github.com/apache/hadoop/blob/rel/release-2.10.0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StreamCapabilities.java#L72].
> It doesn't exist in flink.hadoop.version
> [3.2.3|https://github.com/apache/hadoop/blob/rel/release-3.2.3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StreamCapabilities.java],
> which we are using in end-to-end tests.
> {code:java}
> 00:23:55.093 [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile
> (default-compile) on project flink-hadoop-fs: Compilation failure:
> Compilation failure:
> 00:23:55.093 [ERROR]
> /home/vsts/work/1/s/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopDataInputStream.java:[151,63]
> cannot find symbol
> 00:23:55.094 [ERROR] symbol: variable READBYTEBUFFER
> 00:23:55.094 [ERROR] location: interface
> org.apache.hadoop.fs.StreamCapabilities
> 00:23:55.094 [ERROR]
> /home/vsts/work/1/s/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopDataInputStream.java:[182,63]
> cannot find symbol
> 00:23:55.094 [ERROR] symbol: variable PREADBYTEBUFFER
> 00:23:55.094 [ERROR] location: interface
> org.apache.hadoop.fs.StreamCapabilities
> 00:23:55.094 [ERROR]
> /home/vsts/work/1/s/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopDataInputStream.java:[183,43]
> incompatible types: long cannot be converted to
> org.apache.hadoop.io.ByteBufferPool
> 00:23:55.094 [ERROR] -> [Help 1] {code}
> * 1.20 compile_cron_hadoop313
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59012=logs=87489130-75dc-54e4-1f45-80c30aa367a3=73da6d75-f30d-5d5a-acbe-487a9dcff678=3630
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-35175][filesystem] Avoid reading with ByteBuffer on hadoop version below 3.3.0 [flink]
flinkbot commented on PR #24691: URL: https://github.com/apache/flink/pull/24691#issuecomment-2066992465 ## CI report: * 38765977f787a0f88f247cd9ccfa7660f1c4d80b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-35175) HadoopDataInputStream can't compile with Hadoop 3.2.3
[
https://issues.apache.org/jira/browse/FLINK-35175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35175:
---
Labels: pull-request-available (was: )
> HadoopDataInputStream can't compile with Hadoop 3.2.3
> -
>
> Key: FLINK-35175
> URL: https://issues.apache.org/jira/browse/FLINK-35175
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.20.0
>Reporter: Ryan Skraba
>Priority: Critical
> Labels: pull-request-available
>
> Unfortunately, introduced in FLINK-35045:
> [PREADWRITEBUFFER|https://github.com/apache/flink/commit/a312a3bdd258e0ff7d6f94e979b32e2bc762b82f#diff-3ed57be01895ba0f792110e40f4283427c55528f11a5105b4bf34ebd4e6fef0dR182]
> was added in Hadoop releases
> [3.3.0|https://github.com/apache/hadoop/blob/rel/release-3.3.0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StreamCapabilities.java#L72]
> and
> [2.10.0|https://github.com/apache/hadoop/blob/rel/release-2.10.0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StreamCapabilities.java#L72].
> It doesn't exist in flink.hadoop.version
> [3.2.3|https://github.com/apache/hadoop/blob/rel/release-3.2.3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StreamCapabilities.java],
> which we are using in end-to-end tests.
> {code:java}
> 00:23:55.093 [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile
> (default-compile) on project flink-hadoop-fs: Compilation failure:
> Compilation failure:
> 00:23:55.093 [ERROR]
> /home/vsts/work/1/s/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopDataInputStream.java:[151,63]
> cannot find symbol
> 00:23:55.094 [ERROR] symbol: variable READBYTEBUFFER
> 00:23:55.094 [ERROR] location: interface
> org.apache.hadoop.fs.StreamCapabilities
> 00:23:55.094 [ERROR]
> /home/vsts/work/1/s/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopDataInputStream.java:[182,63]
> cannot find symbol
> 00:23:55.094 [ERROR] symbol: variable PREADBYTEBUFFER
> 00:23:55.094 [ERROR] location: interface
> org.apache.hadoop.fs.StreamCapabilities
> 00:23:55.094 [ERROR]
> /home/vsts/work/1/s/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopDataInputStream.java:[183,43]
> incompatible types: long cannot be converted to
> org.apache.hadoop.io.ByteBufferPool
> 00:23:55.094 [ERROR] -> [Help 1] {code}
> * 1.20 compile_cron_hadoop313
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59012=logs=87489130-75dc-54e4-1f45-80c30aa367a3=73da6d75-f30d-5d5a-acbe-487a9dcff678=3630
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] [FLINK-35175][filesystem] Avoid reading with ByteBuffer on hadoop version below 3.3.0 [flink]
masteryhx opened a new pull request, #24691: URL: https://github.com/apache/flink/pull/24691 ## What is the purpose of the change Hadoop whose version below 3.3.0 doesn't support reading and positionable reading with ByteBuffer, which may cause compile error. See https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59012=logs=87489130-75dc-54e4-1f45-80c30aa367a3=73da6d75-f30d-5d5a-acbe-487a9dcff678=3630 ## Brief change log - Remove support about reading and positionable reading with ByteBuffer, and add TODO. ## Verifying this change This change could be verified by current CI. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-28048][connectors] Introduce Source API alternative to FiniteTestSource [flink]
tweise merged PR #23777: URL: https://github.com/apache/flink/pull/23777 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [FLINK-33460][Connector/JDBC] Support property authentication connection for JDBC catalog & dynamic table [flink-connector-jdbc]
RocMarshal opened a new pull request, #116: URL: https://github.com/apache/flink-connector-jdbc/pull/116 - Support property authentication connection for JDBC catalog & dynamic table -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-34574) Add CPU and memory size autoscaler quota
[ https://issues.apache.org/jira/browse/FLINK-34574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora closed FLINK-34574. -- Fix Version/s: kubernetes-operator-1.9.0 Resolution: Fixed merged to main baad90088ea5b5b240186a530a79b64fb84cc77e > Add CPU and memory size autoscaler quota > > > Key: FLINK-34574 > URL: https://issues.apache.org/jira/browse/FLINK-34574 > Project: Flink > Issue Type: New Feature > Components: Autoscaler >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > Labels: pull-request-available > Fix For: kubernetes-operator-1.9.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34574] Add CPU and memory size autoscaler quota [flink-kubernetes-operator]
gyfora merged PR #789: URL: https://github.com/apache/flink-kubernetes-operator/pull/789 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-35176) Support property authentication connection for JDBC catalog & dynamic table
[ https://issues.apache.org/jira/browse/FLINK-35176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RocMarshal updated FLINK-35176: --- Parent: FLINK-25420 Issue Type: Sub-task (was: Improvement) > Support property authentication connection for JDBC catalog & dynamic table > --- > > Key: FLINK-35176 > URL: https://issues.apache.org/jira/browse/FLINK-35176 > Project: Flink > Issue Type: Sub-task > Components: Connectors / JDBC >Reporter: RocMarshal >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35176) Support property authentication connection for JDBC catalog & dynamic table
RocMarshal created FLINK-35176: -- Summary: Support property authentication connection for JDBC catalog & dynamic table Key: FLINK-35176 URL: https://issues.apache.org/jira/browse/FLINK-35176 Project: Flink Issue Type: Improvement Components: Connectors / JDBC Reporter: RocMarshal -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-35159) CreatingExecutionGraph can leak CheckpointCoordinator and cause JM crash
[
https://issues.apache.org/jira/browse/FLINK-35159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-35159.
Resolution: Fixed
> CreatingExecutionGraph can leak CheckpointCoordinator and cause JM crash
>
>
> Key: FLINK-35159
> URL: https://issues.apache.org/jira/browse/FLINK-35159
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.18.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.2, 1.20.0, 1.19.1
>
>
> When a task manager dies while the JM is generating an ExecutionGraph in the
> background then {{CreatingExecutionGraph#handleExecutionGraphCreation}} can
> transition back into WaitingForResources if the TM hosted one of the slots
> that we planned to use in {{tryToAssignSlots}}.
> At this point the ExecutionGraph was already transitioned to running, which
> implicitly kicks of periodic checkpointing by the CheckpointCoordinator,
> without the operator coordinator holders being initialized yet (as this
> happens after we assigned slots).
> This effectively leaks that CheckpointCoordinator, including the timer thread
> that will continue to try triggering checkpoints, which will naturally fail
> to trigger.
> This can cause a JM crash because it results in
> {{OperatorCoordinatorHolder#abortCurrentTriggering}} to be called, which
> fails with an NPE since the {{mainThreadExecutor}} was not initialized yet.
> {code}
> java.util.concurrent.CompletionException:
> java.util.concurrent.CompletionException: java.lang.NullPointerException
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.lambda$startTriggeringCheckpoint$8(CheckpointCoordinator.java:707)
> at
> java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:986)
> at
> java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:970)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:610)
> at
> java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:910)
> at
> java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: java.util.concurrent.CompletionException:
> java.lang.NullPointerException
> at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
> at
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)
> at
> java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:932)
> at
> java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907)
> ... 7 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder.abortCurrentTriggering(OperatorCoordinatorHolder.java:388)
> at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
> at
> java.base/java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1085)
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.onTriggerFailure(CheckpointCoordinator.java:985)
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.onTriggerFailure(CheckpointCoordinator.java:961)
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.lambda$startTriggeringCheckpoint$7(CheckpointCoordinator.java:693)
> at
> java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930)
> ... 8 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-35159) CreatingExecutionGraph can leak CheckpointCoordinator and cause JM crash
[
https://issues.apache.org/jira/browse/FLINK-35159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17838908#comment-17838908
]
Chesnay Schepler edited comment on FLINK-35159 at 4/19/24 3:03 PM:
---
master: 131358b918ba511064c23dfee96969a628d07d2a
1.19: 87ed9ccc2103457ba91f6ca45adfd2bfcc75c9ac
1.18: aacc735806acf1d63fa732706e079bc2ca1bb4fc
was (Author: zentol):
master: 131358b918ba511064c23dfee96969a628d07d2a
1.19:
1.18: aacc735806acf1d63fa732706e079bc2ca1bb4fc
> CreatingExecutionGraph can leak CheckpointCoordinator and cause JM crash
>
>
> Key: FLINK-35159
> URL: https://issues.apache.org/jira/browse/FLINK-35159
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.18.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.2, 1.20.0, 1.19.1
>
>
> When a task manager dies while the JM is generating an ExecutionGraph in the
> background then {{CreatingExecutionGraph#handleExecutionGraphCreation}} can
> transition back into WaitingForResources if the TM hosted one of the slots
> that we planned to use in {{tryToAssignSlots}}.
> At this point the ExecutionGraph was already transitioned to running, which
> implicitly kicks of periodic checkpointing by the CheckpointCoordinator,
> without the operator coordinator holders being initialized yet (as this
> happens after we assigned slots).
> This effectively leaks that CheckpointCoordinator, including the timer thread
> that will continue to try triggering checkpoints, which will naturally fail
> to trigger.
> This can cause a JM crash because it results in
> {{OperatorCoordinatorHolder#abortCurrentTriggering}} to be called, which
> fails with an NPE since the {{mainThreadExecutor}} was not initialized yet.
> {code}
> java.util.concurrent.CompletionException:
> java.util.concurrent.CompletionException: java.lang.NullPointerException
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.lambda$startTriggeringCheckpoint$8(CheckpointCoordinator.java:707)
> at
> java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:986)
> at
> java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:970)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:610)
> at
> java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:910)
> at
> java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: java.util.concurrent.CompletionException:
> java.lang.NullPointerException
> at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
> at
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)
> at
> java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:932)
> at
> java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907)
> ... 7 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder.abortCurrentTriggering(OperatorCoordinatorHolder.java:388)
> at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
> at
> java.base/java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1085)
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.onTriggerFailure(CheckpointCoordinator.java:985)
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.onTriggerFailure(CheckpointCoordinator.java:961)
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.lambda$startTriggeringCheckpoint$7(CheckpointCoordinator.java:693)
> at
> java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930)
> ... 8 more
> {code}
--
This
Re: [PR] [BP-1.19][FLINK-35159] Transition ExecutionGraph to RUNNING after slot assignment [flink]
zentol merged PR #24686: URL: https://github.com/apache/flink/pull/24686 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-28048][connectors] Introduce Source API alternative to FiniteTestSource [flink]
afedulov commented on PR #23777: URL: https://github.com/apache/flink/pull/23777#issuecomment-2066761675 CI is green: https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58998=results -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33460][Connector/JDBC] Support property authentication connection. [flink-connector-jdbc]
RocMarshal commented on code in PR #115:
URL:
https://github.com/apache/flink-connector-jdbc/pull/115#discussion_r1572398050
##
flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/JdbcConnectionOptions.java:
##
@@ -90,13 +98,31 @@ public JdbcConnectionOptionsBuilder withDriverName(String
driverName) {
return this;
}
+public JdbcConnectionOptionsBuilder withProperties(Properties
properties) {
Review Comment:
Thanks @eskabetxe for the comments.
That sounds nice to me
updated~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33460][Connector/JDBC] Support property authentication connection. [flink-connector-jdbc]
RocMarshal commented on code in PR #115:
URL:
https://github.com/apache/flink-connector-jdbc/pull/115#discussion_r1572398050
##
flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/JdbcConnectionOptions.java:
##
@@ -90,13 +98,31 @@ public JdbcConnectionOptionsBuilder withDriverName(String
driverName) {
return this;
}
+public JdbcConnectionOptionsBuilder withProperties(Properties
properties) {
Review Comment:
Thanks for the comments.
That sounds nice to me
updated~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Comment Edited] (FLINK-35159) CreatingExecutionGraph can leak CheckpointCoordinator and cause JM crash
[
https://issues.apache.org/jira/browse/FLINK-35159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17838908#comment-17838908
]
Chesnay Schepler edited comment on FLINK-35159 at 4/19/24 1:40 PM:
---
master: 131358b918ba511064c23dfee96969a628d07d2a
1.19:
1.18: aacc735806acf1d63fa732706e079bc2ca1bb4fc
was (Author: zentol):
master: 131358b918ba511064c23dfee96969a628d07d2a
1.19:
1.18:
> CreatingExecutionGraph can leak CheckpointCoordinator and cause JM crash
>
>
> Key: FLINK-35159
> URL: https://issues.apache.org/jira/browse/FLINK-35159
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.18.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.2, 1.20.0, 1.19.1
>
>
> When a task manager dies while the JM is generating an ExecutionGraph in the
> background then {{CreatingExecutionGraph#handleExecutionGraphCreation}} can
> transition back into WaitingForResources if the TM hosted one of the slots
> that we planned to use in {{tryToAssignSlots}}.
> At this point the ExecutionGraph was already transitioned to running, which
> implicitly kicks of periodic checkpointing by the CheckpointCoordinator,
> without the operator coordinator holders being initialized yet (as this
> happens after we assigned slots).
> This effectively leaks that CheckpointCoordinator, including the timer thread
> that will continue to try triggering checkpoints, which will naturally fail
> to trigger.
> This can cause a JM crash because it results in
> {{OperatorCoordinatorHolder#abortCurrentTriggering}} to be called, which
> fails with an NPE since the {{mainThreadExecutor}} was not initialized yet.
> {code}
> java.util.concurrent.CompletionException:
> java.util.concurrent.CompletionException: java.lang.NullPointerException
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.lambda$startTriggeringCheckpoint$8(CheckpointCoordinator.java:707)
> at
> java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:986)
> at
> java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:970)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:610)
> at
> java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:910)
> at
> java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: java.util.concurrent.CompletionException:
> java.lang.NullPointerException
> at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
> at
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)
> at
> java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:932)
> at
> java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907)
> ... 7 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder.abortCurrentTriggering(OperatorCoordinatorHolder.java:388)
> at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
> at
> java.base/java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1085)
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.onTriggerFailure(CheckpointCoordinator.java:985)
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.onTriggerFailure(CheckpointCoordinator.java:961)
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.lambda$startTriggeringCheckpoint$7(CheckpointCoordinator.java:693)
> at
> java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930)
> ... 8 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [BP-1.19][FLINK-35159] Transition ExecutionGraph to RUNNING after slot assignment [flink]
zentol commented on PR #24686: URL: https://github.com/apache/flink/pull/24686#issuecomment-2066614601 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [BP-1.18][FLINK-35159] Transition ExecutionGraph to RUNNING after slot assignment [flink]
zentol merged PR #24687: URL: https://github.com/apache/flink/pull/24687 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-35175) HadoopDataInputStream can't compile with Hadoop 3.2.3
[
https://issues.apache.org/jira/browse/FLINK-35175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839001#comment-17839001
]
Ryan Skraba commented on FLINK-35175:
-
1.20 Hadoop 3.1.3 / Compile
https://github.com/apache/flink/actions/runs/8747381080/job/24005737445#step:6:1560
> HadoopDataInputStream can't compile with Hadoop 3.2.3
> -
>
> Key: FLINK-35175
> URL: https://issues.apache.org/jira/browse/FLINK-35175
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.20.0
>Reporter: Ryan Skraba
>Priority: Critical
>
> Unfortunately, introduced in FLINK-35045:
> [PREADWRITEBUFFER|https://github.com/apache/flink/commit/a312a3bdd258e0ff7d6f94e979b32e2bc762b82f#diff-3ed57be01895ba0f792110e40f4283427c55528f11a5105b4bf34ebd4e6fef0dR182]
> was added in Hadoop releases
> [3.3.0|https://github.com/apache/hadoop/blob/rel/release-3.3.0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StreamCapabilities.java#L72]
> and
> [2.10.0|https://github.com/apache/hadoop/blob/rel/release-2.10.0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StreamCapabilities.java#L72].
> It doesn't exist in flink.hadoop.version
> [3.2.3|https://github.com/apache/hadoop/blob/rel/release-3.2.3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StreamCapabilities.java],
> which we are using in end-to-end tests.
> {code:java}
> 00:23:55.093 [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile
> (default-compile) on project flink-hadoop-fs: Compilation failure:
> Compilation failure:
> 00:23:55.093 [ERROR]
> /home/vsts/work/1/s/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopDataInputStream.java:[151,63]
> cannot find symbol
> 00:23:55.094 [ERROR] symbol: variable READBYTEBUFFER
> 00:23:55.094 [ERROR] location: interface
> org.apache.hadoop.fs.StreamCapabilities
> 00:23:55.094 [ERROR]
> /home/vsts/work/1/s/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopDataInputStream.java:[182,63]
> cannot find symbol
> 00:23:55.094 [ERROR] symbol: variable PREADBYTEBUFFER
> 00:23:55.094 [ERROR] location: interface
> org.apache.hadoop.fs.StreamCapabilities
> 00:23:55.094 [ERROR]
> /home/vsts/work/1/s/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopDataInputStream.java:[183,43]
> incompatible types: long cannot be converted to
> org.apache.hadoop.io.ByteBufferPool
> 00:23:55.094 [ERROR] -> [Help 1] {code}
> * 1.20 compile_cron_hadoop313
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59012=logs=87489130-75dc-54e4-1f45-80c30aa367a3=73da6d75-f30d-5d5a-acbe-487a9dcff678=3630
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34227) Job doesn't disconnect from ResourceManager
[
https://issues.apache.org/jira/browse/FLINK-34227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839002#comment-17839002
]
Ryan Skraba commented on FLINK-34227:
-
1.18 AdaptiveScheduler / Test (module: table)
https://github.com/apache/flink/actions/runs/8747381390/job/24006045095#step:10:12211
> Job doesn't disconnect from ResourceManager
> ---
>
> Key: FLINK-34227
> URL: https://issues.apache.org/jira/browse/FLINK-34227
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.19.0, 1.18.1
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Critical
> Labels: github-actions, pull-request-available, test-stability
> Attachments: FLINK-34227.7e7d69daebb438b8d03b7392c9c55115.log,
> FLINK-34227.log
>
>
> https://github.com/XComp/flink/actions/runs/7634987973/job/20800205972#step:10:14557
> {code}
> [...]
> "main" #1 prio=5 os_prio=0 tid=0x7f4b7000 nid=0x24ec0 waiting on
> condition [0x7fccce1eb000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xbdd52618> (a
> java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:2131)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:2099)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:2077)
> at
> org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:876)
> at
> org.apache.flink.table.planner.runtime.stream.sql.WindowDistinctAggregateITCase.testHopWindow_Cube(WindowDistinctAggregateITCase.scala:550)
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-18476) PythonEnvUtilsTest#testStartPythonProcess fails
[
https://issues.apache.org/jira/browse/FLINK-18476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839003#comment-17839003
]
Ryan Skraba commented on FLINK-18476:
-
1.19 Java 11 / Test (module: misc)
https://github.com/apache/flink/actions/runs/8747381464/job/24006015161#step:10:21789
> PythonEnvUtilsTest#testStartPythonProcess fails
> ---
>
> Key: FLINK-18476
> URL: https://issues.apache.org/jira/browse/FLINK-18476
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Tests
>Affects Versions: 1.11.0, 1.15.3, 1.18.0, 1.19.0, 1.20.0
>Reporter: Dawid Wysakowicz
>Priority: Major
> Labels: auto-deprioritized-major, auto-deprioritized-minor,
> test-stability
>
> The
> {{org.apache.flink.client.python.PythonEnvUtilsTest#testStartPythonProcess}}
> failed in my local environment as it assumes the environment has
> {{/usr/bin/python}}.
> I don't know exactly how did I get python in Ubuntu 20.04, but I have only
> alias for {{python = python3}}. Therefore the tests fails.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-35175) HadoopDataInputStream can't compile with Hadoop 3.2.3
[
https://issues.apache.org/jira/browse/FLINK-35175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17838999#comment-17838999
]
Ryan Skraba commented on FLINK-35175:
-
[~hangxiang]: what do you think? Is it better to bump the supported version of
Hadoop or fix the class that relies on this capability?
> HadoopDataInputStream can't compile with Hadoop 3.2.3
> -
>
> Key: FLINK-35175
> URL: https://issues.apache.org/jira/browse/FLINK-35175
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.20.0
>Reporter: Ryan Skraba
>Priority: Critical
>
> Unfortunately, introduced in FLINK-35045:
> [PREADWRITEBUFFER|https://github.com/apache/flink/commit/a312a3bdd258e0ff7d6f94e979b32e2bc762b82f#diff-3ed57be01895ba0f792110e40f4283427c55528f11a5105b4bf34ebd4e6fef0dR182]
> was added in Hadoop releases
> [3.3.0|https://github.com/apache/hadoop/blob/rel/release-3.3.0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StreamCapabilities.java#L72]
> and
> [2.10.0|https://github.com/apache/hadoop/blob/rel/release-2.10.0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StreamCapabilities.java#L72].
> It doesn't exist in flink.hadoop.version
> [3.2.3|https://github.com/apache/hadoop/blob/rel/release-3.2.3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StreamCapabilities.java],
> which we are using in end-to-end tests.
> {code:java}
> 00:23:55.093 [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile
> (default-compile) on project flink-hadoop-fs: Compilation failure:
> Compilation failure:
> 00:23:55.093 [ERROR]
> /home/vsts/work/1/s/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopDataInputStream.java:[151,63]
> cannot find symbol
> 00:23:55.094 [ERROR] symbol: variable READBYTEBUFFER
> 00:23:55.094 [ERROR] location: interface
> org.apache.hadoop.fs.StreamCapabilities
> 00:23:55.094 [ERROR]
> /home/vsts/work/1/s/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopDataInputStream.java:[182,63]
> cannot find symbol
> 00:23:55.094 [ERROR] symbol: variable PREADBYTEBUFFER
> 00:23:55.094 [ERROR] location: interface
> org.apache.hadoop.fs.StreamCapabilities
> 00:23:55.094 [ERROR]
> /home/vsts/work/1/s/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopDataInputStream.java:[183,43]
> incompatible types: long cannot be converted to
> org.apache.hadoop.io.ByteBufferPool
> 00:23:55.094 [ERROR] -> [Help 1] {code}
> * 1.20 compile_cron_hadoop313
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59012=logs=87489130-75dc-54e4-1f45-80c30aa367a3=73da6d75-f30d-5d5a-acbe-487a9dcff678=3630
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-35045][state] Introduce ForStFlinkFileSystem to support reading and writing with ByteBuffer [flink]
RyanSkraba commented on code in PR #24632:
URL: https://github.com/apache/flink/pull/24632#discussion_r1572359539
##
flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopDataInputStream.java:
##
@@ -140,4 +144,56 @@ public void skipFully(long bytes) throws IOException {
bytes -= fsDataInputStream.skip(bytes);
}
}
+
+@Override
+public int read(ByteBuffer byteBuffer) throws IOException {
+// Not all internal stream supports ByteBufferReadable
+if
(fsDataInputStream.hasCapability(StreamCapabilities.READBYTEBUFFER)) {
+return fsDataInputStream.read(byteBuffer);
+} else {
+if (byteBuffer.hasArray()) {
+int len = byteBuffer.remaining();
+fsDataInputStream.readFully(byteBuffer.array(),
byteBuffer.arrayOffset(), len);
+return len;
+} else {
+// Fallback to read byte then put
+int c = read();
+if (c == -1) {
+return -1;
+}
+byteBuffer.put((byte) c);
+
+int n = 1, len = byteBuffer.remaining() + 1;
+for (; n < len; n++) {
+c = read();
+if (c == -1) {
+break;
+}
+byteBuffer.put((byte) c);
+}
+return n;
+}
+}
+}
+
+@Override
+public int read(long position, ByteBuffer byteBuffer) throws IOException {
+// Not all internal stream supports ByteBufferPositionedReadable
+if
(fsDataInputStream.hasCapability(StreamCapabilities.PREADBYTEBUFFER)) {
Review Comment:
Hello! Unfortunately this causes a compile error when using Hadoop 3.2.3
(found in the nightly builds). I created FLINK-35175 to address this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-35175) HadoopDataInputStream can't compile with Hadoop 3.2.3
Ryan Skraba created FLINK-35175:
---
Summary: HadoopDataInputStream can't compile with Hadoop 3.2.3
Key: FLINK-35175
URL: https://issues.apache.org/jira/browse/FLINK-35175
Project: Flink
Issue Type: Bug
Affects Versions: 1.20.0
Reporter: Ryan Skraba
Unfortunately, introduced in FLINK-35045:
[PREADWRITEBUFFER|https://github.com/apache/flink/commit/a312a3bdd258e0ff7d6f94e979b32e2bc762b82f#diff-3ed57be01895ba0f792110e40f4283427c55528f11a5105b4bf34ebd4e6fef0dR182]
was added in Hadoop releases
[3.3.0|https://github.com/apache/hadoop/blob/rel/release-3.3.0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StreamCapabilities.java#L72]
and
[2.10.0|https://github.com/apache/hadoop/blob/rel/release-2.10.0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StreamCapabilities.java#L72].
It doesn't exist in flink.hadoop.version
[3.2.3|https://github.com/apache/hadoop/blob/rel/release-3.2.3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StreamCapabilities.java],
which we are using in end-to-end tests.
{code:java}
00:23:55.093 [ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile)
on project flink-hadoop-fs: Compilation failure: Compilation failure:
00:23:55.093 [ERROR]
/home/vsts/work/1/s/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopDataInputStream.java:[151,63]
cannot find symbol
00:23:55.094 [ERROR] symbol: variable READBYTEBUFFER
00:23:55.094 [ERROR] location: interface
org.apache.hadoop.fs.StreamCapabilities
00:23:55.094 [ERROR]
/home/vsts/work/1/s/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopDataInputStream.java:[182,63]
cannot find symbol
00:23:55.094 [ERROR] symbol: variable PREADBYTEBUFFER
00:23:55.094 [ERROR] location: interface
org.apache.hadoop.fs.StreamCapabilities
00:23:55.094 [ERROR]
/home/vsts/work/1/s/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopDataInputStream.java:[183,43]
incompatible types: long cannot be converted to
org.apache.hadoop.io.ByteBufferPool
00:23:55.094 [ERROR] -> [Help 1] {code}
* 1.20 compile_cron_hadoop313
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59012=logs=87489130-75dc-54e4-1f45-80c30aa367a3=73da6d75-f30d-5d5a-acbe-487a9dcff678=3630
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-35026][runtime][config] Introduce async execution configurations [flink]
yunfengzhou-hub commented on code in PR #24667:
URL: https://github.com/apache/flink/pull/24667#discussion_r1572344539
##
flink-core/src/main/java/org/apache/flink/configuration/ExecutionOptions.java:
##
@@ -181,4 +182,62 @@ public class ExecutionOptions {
+ " operators. NOTE: It takes effect only
in the BATCH runtime mode and requires sorted inputs"
+ SORT_INPUTS.key()
+ " to be enabled.");
+
+// - Async State Execution
--
+
+/**
+ * The max limit of in-flight records number in async state execution,
'in-flight' refers to the
+ * records that have entered the operator but have not yet been processed
and emitted to the
+ * downstream. If the in-flight records number exceeds the limit, the
newly records entering
+ * will be blocked until the in-flight records number drops below the
limit.
+ */
+@Experimental
+@Documentation.ExcludeFromDocumentation(
+"This is an experimental option, internal use only for now.")
+public static final ConfigOption ASYNC_INFLIGHT_RECORDS_LIMIT =
+ConfigOptions.key("execution.async-state.in-flight-records-limit")
+.intType()
+.defaultValue(6000)
+.withDescription(
+"The max limit of in-flight records number in
async state execution, 'in-flight' refers"
++ " to the records that have entered the
operator but have not yet been processed and"
++ " emitted to the downstream. If the
in-flight records number exceeds the limit,"
++ " the newly records entering will be
blocked until the in-flight records number drops below the limit.");
+
+/**
+ * The size of buffer under async state execution. Async state execution
provides a buffer
+ * mechanism to reduce state access. When the number of state requests in
the buffer exceeds the
+ * batch size, a batched state execution would be triggered. Larger batch
sizes will bring
+ * higher end-to-end latency, this option works with {@link
#ASYNC_STATE_BUFFER_TIMEOUT} to
+ * control the frequency of triggering.
+ */
+@Experimental
+@Documentation.ExcludeFromDocumentation(
+"This is an experimental option, internal use only for now.")
+public static final ConfigOption ASYNC_STATE_BUFFER_SIZE =
+ConfigOptions.key("execution.async-state.buffer-size")
+.intType()
+.defaultValue(1000)
+.withDescription(
+"The size of buffer under async state execution.
Async state execution provides a buffer mechanism to reduce state access."
++ " When the number of state requests in
the active buffer exceeds the batch size,"
++ " a batched state execution would be
triggered. Larger batch sizes will bring higher end-to-end latency,"
++ " this option works with
'execution.async-state.buffer-timeout' to control the frequency of
triggering.");
+
+/**
+ * The timeout of buffer triggering in milliseconds. If the buffer has not
reached the {@link
+ * #ASYNC_STATE_BUFFER_SIZE} within 'buffer-timeout' milliseconds, a
trigger will perform
+ * actively.
+ */
+@Experimental
+@Documentation.ExcludeFromDocumentation(
+"This is an experimental option, internal use only for now.")
+public static final ConfigOption ASYNC_STATE_BUFFER_TIMEOUT =
Review Comment:
It might be better to make time configurations as `long` instead of as `int`.
##
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/AsyncExecutionController.java:
##
@@ -50,15 +50,24 @@ public class AsyncExecutionController {
private static final Logger LOG =
LoggerFactory.getLogger(AsyncExecutionController.class);
-public static final int DEFAULT_BATCH_SIZE = 1000;
-public static final int DEFAULT_MAX_IN_FLIGHT_RECORD_NUM = 6000;
+private static final int DEFAULT_BATCH_SIZE = 1000;
+
+private static final int DEFAULT_BUFFER_TIMEOUT = 1000;
+private static final int DEFAULT_MAX_IN_FLIGHT_RECORD_NUM = 6000;
/**
* The batch size. When the number of state requests in the active buffer
exceeds the batch
* size, a batched state execution would be triggered.
*/
private final int batchSize;
+/**
+ * The timeout of {@link StateRequestBuffer#activeQueue} triggering in
milliseconds. If the
+ * activeQueue has not reached the {@link #batchSize} within
'buffer-timeout' milliseconds, a
+ * trigger will perform actively.
+ */
+private final int bufferTimeOut;
Re: [PR] [FLINK-35028][runtime] Timer firing under async execution model [flink]
yunfengzhou-hub commented on code in PR #24672:
URL: https://github.com/apache/flink/pull/24672#discussion_r1572300164
##
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/asyncprocessing/AbstractAsyncStateStreamOperatorV2.java:
##
@@ -113,6 +120,26 @@ public final ThrowingConsumer,
Exception> getRecordProcessor
+ " since this part is handled by the Input.");
}
+@Override
+@SuppressWarnings("unchecked")
+public InternalTimerService getInternalTimerService(
+String name, TypeSerializer namespaceSerializer, Triggerable triggerable) {
Review Comment:
A fallback plan when isAsyncStateProcessingEnabled=false might be needed
here as well.
##
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/InternalTimerServiceAsyncImpl.java:
##
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators;
+
+import org.apache.flink.runtime.asyncprocessing.AsyncExecutionController;
+import org.apache.flink.runtime.asyncprocessing.RecordContext;
+import org.apache.flink.runtime.metrics.groups.TaskIOMetricGroup;
+import org.apache.flink.runtime.state.KeyGroupRange;
+import org.apache.flink.runtime.state.KeyGroupedInternalPriorityQueue;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.flink.streaming.runtime.tasks.StreamTaskCancellationContext;
+import org.apache.flink.util.CloseableIterator;
+import org.apache.flink.util.function.BiConsumerWithException;
+
+import static
org.apache.flink.runtime.asyncprocessing.KeyAccountingUnit.EMPTY_RECORD;
+
+/**
+ * An implementation of {@link InternalTimerService} that is used by {@link
+ *
org.apache.flink.streaming.runtime.operators.asyncprocessing.AbstractAsyncStateStreamOperator}.
+ * The timer service will set {@link RecordContext} for the timers before
invoking action to
+ * preserve the execution order between timer firing and records processing.
+ *
+ * @see https://cwiki.apache.org/confluence/display/FLINK/FLIP-425%3A+Asynchronous+Execution+Model#FLIP425:AsynchronousExecutionModel-Timers>FLIP-425
+ * timers section.
+ * @param Type of timer's key.
+ * @param Type of the namespace to which timers are scoped.
+ */
+public class InternalTimerServiceAsyncImpl extends
InternalTimerServiceImpl {
+
+private AsyncExecutionController asyncExecutionController;
+
+InternalTimerServiceAsyncImpl(
+TaskIOMetricGroup taskIOMetricGroup,
+KeyGroupRange localKeyGroupRange,
+KeyContext keyContext,
+ProcessingTimeService processingTimeService,
+KeyGroupedInternalPriorityQueue processingTimeTimersQueue,
+KeyGroupedInternalPriorityQueue eventTimeTimersQueue,
+StreamTaskCancellationContext cancellationContext,
+AsyncExecutionController asyncExecutionController) {
+super(
+taskIOMetricGroup,
+localKeyGroupRange,
+keyContext,
+processingTimeService,
+processingTimeTimersQueue,
+eventTimeTimersQueue,
+cancellationContext);
+this.asyncExecutionController = asyncExecutionController;
+this.processingTimeCallback = this::onProcessingTime;
+}
+
+private void onProcessingTime(long time) throws Exception {
+// null out the timer in case the Triggerable calls
registerProcessingTimeTimer()
+// inside the callback.
+nextTimer = null;
+
+InternalTimer timer;
+
+while ((timer = processingTimeTimersQueue.peek()) != null
+&& timer.getTimestamp() <= time
+&& !cancellationContext.isCancelled()) {
+RecordContext recordCtx =
+asyncExecutionController.buildContext(EMPTY_RECORD,
timer.getKey());
+recordCtx.retain();
+asyncExecutionController.setCurrentContext(recordCtx);
+keyContext.setCurrentKey(timer.getKey());
+processingTimeTimersQueue.poll();
+final InternalTimer timerToTrigger = timer;
+
Re: [PR] [FLINK-34444] Initial implementation of JM operator metric rest api [flink]
afedulov commented on code in PR #24564:
URL: https://github.com/apache/flink/pull/24564#discussion_r1571313741
##
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/legacy/metrics/MetricStore.java:
##
@@ -184,6 +184,21 @@ public synchronized ComponentMetricStore
getJobManagerMetricStore() {
return ComponentMetricStore.unmodifiable(jobManager);
}
+public synchronized ComponentMetricStore getJobManagerOperatorMetricStore(
+String jobID, String taskID) {
+JobMetricStore job = jobID == null ? null : jobs.get(jobID);
+if (job == null || taskID == null) {
+return null;
+}
+
+TaskMetricStore task = job.getTaskMetricStore(taskID);
+if (task == null) {
+return null;
+}
+
+return
ComponentMetricStore.unmodifiable(task.getJobManagerOperatorMetricStore());
+}
Review Comment:
nit: feels like this could be a bit more readable
```suggestion
public synchronized ComponentMetricStore
getJobManagerOperatorMetricStore(String jobID, String taskID) {
if (jobID == null || taskID == null) {
return null;
}
JobMetricStore job = jobs.get(jobID);
if (job == null) {
return null;
}
TaskMetricStore task = job.getTaskMetricStore(taskID);
if (task == null) {
return null;
}
return
ComponentMetricStore.unmodifiable(task.getJobManagerOperatorMetricStore());
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34444] Initial implementation of JM operator metric rest api [flink]
afedulov commented on code in PR #24564:
URL: https://github.com/apache/flink/pull/24564#discussion_r1572305374
##
docs/layouts/shortcodes/generated/rest_v1_dispatcher.html:
##
@@ -1277,6 +1277,9 @@
"parallelism" : {
"type" : "integer"
},
+ "slotSharingGroupId" : {
Review Comment:
Do I get it correctly that this is the consequence of [this
commit](https://github.com/apache/flink/pull/24380/commits/6dec90667d24b4f197ad9a814f30b99ffe77a6af)
not having regenerated the docs? If so, it could be better to have a separate
[hotfix] commit in this PR for clarity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-32151) 'Run kubernetes pyflink application test' fails while pulling image
[
https://issues.apache.org/jira/browse/FLINK-32151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17838980#comment-17838980
]
Ryan Skraba commented on FLINK-32151:
-
1.20 e2e_1_c1
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59021=logs=bea52777-eaf8-5663-8482-18fbc3630e81=43ba8ce7-ebbf-57cd-9163-444305d74117=10878
> 'Run kubernetes pyflink application test' fails while pulling image
> ---
>
> Key: FLINK-32151
> URL: https://issues.apache.org/jira/browse/FLINK-32151
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Deployment / Kubernetes
>Affects Versions: 1.16.2
>Reporter: Sergey Nuyanzin
>Priority: Minor
> Labels: auto-deprioritized-major, test-stability
>
> {noformat}
> 2023-05-16T13:29:39.0614891Z May 16 13:29:39 Current logs for
> flink-native-k8s-pyflink-application-1-6f4c9bfc56-cstw7:
> 2023-05-16T13:29:39.1253736Z Error from server (BadRequest): container
> "flink-main-container" in pod
> "flink-native-k8s-pyflink-application-1-6f4c9bfc56-cstw7" is waiting to
> start: image can't be pulled
> 2023-05-16T13:29:39.2611218Z May 16 13:29:39 deployment.apps
> "flink-native-k8s-pyflink-application-1" deleted
> 2023-05-16T13:29:39.4214711Z May 16 13:29:39
> clusterrolebinding.rbac.authorization.k8s.io "flink-role-binding-default"
> deleted
> 2023-05-16T13:29:40.2644587Z May 16 13:29:40
> pod/flink-native-k8s-pyflink-application-1-6f4c9bfc56-cstw7 condition met
> 2023-05-16T13:29:40.2664618Z May 16 13:29:40 Stopping minikube ...
> 2023-05-16T13:29:40.3396336Z May 16 13:29:40 * Stopping node "minikube" ...
> 2023-05-16T13:29:50.7499872Z May 16 13:29:50 * 1 node stopped.
> {noformat}
> it's very similar to https://issues.apache.org/jira/browse/FLINK-28226
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35133) Release flink-connector-cassandra v3.2.0 for Flink 1.19
[ https://issues.apache.org/jira/browse/FLINK-35133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35133: --- Labels: pull-request-available (was: ) > Release flink-connector-cassandra v3.2.0 for Flink 1.19 > --- > > Key: FLINK-35133 > URL: https://issues.apache.org/jira/browse/FLINK-35133 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Cassandra >Reporter: Danny Cranmer >Assignee: Danny Cranmer >Priority: Major > Labels: pull-request-available > > https://github.com/apache/flink-connector-cassandra -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-35133][Connectors/Cassandra] Adding support for Flink 1.19 [flink-connector-cassandra]
dannycranmer opened a new pull request, #28: URL: https://github.com/apache/flink-connector-cassandra/pull/28 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-35069) ContinuousProcessingTimeTrigger continuously registers timers in a loop at the end of the window
[ https://issues.apache.org/jira/browse/FLINK-35069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17838962#comment-17838962 ] Hong Liang Teoh commented on FLINK-35069: - Thanks [~lijinzhong] for the fix! Are we planning to backport this fix to 1.18 and 1.19 as well? > ContinuousProcessingTimeTrigger continuously registers timers in a loop at > the end of the window > > > Key: FLINK-35069 > URL: https://issues.apache.org/jira/browse/FLINK-35069 > Project: Flink > Issue Type: Bug > Components: API / DataStream >Affects Versions: 1.17.0 >Reporter: Jinzhong Li >Priority: Major > Labels: pull-request-available > Attachments: image-2024-04-09-20-23-54-415.png > > > In our production environment, when TumblingEventTimeWindows and > ContinuousProcessingTimeTrigger are used in combination within a > WindowOperator, we observe a situation where the timers are continuously > registered in a loop at the end of the window, leading to the job being > perpetually stuck in timer processing. > !image-2024-04-09-20-23-54-415.png|width=516,height=205! > This issue can be reproduced using the > [UT|https://github.com/apache/flink/blob/8e80ff889701ed1abbb5c15cd3943b254f1fb5cc/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/windowing/ContinuousProcessingTimeTriggerTest.java#L177] > provided by the pr. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35168][State] Basic State Iterator for async processing [flink]
flinkbot commented on PR #24690: URL: https://github.com/apache/flink/pull/24690#issuecomment-2066354887 ## CI report: * 5fa3b48c1caed9d9e4ff9b31c1b8640318c18555 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-35168) Basic State Iterator for async processing
[ https://issues.apache.org/jira/browse/FLINK-35168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35168: --- Labels: pull-request-available (was: ) > Basic State Iterator for async processing > - > > Key: FLINK-35168 > URL: https://issues.apache.org/jira/browse/FLINK-35168 > Project: Flink > Issue Type: Sub-task > Components: Runtime / State Backends >Reporter: Zakelly Lan >Assignee: Zakelly Lan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-35168][State] Basic State Iterator for async processing [flink]
Zakelly opened a new pull request, #24690: URL: https://github.com/apache/flink/pull/24690 ## What is the purpose of the change This PR ship the basic logic of state iterator, only leave the necessary part abstract for subclass to implement. Most of the code is future-related. ## Brief change log - Introduce `AbstractStateIterator` and corresponding tests. ## Verifying this change - Run newly added `AbstractStateIteratorTest`. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): yes - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-35174) Bump org.apache.commons:commons-compress from 1.24.0 to 1.26.1 for Flink RabbitMQ connector
[ https://issues.apache.org/jira/browse/FLINK-35174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17838954#comment-17838954 ] Danny Cranmer commented on FLINK-35174: --- Merged commit 66e323a into apache:main > Bump org.apache.commons:commons-compress from 1.24.0 to 1.26.1 for Flink > RabbitMQ connector > --- > > Key: FLINK-35174 > URL: https://issues.apache.org/jira/browse/FLINK-35174 > Project: Flink > Issue Type: Technical Debt > Components: Connectors/ RabbitMQ >Reporter: Danny Cranmer >Assignee: Danny Cranmer >Priority: Major > Labels: pull-request-available > Fix For: rabbitmq-3.1.0 > > > Bump org.apache.commons:commons-compress from 1.24.0 to 1.26.1 for Flink > RabbitMQ connector > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35039][rest] Use PUT method supported by YARN web proxy instead of POST [flink]
Myasuka commented on PR #24689: URL: https://github.com/apache/flink/pull/24689#issuecomment-2066319875 @yeezychao Can your solution also works in standalone environment? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35174][Connectors/RabbitMQ] Bump commons-compress to 1.26.1 [flink-connector-rabbitmq]
dannycranmer merged PR #28: URL: https://github.com/apache/flink-connector-rabbitmq/pull/28 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-35039) Create Profiling JobManager/TaskManager Instance failed
[ https://issues.apache.org/jira/browse/FLINK-35039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17838951#comment-17838951 ] Yun Tang commented on FLINK-35039: -- [~wczhu] Already assigned to you. > Create Profiling JobManager/TaskManager Instance failed > --- > > Key: FLINK-35039 > URL: https://issues.apache.org/jira/browse/FLINK-35039 > Project: Flink > Issue Type: Bug > Components: Runtime / Web Frontend >Affects Versions: 1.19.0 > Environment: Hadoop 3.2.2 > Flink 1.19 >Reporter: ude >Assignee: ude >Priority: Major > Labels: pull-request-available > Attachments: image-2024-04-08-10-21-31-066.png, > image-2024-04-08-10-21-48-417.png, image-2024-04-08-10-30-16-683.png > > > I'm test the "async-profiler" feature in version 1.19, but when I submit a > task in yarn per-job mode, I get an error when I click Create Profiling > Instance on the flink Web UI page. > !image-2024-04-08-10-21-31-066.png! > !image-2024-04-08-10-21-48-417.png! > The error message obviously means that the yarn proxy server does not support > *POST* calls. I checked the code of _*WebAppProxyServlet.java*_ and found > that the *POST* method is indeed not supported, so I changed it to *PUT* > method and the call was successful. > !image-2024-04-08-10-30-16-683.png! > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-35039) Create Profiling JobManager/TaskManager Instance failed
[ https://issues.apache.org/jira/browse/FLINK-35039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Tang reassigned FLINK-35039: Assignee: ude > Create Profiling JobManager/TaskManager Instance failed > --- > > Key: FLINK-35039 > URL: https://issues.apache.org/jira/browse/FLINK-35039 > Project: Flink > Issue Type: Bug > Components: Runtime / Web Frontend >Affects Versions: 1.19.0 > Environment: Hadoop 3.2.2 > Flink 1.19 >Reporter: ude >Assignee: ude >Priority: Major > Labels: pull-request-available > Attachments: image-2024-04-08-10-21-31-066.png, > image-2024-04-08-10-21-48-417.png, image-2024-04-08-10-30-16-683.png > > > I'm test the "async-profiler" feature in version 1.19, but when I submit a > task in yarn per-job mode, I get an error when I click Create Profiling > Instance on the flink Web UI page. > !image-2024-04-08-10-21-31-066.png! > !image-2024-04-08-10-21-48-417.png! > The error message obviously means that the yarn proxy server does not support > *POST* calls. I checked the code of _*WebAppProxyServlet.java*_ and found > that the *POST* method is indeed not supported, so I changed it to *PUT* > method and the call was successful. > !image-2024-04-08-10-30-16-683.png! > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35055][cdc] Exclude connector dependencies with questionable licenses from fat jar packages [flink-cdc]
yuxiqian commented on PR #3212: URL: https://github.com/apache/flink-cdc/pull/3212#issuecomment-2066306357 Seems removing connector jar dependencies break SQL Job E2e tests. Need to fix this first. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35142][Connectors/RabbitMQ] Drop support for Flink 1.17 [flink-connector-rabbitmq]
dannycranmer merged PR #27: URL: https://github.com/apache/flink-connector-rabbitmq/pull/27 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [hotfix] Update copyright year to 2024 [flink-connector-rabbitmq]
dannycranmer merged PR #24: URL: https://github.com/apache/flink-connector-rabbitmq/pull/24 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35174][Connectors/RabbitMQ] Bump commons-compress to 1.26.1 [flink-connector-rabbitmq]
dannycranmer commented on PR #28: URL: https://github.com/apache/flink-connector-rabbitmq/pull/28#issuecomment-2066294353 > Seems like there is a dependency convergence error > > ``` > Warning: > Dependency convergence error for org.apache.commons:commons-lang3:3.14.0 paths to dependency are: > +-org.apache.flink:flink-connector-rabbitmq-parent:3.1-SNAPSHOT > +-org.testcontainers:junit-jupiter:1.19.3 > +-org.testcontainers:testcontainers:1.19.3 > +-org.apache.commons:commons-compress:1.26.1 > +-org.apache.commons:commons-lang3:3.14.0 > and > +-org.apache.flink:flink-connector-rabbitmq-parent:3.1-SNAPSHOT > +-org.apache.flink:flink-architecture-tests-production:1.19-20240418.012917-199 > +-org.apache.flink:flink-core:1.19-SNAPSHOT > +-org.apache.commons:commons-lang3:3.12.0 > and > +-org.apache.flink:flink-connector-rabbitmq-parent:3.1-SNAPSHOT > +-org.apache.flink:flink-architecture-tests-production:1.19-20240418.012917-199 > +-org.apache.flink:flink-core:1.19-SNAPSHOT > +-org.apache.commons:commons-text:1.10.0 > +-org.apache.commons:commons-lang3:3.12.0 > ``` Thanks, fixed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33460][Connector/JDBC] Support property authentication connection. [flink-connector-jdbc]
eskabetxe commented on code in PR #115:
URL:
https://github.com/apache/flink-connector-jdbc/pull/115#discussion_r1572162247
##
flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/JdbcConnectionOptions.java:
##
@@ -90,13 +98,31 @@ public JdbcConnectionOptionsBuilder withDriverName(String
driverName) {
return this;
}
+public JdbcConnectionOptionsBuilder withProperties(Properties
properties) {
Review Comment:
@RocMarshal I would say the easiest way, eliminating the method.
If someone has a properties, they can do:
`properties.forEach((key, value) -> builder.withProperty(key, value));`
Under the hood, the `properties.putAll` is what it does.
I always prefer to keep it simple..
If we have to explain, document or add something to explain what is being
done, we are doing it 'wrong'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Comment Edited] (FLINK-35137) Release flink-connector-jdbc v3.2.0 for Flink 1.19
[ https://issues.apache.org/jira/browse/FLINK-35137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17838591#comment-17838591 ] Danny Cranmer edited comment on FLINK-35137 at 4/19/24 10:28 AM: - RC2 Vote: https://lists.apache.org/thread/b7xbjo4crt1527ldksw4nkwo8vs56csy was (Author: dannycranmer): https://lists.apache.org/thread/b7xbjo4crt1527ldksw4nkwo8vs56csy > Release flink-connector-jdbc v3.2.0 for Flink 1.19 > -- > > Key: FLINK-35137 > URL: https://issues.apache.org/jira/browse/FLINK-35137 > Project: Flink > Issue Type: Sub-task > Components: Connectors / JDBC >Reporter: Danny Cranmer >Assignee: Danny Cranmer >Priority: Major > Fix For: jdbc-3.2.0 > > > https://github.com/apache/flink-connector-jdbc -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-35139) Release flink-connector-mongodb v1.2.0 for Flink 1.19
[ https://issues.apache.org/jira/browse/FLINK-35139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17838086#comment-17838086 ] Danny Cranmer edited comment on FLINK-35139 at 4/19/24 10:28 AM: - RC1 Vote: https://lists.apache.org/thread/s18g7obgp4sbdtl73571976vqvy1ftk8 was (Author: dannycranmer): https://lists.apache.org/thread/s18g7obgp4sbdtl73571976vqvy1ftk8 > Release flink-connector-mongodb v1.2.0 for Flink 1.19 > - > > Key: FLINK-35139 > URL: https://issues.apache.org/jira/browse/FLINK-35139 > Project: Flink > Issue Type: Sub-task > Components: Connectors / MongoDB >Reporter: Danny Cranmer >Assignee: Danny Cranmer >Priority: Major > Labels: pull-request-available > Fix For: mongodb-1.2.0 > > > https://github.com/apache/flink-connector-mongodb -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-35137) Release flink-connector-jdbc v3.2.0 for Flink 1.19
[ https://issues.apache.org/jira/browse/FLINK-35137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17838181#comment-17838181 ] Danny Cranmer edited comment on FLINK-35137 at 4/19/24 10:28 AM: - RC1 Vote: https://lists.apache.org/thread/q6dmc5dbz7kcfvpo99pj2sh5mzhffgl5 was (Author: dannycranmer): https://lists.apache.org/thread/q6dmc5dbz7kcfvpo99pj2sh5mzhffgl5 > Release flink-connector-jdbc v3.2.0 for Flink 1.19 > -- > > Key: FLINK-35137 > URL: https://issues.apache.org/jira/browse/FLINK-35137 > Project: Flink > Issue Type: Sub-task > Components: Connectors / JDBC >Reporter: Danny Cranmer >Assignee: Danny Cranmer >Priority: Major > Fix For: jdbc-3.2.0 > > > https://github.com/apache/flink-connector-jdbc -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-35132) Release flink-connector-aws v4.3.0 for Flink 1.19
[ https://issues.apache.org/jira/browse/FLINK-35132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17838058#comment-17838058 ] Danny Cranmer edited comment on FLINK-35132 at 4/19/24 10:27 AM: - RC1 Vote: [https://lists.apache.org/thread/0nw9smt23crx4gwkf6p1dd4jwvp1g5s0] was (Author: dannycranmer): https://lists.apache.org/thread/0nw9smt23crx4gwkf6p1dd4jwvp1g5s0 > Release flink-connector-aws v4.3.0 for Flink 1.19 > - > > Key: FLINK-35132 > URL: https://issues.apache.org/jira/browse/FLINK-35132 > Project: Flink > Issue Type: Sub-task > Components: Connectors / AWS >Reporter: Danny Cranmer >Assignee: Danny Cranmer >Priority: Major > Fix For: aws-connector-4.3.0 > > > https://github.com/apache/flink-connector-aws -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-35135) Release flink-connector-gcp-pubsub v3.1.0 for Flink 1.19
[ https://issues.apache.org/jira/browse/FLINK-35135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17838590#comment-17838590 ] Danny Cranmer edited comment on FLINK-35135 at 4/19/24 10:28 AM: - RC1 Vote: https://lists.apache.org/thread/b7l1r0y7nwox2vhf2z3kwjn41clf6w1v was (Author: dannycranmer): https://lists.apache.org/thread/b7l1r0y7nwox2vhf2z3kwjn41clf6w1v > Release flink-connector-gcp-pubsub v3.1.0 for Flink 1.19 > > > Key: FLINK-35135 > URL: https://issues.apache.org/jira/browse/FLINK-35135 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Google Cloud PubSub >Reporter: Danny Cranmer >Assignee: Danny Cranmer >Priority: Major > Labels: pull-request-available > Fix For: gcp-pubsub-3.1.0 > > > https://github.com/apache/flink-connector-gcp-pubsub -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-35139) Release flink-connector-mongodb v1.2.0 for Flink 1.19
[ https://issues.apache.org/jira/browse/FLINK-35139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17838592#comment-17838592 ] Danny Cranmer edited comment on FLINK-35139 at 4/19/24 10:28 AM: - RC2 Vote: https://lists.apache.org/thread/2982v6n5q0bgldrp919t5t6d19xsl710 was (Author: dannycranmer): https://lists.apache.org/thread/2982v6n5q0bgldrp919t5t6d19xsl710 > Release flink-connector-mongodb v1.2.0 for Flink 1.19 > - > > Key: FLINK-35139 > URL: https://issues.apache.org/jira/browse/FLINK-35139 > Project: Flink > Issue Type: Sub-task > Components: Connectors / MongoDB >Reporter: Danny Cranmer >Assignee: Danny Cranmer >Priority: Major > Labels: pull-request-available > Fix For: mongodb-1.2.0 > > > https://github.com/apache/flink-connector-mongodb -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35174][Connectors/RabbitMQ] Bump commons-compress to 1.26.1 [flink-connector-rabbitmq]
hlteoh37 commented on PR #28: URL: https://github.com/apache/flink-connector-rabbitmq/pull/28#issuecomment-2066287030 Seems like there is a dependency convergence error ``` Warning: Dependency convergence error for org.apache.commons:commons-lang3:3.14.0 paths to dependency are: +-org.apache.flink:flink-connector-rabbitmq-parent:3.1-SNAPSHOT +-org.testcontainers:junit-jupiter:1.19.3 +-org.testcontainers:testcontainers:1.19.3 +-org.apache.commons:commons-compress:1.26.1 +-org.apache.commons:commons-lang3:3.14.0 and +-org.apache.flink:flink-connector-rabbitmq-parent:3.1-SNAPSHOT +-org.apache.flink:flink-architecture-tests-production:1.19-20240418.012917-199 +-org.apache.flink:flink-core:1.19-SNAPSHOT +-org.apache.commons:commons-lang3:3.12.0 and +-org.apache.flink:flink-connector-rabbitmq-parent:3.1-SNAPSHOT +-org.apache.flink:flink-architecture-tests-production:1.19-20240418.012917-199 +-org.apache.flink:flink-core:1.19-SNAPSHOT +-org.apache.commons:commons-text:1.10.0 +-org.apache.commons:commons-lang3:3.12.0 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-35132) Release flink-connector-aws v4.3.0 for Flink 1.19
[ https://issues.apache.org/jira/browse/FLINK-35132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17838949#comment-17838949 ] Danny Cranmer commented on FLINK-35132: --- RC2 Vote: https://lists.apache.org/thread/dof3tprw5s19jfwhk6yxs5qxbb4x5ss8 > Release flink-connector-aws v4.3.0 for Flink 1.19 > - > > Key: FLINK-35132 > URL: https://issues.apache.org/jira/browse/FLINK-35132 > Project: Flink > Issue Type: Sub-task > Components: Connectors / AWS >Reporter: Danny Cranmer >Assignee: Danny Cranmer >Priority: Major > Fix For: aws-connector-4.3.0 > > > https://github.com/apache/flink-connector-aws -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33460][Connector/JDBC] Support property authentication connection. [flink-connector-jdbc]
eskabetxe commented on code in PR #115:
URL:
https://github.com/apache/flink-connector-jdbc/pull/115#discussion_r1572162247
##
flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/JdbcConnectionOptions.java:
##
@@ -90,13 +98,31 @@ public JdbcConnectionOptionsBuilder withDriverName(String
driverName) {
return this;
}
+public JdbcConnectionOptionsBuilder withProperties(Properties
properties) {
Review Comment:
@RocMarshal I would say the easiest way, eliminating the method.
If someone has a properties, they can do:
`properties.forEach((key, value) -> builder.withProperty(key, value));`
Under the hood, the `properties.putAll` is what it does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-35162][state] Implement WriteBatchOperation and general MultiGetOperation for ForSt [flink]
ljz2051 commented on PR #24681: URL: https://github.com/apache/flink/pull/24681#issuecomment-2066281635 @Zakelly Thanks for your review! I have addressed all comments your mentioned. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35162][state] Implement WriteBatchOperation and general MultiGetOperation for ForSt [flink]
ljz2051 commented on code in PR #24681:
URL: https://github.com/apache/flink/pull/24681#discussion_r1572160266
##
flink-state-backends/flink-statebackend-forst/src/main/java/org/apache/flink/state/forst/ForStDBOperation.java:
##
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.state.forst;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.IOException;
+
+/**
+ * Data access operation to ForStDB. This interface is used to encapsulate the
DB access operations
+ * formed after grouping state access. For more information about “Grouping
state access”, please
+ * refer to FLIP-426.
+ *
+ * @param The type of output for DB access.
+ */
+@Internal
+public interface ForStDBOperation {
+
+/**
+ * Process the ForStDB access requests.
+ *
+ * @return Processing result.
+ * @throws IOException Thrown if ForStDB access encountered an I/O related
error.
+ */
+OUT process() throws IOException;
Review Comment:
Good point. I agree that the future-style interface is more flexible. I have
refined it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-35162][state] Implement WriteBatchOperation and general MultiGetOperation for ForSt [flink]
ljz2051 commented on code in PR #24681:
URL: https://github.com/apache/flink/pull/24681#discussion_r1572156946
##
flink-state-backends/flink-statebackend-forst/src/main/java/org/apache/flink/state/forst/ContextKey.java:
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.state.forst;
+
+import javax.annotation.Nullable;
+
+/**
+ * The composite key which contains some context information.
+ *
+ * @param The type of the raw key.
+ */
+public class ContextKey {
Review Comment:
Yes, the ContextKey can be shared across state requests, and ContextKey
will be attached to the RecordContext in another PR. **So I move the ContextKey
class to flink-runtime module from forst-module**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-35162][state] Implement WriteBatchOperation and general MultiGetOperation for ForSt [flink]
ljz2051 commented on code in PR #24681:
URL: https://github.com/apache/flink/pull/24681#discussion_r1572153463
##
flink-state-backends/flink-statebackend-forst/src/main/java/org/apache/flink/state/forst/ContextKey.java:
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.state.forst;
+
+import javax.annotation.Nullable;
+
+/**
+ * The composite key which contains some context information.
+ *
+ * @param The type of the raw key.
+ */
+public class ContextKey {
+
+private final K rawKey;
+
+private final int keyGroup;
+
+/**
+ * A record in user layer may access the state multiple times. The {@code
serializedKey} can be
+ * used to cache the serialized key bytes after its first serialization,
so that subsequent
+ * state accesses with the same key can avoid being serialized repeatedly.
+ */
+private @Nullable byte[] serializedKey = null;
+
+public ContextKey(K rawKey, int keyGroup) {
Review Comment:
I have remove the "ContextKey#of" static method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-35142) Release flink-connector-rabbitmq v3.1.0 for Flink 1.19
[ https://issues.apache.org/jira/browse/FLINK-35142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Cranmer updated FLINK-35142: -- Summary: Release flink-connector-rabbitmq v3.1.0 for Flink 1.19 (was: Release flink-connector-rabbitmq vX.X.X for Flink 1.19) > Release flink-connector-rabbitmq v3.1.0 for Flink 1.19 > -- > > Key: FLINK-35142 > URL: https://issues.apache.org/jira/browse/FLINK-35142 > Project: Flink > Issue Type: Sub-task > Components: Connectors/ RabbitMQ >Reporter: Danny Cranmer >Assignee: Danny Cranmer >Priority: Major > Labels: pull-request-available > > https://github.com/apache/flink-connector-rabbitmq -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] Fix problem due to HBaseRowDataLookupFunction returned to reuse objects [flink-connector-hbase]
Tan-JiaLiang commented on PR #20: URL: https://github.com/apache/flink-connector-hbase/pull/20#issuecomment-2066268025 @Jackson-xky This is a serious problem in the hbase lookup join. Are you still working on it? I'd be happy to help. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Bump org.apache.commons:commons-compress from 1.24.0 to 1.26.0 [flink-connector-rabbitmq]
dependabot[bot] commented on PR #25: URL: https://github.com/apache/flink-connector-rabbitmq/pull/25#issuecomment-2066266561 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let me know by commenting `@dependabot ignore this major version` or `@dependabot ignore this minor version`. If you change your mind, just re-open this PR and I'll resolve any conflicts on it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Bump org.apache.commons:commons-compress from 1.24.0 to 1.26.0 [flink-connector-rabbitmq]
dannycranmer commented on PR #25: URL: https://github.com/apache/flink-connector-rabbitmq/pull/25#issuecomment-2066266504 https://issues.apache.org/jira/browse/FLINK-35174 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35162][state] Implement WriteBatchOperation and general MultiGetOperation for ForSt [flink]
ljz2051 commented on code in PR #24681:
URL: https://github.com/apache/flink/pull/24681#discussion_r1572151847
##
flink-state-backends/flink-statebackend-forst/src/main/java/org/apache/flink/state/forst/ForStValueState.java:
##
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.state.forst;
+
+import org.apache.flink.api.common.state.v2.ValueState;
+import org.apache.flink.core.memory.DataInputDeserializer;
+import org.apache.flink.core.memory.DataOutputSerializer;
+import org.apache.flink.runtime.asyncprocessing.StateRequestHandler;
+import org.apache.flink.runtime.state.SerializedCompositeKeyBuilder;
+import org.apache.flink.runtime.state.v2.InternalValueState;
+import org.apache.flink.runtime.state.v2.ValueStateDescriptor;
+
+import org.rocksdb.ColumnFamilyHandle;
+
+import java.io.IOException;
+
+/**
+ * The {@link InternalValueState} implement for ForStDB.
+ *
+ * @param The type of the key.
+ * @param The type of the value.
+ */
+public class ForStValueState extends InternalValueState
+implements ValueState, ForStInnerTable, V> {
+
+/** The column family which this internal value state belongs to. */
+private final ColumnFamilyHandle columnFamilyHandle;
+
+/** The serialized key builder which need be thread-safe. */
+private final ThreadLocal>
serializedKeyBuilder;
+
+/** The data outputStream used for value serializer, which need be
thread-safe. */
+private final ThreadLocal valueSerializerView;
+
+/** The data inputStream used for value deserializer, which need be
thread-safe. */
+private final ThreadLocal valueDeserializerView;
+
+public ForStValueState(
+StateRequestHandler stateRequestHandler,
+ColumnFamilyHandle columnFamily,
+ValueStateDescriptor valueStateDescriptor,
+ThreadLocal> serializedKeyBuilder,
+ThreadLocal valueSerializerView,
+ThreadLocal valueDeserializerView) {
+super(stateRequestHandler, valueStateDescriptor);
+this.columnFamilyHandle = columnFamily;
+this.serializedKeyBuilder = serializedKeyBuilder;
+this.valueSerializerView = valueSerializerView;
+this.valueDeserializerView = valueDeserializerView;
+}
+
+@Override
+public ColumnFamilyHandle getColumnFamilyHandle() {
+return columnFamilyHandle;
+}
+
+@Override
+public byte[] serializeKey(ContextKey contextKey) throws IOException {
Review Comment:
Yes. I have marked the ContextKey with @ThreadSafe annotation, and refactor
the 'read cache or serialize' logic. Please review the
ContextKey#getOrCreateSerializedKey method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-35174) Bump org.apache.commons:commons-compress from 1.24.0 to 1.26.1 for Flink RabbitMQ connector
[ https://issues.apache.org/jira/browse/FLINK-35174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Cranmer updated FLINK-35174: -- Description: Bump org.apache.commons:commons-compress from 1.24.0 to 1.26.1 for Flink RabbitMQ connector was:Bump org.apache.commons:commons-compress from 1.25.0 to 1.26.1 for Flink AWS connectors > Bump org.apache.commons:commons-compress from 1.24.0 to 1.26.1 for Flink > RabbitMQ connector > --- > > Key: FLINK-35174 > URL: https://issues.apache.org/jira/browse/FLINK-35174 > Project: Flink > Issue Type: Technical Debt > Components: Connectors / AWS >Reporter: Danny Cranmer >Assignee: Danny Cranmer >Priority: Major > Labels: pull-request-available > Fix For: aws-connector-4.3.0 > > > Bump org.apache.commons:commons-compress from 1.24.0 to 1.26.1 for Flink > RabbitMQ connector > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35174) Bump org.apache.commons:commons-compress from 1.24.0 to 1.26.1 for Flink RabbitMQ connector
[ https://issues.apache.org/jira/browse/FLINK-35174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Cranmer updated FLINK-35174: -- Fix Version/s: rabbitmq-3.1.0 (was: aws-connector-4.3.0) > Bump org.apache.commons:commons-compress from 1.24.0 to 1.26.1 for Flink > RabbitMQ connector > --- > > Key: FLINK-35174 > URL: https://issues.apache.org/jira/browse/FLINK-35174 > Project: Flink > Issue Type: Technical Debt > Components: Connectors / AWS >Reporter: Danny Cranmer >Assignee: Danny Cranmer >Priority: Major > Labels: pull-request-available > Fix For: rabbitmq-3.1.0 > > > Bump org.apache.commons:commons-compress from 1.24.0 to 1.26.1 for Flink > RabbitMQ connector > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35174) Bump org.apache.commons:commons-compress from 1.24.0 to 1.26.1 for Flink RabbitMQ connector
[ https://issues.apache.org/jira/browse/FLINK-35174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Cranmer updated FLINK-35174: -- Component/s: Connectors/ RabbitMQ (was: Connectors / AWS) > Bump org.apache.commons:commons-compress from 1.24.0 to 1.26.1 for Flink > RabbitMQ connector > --- > > Key: FLINK-35174 > URL: https://issues.apache.org/jira/browse/FLINK-35174 > Project: Flink > Issue Type: Technical Debt > Components: Connectors/ RabbitMQ >Reporter: Danny Cranmer >Assignee: Danny Cranmer >Priority: Major > Labels: pull-request-available > Fix For: rabbitmq-3.1.0 > > > Bump org.apache.commons:commons-compress from 1.24.0 to 1.26.1 for Flink > RabbitMQ connector > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35174) Bump org.apache.commons:commons-compress from 1.24.0 to 1.26.1 for Flink RabbitMQ connector
[ https://issues.apache.org/jira/browse/FLINK-35174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Cranmer updated FLINK-35174: -- Summary: Bump org.apache.commons:commons-compress from 1.24.0 to 1.26.1 for Flink RabbitMQ connector (was: Bump org.apache.commons:commons-compress from 1.25.0 to 1.26.1 for Flink RabbitMQ connector) > Bump org.apache.commons:commons-compress from 1.24.0 to 1.26.1 for Flink > RabbitMQ connector > --- > > Key: FLINK-35174 > URL: https://issues.apache.org/jira/browse/FLINK-35174 > Project: Flink > Issue Type: Technical Debt > Components: Connectors / AWS >Reporter: Danny Cranmer >Assignee: Danny Cranmer >Priority: Major > Labels: pull-request-available > Fix For: aws-connector-4.3.0 > > > Bump org.apache.commons:commons-compress from 1.25.0 to 1.26.1 for Flink AWS > connectors -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33460][Connector/JDBC] Support property authentication connection. [flink-connector-jdbc]
RocMarshal commented on code in PR #115:
URL:
https://github.com/apache/flink-connector-jdbc/pull/115#discussion_r1572083767
##
flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/JdbcConnectionOptions.java:
##
@@ -90,13 +98,31 @@ public JdbcConnectionOptionsBuilder withDriverName(String
driverName) {
return this;
}
+public JdbcConnectionOptionsBuilder withProperties(Properties
properties) {
Review Comment:
hi, @eskabetxe Thanks a lot for your quick-review .
```
.withProperty("a", "a1")
.withProperties(new properties())
.withProperty("b", "b1")
```
> .
> that a and b should be in the properties, or just b...
It's right.
It's difficult to express , which depends the what cases devs need and I
have to admit that the lines referenced above is ambiguous now.
I have come up with two temporary ways to alleviate this ambiguity:
- Keep the current logic, and then we add comment to describe semantics for
these two methods.
OR
- Add a flag parameter to method `withProperties(new properties(),
true/false)` , indicating whether to force assign entire `properties`, rather
than a part of the `properties` by `upsert` mode.
Of course, I'd appreciated to hear any ideas form you / community devs !
Please let me know what's your opinion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-35142) Release flink-connector-rabbitmq v3.1.0 for Flink 1.19
[ https://issues.apache.org/jira/browse/FLINK-35142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Cranmer updated FLINK-35142: -- Fix Version/s: rabbitmq-3.1.0 > Release flink-connector-rabbitmq v3.1.0 for Flink 1.19 > -- > > Key: FLINK-35142 > URL: https://issues.apache.org/jira/browse/FLINK-35142 > Project: Flink > Issue Type: Sub-task > Components: Connectors/ RabbitMQ >Reporter: Danny Cranmer >Assignee: Danny Cranmer >Priority: Major > Labels: pull-request-available > Fix For: rabbitmq-3.1.0 > > > https://github.com/apache/flink-connector-rabbitmq -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] Bump org.apache.commons:commons-compress from 1.24.0 to 1.26.0 [flink-connector-rabbitmq]
dannycranmer closed pull request #25: Bump org.apache.commons:commons-compress from 1.24.0 to 1.26.0 URL: https://github.com/apache/flink-connector-rabbitmq/pull/25 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-35174) Bump org.apache.commons:commons-compress from 1.25.0 to 1.26.1 for Flink RabbitMQ connector
Danny Cranmer created FLINK-35174: - Summary: Bump org.apache.commons:commons-compress from 1.25.0 to 1.26.1 for Flink RabbitMQ connector Key: FLINK-35174 URL: https://issues.apache.org/jira/browse/FLINK-35174 Project: Flink Issue Type: Technical Debt Components: Connectors / AWS Reporter: Danny Cranmer Assignee: Danny Cranmer Fix For: aws-connector-4.3.0 Bump org.apache.commons:commons-compress from 1.25.0 to 1.26.1 for Flink AWS connectors -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-35142) Release flink-connector-rabbitmq vX.X.X for Flink 1.19
[ https://issues.apache.org/jira/browse/FLINK-35142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Cranmer reassigned FLINK-35142: - Assignee: Danny Cranmer > Release flink-connector-rabbitmq vX.X.X for Flink 1.19 > -- > > Key: FLINK-35142 > URL: https://issues.apache.org/jira/browse/FLINK-35142 > Project: Flink > Issue Type: Sub-task > Components: Connectors/ RabbitMQ >Reporter: Danny Cranmer >Assignee: Danny Cranmer >Priority: Major > Labels: pull-request-available > > https://github.com/apache/flink-connector-rabbitmq -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35142) Release flink-connector-rabbitmq vX.X.X for Flink 1.19
[ https://issues.apache.org/jira/browse/FLINK-35142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35142: --- Labels: pull-request-available (was: ) > Release flink-connector-rabbitmq vX.X.X for Flink 1.19 > -- > > Key: FLINK-35142 > URL: https://issues.apache.org/jira/browse/FLINK-35142 > Project: Flink > Issue Type: Sub-task > Components: Connectors/ RabbitMQ >Reporter: Danny Cranmer >Priority: Major > Labels: pull-request-available > > https://github.com/apache/flink-connector-rabbitmq -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34954) Kryo input implementation NoFetchingInput fails to handle zero length bytes
[
https://issues.apache.org/jira/browse/FLINK-34954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17838937#comment-17838937
]
Danny Cranmer commented on FLINK-34954:
---
Thanks for the bug report and fix, [~q.xu] .
> Kryo input implementation NoFetchingInput fails to handle zero length bytes
> ---
>
> Key: FLINK-34954
> URL: https://issues.apache.org/jira/browse/FLINK-34954
> Project: Flink
> Issue Type: Bug
> Components: API / Type Serialization System
>Affects Versions: 1.19.0
>Reporter: Qinghui Xu
>Assignee: Qinghui Xu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
> If the serailized bytes are empty, `NoFetchingInput` will run into error when
> Kryo tries to deserialize it.
> Example: a protobuf 3 object that contains only default values will be
> serialized as 0 length byte array, and the deserialization later will fail.
> Illustration:
> {noformat}
> import com.esotericsoftware.kryo.Kryo
> import com.esotericsoftware.kryo.io.{ByteBufferInput, ByteBufferOutput,
> Input, Output}
> import com.google.protobuf.{DescriptorProtos, Message}import
> com.twitter.chill.protobuf.ProtobufSerializer
> import org.apache.flink.api.java.typeutils.runtime.NoFetchingInput
> import java.io.ByteArrayInputStream
>
> object ProtoSerializationTest {
> def main(args: Array[String]) = {
> val chillProtoSerializer = new ProtobufSerializer
> val protomessage = DescriptorProtos.DescriptorProto.getDefaultInstance
> val output: Output = new ByteBufferOutput(1000)
> chillProtoSerializer.write(null, output, protomessage)
> val serialized: Array[Byte] = output.toBytes
> println(s"Serialized : $serialized")
> val input: Input = new NoFetchingInput(new
> ByteArrayInputStream(serialized))
> val deserialized = chillProtoSerializer.read(null, input,
> classOf[BillableClick].asInstanceOf[Class[Message]])
> println(deserialized)
> }
> }
> {noformat}
>
> Error
> {noformat}
> Exception in thread "main" java.lang.RuntimeException: Could not create class
> com.criteo.glup.BillableClickProto$BillableClick
> at
> com.twitter.chill.protobuf.ProtobufSerializer.read(ProtobufSerializer.java:76)
> at
> com.criteo.streaming.common.bootstrap.ProtoSerialization$.main(ProtoSerialization.scala:22)
> at ProtoSerialization.main(ProtoSerialization.scala)
> Caused by: com.esotericsoftware.kryo.KryoException: java.io.EOFException: No
> more bytes left.
> at
> org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.readBytes(NoFetchingInput.java:128)
> at com.esotericsoftware.kryo.io.Input.readBytes(Input.java:332)
> at
> com.twitter.chill.protobuf.ProtobufSerializer.read(ProtobufSerializer.java:73)
> ... 2 more
> Caused by: java.io.EOFException: No more bytes left.
> ... 5 more{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-34954) Kryo input implementation NoFetchingInput fails to handle zero length bytes
[
https://issues.apache.org/jira/browse/FLINK-34954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Cranmer updated FLINK-34954:
--
Component/s: API / Type Serialization System
> Kryo input implementation NoFetchingInput fails to handle zero length bytes
> ---
>
> Key: FLINK-34954
> URL: https://issues.apache.org/jira/browse/FLINK-34954
> Project: Flink
> Issue Type: Bug
> Components: API / Type Serialization System
>Affects Versions: 1.19.0
>Reporter: Qinghui Xu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
> If the serailized bytes are empty, `NoFetchingInput` will run into error when
> Kryo tries to deserialize it.
> Example: a protobuf 3 object that contains only default values will be
> serialized as 0 length byte array, and the deserialization later will fail.
> Illustration:
> {noformat}
> import com.esotericsoftware.kryo.Kryo
> import com.esotericsoftware.kryo.io.{ByteBufferInput, ByteBufferOutput,
> Input, Output}
> import com.google.protobuf.{DescriptorProtos, Message}import
> com.twitter.chill.protobuf.ProtobufSerializer
> import org.apache.flink.api.java.typeutils.runtime.NoFetchingInput
> import java.io.ByteArrayInputStream
>
> object ProtoSerializationTest {
> def main(args: Array[String]) = {
> val chillProtoSerializer = new ProtobufSerializer
> val protomessage = DescriptorProtos.DescriptorProto.getDefaultInstance
> val output: Output = new ByteBufferOutput(1000)
> chillProtoSerializer.write(null, output, protomessage)
> val serialized: Array[Byte] = output.toBytes
> println(s"Serialized : $serialized")
> val input: Input = new NoFetchingInput(new
> ByteArrayInputStream(serialized))
> val deserialized = chillProtoSerializer.read(null, input,
> classOf[BillableClick].asInstanceOf[Class[Message]])
> println(deserialized)
> }
> }
> {noformat}
>
> Error
> {noformat}
> Exception in thread "main" java.lang.RuntimeException: Could not create class
> com.criteo.glup.BillableClickProto$BillableClick
> at
> com.twitter.chill.protobuf.ProtobufSerializer.read(ProtobufSerializer.java:76)
> at
> com.criteo.streaming.common.bootstrap.ProtoSerialization$.main(ProtoSerialization.scala:22)
> at ProtoSerialization.main(ProtoSerialization.scala)
> Caused by: com.esotericsoftware.kryo.KryoException: java.io.EOFException: No
> more bytes left.
> at
> org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.readBytes(NoFetchingInput.java:128)
> at com.esotericsoftware.kryo.io.Input.readBytes(Input.java:332)
> at
> com.twitter.chill.protobuf.ProtobufSerializer.read(ProtobufSerializer.java:73)
> ... 2 more
> Caused by: java.io.EOFException: No more bytes left.
> ... 5 more{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-34954) Kryo input implementation NoFetchingInput fails to handle zero length bytes
[
https://issues.apache.org/jira/browse/FLINK-34954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Cranmer reassigned FLINK-34954:
-
Assignee: Qinghui Xu
> Kryo input implementation NoFetchingInput fails to handle zero length bytes
> ---
>
> Key: FLINK-34954
> URL: https://issues.apache.org/jira/browse/FLINK-34954
> Project: Flink
> Issue Type: Bug
> Components: API / Type Serialization System
>Affects Versions: 1.19.0
>Reporter: Qinghui Xu
>Assignee: Qinghui Xu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
> If the serailized bytes are empty, `NoFetchingInput` will run into error when
> Kryo tries to deserialize it.
> Example: a protobuf 3 object that contains only default values will be
> serialized as 0 length byte array, and the deserialization later will fail.
> Illustration:
> {noformat}
> import com.esotericsoftware.kryo.Kryo
> import com.esotericsoftware.kryo.io.{ByteBufferInput, ByteBufferOutput,
> Input, Output}
> import com.google.protobuf.{DescriptorProtos, Message}import
> com.twitter.chill.protobuf.ProtobufSerializer
> import org.apache.flink.api.java.typeutils.runtime.NoFetchingInput
> import java.io.ByteArrayInputStream
>
> object ProtoSerializationTest {
> def main(args: Array[String]) = {
> val chillProtoSerializer = new ProtobufSerializer
> val protomessage = DescriptorProtos.DescriptorProto.getDefaultInstance
> val output: Output = new ByteBufferOutput(1000)
> chillProtoSerializer.write(null, output, protomessage)
> val serialized: Array[Byte] = output.toBytes
> println(s"Serialized : $serialized")
> val input: Input = new NoFetchingInput(new
> ByteArrayInputStream(serialized))
> val deserialized = chillProtoSerializer.read(null, input,
> classOf[BillableClick].asInstanceOf[Class[Message]])
> println(deserialized)
> }
> }
> {noformat}
>
> Error
> {noformat}
> Exception in thread "main" java.lang.RuntimeException: Could not create class
> com.criteo.glup.BillableClickProto$BillableClick
> at
> com.twitter.chill.protobuf.ProtobufSerializer.read(ProtobufSerializer.java:76)
> at
> com.criteo.streaming.common.bootstrap.ProtoSerialization$.main(ProtoSerialization.scala:22)
> at ProtoSerialization.main(ProtoSerialization.scala)
> Caused by: com.esotericsoftware.kryo.KryoException: java.io.EOFException: No
> more bytes left.
> at
> org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.readBytes(NoFetchingInput.java:128)
> at com.esotericsoftware.kryo.io.Input.readBytes(Input.java:332)
> at
> com.twitter.chill.protobuf.ProtobufSerializer.read(ProtobufSerializer.java:73)
> ... 2 more
> Caused by: java.io.EOFException: No more bytes left.
> ... 5 more{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-34954) Kryo input implementation NoFetchingInput fails to handle zero length bytes
[
https://issues.apache.org/jira/browse/FLINK-34954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Cranmer resolved FLINK-34954.
---
Resolution: Fixed
> Kryo input implementation NoFetchingInput fails to handle zero length bytes
> ---
>
> Key: FLINK-34954
> URL: https://issues.apache.org/jira/browse/FLINK-34954
> Project: Flink
> Issue Type: Bug
> Components: API / Type Serialization System
>Affects Versions: 1.19.0
>Reporter: Qinghui Xu
>Assignee: Qinghui Xu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
> If the serailized bytes are empty, `NoFetchingInput` will run into error when
> Kryo tries to deserialize it.
> Example: a protobuf 3 object that contains only default values will be
> serialized as 0 length byte array, and the deserialization later will fail.
> Illustration:
> {noformat}
> import com.esotericsoftware.kryo.Kryo
> import com.esotericsoftware.kryo.io.{ByteBufferInput, ByteBufferOutput,
> Input, Output}
> import com.google.protobuf.{DescriptorProtos, Message}import
> com.twitter.chill.protobuf.ProtobufSerializer
> import org.apache.flink.api.java.typeutils.runtime.NoFetchingInput
> import java.io.ByteArrayInputStream
>
> object ProtoSerializationTest {
> def main(args: Array[String]) = {
> val chillProtoSerializer = new ProtobufSerializer
> val protomessage = DescriptorProtos.DescriptorProto.getDefaultInstance
> val output: Output = new ByteBufferOutput(1000)
> chillProtoSerializer.write(null, output, protomessage)
> val serialized: Array[Byte] = output.toBytes
> println(s"Serialized : $serialized")
> val input: Input = new NoFetchingInput(new
> ByteArrayInputStream(serialized))
> val deserialized = chillProtoSerializer.read(null, input,
> classOf[BillableClick].asInstanceOf[Class[Message]])
> println(deserialized)
> }
> }
> {noformat}
>
> Error
> {noformat}
> Exception in thread "main" java.lang.RuntimeException: Could not create class
> com.criteo.glup.BillableClickProto$BillableClick
> at
> com.twitter.chill.protobuf.ProtobufSerializer.read(ProtobufSerializer.java:76)
> at
> com.criteo.streaming.common.bootstrap.ProtoSerialization$.main(ProtoSerialization.scala:22)
> at ProtoSerialization.main(ProtoSerialization.scala)
> Caused by: com.esotericsoftware.kryo.KryoException: java.io.EOFException: No
> more bytes left.
> at
> org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.readBytes(NoFetchingInput.java:128)
> at com.esotericsoftware.kryo.io.Input.readBytes(Input.java:332)
> at
> com.twitter.chill.protobuf.ProtobufSerializer.read(ProtobufSerializer.java:73)
> ... 2 more
> Caused by: java.io.EOFException: No more bytes left.
> ... 5 more{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-34954) Kryo input implementation NoFetchingInput fails to handle zero length bytes
[
https://issues.apache.org/jira/browse/FLINK-34954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Cranmer updated FLINK-34954:
--
Fix Version/s: 1.20.0
> Kryo input implementation NoFetchingInput fails to handle zero length bytes
> ---
>
> Key: FLINK-34954
> URL: https://issues.apache.org/jira/browse/FLINK-34954
> Project: Flink
> Issue Type: Bug
>Reporter: Qinghui Xu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
> If the serailized bytes are empty, `NoFetchingInput` will run into error when
> Kryo tries to deserialize it.
> Example: a protobuf 3 object that contains only default values will be
> serialized as 0 length byte array, and the deserialization later will fail.
> Illustration:
> {noformat}
> import com.esotericsoftware.kryo.Kryo
> import com.esotericsoftware.kryo.io.{ByteBufferInput, ByteBufferOutput,
> Input, Output}
> import com.google.protobuf.{DescriptorProtos, Message}import
> com.twitter.chill.protobuf.ProtobufSerializer
> import org.apache.flink.api.java.typeutils.runtime.NoFetchingInput
> import java.io.ByteArrayInputStream
>
> object ProtoSerializationTest {
> def main(args: Array[String]) = {
> val chillProtoSerializer = new ProtobufSerializer
> val protomessage = DescriptorProtos.DescriptorProto.getDefaultInstance
> val output: Output = new ByteBufferOutput(1000)
> chillProtoSerializer.write(null, output, protomessage)
> val serialized: Array[Byte] = output.toBytes
> println(s"Serialized : $serialized")
> val input: Input = new NoFetchingInput(new
> ByteArrayInputStream(serialized))
> val deserialized = chillProtoSerializer.read(null, input,
> classOf[BillableClick].asInstanceOf[Class[Message]])
> println(deserialized)
> }
> }
> {noformat}
>
> Error
> {noformat}
> Exception in thread "main" java.lang.RuntimeException: Could not create class
> com.criteo.glup.BillableClickProto$BillableClick
> at
> com.twitter.chill.protobuf.ProtobufSerializer.read(ProtobufSerializer.java:76)
> at
> com.criteo.streaming.common.bootstrap.ProtoSerialization$.main(ProtoSerialization.scala:22)
> at ProtoSerialization.main(ProtoSerialization.scala)
> Caused by: com.esotericsoftware.kryo.KryoException: java.io.EOFException: No
> more bytes left.
> at
> org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.readBytes(NoFetchingInput.java:128)
> at com.esotericsoftware.kryo.io.Input.readBytes(Input.java:332)
> at
> com.twitter.chill.protobuf.ProtobufSerializer.read(ProtobufSerializer.java:73)
> ... 2 more
> Caused by: java.io.EOFException: No more bytes left.
> ... 5 more{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-34954][core] Kryo Input bug fix [flink]
dannycranmer merged PR #24586: URL: https://github.com/apache/flink/pull/24586 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-35039) Create Profiling JobManager/TaskManager Instance failed
[ https://issues.apache.org/jira/browse/FLINK-35039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35039: --- Labels: pull-request-available (was: ) > Create Profiling JobManager/TaskManager Instance failed > --- > > Key: FLINK-35039 > URL: https://issues.apache.org/jira/browse/FLINK-35039 > Project: Flink > Issue Type: Bug > Components: Runtime / Web Frontend >Affects Versions: 1.19.0 > Environment: Hadoop 3.2.2 > Flink 1.19 >Reporter: ude >Priority: Major > Labels: pull-request-available > Attachments: image-2024-04-08-10-21-31-066.png, > image-2024-04-08-10-21-48-417.png, image-2024-04-08-10-30-16-683.png > > > I'm test the "async-profiler" feature in version 1.19, but when I submit a > task in yarn per-job mode, I get an error when I click Create Profiling > Instance on the flink Web UI page. > !image-2024-04-08-10-21-31-066.png! > !image-2024-04-08-10-21-48-417.png! > The error message obviously means that the yarn proxy server does not support > *POST* calls. I checked the code of _*WebAppProxyServlet.java*_ and found > that the *POST* method is indeed not supported, so I changed it to *PUT* > method and the call was successful. > !image-2024-04-08-10-30-16-683.png! > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35039][rest] Use PUT method supported by YARN web proxy instead of POST [flink]
flinkbot commented on PR #24689: URL: https://github.com/apache/flink/pull/24689#issuecomment-2066235806 ## CI report: * 5179e513e55a60de15d50b32497442d432cf051b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-34954) Kryo input implementation NoFetchingInput fails to handle zero length bytes
[
https://issues.apache.org/jira/browse/FLINK-34954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17838938#comment-17838938
]
Danny Cranmer commented on FLINK-34954:
---
Merged commit
[{{3977982}}|https://github.com/apache/flink/commit/39779829b8853c61c9da9eaf193f0fedf5857f7b]
into apache:master
> Kryo input implementation NoFetchingInput fails to handle zero length bytes
> ---
>
> Key: FLINK-34954
> URL: https://issues.apache.org/jira/browse/FLINK-34954
> Project: Flink
> Issue Type: Bug
> Components: API / Type Serialization System
>Affects Versions: 1.19.0
>Reporter: Qinghui Xu
>Assignee: Qinghui Xu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
> If the serailized bytes are empty, `NoFetchingInput` will run into error when
> Kryo tries to deserialize it.
> Example: a protobuf 3 object that contains only default values will be
> serialized as 0 length byte array, and the deserialization later will fail.
> Illustration:
> {noformat}
> import com.esotericsoftware.kryo.Kryo
> import com.esotericsoftware.kryo.io.{ByteBufferInput, ByteBufferOutput,
> Input, Output}
> import com.google.protobuf.{DescriptorProtos, Message}import
> com.twitter.chill.protobuf.ProtobufSerializer
> import org.apache.flink.api.java.typeutils.runtime.NoFetchingInput
> import java.io.ByteArrayInputStream
>
> object ProtoSerializationTest {
> def main(args: Array[String]) = {
> val chillProtoSerializer = new ProtobufSerializer
> val protomessage = DescriptorProtos.DescriptorProto.getDefaultInstance
> val output: Output = new ByteBufferOutput(1000)
> chillProtoSerializer.write(null, output, protomessage)
> val serialized: Array[Byte] = output.toBytes
> println(s"Serialized : $serialized")
> val input: Input = new NoFetchingInput(new
> ByteArrayInputStream(serialized))
> val deserialized = chillProtoSerializer.read(null, input,
> classOf[BillableClick].asInstanceOf[Class[Message]])
> println(deserialized)
> }
> }
> {noformat}
>
> Error
> {noformat}
> Exception in thread "main" java.lang.RuntimeException: Could not create class
> com.criteo.glup.BillableClickProto$BillableClick
> at
> com.twitter.chill.protobuf.ProtobufSerializer.read(ProtobufSerializer.java:76)
> at
> com.criteo.streaming.common.bootstrap.ProtoSerialization$.main(ProtoSerialization.scala:22)
> at ProtoSerialization.main(ProtoSerialization.scala)
> Caused by: com.esotericsoftware.kryo.KryoException: java.io.EOFException: No
> more bytes left.
> at
> org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.readBytes(NoFetchingInput.java:128)
> at com.esotericsoftware.kryo.io.Input.readBytes(Input.java:332)
> at
> com.twitter.chill.protobuf.ProtobufSerializer.read(ProtobufSerializer.java:73)
> ... 2 more
> Caused by: java.io.EOFException: No more bytes left.
> ... 5 more{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-34954) Kryo input implementation NoFetchingInput fails to handle zero length bytes
[
https://issues.apache.org/jira/browse/FLINK-34954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Cranmer updated FLINK-34954:
--
Affects Version/s: 1.19.0
> Kryo input implementation NoFetchingInput fails to handle zero length bytes
> ---
>
> Key: FLINK-34954
> URL: https://issues.apache.org/jira/browse/FLINK-34954
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.19.0
>Reporter: Qinghui Xu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
> If the serailized bytes are empty, `NoFetchingInput` will run into error when
> Kryo tries to deserialize it.
> Example: a protobuf 3 object that contains only default values will be
> serialized as 0 length byte array, and the deserialization later will fail.
> Illustration:
> {noformat}
> import com.esotericsoftware.kryo.Kryo
> import com.esotericsoftware.kryo.io.{ByteBufferInput, ByteBufferOutput,
> Input, Output}
> import com.google.protobuf.{DescriptorProtos, Message}import
> com.twitter.chill.protobuf.ProtobufSerializer
> import org.apache.flink.api.java.typeutils.runtime.NoFetchingInput
> import java.io.ByteArrayInputStream
>
> object ProtoSerializationTest {
> def main(args: Array[String]) = {
> val chillProtoSerializer = new ProtobufSerializer
> val protomessage = DescriptorProtos.DescriptorProto.getDefaultInstance
> val output: Output = new ByteBufferOutput(1000)
> chillProtoSerializer.write(null, output, protomessage)
> val serialized: Array[Byte] = output.toBytes
> println(s"Serialized : $serialized")
> val input: Input = new NoFetchingInput(new
> ByteArrayInputStream(serialized))
> val deserialized = chillProtoSerializer.read(null, input,
> classOf[BillableClick].asInstanceOf[Class[Message]])
> println(deserialized)
> }
> }
> {noformat}
>
> Error
> {noformat}
> Exception in thread "main" java.lang.RuntimeException: Could not create class
> com.criteo.glup.BillableClickProto$BillableClick
> at
> com.twitter.chill.protobuf.ProtobufSerializer.read(ProtobufSerializer.java:76)
> at
> com.criteo.streaming.common.bootstrap.ProtoSerialization$.main(ProtoSerialization.scala:22)
> at ProtoSerialization.main(ProtoSerialization.scala)
> Caused by: com.esotericsoftware.kryo.KryoException: java.io.EOFException: No
> more bytes left.
> at
> org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.readBytes(NoFetchingInput.java:128)
> at com.esotericsoftware.kryo.io.Input.readBytes(Input.java:332)
> at
> com.twitter.chill.protobuf.ProtobufSerializer.read(ProtobufSerializer.java:73)
> ... 2 more
> Caused by: java.io.EOFException: No more bytes left.
> ... 5 more{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] Flink 35039 [flink]
yeezychao opened a new pull request, #24689: URL: https://github.com/apache/flink/pull/24689 ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change Please make sure both new and modified tests in this PR follow [the conventions for tests defined in our code quality guide](https://flink.apache.org/how-to-contribute/code-style-and-quality-common/#7-testing). *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-35115) Kinesis connector writes wrong Kinesis sequence number at stop with savepoint
[
https://issues.apache.org/jira/browse/FLINK-35115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Cranmer resolved FLINK-35115.
---
Resolution: Fixed
> Kinesis connector writes wrong Kinesis sequence number at stop with savepoint
> -
>
> Key: FLINK-35115
> URL: https://issues.apache.org/jira/browse/FLINK-35115
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: 1.15.4, aws-connector-4.0.0, aws-connector-4.1.0,
> aws-connector-4.2.0, 1.16.3, 1.17.2, 1.18.1
> Environment: The issue happens in a *Kinesis -> Flink -> Kafka*
> exactly-once setup with:
> * Flink versions checked 1.16.3 and 1.18.1
> * Kinesis connector checked 1.16.3 and 4.2.0-1.18
> * checkpointing configured at 1 minute with EXACTLY_ONCE mode:
> {code:java}
> StreamExecutionEnvironment execEnv =
> StreamExecutionEnvironment.getExecutionEnvironment ();
> execEnv.enableCheckpointing (6,EXACTLY_ONCE); execEnv.getCheckpointConfig
> ().setCheckpointTimeout (9); execEnv.getCheckpointConfig
> ().setCheckpointStorage (CHECKPOINTS_PATH); {code}
> * Kafka sink configured with EXACTLY_ONCE semantic/delivery guarantee:
> {code:java}
> Properties sinkConfig = new Properties ();
> sinkConfig.put ("transaction.timeout.ms", 48);
> KafkaSink sink = KafkaSink.builder ()
> .setBootstrapServers ("localhost:9092")
> .setTransactionalIdPrefix ("test-prefix")
> .setDeliverGuarantee (EXACTLY_ONCE)
> .setKafkaProducerConfig (sinkConfig)
> .setRecordSerializer (
> (KafkaRecordSerializationSchema) (element, context,
> timestamp) -> new ProducerRecord<> (
> "test-output-topic", null, element.getBytes ()))
> .build (); {code}
> * Kinesis consumer defined as:
> {code:java}
> FlinkKinesisConsumer flinkKinesisConsumer = new
> FlinkKinesisConsumer<> ("test-stream",
> new AbstractDeserializationSchema<> () {
> @Override
> public ByteBuffer deserialize (byte[] bytes) {
> // Return
> return ByteBuffer.wrap (bytes);
> }
> }, props); {code}
>
>Reporter: Vadim Vararu
>Assignee: Aleksandr Pilipenko
>Priority: Blocker
> Labels: kinesis, pull-request-available
> Fix For: aws-connector-4.3.0
>
>
> Having an exactly-once Kinesis -> Flink -> Kafka job and triggering a
> stop-with-savepoint, Flink duplicates in Kafka all the records between the
> last checkpoint and the savepoint at resume:
> * Event1 is written to Kinesis
> * Event1 is processed by Flink
> * Event1 is committed to Kafka at the checkpoint
> *
>
> * Event2 is written to Kinesis
> * Event2 is processed by Flink
> * Stop with savepoint is triggered manually
> * Event2 is committed to Kafka
> *
>
> * Job is resumed from the savepoint
> * *{color:#FF}Event2 is written again to Kafka at the first
> checkpoint{color}*
>
> {color:#172b4d}I believe that it's a Kinesis connector issue for 2
> reasons:{color}
> * I've checked the actual Kinesis sequence number in the _metadata file
> generated at stop-with-savepoint and it's the one from the checkpoint before
> the savepoint instead of being the one of the last record committed to Kafka.
> * I've tested exactly the save job with Kafka as source instead of Kinesis
> as source and the behaviour does not reproduce.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-35173) Debezium Custom Time Serializer
ZhengYu Chen created FLINK-35173:
Summary: Debezium Custom Time Serializer
Key: FLINK-35173
URL: https://issues.apache.org/jira/browse/FLINK-35173
Project: Flink
Issue Type: Improvement
Components: Flink CDC
Affects Versions: 3.1.0
Reporter: ZhengYu Chen
Fix For: 3.1.0
Currently, Flink CDC Time encounters time type errors (including DateTime,
Time, Date, TimeStamp) when using MySQL Connector
(JsonDebeziumDeserializationSchema) as deserialization, and the converted time
is wrong. The essential reason is that the timestamp returned by the bottom
layer of debezium is UTC (such as io.debezium.time.Timestamp). The community
has already had some
[PR|https://github.com/apache/flink-cdc/pull/1366/files#diff-e129e9fae3eea0bb32f0019debb4932413c91088d6dae656e2ecb63913badae4],
but they are not work.
Now a way is provided to provide a solution based on Debezium's custom Convert
interface
(https://debezium.io/documentation/reference/1.9/development/converters.html),
Users can choose to convert the above four time types into STRING according to
the specified time format to ensure that users can correctly convert JSON when
using the Flink DataStream API.
When the user enables this converter, we need to configure it according to the
parameters, That's some datastream use case:
{code:java}
Properties debeziumProperties = new Properties();
debeziumProperties.setProperty("converters", "datetime");
debeziumProperties.setProperty("datetime.database.type",
DataBaseType.MYSQL.getType());
debeziumProperties.setProperty("datetime.type",
"cn.xxx.sources.cdc.MysqlDebeziumConverter");
debeziumProperties.setProperty("datetime.format.date", "-MM-dd");
debeziumProperties.setProperty("datetime.format.time", "HH:mm:ss");
debeziumProperties.setProperty("datetime.format.datetime", "-MM-dd
HH:mm:ss");
debeziumProperties.setProperty("datetime.format.timestamp", "-MM-dd
HH:mm:ss");
debeziumProperties.setProperty("datetime.format.timestamp.zone", "UTC+8");
MySqlSourceBuilder builder = MySqlSource.builder()
.hostname(url[0])
.port(Integer.parseInt(url[1]))
.databaseList(table.getDatabase())
.tableList(getTablePattern(table))
.username(table.getUserName())
.password(table.getPassword())
.debeziumProperties(debeziumProperties); {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-35028][runtime] Timer firing under async execution model [flink]
fredia commented on PR #24672: URL: https://github.com/apache/flink/pull/24672#issuecomment-2066225062 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35156][Runtime] Make operators of DataStream V2 integrate with async state processing framework [flink]
reswqa commented on code in PR #24678:
URL: https://github.com/apache/flink/pull/24678#discussion_r1572117471
##
flink-datastream/src/main/java/org/apache/flink/datastream/impl/operators/ProcessOperator.java:
##
@@ -23,15 +23,15 @@
import org.apache.flink.datastream.impl.common.TimestampCollector;
import org.apache.flink.datastream.impl.context.DefaultNonPartitionedContext;
import org.apache.flink.datastream.impl.context.DefaultRuntimeContext;
-import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
import org.apache.flink.streaming.api.operators.BoundedOneInput;
import org.apache.flink.streaming.api.operators.ChainingStrategy;
import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import
org.apache.flink.streaming.runtime.operators.asyncprocessing.AbstractAsyncStateUdfStreamOperator;
import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
/** Operator for {@link OneInputStreamProcessFunction}. */
public class ProcessOperator
-extends AbstractUdfStreamOperator>
+extends AbstractAsyncStateUdfStreamOperator>
Review Comment:
I think we should also do this for
`TwoInputBroadcastProcessOperator`/`TwoInputNonBroadcastProcessOperator` and
`TwoOutputProcessOperator` 樂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-35115) Kinesis connector writes wrong Kinesis sequence number at stop with savepoint
[
https://issues.apache.org/jira/browse/FLINK-35115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17838934#comment-17838934
]
Danny Cranmer commented on FLINK-35115:
---
Merged commit
[{{8d29147}}|https://github.com/apache/flink-connector-aws/commit/8d29147b9e6c0a7d27399662c6023ad634363764]
into apache:main
> Kinesis connector writes wrong Kinesis sequence number at stop with savepoint
> -
>
> Key: FLINK-35115
> URL: https://issues.apache.org/jira/browse/FLINK-35115
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: 1.15.4, aws-connector-4.0.0, aws-connector-4.1.0,
> aws-connector-4.2.0, 1.16.3, 1.17.2, 1.18.1
> Environment: The issue happens in a *Kinesis -> Flink -> Kafka*
> exactly-once setup with:
> * Flink versions checked 1.16.3 and 1.18.1
> * Kinesis connector checked 1.16.3 and 4.2.0-1.18
> * checkpointing configured at 1 minute with EXACTLY_ONCE mode:
> {code:java}
> StreamExecutionEnvironment execEnv =
> StreamExecutionEnvironment.getExecutionEnvironment ();
> execEnv.enableCheckpointing (6,EXACTLY_ONCE); execEnv.getCheckpointConfig
> ().setCheckpointTimeout (9); execEnv.getCheckpointConfig
> ().setCheckpointStorage (CHECKPOINTS_PATH); {code}
> * Kafka sink configured with EXACTLY_ONCE semantic/delivery guarantee:
> {code:java}
> Properties sinkConfig = new Properties ();
> sinkConfig.put ("transaction.timeout.ms", 48);
> KafkaSink sink = KafkaSink.builder ()
> .setBootstrapServers ("localhost:9092")
> .setTransactionalIdPrefix ("test-prefix")
> .setDeliverGuarantee (EXACTLY_ONCE)
> .setKafkaProducerConfig (sinkConfig)
> .setRecordSerializer (
> (KafkaRecordSerializationSchema) (element, context,
> timestamp) -> new ProducerRecord<> (
> "test-output-topic", null, element.getBytes ()))
> .build (); {code}
> * Kinesis consumer defined as:
> {code:java}
> FlinkKinesisConsumer flinkKinesisConsumer = new
> FlinkKinesisConsumer<> ("test-stream",
> new AbstractDeserializationSchema<> () {
> @Override
> public ByteBuffer deserialize (byte[] bytes) {
> // Return
> return ByteBuffer.wrap (bytes);
> }
> }, props); {code}
>
>Reporter: Vadim Vararu
>Assignee: Aleksandr Pilipenko
>Priority: Blocker
> Labels: kinesis, pull-request-available
> Fix For: aws-connector-4.3.0
>
>
> Having an exactly-once Kinesis -> Flink -> Kafka job and triggering a
> stop-with-savepoint, Flink duplicates in Kafka all the records between the
> last checkpoint and the savepoint at resume:
> * Event1 is written to Kinesis
> * Event1 is processed by Flink
> * Event1 is committed to Kafka at the checkpoint
> *
>
> * Event2 is written to Kinesis
> * Event2 is processed by Flink
> * Stop with savepoint is triggered manually
> * Event2 is committed to Kafka
> *
>
> * Job is resumed from the savepoint
> * *{color:#FF}Event2 is written again to Kafka at the first
> checkpoint{color}*
>
> {color:#172b4d}I believe that it's a Kinesis connector issue for 2
> reasons:{color}
> * I've checked the actual Kinesis sequence number in the _metadata file
> generated at stop-with-savepoint and it's the one from the checkpoint before
> the savepoint instead of being the one of the last record committed to Kafka.
> * I've tested exactly the save job with Kafka as source instead of Kinesis
> as source and the behaviour does not reproduce.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
