Re: [PR] [FLINK-26948][table] Add-ARRAY_SORT-function. [flink]
hanyuzheng7 commented on PR #22951: URL: https://github.com/apache/flink/pull/22951#issuecomment-1936932786 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-34420) Various YARN tests fail after failing to download hadoop.tar.gz
Roman Khachatryan created FLINK-34420: - Summary: Various YARN tests fail after failing to download hadoop.tar.gz Key: FLINK-34420 URL: https://issues.apache.org/jira/browse/FLINK-34420 Project: Flink Issue Type: Bug Components: Tests Affects Versions: 1.19.0, 1.18.2, 1.20.0 Reporter: Roman Khachatryan Assignee: Roman Khachatryan Fix For: 1.19.0, 1.18.2, 1.20.0 https://dev.azure.com/khachatryanroman/flink/_build/results?buildId=1702&view=logs&j=0e31ee24-31a6-528c-a4bf-45cde9b2a14e&t=696bc156-f753-5888-468e-42d78df39222&l=11334 https://dev.azure.com/khachatryanroman/flink/_build/results?buildId=1700&view=logs&j=bbbd0720-137e-5f59-95a5-b5d332f196d3&t=4769aa47-e87b-5ecd-1fb2-14d52396866d&l=9937 ``` 2024-02-09T19:21:57.8947690Z Feb 09 19:21:57 Pre-downloading Hadoop tarball 2024-02-09T19:21:57.9250518Z % Total% Received % Xferd Average Speed TimeTime Time Current 2024-02-09T19:21:57.9250844Z Dload Upload Total SpentLeft Speed 2024-02-09T19:21:57.9251910Z 2024-02-09T19:21:58.0005684Z 0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-02-09T19:21:58.0006583Z 100 288 100 2880 0 3789 0 --:--:-- --:--:-- --:--:-- 3789 ``` which is way too small - meaning we got redirection (https://dev.azure.com/khachatryanroman/flink/_build/results?buildId=1700&view=logs&j=bbbd0720-137e-5f59-95a5-b5d332f196d3&t=4769aa47-e87b-5ecd-1fb2-14d52396866d&l=8056 ) Later, it can't be unpacked: https://dev.azure.com/khachatryanroman/flink/_build/results?buildId=1700&view=logs&j=bbbd0720-137e-5f59-95a5-b5d332f196d3&t=4769aa47-e87b-5ecd-1fb2-14d52396866d&l=9657 ``` #11 [ 7/28] COPY hadoop.tar.gz /tmp/hadoop.tar.gz #11 CACHED #12 [ 8/28] RUN set -x && mkdir -p /usr/local/hadoop && tar -xf /tmp/hadoop.tar.gz --strip-components=1 -C /usr/local/hadoop && rm /tmp/hadoop.tar.gz* #12 0.175 + mkdir -p /usr/local/hadoop #12 0.177 + tar -xf /tmp/hadoop.tar.gz --strip-components=1 -C /usr/local/hadoop #12 0.178 tar: This does not look like a tar archive #12 0.179 #12 0.179 gzip: stdin: not in gzip format #12 0.179 tar: Child returned status 1 #12 0.179 tar: Error is not recoverable: exiting now ``` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34420) Various YARN tests fail after failing to download hadoop.tar.gz
[ https://issues.apache.org/jira/browse/FLINK-34420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Khachatryan updated FLINK-34420: -- Priority: Blocker (was: Major) > Various YARN tests fail after failing to download hadoop.tar.gz > --- > > Key: FLINK-34420 > URL: https://issues.apache.org/jira/browse/FLINK-34420 > Project: Flink > Issue Type: Bug > Components: Tests >Affects Versions: 1.19.0, 1.18.2, 1.20.0 >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Blocker > Fix For: 1.19.0, 1.18.2, 1.20.0 > > > https://dev.azure.com/khachatryanroman/flink/_build/results?buildId=1702&view=logs&j=0e31ee24-31a6-528c-a4bf-45cde9b2a14e&t=696bc156-f753-5888-468e-42d78df39222&l=11334 > https://dev.azure.com/khachatryanroman/flink/_build/results?buildId=1700&view=logs&j=bbbd0720-137e-5f59-95a5-b5d332f196d3&t=4769aa47-e87b-5ecd-1fb2-14d52396866d&l=9937 > ``` > 2024-02-09T19:21:57.8947690Z Feb 09 19:21:57 Pre-downloading Hadoop tarball > 2024-02-09T19:21:57.9250518Z % Total% Received % Xferd Average Speed > TimeTime Time Current > 2024-02-09T19:21:57.9250844Z Dload Upload > Total SpentLeft Speed > 2024-02-09T19:21:57.9251910Z > 2024-02-09T19:21:58.0005684Z 0 00 00 0 0 0 > --:--:-- --:--:-- --:--:-- 0 > 2024-02-09T19:21:58.0006583Z 100 288 100 2880 0 3789 0 > --:--:-- --:--:-- --:--:-- 3789 > ``` > which is way too small - meaning we got redirection > (https://dev.azure.com/khachatryanroman/flink/_build/results?buildId=1700&view=logs&j=bbbd0720-137e-5f59-95a5-b5d332f196d3&t=4769aa47-e87b-5ecd-1fb2-14d52396866d&l=8056 > ) > Later, it can't be unpacked: > https://dev.azure.com/khachatryanroman/flink/_build/results?buildId=1700&view=logs&j=bbbd0720-137e-5f59-95a5-b5d332f196d3&t=4769aa47-e87b-5ecd-1fb2-14d52396866d&l=9657 > ``` > #11 [ 7/28] COPY hadoop.tar.gz /tmp/hadoop.tar.gz > #11 CACHED > #12 [ 8/28] RUN set -x && mkdir -p /usr/local/hadoop && tar -xf > /tmp/hadoop.tar.gz --strip-components=1 -C /usr/local/hadoop && rm > /tmp/hadoop.tar.gz* > #12 0.175 + mkdir -p /usr/local/hadoop > #12 0.177 + tar -xf /tmp/hadoop.tar.gz --strip-components=1 -C > /usr/local/hadoop > #12 0.178 tar: This does not look like a tar archive > #12 0.179 > #12 0.179 gzip: stdin: not in gzip format > #12 0.179 tar: Child returned status 1 > #12 0.179 tar: Error is not recoverable: exiting now > ``` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-34410] Correct hadoop.tar.gz download url [flink]
rkhachatryan opened a new pull request, #24293: URL: https://github.com/apache/flink/pull/24293 ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34410] Correct hadoop.tar.gz download url [flink]
flinkbot commented on PR #24293: URL: https://github.com/apache/flink/pull/24293#issuecomment-1936939562 ## CI report: * da9837b1ec758d5976a6d64431421101e56e90e7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34417] Log Job ID via MDC [flink]
rkhachatryan commented on PR #24292: URL: https://github.com/apache/flink/pull/24292#issuecomment-1936944954 The failure is unrelated: FLINK-34420 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [BP-1.18][FLINK-34410] Correct hadoop.tar.gz download url [flink]
flinkbot commented on PR #24294: URL: https://github.com/apache/flink/pull/24294#issuecomment-1936945024 ## CI report: * 7b6be65db700d3e7c674008f72c31f48a28eaad2 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34410] Correct hadoop.tar.gz download url [flink]
flinkbot commented on PR #24295: URL: https://github.com/apache/flink/pull/24295#issuecomment-1936945066 ## CI report: * e189d658d5d19062c88b817b3a15938349f97598 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34401][docs-zh] Translate "Flame Graphs" page into Chinese [flink]
lxliyou001 commented on PR #24279: URL: https://github.com/apache/flink/pull/24279#issuecomment-1936962285 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18356) flink-table-planner Exit code 137 returned from process
[
https://issues.apache.org/jira/browse/FLINK-18356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17816293#comment-17816293
]
Chesnay Schepler commented on FLINK-18356:
--
Can we re-enable fork-reuse for the planner tests then?
{code}
false
{code}
> flink-table-planner Exit code 137 returned from process
> ---
>
> Key: FLINK-18356
> URL: https://issues.apache.org/jira/browse/FLINK-18356
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines, Tests
>Affects Versions: 1.12.0, 1.13.0, 1.14.0, 1.15.0, 1.16.0, 1.17.0, 1.18.0,
> 1.19.0
>Reporter: Piotr Nowojski
>Assignee: Yunhong Zheng
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.19.0, 1.18.1, 1.17.3
>
> Attachments: 1234.jpg, app-profiling_4.gif,
> image-2023-01-11-22-21-57-784.png, image-2023-01-11-22-22-32-124.png,
> image-2023-02-16-20-18-09-431.png, image-2023-07-11-19-28-52-851.png,
> image-2023-07-11-19-35-54-530.png, image-2023-07-11-19-41-18-626.png,
> image-2023-07-11-19-41-37-105.png
>
>
> {noformat}
> = test session starts
> ==
> platform linux -- Python 3.7.3, pytest-5.4.3, py-1.8.2, pluggy-0.13.1
> cachedir: .tox/py37-cython/.pytest_cache
> rootdir: /__w/3/s/flink-python
> collected 568 items
> pyflink/common/tests/test_configuration.py ..[
> 1%]
> pyflink/common/tests/test_execution_config.py ...[
> 5%]
> pyflink/dataset/tests/test_execution_environment.py .
> ##[error]Exit code 137 returned from process: file name '/bin/docker',
> arguments 'exec -i -u 1002
> 97fc4e22522d2ced1f4d23096b8929045d083dd0a99a4233a8b20d0489e9bddb
> /__a/externals/node/bin/node /__w/_temp/containerHandlerInvoker.js'.
> Finishing: Test - python
> {noformat}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=3729&view=logs&j=9cada3cb-c1d3-5621-16da-0f718fb86602&t=8d78fe4f-d658-5c70-12f8-4921589024c3
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34421) Skip post-compile checks in compile.sh if fast profile is active
Chesnay Schepler created FLINK-34421: Summary: Skip post-compile checks in compile.sh if fast profile is active Key: FLINK-34421 URL: https://issues.apache.org/jira/browse/FLINK-34421 Project: Flink Issue Type: Improvement Components: Build System / CI Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.19.0, 1.18.2, 1.20.0 We currently waste time in our e2e tests, re-running a bunch of post-compile checks (like packaging/licensing). Let's couple this to the -Dfast/-Pfast switches. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34421) Skip post-compile checks in compile.sh if fast profile is active
[ https://issues.apache.org/jira/browse/FLINK-34421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34421: --- Labels: pull-request-available (was: ) > Skip post-compile checks in compile.sh if fast profile is active > > > Key: FLINK-34421 > URL: https://issues.apache.org/jira/browse/FLINK-34421 > Project: Flink > Issue Type: Improvement > Components: Build System / CI >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0, 1.18.2, 1.20.0 > > > We currently waste time in our e2e tests, re-running a bunch of post-compile > checks (like packaging/licensing). > Let's couple this to the -Dfast/-Pfast switches. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34421] Skip post-compile checks if 'fast' profile is active [flink]
flinkbot commented on PR #24296: URL: https://github.com/apache/flink/pull/24296#issuecomment-1936990365 ## CI report: * e8ed5e8fd96ad32030b0f34017b0871a4ef0338f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34407][Connectors/Kinesis] Fix unstable test [flink-connector-aws]
hlteoh37 commented on PR #128: URL: https://github.com/apache/flink-connector-aws/pull/128#issuecomment-1937005215 LGTM thanks @z3d1k -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34407][Connectors/Kinesis] Fix unstable test [flink-connector-aws]
hlteoh37 merged PR #128: URL: https://github.com/apache/flink-connector-aws/pull/128 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-34407) Flaky tests causing workflow timeout
[
https://issues.apache.org/jira/browse/FLINK-34407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hong Liang Teoh resolved FLINK-34407.
-
Resolution: Fixed
> Flaky tests causing workflow timeout
>
>
> Key: FLINK-34407
> URL: https://issues.apache.org/jira/browse/FLINK-34407
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: aws-connector-4.2.0
>Reporter: Aleksandr Pilipenko
>Assignee: Aleksandr Pilipenko
>Priority: Major
> Labels: pull-request-available
>
> Example build:
> [https://github.com/apache/flink-connector-aws/actions/runs/7735404733]
> Tests are stuck retrying due to the following exception:
> {code:java}
> 797445 [main] WARN
> org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutRecordPublisher
> [] - Encountered recoverable error TimeoutException. Backing off for 0
> millis 00 (arn)
> org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutShardSubscriber$RecoverableFanOutSubscriberException:
> java.util.concurrent.TimeoutException: Timed out acquiring subscription -
> 00 (arn)
> at
> org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutShardSubscriber.handleErrorAndRethrow(FanOutShardSubscriber.java:327)
> ~[classes/:?]
> at
> org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutShardSubscriber.openSubscriptionToShard(FanOutShardSubscriber.java:283)
> ~[classes/:?]
> at
> org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutShardSubscriber.subscribeToShardAndConsumeRecords(FanOutShardSubscriber.java:210)
> ~[classes/:?]
> at
> org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutRecordPublisher.runWithBackoff(FanOutRecordPublisher.java:177)
> ~[classes/:?]
> at
> org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutRecordPublisher.run(FanOutRecordPublisher.java:130)
> ~[classes/:?]
> at
> org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutRecordPublisherTest.testCancelExitsGracefully(FanOutRecordPublisherTest.java:595)
> ~[test-classes/:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_402]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34407) Flaky tests causing workflow timeout
[
https://issues.apache.org/jira/browse/FLINK-34407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17816313#comment-17816313
]
Hong Liang Teoh commented on FLINK-34407:
-
merged commit
[{{b207606}}|https://github.com/apache/flink-connector-aws/commit/b207606a95d0ce508c55e69dd0dc6c598eb2fb3c]
into apache:main
> Flaky tests causing workflow timeout
>
>
> Key: FLINK-34407
> URL: https://issues.apache.org/jira/browse/FLINK-34407
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: aws-connector-4.2.0
>Reporter: Aleksandr Pilipenko
>Assignee: Aleksandr Pilipenko
>Priority: Major
> Labels: pull-request-available
>
> Example build:
> [https://github.com/apache/flink-connector-aws/actions/runs/7735404733]
> Tests are stuck retrying due to the following exception:
> {code:java}
> 797445 [main] WARN
> org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutRecordPublisher
> [] - Encountered recoverable error TimeoutException. Backing off for 0
> millis 00 (arn)
> org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutShardSubscriber$RecoverableFanOutSubscriberException:
> java.util.concurrent.TimeoutException: Timed out acquiring subscription -
> 00 (arn)
> at
> org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutShardSubscriber.handleErrorAndRethrow(FanOutShardSubscriber.java:327)
> ~[classes/:?]
> at
> org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutShardSubscriber.openSubscriptionToShard(FanOutShardSubscriber.java:283)
> ~[classes/:?]
> at
> org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutShardSubscriber.subscribeToShardAndConsumeRecords(FanOutShardSubscriber.java:210)
> ~[classes/:?]
> at
> org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutRecordPublisher.runWithBackoff(FanOutRecordPublisher.java:177)
> ~[classes/:?]
> at
> org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutRecordPublisher.run(FanOutRecordPublisher.java:130)
> ~[classes/:?]
> at
> org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutRecordPublisherTest.testCancelExitsGracefully(FanOutRecordPublisherTest.java:595)
> ~[test-classes/:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_402]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-34260][Connectors/AWS] Update flink-connector-aws to be compatible with updated SinkV2 interfaces [flink-connector-aws]
hlteoh37 merged PR #127: URL: https://github.com/apache/flink-connector-aws/pull/127 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-34260) Update flink-connector-aws to be compatible with updated SinkV2 interfaces
[
https://issues.apache.org/jira/browse/FLINK-34260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17816317#comment-17816317
]
Hong Liang Teoh commented on FLINK-34260:
-
merged commit
[{{5b6f087}}|https://github.com/apache/flink-connector-aws/commit/5b6f087815bcf18cf62ba39b2ac1f84f5e72f951]
into apache:main
> Update flink-connector-aws to be compatible with updated SinkV2 interfaces
> --
>
> Key: FLINK-34260
> URL: https://issues.apache.org/jira/browse/FLINK-34260
> Project: Flink
> Issue Type: Technical Debt
> Components: Connectors / AWS
>Affects Versions: aws-connector-4.3.0
>Reporter: Martijn Visser
>Assignee: Aleksandr Pilipenko
>Priority: Blocker
> Labels: pull-request-available
>
> https://github.com/apache/flink-connector-aws/actions/runs/7689300085/job/20951547366#step:9:798
> {code:java}
> Error: Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.8.0:testCompile
> (default-testCompile) on project flink-connector-dynamodb: Compilation failure
> Error:
> /home/runner/work/flink-connector-aws/flink-connector-aws/flink-connector-aws/flink-connector-dynamodb/src/test/java/org/apache/flink/connector/dynamodb/sink/DynamoDbSinkWriterTest.java:[357,40]
> incompatible types:
> org.apache.flink.connector.base.sink.writer.TestSinkInitContext cannot be
> converted to org.apache.flink.api.connector.sink2.Sink.InitContext
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-34260) Update flink-connector-aws to be compatible with updated SinkV2 interfaces
[
https://issues.apache.org/jira/browse/FLINK-34260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hong Liang Teoh resolved FLINK-34260.
-
Resolution: Fixed
> Update flink-connector-aws to be compatible with updated SinkV2 interfaces
> --
>
> Key: FLINK-34260
> URL: https://issues.apache.org/jira/browse/FLINK-34260
> Project: Flink
> Issue Type: Technical Debt
> Components: Connectors / AWS
>Affects Versions: aws-connector-4.3.0
>Reporter: Martijn Visser
>Assignee: Aleksandr Pilipenko
>Priority: Blocker
> Labels: pull-request-available
>
> https://github.com/apache/flink-connector-aws/actions/runs/7689300085/job/20951547366#step:9:798
> {code:java}
> Error: Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.8.0:testCompile
> (default-testCompile) on project flink-connector-dynamodb: Compilation failure
> Error:
> /home/runner/work/flink-connector-aws/flink-connector-aws/flink-connector-aws/flink-connector-dynamodb/src/test/java/org/apache/flink/connector/dynamodb/sink/DynamoDbSinkWriterTest.java:[357,40]
> incompatible types:
> org.apache.flink.connector.base.sink.writer.TestSinkInitContext cannot be
> converted to org.apache.flink.api.connector.sink2.Sink.InitContext
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] Update sql_functions.yml [flink]
hanyuzheng7 opened a new pull request, #24297: URL: https://github.com/apache/flink/pull/24297 ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Update sql_functions.yml [flink]
hanyuzheng7 commented on PR #24297: URL: https://github.com/apache/flink/pull/24297#issuecomment-1937064943 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Update sql_functions.yml [flink]
flinkbot commented on PR #24297: URL: https://github.com/apache/flink/pull/24297#issuecomment-1937068469 ## CI report: * 59daf5cdcee12104381e97d28c03f96a2169fe23 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Update sql_functions.yml [flink]
hanyuzheng7 commented on PR #24297: URL: https://github.com/apache/flink/pull/24297#issuecomment-1937080922 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34421] Skip post-compile checks if 'fast' profile is active [flink]
hanyuzheng7 commented on PR #24296: URL: https://github.com/apache/flink/pull/24296#issuecomment-1937097943 Hi, @zentol, it appears that Azure is failing the CI test, similar to the issue I'm encountering with my PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33936][table] Outputting Identical Results in Mini-Batch Aggregation with Set TTL [flink]
jeyhunkarimov commented on code in PR #24290:
URL: https://github.com/apache/flink/pull/24290#discussion_r1485270891
##
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/runtime/harness/GroupAggregateHarnessTest.scala:
##
@@ -220,6 +225,63 @@ class GroupAggregateHarnessTest(mode: StateBackendMode,
miniBatch: MiniBatchMode
testHarness.close()
}
+ @TestTemplate
+ def testGlobalAggregateWithRetraction(): Unit = {
+if (!this.miniBatch.on) {
+ return
+}
+val data = new mutable.MutableList[(String, String, Long)]
+val t = env.fromCollection(data).toTable(tEnv, 'a, 'b, 'c)
+tEnv.createTemporaryView("T", t)
+
+val sql =
+ """
+|SELECT a, SUM(c)
+|FROM (
+| SELECT a, b, SUM(c) as c
+| FROM T GROUP BY a, b
+|)GROUP BY a
+ """.stripMargin
+val t1 = tEnv.sqlQuery(sql)
+
+tEnv.getConfig.setIdleStateRetention(Duration.ofSeconds(2))
+tEnv.getConfig.set("table.optimizer.agg-phase-strategy", "TWO_PHASE")
Review Comment:
Does this also cover the changes in `MiniBatchGroupAggFunction` ?
##
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/aggregate/MiniBatchGlobalGroupAggFunction.java:
##
@@ -193,7 +193,7 @@ public void finishBundle(Map buffer,
Collector out)
// if this was not the first row and we have to emit
retractions
if (!firstRow) {
-if (!equaliser.equals(prevAggValue, newAggValue)) {
+if (stateRetentionTime > 0 ||
!equaliser.equals(prevAggValue, newAggValue)) {
Review Comment:
IMO, this logic should be ideally embedded in `globalAgg`. Since both
`prevAggValue ` and `newAggValue ` are derived from `globalAgg` and since
`globalAgg` has knowledge about ttl (`globalAgg.open(new
PerKeyStateDataViewStore(ctx.getRuntimeContext(), ttlConfig))`) it should be
better to have all the necessary logic in one place. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33936][table] Outputting Identical Results in Mini-Batch Aggregation with Set TTL [flink]
jeyhunkarimov commented on code in PR #24290:
URL: https://github.com/apache/flink/pull/24290#discussion_r1485282227
##
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/aggregate/MiniBatchGlobalGroupAggFunction.java:
##
@@ -193,7 +193,7 @@ public void finishBundle(Map buffer,
Collector out)
// if this was not the first row and we have to emit
retractions
if (!firstRow) {
-if (!equaliser.equals(prevAggValue, newAggValue)) {
+if (stateRetentionTime > 0 ||
!equaliser.equals(prevAggValue, newAggValue)) {
Review Comment:
IMO, this logic should be ideally embedded in `globalAgg`. Since both
`prevAggValue ` and `newAggValue ` are derived from `globalAgg` and since
`globalAgg` has knowledge about ttl (`globalAgg.open(new
PerKeyStateDataViewStore(ctx.getRuntimeContext(), ttlConfig))`) it should be
better to have all the necessary logic in one place. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-34422) BatchTestBase doesn't actually use MiniClusterExtension
[ https://issues.apache.org/jira/browse/FLINK-34422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-34422: - Description: BatchTestBase sets up a table environment in instance fields, which runs before the BeforeEachCallback from the MiniClusterExtension has time to run. As a result the table environment internally creates a local stream environment, due to which _all_ test extending the BatchTestBase are spawning separate mini clusters for every single job. was: BatchTestBase sets up a table environment in instance fields, which runs before the BeforeEachCallback from the MiniClusterExtension has time to run. As a result _all_ test extending the BatchTestBase are spawning separate mini clusters for every single job. > BatchTestBase doesn't actually use MiniClusterExtension > --- > > Key: FLINK-34422 > URL: https://issues.apache.org/jira/browse/FLINK-34422 > Project: Flink > Issue Type: Technical Debt > Components: Test Infrastructure >Affects Versions: 1.18.1 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.19.0, 1.18.2, 1.20.0 > > > BatchTestBase sets up a table environment in instance fields, which runs > before the BeforeEachCallback from the MiniClusterExtension has time to run. > As a result the table environment internally creates a local stream > environment, due to which _all_ test extending the BatchTestBase are spawning > separate mini clusters for every single job. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34422) BatchTestBase doesn't actually use MiniClusterExtension
Chesnay Schepler created FLINK-34422: Summary: BatchTestBase doesn't actually use MiniClusterExtension Key: FLINK-34422 URL: https://issues.apache.org/jira/browse/FLINK-34422 Project: Flink Issue Type: Technical Debt Components: Test Infrastructure Affects Versions: 1.18.1 Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.19.0, 1.18.2, 1.20.0 BatchTestBase sets up a table environment in instance fields, which runs before the BeforeEachCallback from the MiniClusterExtension has time to run. As a result _all_ test extending the BatchTestBase are spawning separate mini clusters for every single job. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34422) BatchTestBase doesn't actually use MiniClusterExtension
[ https://issues.apache.org/jira/browse/FLINK-34422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-34422: - Description: BatchTestBase sets up a table environment in instance fields, which runs before the BeforeEachCallback from the MiniClusterExtension has time to run. As a result the table environment internally creates a local stream environment, due to which _all_ test extending the BatchTestBase are spawning separate mini clusters for every single job. I believe this is on reason why we had troubles with enabling fork-reuse in the planner module was: BatchTestBase sets up a table environment in instance fields, which runs before the BeforeEachCallback from the MiniClusterExtension has time to run. As a result the table environment internally creates a local stream environment, due to which _all_ test extending the BatchTestBase are spawning separate mini clusters for every single job. > BatchTestBase doesn't actually use MiniClusterExtension > --- > > Key: FLINK-34422 > URL: https://issues.apache.org/jira/browse/FLINK-34422 > Project: Flink > Issue Type: Technical Debt > Components: Test Infrastructure >Affects Versions: 1.18.1 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.19.0, 1.18.2, 1.20.0 > > > BatchTestBase sets up a table environment in instance fields, which runs > before the BeforeEachCallback from the MiniClusterExtension has time to run. > As a result the table environment internally creates a local stream > environment, due to which _all_ test extending the BatchTestBase are spawning > separate mini clusters for every single job. > I believe this is on reason why we had troubles with enabling fork-reuse in > the planner module -- This message was sent by Atlassian Jira (v8.20.10#820010)
