[GitHub] [flink] zhongqishang commented on pull request #20563: [FLINK-28817][connector/common] NullPointerException in HybridSource when restoring from checkpoint
zhongqishang commented on PR #20563: URL: https://github.com/apache/flink/pull/20563#issuecomment-1213646006 @tweise -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #20563: [FLINK-28817][connector/common] NullPointerException in HybridSource when restoring from checkpoint
flinkbot commented on PR #20563: URL: https://github.com/apache/flink/pull/20563#issuecomment-1213645652 ## CI report: * c308ad59df76d5336b659756ace87e3c10d519e7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhongqishang opened a new pull request, #20563: [FLINK-28817][connector/common] NullPointerException in HybridSource when restoring from checkpoint
zhongqishang opened a new pull request, #20563: URL: https://github.com/apache/flink/pull/20563 ## What is the purpose of the change This is a backport of #20530 ## Brief change log For the new SourceReaderFinishedEvent, get an available source. ## Verifying this change This change added tests and can be verified as follows: - Added testRestoreEnumeratorWith2ndSource in HybridSourceSplitEnumeratorTest ## Does this pull request potentially affect one of the following parts: - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes) ## Documentation - Does this pull request introduce a new feature? (no) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-28855) ThriftObjectConversions compile failed
[
https://issues.apache.org/jira/browse/FLINK-28855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579206#comment-17579206
]
Shengkai Fang commented on FLINK-28855:
---
Merged into master: 1232629c80cbb64eb4ca9f6c95d6c5c1a2e8e82d
> ThriftObjectConversions compile failed
> --
>
> Key: FLINK-28855
> URL: https://issues.apache.org/jira/browse/FLINK-28855
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive, Table SQL / Gateway
>Affects Versions: 1.16.0
>Reporter: Huang Xingbo
>Assignee: yuzelin
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.16.0
>
>
> {code:java}
> 2022-08-08T00:32:45.5104326Z [ERROR]
> /home/vsts/work/1/s/flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/table/endpoint/hive/util/ThriftObjectConversions.java:[615,31]
> cannot find symbol
> 2022-08-08T00:32:45.5105191Z symbol: variable INDEX_TABLE
> 2022-08-08T00:32:45.5107273Z location: class
> org.apache.hadoop.hive.metastore.TableType
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=39514=logs=87489130-75dc-54e4-1f45-80c30aa367a3=4632ba9d-f1f2-5ad2-13fc-828d0e28bac4
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-28855) ThriftObjectConversions compile failed
[
https://issues.apache.org/jira/browse/FLINK-28855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shengkai Fang closed FLINK-28855.
-
Resolution: Fixed
> ThriftObjectConversions compile failed
> --
>
> Key: FLINK-28855
> URL: https://issues.apache.org/jira/browse/FLINK-28855
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive, Table SQL / Gateway
>Affects Versions: 1.16.0
>Reporter: Huang Xingbo
>Assignee: yuzelin
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.16.0
>
>
> {code:java}
> 2022-08-08T00:32:45.5104326Z [ERROR]
> /home/vsts/work/1/s/flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/table/endpoint/hive/util/ThriftObjectConversions.java:[615,31]
> cannot find symbol
> 2022-08-08T00:32:45.5105191Z symbol: variable INDEX_TABLE
> 2022-08-08T00:32:45.5107273Z location: class
> org.apache.hadoop.hive.metastore.TableType
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=39514=logs=87489130-75dc-54e4-1f45-80c30aa367a3=4632ba9d-f1f2-5ad2-13fc-828d0e28bac4
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] fsk119 merged pull request #20514: [FLINK-28855][hive] Fix 'Table.INDEX_TABLE' not found in the Hive3
fsk119 merged PR #20514: URL: https://github.com/apache/flink/pull/20514 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-28817) NullPointerException in HybridSource when restoring from checkpoint
[
https://issues.apache.org/jira/browse/FLINK-28817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579192#comment-17579192
]
Thomas Weise commented on FLINK-28817:
--
[~Benenson] thank you for the thorough investigation!
> NullPointerException in HybridSource when restoring from checkpoint
> ---
>
> Key: FLINK-28817
> URL: https://issues.apache.org/jira/browse/FLINK-28817
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.14.4, 1.15.1
>Reporter: Michael
>Assignee: Qishang Zhong
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.16.0
>
> Attachments: Preconditions-checkNotNull-error.zip,
> bf-29-JM-err-analysis.log
>
>
> Scenario:
> # CheckpointCoordinator - Completed checkpoint 14 for job
>
> # HybridSource successfully completed processing a few SourceFactories, that
> reads from s3
> # HybridSourceSplitEnumerator.switchEnumerator failed with
> com.amazonaws.SdkClientException: Unable to execute HTTP request: Read timed
> out. This is intermittent error, it is usually fixed, when Flink recover from
> checkpoint & repeat the operation.
> # Flink starts recovering from checkpoint,
> # HybridSourceSplitEnumerator receives
> SourceReaderFinishedEvent\{sourceIndex=-1}
> # Processing this event cause
> 2022/08/08 08:39:34.862 ERROR o.a.f.r.s.c.SourceCoordinator - Uncaught
> exception in the SplitEnumerator for Source Source: hybrid-source while
> handling operator event SourceEventWrapper[SourceReaderFinishedEvent
> {sourceIndex=-1}
> ] from subtask 6. Triggering job failover.
> java.lang.NullPointerException: Source for index=0 is not available from
> sources: \{788=org.apache.flink.connector.file.src.SppFileSource@5a3803f3}
> at org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:104)
> at
> org.apache.flink.connector.base.source.hybridspp.SwitchedSources.sourceOf(SwitchedSources.java:36)
> at
> org.apache.flink.connector.base.source.hybridspp.HybridSourceSplitEnumerator.sendSwitchSourceEvent(HybridSourceSplitEnumerator.java:152)
> at
> org.apache.flink.connector.base.source.hybridspp.HybridSourceSplitEnumerator.handleSourceEvent(HybridSourceSplitEnumerator.java:226)
> ...
> I'm running my version of the Hybrid Sources with additional logging, so line
> numbers & some names could be different from Flink Github.
> My Observation: the problem is intermittent, sometimes it works ok, i.e.
> SourceReaderFinishedEvent comes with correct sourceIndex. As I see from my
> log, it happens if my SourceFactory.create() is executed BEFORE
> HybridSourceSplitEnumerator - handleSourceEvent
> SourceReaderFinishedEvent\{sourceIndex=-1}.
> If HybridSourceSplitEnumerator - handleSourceEvent is executed before my
> SourceFactory.create(), then sourceIndex=-1 in SourceReaderFinishedEvent
> Preconditions-checkNotNull-error log from JobMgr is attached
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-28817) NullPointerException in HybridSource when restoring from checkpoint
[
https://issues.apache.org/jira/browse/FLINK-28817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Weise reassigned FLINK-28817:
Assignee: Qishang Zhong
> NullPointerException in HybridSource when restoring from checkpoint
> ---
>
> Key: FLINK-28817
> URL: https://issues.apache.org/jira/browse/FLINK-28817
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.14.4, 1.15.1
>Reporter: Michael
>Assignee: Qishang Zhong
>Priority: Major
> Labels: pull-request-available
> Attachments: Preconditions-checkNotNull-error.zip,
> bf-29-JM-err-analysis.log
>
>
> Scenario:
> # CheckpointCoordinator - Completed checkpoint 14 for job
>
> # HybridSource successfully completed processing a few SourceFactories, that
> reads from s3
> # HybridSourceSplitEnumerator.switchEnumerator failed with
> com.amazonaws.SdkClientException: Unable to execute HTTP request: Read timed
> out. This is intermittent error, it is usually fixed, when Flink recover from
> checkpoint & repeat the operation.
> # Flink starts recovering from checkpoint,
> # HybridSourceSplitEnumerator receives
> SourceReaderFinishedEvent\{sourceIndex=-1}
> # Processing this event cause
> 2022/08/08 08:39:34.862 ERROR o.a.f.r.s.c.SourceCoordinator - Uncaught
> exception in the SplitEnumerator for Source Source: hybrid-source while
> handling operator event SourceEventWrapper[SourceReaderFinishedEvent
> {sourceIndex=-1}
> ] from subtask 6. Triggering job failover.
> java.lang.NullPointerException: Source for index=0 is not available from
> sources: \{788=org.apache.flink.connector.file.src.SppFileSource@5a3803f3}
> at org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:104)
> at
> org.apache.flink.connector.base.source.hybridspp.SwitchedSources.sourceOf(SwitchedSources.java:36)
> at
> org.apache.flink.connector.base.source.hybridspp.HybridSourceSplitEnumerator.sendSwitchSourceEvent(HybridSourceSplitEnumerator.java:152)
> at
> org.apache.flink.connector.base.source.hybridspp.HybridSourceSplitEnumerator.handleSourceEvent(HybridSourceSplitEnumerator.java:226)
> ...
> I'm running my version of the Hybrid Sources with additional logging, so line
> numbers & some names could be different from Flink Github.
> My Observation: the problem is intermittent, sometimes it works ok, i.e.
> SourceReaderFinishedEvent comes with correct sourceIndex. As I see from my
> log, it happens if my SourceFactory.create() is executed BEFORE
> HybridSourceSplitEnumerator - handleSourceEvent
> SourceReaderFinishedEvent\{sourceIndex=-1}.
> If HybridSourceSplitEnumerator - handleSourceEvent is executed before my
> SourceFactory.create(), then sourceIndex=-1 in SourceReaderFinishedEvent
> Preconditions-checkNotNull-error log from JobMgr is attached
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28817) NullPointerException in HybridSource when restoring from checkpoint
[
https://issues.apache.org/jira/browse/FLINK-28817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Weise updated FLINK-28817:
-
Fix Version/s: 1.16.0
> NullPointerException in HybridSource when restoring from checkpoint
> ---
>
> Key: FLINK-28817
> URL: https://issues.apache.org/jira/browse/FLINK-28817
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.14.4, 1.15.1
>Reporter: Michael
>Assignee: Qishang Zhong
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.16.0
>
> Attachments: Preconditions-checkNotNull-error.zip,
> bf-29-JM-err-analysis.log
>
>
> Scenario:
> # CheckpointCoordinator - Completed checkpoint 14 for job
>
> # HybridSource successfully completed processing a few SourceFactories, that
> reads from s3
> # HybridSourceSplitEnumerator.switchEnumerator failed with
> com.amazonaws.SdkClientException: Unable to execute HTTP request: Read timed
> out. This is intermittent error, it is usually fixed, when Flink recover from
> checkpoint & repeat the operation.
> # Flink starts recovering from checkpoint,
> # HybridSourceSplitEnumerator receives
> SourceReaderFinishedEvent\{sourceIndex=-1}
> # Processing this event cause
> 2022/08/08 08:39:34.862 ERROR o.a.f.r.s.c.SourceCoordinator - Uncaught
> exception in the SplitEnumerator for Source Source: hybrid-source while
> handling operator event SourceEventWrapper[SourceReaderFinishedEvent
> {sourceIndex=-1}
> ] from subtask 6. Triggering job failover.
> java.lang.NullPointerException: Source for index=0 is not available from
> sources: \{788=org.apache.flink.connector.file.src.SppFileSource@5a3803f3}
> at org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:104)
> at
> org.apache.flink.connector.base.source.hybridspp.SwitchedSources.sourceOf(SwitchedSources.java:36)
> at
> org.apache.flink.connector.base.source.hybridspp.HybridSourceSplitEnumerator.sendSwitchSourceEvent(HybridSourceSplitEnumerator.java:152)
> at
> org.apache.flink.connector.base.source.hybridspp.HybridSourceSplitEnumerator.handleSourceEvent(HybridSourceSplitEnumerator.java:226)
> ...
> I'm running my version of the Hybrid Sources with additional logging, so line
> numbers & some names could be different from Flink Github.
> My Observation: the problem is intermittent, sometimes it works ok, i.e.
> SourceReaderFinishedEvent comes with correct sourceIndex. As I see from my
> log, it happens if my SourceFactory.create() is executed BEFORE
> HybridSourceSplitEnumerator - handleSourceEvent
> SourceReaderFinishedEvent\{sourceIndex=-1}.
> If HybridSourceSplitEnumerator - handleSourceEvent is executed before my
> SourceFactory.create(), then sourceIndex=-1 in SourceReaderFinishedEvent
> Preconditions-checkNotNull-error log from JobMgr is attached
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-28817) NullPointerException in HybridSource when restoring from checkpoint
[
https://issues.apache.org/jira/browse/FLINK-28817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Weise resolved FLINK-28817.
--
Resolution: Fixed
> NullPointerException in HybridSource when restoring from checkpoint
> ---
>
> Key: FLINK-28817
> URL: https://issues.apache.org/jira/browse/FLINK-28817
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.14.4, 1.15.1
>Reporter: Michael
>Assignee: Qishang Zhong
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.16.0
>
> Attachments: Preconditions-checkNotNull-error.zip,
> bf-29-JM-err-analysis.log
>
>
> Scenario:
> # CheckpointCoordinator - Completed checkpoint 14 for job
>
> # HybridSource successfully completed processing a few SourceFactories, that
> reads from s3
> # HybridSourceSplitEnumerator.switchEnumerator failed with
> com.amazonaws.SdkClientException: Unable to execute HTTP request: Read timed
> out. This is intermittent error, it is usually fixed, when Flink recover from
> checkpoint & repeat the operation.
> # Flink starts recovering from checkpoint,
> # HybridSourceSplitEnumerator receives
> SourceReaderFinishedEvent\{sourceIndex=-1}
> # Processing this event cause
> 2022/08/08 08:39:34.862 ERROR o.a.f.r.s.c.SourceCoordinator - Uncaught
> exception in the SplitEnumerator for Source Source: hybrid-source while
> handling operator event SourceEventWrapper[SourceReaderFinishedEvent
> {sourceIndex=-1}
> ] from subtask 6. Triggering job failover.
> java.lang.NullPointerException: Source for index=0 is not available from
> sources: \{788=org.apache.flink.connector.file.src.SppFileSource@5a3803f3}
> at org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:104)
> at
> org.apache.flink.connector.base.source.hybridspp.SwitchedSources.sourceOf(SwitchedSources.java:36)
> at
> org.apache.flink.connector.base.source.hybridspp.HybridSourceSplitEnumerator.sendSwitchSourceEvent(HybridSourceSplitEnumerator.java:152)
> at
> org.apache.flink.connector.base.source.hybridspp.HybridSourceSplitEnumerator.handleSourceEvent(HybridSourceSplitEnumerator.java:226)
> ...
> I'm running my version of the Hybrid Sources with additional logging, so line
> numbers & some names could be different from Flink Github.
> My Observation: the problem is intermittent, sometimes it works ok, i.e.
> SourceReaderFinishedEvent comes with correct sourceIndex. As I see from my
> log, it happens if my SourceFactory.create() is executed BEFORE
> HybridSourceSplitEnumerator - handleSourceEvent
> SourceReaderFinishedEvent\{sourceIndex=-1}.
> If HybridSourceSplitEnumerator - handleSourceEvent is executed before my
> SourceFactory.create(), then sourceIndex=-1 in SourceReaderFinishedEvent
> Preconditions-checkNotNull-error log from JobMgr is attached
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-28817) NullPointerException in HybridSource when restoring from checkpoint
[
https://issues.apache.org/jira/browse/FLINK-28817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579190#comment-17579190
]
Thomas Weise commented on FLINK-28817:
--
[~nicholasjiang] I believe it does. Can you please verify and close FLINK-26938
if so?
> NullPointerException in HybridSource when restoring from checkpoint
> ---
>
> Key: FLINK-28817
> URL: https://issues.apache.org/jira/browse/FLINK-28817
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.14.4, 1.15.1
>Reporter: Michael
>Priority: Major
> Labels: pull-request-available
> Attachments: Preconditions-checkNotNull-error.zip,
> bf-29-JM-err-analysis.log
>
>
> Scenario:
> # CheckpointCoordinator - Completed checkpoint 14 for job
>
> # HybridSource successfully completed processing a few SourceFactories, that
> reads from s3
> # HybridSourceSplitEnumerator.switchEnumerator failed with
> com.amazonaws.SdkClientException: Unable to execute HTTP request: Read timed
> out. This is intermittent error, it is usually fixed, when Flink recover from
> checkpoint & repeat the operation.
> # Flink starts recovering from checkpoint,
> # HybridSourceSplitEnumerator receives
> SourceReaderFinishedEvent\{sourceIndex=-1}
> # Processing this event cause
> 2022/08/08 08:39:34.862 ERROR o.a.f.r.s.c.SourceCoordinator - Uncaught
> exception in the SplitEnumerator for Source Source: hybrid-source while
> handling operator event SourceEventWrapper[SourceReaderFinishedEvent
> {sourceIndex=-1}
> ] from subtask 6. Triggering job failover.
> java.lang.NullPointerException: Source for index=0 is not available from
> sources: \{788=org.apache.flink.connector.file.src.SppFileSource@5a3803f3}
> at org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:104)
> at
> org.apache.flink.connector.base.source.hybridspp.SwitchedSources.sourceOf(SwitchedSources.java:36)
> at
> org.apache.flink.connector.base.source.hybridspp.HybridSourceSplitEnumerator.sendSwitchSourceEvent(HybridSourceSplitEnumerator.java:152)
> at
> org.apache.flink.connector.base.source.hybridspp.HybridSourceSplitEnumerator.handleSourceEvent(HybridSourceSplitEnumerator.java:226)
> ...
> I'm running my version of the Hybrid Sources with additional logging, so line
> numbers & some names could be different from Flink Github.
> My Observation: the problem is intermittent, sometimes it works ok, i.e.
> SourceReaderFinishedEvent comes with correct sourceIndex. As I see from my
> log, it happens if my SourceFactory.create() is executed BEFORE
> HybridSourceSplitEnumerator - handleSourceEvent
> SourceReaderFinishedEvent\{sourceIndex=-1}.
> If HybridSourceSplitEnumerator - handleSourceEvent is executed before my
> SourceFactory.create(), then sourceIndex=-1 in SourceReaderFinishedEvent
> Preconditions-checkNotNull-error log from JobMgr is attached
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] tweise commented on pull request #20530: [FLINK-28817] NullPointerException in HybridSource when restoring from checkpoint
tweise commented on PR #20530: URL: https://github.com/apache/flink/pull/20530#issuecomment-1213595465 @zhongqishang thanks for the contribution! Can you please open a backport PR for release-1.15 also? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tweise merged pull request #20530: [FLINK-28817] NullPointerException in HybridSource when restoring from checkpoint
tweise merged PR #20530: URL: https://github.com/apache/flink/pull/20530 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25122) flink-dist/src/main/flink-bin/bin/flink does not expand variable for FLINK_ENV_JAVA_OPTS
[
https://issues.apache.org/jira/browse/FLINK-25122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-25122:
---
Labels: auto-deprioritized-major pull-request-available stale-minor (was:
auto-deprioritized-major pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> flink-dist/src/main/flink-bin/bin/flink does not expand variable for
> FLINK_ENV_JAVA_OPTS
>
>
> Key: FLINK-25122
> URL: https://issues.apache.org/jira/browse/FLINK-25122
> Project: Flink
> Issue Type: Bug
> Components: Command Line Client
>Affects Versions: 1.14.0, 1.12.5
>Reporter: L Z
>Priority: Minor
> Labels: auto-deprioritized-major, pull-request-available,
> stale-minor
>
> According to the suggestion on
> [https://nightlies.apache.org/flink/flink-docs-master/docs/ops/debugging/application_profiling/]
> or
> [https://nightlies.apache.org/flink/flink-docs-release-1.12/ops/debugging/application_profiling.html]
>
> after adding
> {code:yaml}
> env.java.opts: "-Xloggc:${FLINK_LOG_PREFIX}.gc.log <...omitted chars...>"
> {code}
> to flink-conf.yaml.
> flink CLI `./bin/flink run ` fails with
> {code:bash}
> Invalid file name for use with -Xloggc: Filename can only contain the
> characters [A-Z][a-z][0-9]-_.%[p|t] but it has been
> ${FLINK_LOG_PREFIX}.gc.log Note %p or %t can only be used once Error: Could
> not create the Java Virtual Machine. Error: A fatal exception has occurred.
> Program will exit.{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28074) show statistics details for DESCRIBE EXTENDED
[ https://issues.apache.org/jira/browse/FLINK-28074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-28074: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > show statistics details for DESCRIBE EXTENDED > - > > Key: FLINK-28074 > URL: https://issues.apache.org/jira/browse/FLINK-28074 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Planner >Reporter: godfrey he >Assignee: Yunhong Zheng >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.16.0 > > > Currently, DESCRIBE command only show the schema of a given table, EXTENDED > does not work. so for EXTENDED mode, the statistics details can also be shown. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-25910) Propagate obtained delegation tokens to TaskManagers
[ https://issues.apache.org/jira/browse/FLINK-25910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-25910: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > Propagate obtained delegation tokens to TaskManagers > > > Key: FLINK-25910 > URL: https://issues.apache.org/jira/browse/FLINK-25910 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.15.0 >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > Labels: pull-request-available, stale-assigned > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-22441) In Flink v1.11.3 contains netty(version:3.10.6) netty(version:4.1.60) . There are many vulnerabilities, like CVE-2021-21409 etc. please confirm these version and
[ https://issues.apache.org/jira/browse/FLINK-22441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579116#comment-17579116 ] Hongbo edited comment on FLINK-22441 at 8/12/22 7:41 PM: - [~martijnvisser] How is it fixed in Flink 1.14? We can still see Netty 3.10.6 is used in the latest version: [https://github.com/apache/flink/blob/master/flink-rpc/flink-rpc-akka/pom.xml#L102] and it show up in the scan results: |etty Project|3.10.6.Final|BDSA-2018-4022|MEDIUM|4.7| |Netty Project|3.10.6.Final|BDSA-2019-2642|MEDIUM|6.5| |Netty Project|3.10.6.Final|BDSA-2019-2643|MEDIUM|6.7| |Netty Project|3.10.6.Final|BDSA-2019-2649|MEDIUM|6.5| |Netty Project|3.10.6.Final|BDSA-2019-2610|HIGH|7.2| |Netty Project|3.10.6.Final|CVE-2019-16869 (BDSA-2019-3119)|HIGH|7.5| |Netty Project|3.10.6.Final|BDSA-2020-0130|HIGH|8.8| |Netty Project|3.10.6.Final|CVE-2019-20444 (BDSA-2019-4231)|CRITICAL|9.1| |Netty Project|3.10.6.Final|CVE-2019-20445 (BDSA-2019-4230)|CRITICAL|9.1| |Netty Project|3.10.6.Final|BDSA-2020-0666|MEDIUM|6.5| |Netty Project|3.10.6.Final|CVE-2021-21290 (BDSA-2021-0311)|MEDIUM|5.5| |Netty Project|3.10.6.Final|CVE-2021-21295 (BDSA-2021-0589)|MEDIUM|5.9| |Netty Project|3.10.6.Final|CVE-2021-21409 (BDSA-2021-0828)|MEDIUM|5.9| |Netty Project|3.10.6.Final|CVE-2021-37136|HIGH|7.5| |Netty Project|3.10.6.Final|CVE-2021-37137|HIGH|7.5| |Netty Project|3.10.6.Final|CVE-2021-43797 (BDSA-2021-3741)|MEDIUM|6.5| |Netty Project|3.10.6.Final|CVE-2022-24823|MEDIUM|5.5| was (Author: liuhb86): [~martijnvisser] How is it fixed in Flink 1.14? We can still see Netty 3.10.6 is used in the latest version: [https://github.com/apache/flink/blob/master/flink-rpc/flink-rpc-akka/pom.xml#L102] and it show up in the scan results: |etty Project|3.10.6.Final|BDSA-2018-4022|MEDIUM|4.7| |Netty Project|3.10.6.Final|BDSA-2019-2642|MEDIUM|6.5| |Netty Project|3.10.6.Final|BDSA-2019-2643|MEDIUM|6.7| |Netty Project|3.10.6.Final|BDSA-2019-2649|MEDIUM|6.5| |Netty Project|3.10.6.Final|BDSA-2019-2610|HIGH|7.2| |Netty Project|3.10.6.Final|CVE-2019-16869 (BDSA-2019-3119)|HIGH|7.5| |Netty Project|3.10.6.Final|BDSA-2020-0130|HIGH|8.8| |Netty Project|3.10.6.Final|CVE-2019-20444 (BDSA-2019-4231)|CRITICAL|9.1| |Netty Project|3.10.6.Final|CVE-2019-20445 (BDSA-2019-4230)|CRITICAL|9.1| |Netty Project|3.10.6.Final|BDSA-2020-0666|MEDIUM|6.5| |Netty Project|3.10.6.Final|CVE-2021-21290 (BDSA-2021-0311)|MEDIUM|5.5| |Netty Project|3.10.6.Final|CVE-2021-21295 (BDSA-2021-0589)|MEDIUM|5.9| |Netty Project|3.10.6.Final|CVE-2021-21409 (BDSA-2021-0828)|MEDIUM|5.9| |Netty Project|3.10.6.Final|CVE-2021-37136|HIGH|7.5| |Netty Project|3.10.6.Final|CVE-2021-37137|HIGH|7.5| |Netty Project|3.10.6.Final|CVE-2021-43797 (BDSA-2021-3741)|MEDIUM|6.5| > In Flink v1.11.3 contains netty(version:3.10.6) netty(version:4.1.60) . There > are many vulnerabilities, like CVE-2021-21409 etc. please confirm these > version and fix. thx > -- > > Key: FLINK-22441 > URL: https://issues.apache.org/jira/browse/FLINK-22441 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.11.3, 1.12.2, 1.13.0 >Reporter: 张健 >Priority: Not a Priority > Labels: auto-deprioritized-major, auto-deprioritized-minor > > In Flink v1.11.3 contains netty(version:3.10.6) netty(version:4.1.60) . There > are many vulnerabilities, like CVE-2021-21409 CVE-2021-21295 etc. please > confirm these version and fix. thx -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-22441) In Flink v1.11.3 contains netty(version:3.10.6) netty(version:4.1.60) . There are many vulnerabilities, like CVE-2021-21409 etc. please confirm these version and fix.
[ https://issues.apache.org/jira/browse/FLINK-22441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579116#comment-17579116 ] Hongbo commented on FLINK-22441: [~martijnvisser] How is it fixed in Flink 1.14? We can still see Netty 3.10.6 is used in the latest version: [https://github.com/apache/flink/blob/master/flink-rpc/flink-rpc-akka/pom.xml#L102] and it show up in the scan results: |etty Project|3.10.6.Final|BDSA-2018-4022|MEDIUM|4.7| |Netty Project|3.10.6.Final|BDSA-2019-2642|MEDIUM|6.5| |Netty Project|3.10.6.Final|BDSA-2019-2643|MEDIUM|6.7| |Netty Project|3.10.6.Final|BDSA-2019-2649|MEDIUM|6.5| |Netty Project|3.10.6.Final|BDSA-2019-2610|HIGH|7.2| |Netty Project|3.10.6.Final|CVE-2019-16869 (BDSA-2019-3119)|HIGH|7.5| |Netty Project|3.10.6.Final|BDSA-2020-0130|HIGH|8.8| |Netty Project|3.10.6.Final|CVE-2019-20444 (BDSA-2019-4231)|CRITICAL|9.1| |Netty Project|3.10.6.Final|CVE-2019-20445 (BDSA-2019-4230)|CRITICAL|9.1| |Netty Project|3.10.6.Final|BDSA-2020-0666|MEDIUM|6.5| |Netty Project|3.10.6.Final|CVE-2021-21290 (BDSA-2021-0311)|MEDIUM|5.5| |Netty Project|3.10.6.Final|CVE-2021-21295 (BDSA-2021-0589)|MEDIUM|5.9| |Netty Project|3.10.6.Final|CVE-2021-21409 (BDSA-2021-0828)|MEDIUM|5.9| |Netty Project|3.10.6.Final|CVE-2021-37136|HIGH|7.5| |Netty Project|3.10.6.Final|CVE-2021-37137|HIGH|7.5| |Netty Project|3.10.6.Final|CVE-2021-43797 (BDSA-2021-3741)|MEDIUM|6.5| > In Flink v1.11.3 contains netty(version:3.10.6) netty(version:4.1.60) . There > are many vulnerabilities, like CVE-2021-21409 etc. please confirm these > version and fix. thx > -- > > Key: FLINK-22441 > URL: https://issues.apache.org/jira/browse/FLINK-22441 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.11.3, 1.12.2, 1.13.0 >Reporter: 张健 >Priority: Not a Priority > Labels: auto-deprioritized-major, auto-deprioritized-minor > > In Flink v1.11.3 contains netty(version:3.10.6) netty(version:4.1.60) . There > are many vulnerabilities, like CVE-2021-21409 CVE-2021-21295 etc. please > confirm these version and fix. thx -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] tweise commented on pull request #20530: [FLINK-28817] NullPointerException in HybridSource when restoring from checkpoint
tweise commented on PR #20530: URL: https://github.com/apache/flink/pull/20530#issuecomment-1213280921 > @zhongqishang, it's better to add the test case for the situation described in [FLINK-26938](https://issues.apache.org/jira/browse/FLINK-26938). cc @tweise Let's address that in a separate PR, since it's a separate JIRA also. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhongqishang commented on a diff in pull request #20530: [FLINK-28817] NullPointerException in HybridSource when restoring from checkpoint
zhongqishang commented on code in PR #20530:
URL: https://github.com/apache/flink/pull/20530#discussion_r944612708
##
flink-connectors/flink-connector-base/src/test/java/org/apache/flink/connector/base/source/hybrid/HybridSourceSplitEnumeratorTest.java:
##
@@ -177,6 +177,26 @@ public void testRestoreEnumerator() throws Exception {
.hasSize(1);
}
+@Test
+public void testRestoreEnumeratorAfterFirstSourceWithoutRestoredSplits()
throws Exception {
+setupEnumeratorAndTriggerSourceSwitch();
+HybridSourceEnumeratorState enumeratorState =
enumerator.snapshotState(0);
+MockSplitEnumerator underlyingEnumerator =
getCurrentEnumerator(enumerator);
+assertThat(
+(List)
+
Whitebox.getInternalState(underlyingEnumerator, "splits"))
+.hasSize(0);
+enumerator =
+(HybridSourceSplitEnumerator)
source.restoreEnumerator(context, enumeratorState);
+enumerator.start();
+enumerator.handleSourceEvent(SUBTASK0, new
SourceReaderFinishedEvent(-1));
Review Comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] zhongqishang commented on a diff in pull request #20530: [FLINK-28817] NullPointerException in HybridSource when restoring from checkpoint
zhongqishang commented on code in PR #20530:
URL: https://github.com/apache/flink/pull/20530#discussion_r944611038
##
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/hybrid/SwitchedSources.java:
##
@@ -45,4 +47,8 @@ public SimpleVersionedSerializer
serializerOf(int sourceIndex) {
public void put(int sourceIndex, Source source) {
sources.put(sourceIndex, Preconditions.checkNotNull(source));
}
+
+public Integer getNextSubtaskSourceIndex(int previousSubtaskSourceIndex) {
Review Comment:
I agree to change it to `getFirstSourceIndex()` and keep the rest of the
logic in handleSourceEvent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] zhongqishang commented on a diff in pull request #20530: [FLINK-28817] NullPointerException in HybridSource when restoring from checkpoint
zhongqishang commented on code in PR #20530:
URL: https://github.com/apache/flink/pull/20530#discussion_r944611038
##
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/hybrid/SwitchedSources.java:
##
@@ -45,4 +47,8 @@ public SimpleVersionedSerializer
serializerOf(int sourceIndex) {
public void put(int sourceIndex, Source source) {
sources.put(sourceIndex, Preconditions.checkNotNull(source));
}
+
+public Integer getNextSubtaskSourceIndex(int previousSubtaskSourceIndex) {
Review Comment:
I agree to change to `getFirstSourceIndex()` and keep the rest of the logic
in handleSourceEvent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] zhongqishang commented on a diff in pull request #20530: [FLINK-28817] NullPointerException in HybridSource when restoring from checkpoint
zhongqishang commented on code in PR #20530:
URL: https://github.com/apache/flink/pull/20530#discussion_r944611038
##
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/hybrid/SwitchedSources.java:
##
@@ -45,4 +47,8 @@ public SimpleVersionedSerializer
serializerOf(int sourceIndex) {
public void put(int sourceIndex, Source source) {
sources.put(sourceIndex, Preconditions.checkNotNull(source));
}
+
+public Integer getNextSubtaskSourceIndex(int previousSubtaskSourceIndex) {
Review Comment:
I agree to change to getFirstSourceInde() and keep the rest of the logic in
handleSourceEvent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] tweise commented on a diff in pull request #20530: [FLINK-28817] NullPointerException in HybridSource when restoring from checkpoint
tweise commented on code in PR #20530:
URL: https://github.com/apache/flink/pull/20530#discussion_r944566339
##
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/hybrid/SwitchedSources.java:
##
@@ -45,4 +47,8 @@ public SimpleVersionedSerializer
serializerOf(int sourceIndex) {
public void put(int sourceIndex, Source source) {
sources.put(sourceIndex, Preconditions.checkNotNull(source));
}
+
+public Integer getNextSubtaskSourceIndex(int previousSubtaskSourceIndex) {
Review Comment:
This would be misleading as the very purpose is to determine the source
index for a specific subtask. Probably it is better to just have
`getFirstSourceIndex()` here and keep the rest of the logic in
`handleSourceEvent`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] tweise commented on a diff in pull request #20530: [FLINK-28817] NullPointerException in HybridSource when restoring from checkpoint
tweise commented on code in PR #20530:
URL: https://github.com/apache/flink/pull/20530#discussion_r944562598
##
flink-connectors/flink-connector-base/src/test/java/org/apache/flink/connector/base/source/hybrid/HybridSourceSplitEnumeratorTest.java:
##
@@ -177,6 +177,26 @@ public void testRestoreEnumerator() throws Exception {
.hasSize(1);
}
+@Test
+public void testRestoreEnumeratorAfterFirstSourceWithoutRestoredSplits()
throws Exception {
+setupEnumeratorAndTriggerSourceSwitch();
+HybridSourceEnumeratorState enumeratorState =
enumerator.snapshotState(0);
+MockSplitEnumerator underlyingEnumerator =
getCurrentEnumerator(enumerator);
+assertThat(
+(List)
+
Whitebox.getInternalState(underlyingEnumerator, "splits"))
+.hasSize(0);
+enumerator =
+(HybridSourceSplitEnumerator)
source.restoreEnumerator(context, enumeratorState);
+enumerator.start();
+enumerator.handleSourceEvent(SUBTASK0, new
SourceReaderFinishedEvent(-1));
Review Comment:
```suggestion
// subtask starts at -1 since it has no splits after restore
enumerator.handleSourceEvent(SUBTASK0, new
SourceReaderFinishedEvent(-1));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Closed] (FLINK-26771) Fix incomparable exception between boolean type and numeric type in Hive dialect
[
https://issues.apache.org/jira/browse/FLINK-26771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu closed FLINK-26771.
---
Resolution: Fixed
Fixed in master: 1455e51c7cd64264f5ddebd0b768efc109fab74f
> Fix incomparable exception between boolean type and numeric type in Hive
> dialect
>
>
> Key: FLINK-26771
> URL: https://issues.apache.org/jira/browse/FLINK-26771
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Hive
>Reporter: luoyuxia
>Assignee: luoyuxia
>Priority: Critical
> Labels: pull-request-available, stale-assigned
> Fix For: 1.16.0
>
>
> Hive support compare boolean type with numeric type, for example such sql can
> be excuted in Hive:
> {code:java}
> // the data type for `status` is `int`
> select * from employee where status = true; {code}
> But in Flink, with Hive dialect, it'll throw "Incomparable types: BOOLEAN
> and INT NOT NULL" exception.
> For such case, it should be consistent with Hive while using Hive dialect in
> Flink.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] wuchong merged pull request #19182: [FLINK-26771][Hive] Fix incomparable exception between boolean type and numeric type in Hive dialect
wuchong merged PR #19182: URL: https://github.com/apache/flink/pull/19182 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #20562: [FLINK-28950] Disable HBase2 version check in hbase-default.xml
flinkbot commented on PR #20562: URL: https://github.com/apache/flink/pull/20562#issuecomment-1213172180 ## CI report: * f13abd5ad5581231e03f9a5b998e9ba1b74a5ca8 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28950) Disable HBase2 version check in hbase-default.xml
[ https://issues.apache.org/jira/browse/FLINK-28950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-28950: --- Labels: pull-request-available (was: ) > Disable HBase2 version check in hbase-default.xml > - > > Key: FLINK-28950 > URL: https://issues.apache.org/jira/browse/FLINK-28950 > Project: Flink > Issue Type: Improvement > Components: Connectors / HBase >Reporter: xi chaomin >Priority: Major > Labels: pull-request-available > > With the version check enabled, hbase connector can write to hbase 2.2.3 > only. Disable the version check to support hbase 2.X. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] xicm opened a new pull request, #20562: [FLINK-28950] Disable HBase2 version check in hbase-default.xml
xicm opened a new pull request, #20562: URL: https://github.com/apache/flink/pull/20562 ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-28950) Disable HBase2 version check in hbase-default.xml
xi chaomin created FLINK-28950: -- Summary: Disable HBase2 version check in hbase-default.xml Key: FLINK-28950 URL: https://issues.apache.org/jira/browse/FLINK-28950 Project: Flink Issue Type: Improvement Components: Connectors / HBase Reporter: xi chaomin With the version check enabled, hbase connector can write to hbase 2.2.3 only. Disable the version check to support hbase 2.X. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] ganlute commented on pull request #20542: [FLINK-28910][connectors/hbase]CDC From Mysql To Hbase Bugs
ganlute commented on PR #20542: URL: https://github.com/apache/flink/pull/20542#issuecomment-1213132485 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-28949) Secondary job manager fails to retrieve savepoint
[
https://issues.apache.org/jira/browse/FLINK-28949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-28949.
Resolution: Duplicate
This is fixed in 1.15.
> Secondary job manager fails to retrieve savepoint
> -
>
> Key: FLINK-28949
> URL: https://issues.apache.org/jira/browse/FLINK-28949
> Project: Flink
> Issue Type: Bug
> Components: Test Infrastructure
>Affects Versions: 1.14.4
>Reporter: Tiago Rodrigues
>Priority: Minor
>
> After creating a savepoint, its information cannot be retrieved unless the
> API call is made to the active job manager.
> Active job manager response:
> {code:java}
> >>> curl http://:8081/v1/jobs//savepoints/ |
> >>> jq
> {
> "status": {
> "id": "COMPLETED"
> },
> "operation": {
> "location": "file:/srv/flink/savepoints/"
> }
> }{code}
> Secondary job manager response:
> {code:java}
> >>> curl
> >>> http://:8081/v1/jobs//savepoints/ |
> >>> jq
> {
> "errors": [
> "org.apache.flink.runtime.rest.NotFoundException: Operation not found
> under key:
> org.apache.flink.runtime.rest.handler.job.AsynchronousJobOperationKey@e25522a4\n\tat
>
> org.apache.flink.runtime.rest.handler.async.AbstractAsynchronousOperationHandlers$StatusHandler.handleRequest(AbstractAsynchronousOperationHandlers.java:182)\n\tat
>
> org.apache.flink.runtime.rest.handler.job.savepoints.SavepointHandlers$SavepointStatusHandler.handleRequest(SavepointHandlers.java:219)\n\tat
>
> org.apache.flink.runtime.rest.handler.AbstractRestHandler.respondToRequest(AbstractRestHandler.java:83)\n\tat
>
> org.apache.flink.runtime.rest.handler.AbstractHandler.respondAsLeader(AbstractHandler.java:195)\n\tat
>
> org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.lambda$channelRead0$0(LeaderRetrievalHandler.java:83)\n\tat
> java.base/java.util.Optional.ifPresent(Optional.java:183)\n\tat
> org.apache.flink.util.OptionalConsumer.ifPresent(OptionalConsumer.java:45)\n\tat
>
> org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:80)\n\tat
>
> org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:49)\n\tat
>
> org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)\n\tat
>
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat
>
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat
>
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat
>
> org.apache.flink.runtime.rest.handler.router.RouterHandler.routed(RouterHandler.java:115)\n\tat
>
> org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:94)\n\tat
>
> org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:55)\n\tat
>
> org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)\n\tat
>
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat
>
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat
>
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat
>
> org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat
>
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat
>
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat
>
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat
>
> org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:238)\n\tat
>
> org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:71)\n\tat
>
> org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)\n\tat
>
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat
>
>
[jira] [Updated] (FLINK-28949) Secondary job manager fails to retrieve savepoint
[
https://issues.apache.org/jira/browse/FLINK-28949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tiago Rodrigues updated FLINK-28949:
Description:
After creating a savepoint, its information cannot be retrieved unless the API
call is made to the active job manager.
Active job manager response:
{code:java}
>>> curl http://:8081/v1/jobs//savepoints/ |
>>> jq
{
"status": {
"id": "COMPLETED"
},
"operation": {
"location": "file:/srv/flink/savepoints/"
}
}{code}
Secondary job manager response:
{code:java}
>>> curl http://:8081/v1/jobs//savepoints/
>>> | jq
{
"errors": [
"org.apache.flink.runtime.rest.NotFoundException: Operation not found under
key:
org.apache.flink.runtime.rest.handler.job.AsynchronousJobOperationKey@e25522a4\n\tat
org.apache.flink.runtime.rest.handler.async.AbstractAsynchronousOperationHandlers$StatusHandler.handleRequest(AbstractAsynchronousOperationHandlers.java:182)\n\tat
org.apache.flink.runtime.rest.handler.job.savepoints.SavepointHandlers$SavepointStatusHandler.handleRequest(SavepointHandlers.java:219)\n\tat
org.apache.flink.runtime.rest.handler.AbstractRestHandler.respondToRequest(AbstractRestHandler.java:83)\n\tat
org.apache.flink.runtime.rest.handler.AbstractHandler.respondAsLeader(AbstractHandler.java:195)\n\tat
org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.lambda$channelRead0$0(LeaderRetrievalHandler.java:83)\n\tat
java.base/java.util.Optional.ifPresent(Optional.java:183)\n\tat
org.apache.flink.util.OptionalConsumer.ifPresent(OptionalConsumer.java:45)\n\tat
org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:80)\n\tat
org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:49)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat
org.apache.flink.runtime.rest.handler.router.RouterHandler.routed(RouterHandler.java:115)\n\tat
org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:94)\n\tat
org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:55)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat
org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat
org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:238)\n\tat
org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:71)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)\n\tat
org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)\n\tat
[jira] [Created] (FLINK-28949) Secondary job manager fails to retrieve savepoint
Tiago Rodrigues created FLINK-28949:
---
Summary: Secondary job manager fails to retrieve savepoint
Key: FLINK-28949
URL: https://issues.apache.org/jira/browse/FLINK-28949
Project: Flink
Issue Type: Bug
Components: Test Infrastructure
Affects Versions: 1.14.4
Reporter: Tiago Rodrigues
After creating a savepoint, its information cannot be retrieved with Flink's
API if the call is made to a non-active job manager.
Active job manager response:
{code:java}
>>> curl http://:8081/v1/jobs//savepoints/ |
>>> jq
{
"status": {
"id": "COMPLETED"
},
"operation": {
"location": "file:/srv/flink/savepoints/"
}
}{code}
Secondary job manager response:
{code:java}
>>> curl http://:8081/v1/jobs//savepoints/
>>> | jq
{
"errors": [
"org.apache.flink.runtime.rest.NotFoundException: Operation not found under
key:
org.apache.flink.runtime.rest.handler.job.AsynchronousJobOperationKey@e25522a4\n\tat
org.apache.flink.runtime.rest.handler.async.AbstractAsynchronousOperationHandlers$StatusHandler.handleRequest(AbstractAsynchronousOperationHandlers.java:182)\n\tat
org.apache.flink.runtime.rest.handler.job.savepoints.SavepointHandlers$SavepointStatusHandler.handleRequest(SavepointHandlers.java:219)\n\tat
org.apache.flink.runtime.rest.handler.AbstractRestHandler.respondToRequest(AbstractRestHandler.java:83)\n\tat
org.apache.flink.runtime.rest.handler.AbstractHandler.respondAsLeader(AbstractHandler.java:195)\n\tat
org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.lambda$channelRead0$0(LeaderRetrievalHandler.java:83)\n\tat
java.base/java.util.Optional.ifPresent(Optional.java:183)\n\tat
org.apache.flink.util.OptionalConsumer.ifPresent(OptionalConsumer.java:45)\n\tat
org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:80)\n\tat
org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:49)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat
org.apache.flink.runtime.rest.handler.router.RouterHandler.routed(RouterHandler.java:115)\n\tat
org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:94)\n\tat
org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:55)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat
org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat
org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:238)\n\tat
org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:71)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat
org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)\n\tat
[jira] [Closed] (FLINK-28932) Use Builder#withConfiguration instead of deprecated EnvironmentSettings#fromConfiguration
[ https://issues.apache.org/jira/browse/FLINK-28932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-28932. Fix Version/s: 1.16.0 Assignee: Sergey Nuyanzin Resolution: Fixed master: ea3b44183a876e2154f9a3a1423dee688ba4eb0e > Use Builder#withConfiguration instead of deprecated > EnvironmentSettings#fromConfiguration > - > > Key: FLINK-28932 > URL: https://issues.apache.org/jira/browse/FLINK-28932 > Project: Flink > Issue Type: Technical Debt > Components: Table SQL / API >Reporter: Sergey Nuyanzin >Assignee: Sergey Nuyanzin >Priority: Minor > Labels: pull-request-available > Fix For: 1.16.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] zentol merged pull request #20556: [FLINK-28932][Table/SQL] Use Builder#withConfiguration instead of deprecated EnvironmentSettings#fromConfiguration
zentol merged PR #20556: URL: https://github.com/apache/flink/pull/20556 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-28931) BlockingPartitionBenchmark doesn't compile
[
https://issues.apache.org/jira/browse/FLINK-28931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578956#comment-17578956
]
Chesnay Schepler commented on FLINK-28931:
--
To me a bug has to directly impact the user experience in some way, which this
issue did not.
I will readily admit that I attribute a lot of things to technical debt; pretty
much anything that users don't see, in an attempt to make the release notes
more useful.
However, I'd argue that according to your definition anything that impacts the
development can not be technical debt, but technical debt is usually defined as
exactly that. It would reduce the category to tickets like "removing unused
method", which can't be the goal, because it'd be rather pointless.
> BlockingPartitionBenchmark doesn't compile
> --
>
> Key: FLINK-28931
> URL: https://issues.apache.org/jira/browse/FLINK-28931
> Project: Flink
> Issue Type: Technical Debt
> Components: Benchmarks
>Affects Versions: 1.16.0
>Reporter: Roman Khachatryan
>Assignee: Roman Khachatryan
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.16.0
>
>
> {code}
> 10:15:12 [ERROR]
> /home/jenkins/workspace/flink-master-benchmarks-java8/flink-benchmarks/src/main/java/org/apache/flink/benchmark/BlockingPartitionBenchmark.java:117:50:
> error: cannot find symbol
> {code}
> Caused by
> https://github.com/apache/flink/commit/9f5d0c48f198ff69a175f630832687ba02cf4c3e#diff-f72e79ebd747b6fde91988d65de9121a5907c97e4630cb1e30ab65601b4d9753R79
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-28931) BlockingPartitionBenchmark doesn't compile
[
https://issues.apache.org/jira/browse/FLINK-28931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578953#comment-17578953
]
Roman Khachatryan commented on FLINK-28931:
---
I don't think this was a technical debt because a major performance regression
in Flink would be a bug; and this issue prevented any preformance checks.
> BlockingPartitionBenchmark doesn't compile
> --
>
> Key: FLINK-28931
> URL: https://issues.apache.org/jira/browse/FLINK-28931
> Project: Flink
> Issue Type: Technical Debt
> Components: Benchmarks
>Affects Versions: 1.16.0
>Reporter: Roman Khachatryan
>Assignee: Roman Khachatryan
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.16.0
>
>
> {code}
> 10:15:12 [ERROR]
> /home/jenkins/workspace/flink-master-benchmarks-java8/flink-benchmarks/src/main/java/org/apache/flink/benchmark/BlockingPartitionBenchmark.java:117:50:
> error: cannot find symbol
> {code}
> Caused by
> https://github.com/apache/flink/commit/9f5d0c48f198ff69a175f630832687ba02cf4c3e#diff-f72e79ebd747b6fde91988d65de9121a5907c97e4630cb1e30ab65601b4d9753R79
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] tisonkun merged pull request #19972: [FLINK-27399][Connector/Pulsar] Change initial consuming position setting logic for better handle the checkpoint.
tisonkun merged PR #19972: URL: https://github.com/apache/flink/pull/19972 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-26958) Add Github Actions build pipeline for flink-connector-elasticsearch
[ https://issues.apache.org/jira/browse/FLINK-26958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-26958: - Issue Type: Technical Debt (was: Improvement) > Add Github Actions build pipeline for flink-connector-elasticsearch > --- > > Key: FLINK-26958 > URL: https://issues.apache.org/jira/browse/FLINK-26958 > Project: Flink > Issue Type: Technical Debt > Components: Build System, Connectors / Common, Connectors / > ElasticSearch >Reporter: Martijn Visser >Assignee: Martijn Visser >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0 > > > With connectors being moved to their individual repository, we need to have a > pipeline that can run the necessary compile and test steps. > With Elasticsearch in the process of being moved out (see FLINK-26884) we > should add this to make sure that on pushes and pull requests this pipeline > is executed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-25806) Remove legacy high availability services
[
https://issues.apache.org/jira/browse/FLINK-25806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-25806:
-
Issue Type: Technical Debt (was: Improvement)
> Remove legacy high availability services
>
>
> Key: FLINK-25806
> URL: https://issues.apache.org/jira/browse/FLINK-25806
> Project: Flink
> Issue Type: Technical Debt
> Components: Runtime / Coordination
>Affects Versions: 1.16.0
>Reporter: Till Rohrmann
>Assignee: Chesnay Schepler
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.16.0
>
>
> After FLINK-24038, we should consider removing the legacy high availability
> services {{ZooKeeperHaServices}} and {{KubernetesHaServices}} since they are
> now subsumed by the multiple component leader election service that only uses
> a single leader election per component.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28722) Hybrid Source should use .equals() for Integer comparison
[ https://issues.apache.org/jira/browse/FLINK-28722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-28722: - Issue Type: Bug (was: Improvement) > Hybrid Source should use .equals() for Integer comparison > - > > Key: FLINK-28722 > URL: https://issues.apache.org/jira/browse/FLINK-28722 > Project: Flink > Issue Type: Bug > Components: Connectors / Common >Affects Versions: 1.15.1 >Reporter: Mason Chen >Priority: Major > Fix For: 1.16.0, 1.15.2 > > > HybridSource should use .equals() for Integer comparison in filtering out the > underlying sources. This causes the HybridSource to stop working when it hits > the 128th source (would not work for anything past 127 sources). > https://github.com/apache/flink/blob/release-1.14.3-rc1/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/hybrid/HybridSourceSplitEnumerator.java#L358 > > A user reported this issue here: > https://lists.apache.org/thread/7h2rblsdt7rjf85q9mhfht77bghtbswh -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-19358) when submit job on application mode with HA,the jobid will be 0000000000
[ https://issues.apache.org/jira/browse/FLINK-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-19358: - Issue Type: Improvement (was: Bug) > when submit job on application mode with HA,the jobid will be 00 > > > Key: FLINK-19358 > URL: https://issues.apache.org/jira/browse/FLINK-19358 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.11.0 >Reporter: Jun Zhang >Assignee: Yangze Guo >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available, > usability > Fix For: 1.16.0 > > > when submit a flink job on application mode with HA ,the flink job id will be > , when I have many jobs ,they have the same > job id , it will be lead to a checkpoint error -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-19358) HA should not always use jobid 0000000000
[ https://issues.apache.org/jira/browse/FLINK-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-19358: - Summary: HA should not always use jobid 00 (was: when submit job on application mode with HA,the jobid will be 00) > HA should not always use jobid 00 > - > > Key: FLINK-19358 > URL: https://issues.apache.org/jira/browse/FLINK-19358 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.11.0 >Reporter: Jun Zhang >Assignee: Yangze Guo >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available, > usability > Fix For: 1.16.0 > > > when submit a flink job on application mode with HA ,the flink job id will be > , when I have many jobs ,they have the same > job id , it will be lead to a checkpoint error -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-14998) Remove FileUtils#deletePathIfEmpty
[
https://issues.apache.org/jira/browse/FLINK-14998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-14998:
-
Issue Type: Technical Debt (was: Bug)
> Remove FileUtils#deletePathIfEmpty
> --
>
> Key: FLINK-14998
> URL: https://issues.apache.org/jira/browse/FLINK-14998
> Project: Flink
> Issue Type: Technical Debt
> Components: FileSystems
>Reporter: Yun Tang
>Assignee: jay li
>Priority: Minor
> Labels: auto-deprioritized-major, pull-request-available, starter
> Fix For: 1.16.0
>
>
> With the lesson learned from FLINK-7266, and the refactor of FLINK-8540,
> method of {{FileUtils#deletePathIfEmpty}} has been totally useless in Flink
> production code. From my point of view, it's not wise to provide a method
> with already known high-risk defect in Flink official code. I suggest to
> remove this part of code.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-26553) Enable scalafmt for scala codebase
[ https://issues.apache.org/jira/browse/FLINK-26553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-26553: - Issue Type: Technical Debt (was: Bug) > Enable scalafmt for scala codebase > -- > > Key: FLINK-26553 > URL: https://issues.apache.org/jira/browse/FLINK-26553 > Project: Flink > Issue Type: Technical Debt > Components: Build System >Reporter: Francesco Guardiani >Assignee: Francesco Guardiani >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0 > > > As discussed in > https://lists.apache.org/thread/97398pc9cb8y922xlb6mzlsbjtjf5jnv, we should > enable scalafmt in our codebase -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-26710) TestLoggerResource hides log lines
[
https://issues.apache.org/jira/browse/FLINK-26710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-26710:
-
Issue Type: Technical Debt (was: Bug)
> TestLoggerResource hides log lines
> --
>
> Key: FLINK-26710
> URL: https://issues.apache.org/jira/browse/FLINK-26710
> Project: Flink
> Issue Type: Technical Debt
> Components: Test Infrastructure
>Affects Versions: 1.15.0, 1.16.0
>Reporter: Niklas Semmler
>Assignee: Niklas Semmler
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.16.0
>
>
> {{org.apache.flink.testutils.logging.TestLoggerResource}} makes log lines
> accessible to tests. It extends {{org.junit.rules.ExternalResource}} and thus
> can be used as a rule. Example for its use can be found
> [here|https://github.com/apache/flink/blob/c6997c97c575d334679915c328792b8a3067cfb5/flink-runtime/src/test/java/org/apache/flink/runtime/jobmanager/JobManagerProcessUtilsTest.java#L145].
> Unfortunately, the current implementation consumes *all* log lines of a
> {{Logger}} in such a way that they are not forwarded to the general log
> output. As such, someone debugging a test will not have access to log lines
> and this can complicate debugging. The implementation needs to be changed to
> non-exclusively consume log lines.
> In a first attempt, we enabled the {{additivity}} of the {{Logger}} created
> by {{TestLoggerResource}}. This works only for the case were both the
> {{Logger}} created by {{TestLoggerResource}} and the parent {{Logger}} use
> the same log level. When they use different log levels, the
> {{TestLoggerResource}} {{Logger}} overwrites the log level.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-27163) Fix typo issue in Flink Metrics documentation
[ https://issues.apache.org/jira/browse/FLINK-27163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-27163: - Issue Type: Technical Debt (was: Bug) > Fix typo issue in Flink Metrics documentation > - > > Key: FLINK-27163 > URL: https://issues.apache.org/jira/browse/FLINK-27163 > Project: Flink > Issue Type: Technical Debt > Components: Documentation >Affects Versions: 1.15.0, 1.16.0 >Reporter: hao wang >Assignee: hao wang >Priority: Minor > Labels: pull-request-available > Fix For: 1.16.0 > > Attachments: 20220411145958283.png > > > The Cluster module in the metrics documentation has five items，but only four > are specified. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-27485) Documentation build pipeline is broken
[
https://issues.apache.org/jira/browse/FLINK-27485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-27485:
-
Issue Type: Technical Debt (was: Bug)
> Documentation build pipeline is broken
> ---
>
> Key: FLINK-27485
> URL: https://issues.apache.org/jira/browse/FLINK-27485
> Project: Flink
> Issue Type: Technical Debt
> Components: Build System, Documentation
>Reporter: Martijn Visser
>Assignee: Martijn Visser
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.16.0
>
>
> The current documentation build pipeline is broken due to two failures:
> - It uses git command {{git branch --show-current}} which isn't supported by
> the installed Git version on the Docker image. We can switch to {{git
> rev-parse --abbrev-ref HEAD}} as an alternative
> - The manual Hugo download and installation is outdated and doesn't add Hugo
> to the PATH
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-27751) Dependency resolution from repository.jboss.org fails on CI
[
https://issues.apache.org/jira/browse/FLINK-27751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-27751:
-
Issue Type: Technical Debt (was: Improvement)
> Dependency resolution from repository.jboss.org fails on CI
> ---
>
> Key: FLINK-27751
> URL: https://issues.apache.org/jira/browse/FLINK-27751
> Project: Flink
> Issue Type: Technical Debt
> Components: Build System / Azure Pipelines
>Affects Versions: 1.15.0, 1.16.0
>Reporter: Huang Xingbo
>Assignee: Chesnay Schepler
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.14.5, 1.15.1, 1.16.0
>
>
> {code:java}
> 2022-05-24T03:50:20.5443243Z 03:50:20,543 ERROR
> org.apache.flink.tools.ci.suffixcheck.ScalaSuffixChecker [] - Violations
> found:
> 2022-05-24T03:50:20.5444210Z Scala-free module 'flink-hadoop-compatibility'
> is referenced with scala suffix in
> 'flink-formats/flink-sequence-file/pom.xml'.
> 2022-05-24T03:50:20.5445185Z Scala-free module 'flink-hadoop-compatibility'
> is referenced with scala suffix in
> 'flink-connectors/flink-hadoop-compatibility/pom.xml'.
> 2022-05-24T03:50:20.5446207Z Scala-free module 'flink-hadoop-compatibility'
> is referenced with scala suffix in
> 'flink-connectors/flink-connector-hbase-1.4/pom.xml'.
> 2022-05-24T03:50:20.5447186Z Scala-free module 'flink-hadoop-compatibility'
> is referenced with scala suffix in
> 'flink-connectors/flink-connector-hbase-2.2/pom.xml'.
> 2022-05-24T03:50:20.5448135Z Scala-free module 'flink-hadoop-compatibility'
> is referenced with scala suffix in 'flink-connectors/flink-hcatalog/pom.xml'.
> 2022-05-24T03:50:20.5449237Z Scala-free module 'flink-hadoop-compatibility'
> is referenced with scala suffix in
> 'flink-connectors/flink-connector-hive/pom.xml'.
> 2022-05-24T03:50:20.5450180Z Scala-free module 'flink-hadoop-compatibility'
> is referenced with scala suffix in 'flink-table/flink-sql-client/pom.xml'.
> 2022-05-24T03:50:20.5451049Z Scala-free module 'flink-hadoop-compatibility'
> is referenced with scala suffix in 'flink-tests/pom.xml'.
> 2022-05-24T03:50:20.5452020Z Scala-free module 'flink-hcatalog' is
> referenced with scala suffix in 'flink-connectors/flink-hcatalog/pom.xml'.
> 2022-05-24T03:50:20.5453369Z Scala-free module
> 'flink-sql-connector-hive-2.3.9' is referenced with scala suffix in
> 'flink-connectors/flink-sql-connector-hive-2.3.9/pom.xml'.
> 2022-05-24T03:50:20.5454388Z Scala-free module
> 'flink-sql-connector-hive-3.1.2' is referenced with scala suffix in
> 'flink-connectors/flink-sql-connector-hive-3.1.2/pom.xml'.
> 2022-05-24T03:50:20.6860887Z
> ==
> 2022-05-24T03:50:20.6861601Z Suffix Check failed. See previous output for
> details.
> 2022-05-24T03:50:20.6862335Z
> ==
> 2022-05-24T03:50:20.6942100Z ##[error]Bash exited with code '1'.
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=35994=logs=52b61abe-a3cc-5bde-cc35-1bbe89bb7df5=54421a62-0c80-5aad-3319-094ff69180bb
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-27751) Dependency resolution from repository.jboss.org fails on CI
[
https://issues.apache.org/jira/browse/FLINK-27751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-27751:
-
Issue Type: Improvement (was: Bug)
> Dependency resolution from repository.jboss.org fails on CI
> ---
>
> Key: FLINK-27751
> URL: https://issues.apache.org/jira/browse/FLINK-27751
> Project: Flink
> Issue Type: Improvement
> Components: Build System / Azure Pipelines
>Affects Versions: 1.15.0, 1.16.0
>Reporter: Huang Xingbo
>Assignee: Chesnay Schepler
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.14.5, 1.15.1, 1.16.0
>
>
> {code:java}
> 2022-05-24T03:50:20.5443243Z 03:50:20,543 ERROR
> org.apache.flink.tools.ci.suffixcheck.ScalaSuffixChecker [] - Violations
> found:
> 2022-05-24T03:50:20.5444210Z Scala-free module 'flink-hadoop-compatibility'
> is referenced with scala suffix in
> 'flink-formats/flink-sequence-file/pom.xml'.
> 2022-05-24T03:50:20.5445185Z Scala-free module 'flink-hadoop-compatibility'
> is referenced with scala suffix in
> 'flink-connectors/flink-hadoop-compatibility/pom.xml'.
> 2022-05-24T03:50:20.5446207Z Scala-free module 'flink-hadoop-compatibility'
> is referenced with scala suffix in
> 'flink-connectors/flink-connector-hbase-1.4/pom.xml'.
> 2022-05-24T03:50:20.5447186Z Scala-free module 'flink-hadoop-compatibility'
> is referenced with scala suffix in
> 'flink-connectors/flink-connector-hbase-2.2/pom.xml'.
> 2022-05-24T03:50:20.5448135Z Scala-free module 'flink-hadoop-compatibility'
> is referenced with scala suffix in 'flink-connectors/flink-hcatalog/pom.xml'.
> 2022-05-24T03:50:20.5449237Z Scala-free module 'flink-hadoop-compatibility'
> is referenced with scala suffix in
> 'flink-connectors/flink-connector-hive/pom.xml'.
> 2022-05-24T03:50:20.5450180Z Scala-free module 'flink-hadoop-compatibility'
> is referenced with scala suffix in 'flink-table/flink-sql-client/pom.xml'.
> 2022-05-24T03:50:20.5451049Z Scala-free module 'flink-hadoop-compatibility'
> is referenced with scala suffix in 'flink-tests/pom.xml'.
> 2022-05-24T03:50:20.5452020Z Scala-free module 'flink-hcatalog' is
> referenced with scala suffix in 'flink-connectors/flink-hcatalog/pom.xml'.
> 2022-05-24T03:50:20.5453369Z Scala-free module
> 'flink-sql-connector-hive-2.3.9' is referenced with scala suffix in
> 'flink-connectors/flink-sql-connector-hive-2.3.9/pom.xml'.
> 2022-05-24T03:50:20.5454388Z Scala-free module
> 'flink-sql-connector-hive-3.1.2' is referenced with scala suffix in
> 'flink-connectors/flink-sql-connector-hive-3.1.2/pom.xml'.
> 2022-05-24T03:50:20.6860887Z
> ==
> 2022-05-24T03:50:20.6861601Z Suffix Check failed. See previous output for
> details.
> 2022-05-24T03:50:20.6862335Z
> ==
> 2022-05-24T03:50:20.6942100Z ##[error]Bash exited with code '1'.
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=35994=logs=52b61abe-a3cc-5bde-cc35-1bbe89bb7df5=54421a62-0c80-5aad-3319-094ff69180bb
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28198) CassandraConnectorITCase fails with timeout
[
https://issues.apache.org/jira/browse/FLINK-28198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-28198:
-
Issue Type: Technical Debt (was: Bug)
> CassandraConnectorITCase fails with timeout
> ---
>
> Key: FLINK-28198
> URL: https://issues.apache.org/jira/browse/FLINK-28198
> Project: Flink
> Issue Type: Technical Debt
> Components: Connectors / Cassandra
>Affects Versions: 1.16.0
>Reporter: Martijn Visser
>Assignee: Etienne Chauchot
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.16.0
>
>
> {code:java}
> Jun 22 07:57:37 [ERROR]
> org.apache.flink.streaming.connectors.cassandra.CassandraConnectorITCase.testRaiseCassandraRequestsTimeouts
> Time elapsed: 12.067 s <<< ERROR!
> Jun 22 07:57:37
> com.datastax.driver.core.exceptions.OperationTimedOutException:
> [/172.17.0.1:59915] Timed out waiting for server response
> Jun 22 07:57:37 at
> com.datastax.driver.core.exceptions.OperationTimedOutException.copy(OperationTimedOutException.java:43)
> Jun 22 07:57:37 at
> com.datastax.driver.core.exceptions.OperationTimedOutException.copy(OperationTimedOutException.java:25)
> Jun 22 07:57:37 at
> com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:35)
> Jun 22 07:57:37 at
> com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:293)
> Jun 22 07:57:37 at
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:58)
> Jun 22 07:57:37 at
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:39)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37037=logs=4eda0b4a-bd0d-521a-0916-8285b9be9bb5=2ff6d5fa-53a6-53ac-bff7-fa524ea361a9=13736
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28263) TPC-DS Bash e2e tests don't clean-up after completing
[
https://issues.apache.org/jira/browse/FLINK-28263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-28263:
-
Issue Type: Technical Debt (was: Bug)
> TPC-DS Bash e2e tests don't clean-up after completing
> -
>
> Key: FLINK-28263
> URL: https://issues.apache.org/jira/browse/FLINK-28263
> Project: Flink
> Issue Type: Technical Debt
> Components: Tests
>Affects Versions: 1.16.0
>Reporter: Martijn Visser
>Assignee: Martijn Visser
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.16.0, 1.15.2, 1.14.6
>
>
> When debugging the disk space usage for the e2e tests, the top 20 folders
> with the largest file size are:
> {code:java}
> 2022-06-27T09:32:59.8000587Z Jun 27 09:32:59 List top 20 directories with
> largest file size
> 2022-06-27T09:33:00.9811803Z Jun 27 09:33:00 4088524 .
> 2022-06-27T09:33:00.9813428Z Jun 27 09:33:00 1277080 ./flink-end-to-end-tests
> 2022-06-27T09:33:00.9814324Z Jun 27 09:33:00 624512 ./flink-dist
> 2022-06-27T09:33:00.9815152Z Jun 27 09:33:00 624124 ./flink-dist/target
> 2022-06-27T09:33:00.9816093Z Jun 27 09:33:00 500032
> ./flink-dist/target/flink-1.16-SNAPSHOT-bin
> 2022-06-27T09:33:00.9817429Z Jun 27 09:33:00 500028
> ./flink-dist/target/flink-1.16-SNAPSHOT-bin/flink-1.16-SNAPSHOT
> 2022-06-27T09:33:00.9818167Z Jun 27 09:33:00 486412 ./.git
> 2022-06-27T09:33:00.9819096Z Jun 27 09:33:00 479416 ./.git/objects
> 2022-06-27T09:33:00.9819512Z Jun 27 09:33:00 479408 ./.git/objects/pack
> 2022-06-27T09:33:00.9820584Z Jun 27 09:33:00 461456 ./flink-connectors
> 2022-06-27T09:33:00.9821403Z Jun 27 09:33:00 449832
> ./.git/objects/pack/pack-0bdd9e3186d0cb404910c5843d19b5cb80b84fe0.pack
> 2022-06-27T09:33:00.9821992Z Jun 27 09:33:00 349236 ./flink-table
> 2022-06-27T09:33:00.9822631Z Jun 27 09:33:00 293008
> ./flink-dist/target/flink-1.16-SNAPSHOT-bin/flink-1.16-SNAPSHOT/opt
> 2022-06-27T09:33:00.9823233Z Jun 27 09:33:00 251272 ./flink-filesystems
> 2022-06-27T09:33:00.9823818Z Jun 27 09:33:00 246588
> ./flink-end-to-end-tests/flink-streaming-kinesis-test
> 2022-06-27T09:33:00.9824502Z Jun 27 09:33:00 246464
> ./flink-end-to-end-tests/flink-streaming-kinesis-test/target
> 2022-06-27T09:33:00.9825210Z Jun 27 09:33:00 196656
> ./flink-dist/target/flink-1.16-SNAPSHOT-bin/flink-1.16-SNAPSHOT/lib
> 2022-06-27T09:33:00.9825966Z Jun 27 09:33:00 184364
> ./flink-end-to-end-tests/flink-streaming-kinesis-test/target/KinesisExample.jar
> 2022-06-27T09:33:00.9826652Z Jun 27 09:33:00 156136
> ./flink-end-to-end-tests/flink-tpcds-test
> 2022-06-27T09:33:00.9827284Z Jun 27 09:33:00 151180
> ./flink-end-to-end-tests/flink-tpcds-test/target
> {code}
> See
> https://dev.azure.com/martijn0323/Flink/_build/results?buildId=2732=logs=0e31ee24-31a6-528c-a4bf-45cde9b2a14e=ff03a8fa-e84e-5199-efb2-5433077ce8e2=5093
> After running {{TPC-DS end-to-end test}} and after the clean-up, the
> following directories are listed in the top 20:
> {code:java}
> 2022-06-27T09:49:51.7694429Z Jun 27 09:49:51 List top 20 directories with
> largest file size AFTER cleaning temorary folders and files
> 2022-06-27T09:49:52.9617221Z Jun 27 09:49:52 5315996 .
> 2022-06-27T09:49:52.9618830Z Jun 27 09:49:52 2504556 ./flink-end-to-end-tests
> 2022-06-27T09:49:52.9619848Z Jun 27 09:49:52 1383612
> ./flink-end-to-end-tests/flink-tpcds-test
> 2022-06-27T09:49:52.9620796Z Jun 27 09:49:52 1378656
> ./flink-end-to-end-tests/flink-tpcds-test/target
> 2022-06-27T09:49:52.9621730Z Jun 27 09:49:52 1223944
> ./flink-end-to-end-tests/flink-tpcds-test/target/table
> 2022-06-27T09:49:52.9622844Z Jun 27 09:49:52 624508 ./flink-dist
> 2022-06-27T09:49:52.9623585Z Jun 27 09:49:52 624120 ./flink-dist/target
> 2022-06-27T09:49:52.9624398Z Jun 27 09:49:52 500028
> ./flink-dist/target/flink-1.16-SNAPSHOT-bin
> 2022-06-27T09:49:52.9625366Z Jun 27 09:49:52 500024
> ./flink-dist/target/flink-1.16-SNAPSHOT-bin/flink-1.16-SNAPSHOT
> 2022-06-27T09:49:52.9625994Z Jun 27 09:49:52 486412 ./.git
> 2022-06-27T09:49:52.9626514Z Jun 27 09:49:52 479416 ./.git/objects
> 2022-06-27T09:49:52.9631740Z Jun 27 09:49:52 479408 ./.git/objects/pack
> 2022-06-27T09:49:52.9632755Z Jun 27 09:49:52 461456 ./flink-connectors
> 2022-06-27T09:49:52.9633717Z Jun 27 09:49:52 449832
> ./.git/objects/pack/pack-0bdd9e3186d0cb404910c5843d19b5cb80b84fe0.pack
> 2022-06-27T09:49:52.9634769Z Jun 27 09:49:52 379348
> ./flink-end-to-end-tests/flink-tpcds-test/target/table/store_sales.dat
> 2022-06-27T09:49:52.9635596Z Jun 27 09:49:52 349236 ./flink-table
> 2022-06-27T09:49:52.9636489Z Jun 27 09:49:52 293008
> ./flink-dist/target/flink-1.16-SNAPSHOT-bin/flink-1.16-SNAPSHOT/opt
> 2022-06-27T09:49:52.9637526Z
[jira] [Updated] (FLINK-28388) Python doc build breaking nightly docs
[
https://issues.apache.org/jira/browse/FLINK-28388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-28388:
-
Issue Type: Technical Debt (was: Bug)
> Python doc build breaking nightly docs
> --
>
> Key: FLINK-28388
> URL: https://issues.apache.org/jira/browse/FLINK-28388
> Project: Flink
> Issue Type: Technical Debt
> Components: API / Python, Documentation
>Affects Versions: 1.16.0
>Reporter: Márton Balassi
>Assignee: Dian Fu
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.16.0
>
>
> For the past 5 days the nightly doc builds via GHA are broken:
> https://github.com/apache/flink/actions/workflows/docs.yml
> {noformat}
> Exception occurred:
> File "/root/flink/flink-python/pyflink/java_gateway.py", line 86, in
> launch_gateway
> raise Exception("It's launching the PythonGatewayServer during Python UDF
> execution "
> Exception: It's launching the PythonGatewayServer during Python UDF execution
> which is unexpected. It usually happens when the job codes are in the top
> level of the Python script file and are not enclosed in a `if name == 'main'`
> statement.
> The full traceback has been saved in /tmp/sphinx-err-3thh_wi2.log, if you
> want to report the issue to the developers.
> Please also report this if it was a user error, so that a better error
> message can be provided next time.
> A bug report can be filed in the tracker at
> . Thanks!
> Makefile:76: recipe for target 'html' failed
> make: *** [html] Error 2
> ==sphinx checks... [FAILED]===
> Error: Process completed with exit code 1.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28781) Hybrid Shuffle should support compression
[ https://issues.apache.org/jira/browse/FLINK-28781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-28781: - Issue Type: Improvement (was: Bug) > Hybrid Shuffle should support compression > - > > Key: FLINK-28781 > URL: https://issues.apache.org/jira/browse/FLINK-28781 > Project: Flink > Issue Type: Improvement > Components: Runtime / Network >Reporter: Weijie Guo >Assignee: Weijie Guo >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0 > > > Compression is a useful feature for batch jobs, which can significantly > reduce disk load and the amount of data transferred over the network. Hybrid > shuffle should also support the compression of spilled data, especially under > the full spilling strategy. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28844) YARNHighAvailabilityITCase fails with NoSuchMethod of org.apache.curator
[
https://issues.apache.org/jira/browse/FLINK-28844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-28844:
-
Issue Type: Technical Debt (was: Bug)
> YARNHighAvailabilityITCase fails with NoSuchMethod of org.apache.curator
>
>
> Key: FLINK-28844
> URL: https://issues.apache.org/jira/browse/FLINK-28844
> Project: Flink
> Issue Type: Technical Debt
> Components: Test Infrastructure
>Affects Versions: 1.16.0
>Reporter: Jark Wu
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.16.0
>
>
> This is keep failing on master since the commit of
> https://github.com/flink-ci/flink-mirror/commit/6335b573863af2b30a6541f910be96ddf61f9c84
> which removes curator-test dependency from the flink-test-utils module.
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=39394=logs=fc5181b0-e452-5c8f-68de-1097947f6483=995c650b-6573-581c-9ce6-7ad4cc038461
> {code}
> 2022-08-05T18:31:47.0438160Z Aug 05 18:31:47 [ERROR] Tests run: 1, Failures:
> 0, Errors: 1, Skipped: 0, Time elapsed: 5.237 s <<< FAILURE! - in
> org.apache.flink.yarn.YARNHighAvailabilityITCase
> 2022-08-05T18:31:47.0439564Z Aug 05 18:31:47 [ERROR]
> org.apache.flink.yarn.YARNHighAvailabilityITCase Time elapsed: 5.237 s <<<
> ERROR!
> 2022-08-05T18:31:47.0440370Z Aug 05 18:31:47 java.lang.NoSuchMethodError:
> org.apache.curator.test.InstanceSpec.getHostname()Ljava/lang/String;
> 2022-08-05T18:31:47.0441582Z Aug 05 18:31:47 at
> org.apache.flink.runtime.testutils.ZooKeeperTestUtils.getZookeeperInstanceSpecWithIncreasedSessionTimeout(ZooKeeperTestUtils.java:71)
> 2022-08-05T18:31:47.0442643Z Aug 05 18:31:47 at
> org.apache.flink.runtime.testutils.ZooKeeperTestUtils.createAndStartZookeeperTestingServer(ZooKeeperTestUtils.java:49)
> 2022-08-05T18:31:47.0443461Z Aug 05 18:31:47 at
> org.apache.flink.yarn.YARNHighAvailabilityITCase.setup(YARNHighAvailabilityITCase.java:114)
> 2022-08-05T18:31:47.0444094Z Aug 05 18:31:47 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2022-08-05T18:31:47.0444717Z Aug 05 18:31:47 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2022-08-05T18:31:47.0445424Z Aug 05 18:31:47 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-08-05T18:31:47.0446063Z Aug 05 18:31:47 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2022-08-05T18:31:47.0446818Z Aug 05 18:31:47 at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
> 2022-08-05T18:31:47.0447822Z Aug 05 18:31:47 at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> 2022-08-05T18:31:47.0448657Z Aug 05 18:31:47 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> 2022-08-05T18:31:47.0449692Z Aug 05 18:31:47 at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
> 2022-08-05T18:31:47.0450637Z Aug 05 18:31:47 at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptLifecycleMethod(TimeoutExtension.java:126)
> 2022-08-05T18:31:47.0451443Z Aug 05 18:31:47 at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptBeforeAllMethod(TimeoutExtension.java:68)
> 2022-08-05T18:31:47.0452304Z Aug 05 18:31:47 at
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
> 2022-08-05T18:31:47.0453162Z Aug 05 18:31:47 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
> 2022-08-05T18:31:47.0454013Z Aug 05 18:31:47 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> 2022-08-05T18:31:47.0454882Z Aug 05 18:31:47 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> 2022-08-05T18:31:47.0455716Z Aug 05 18:31:47 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> 2022-08-05T18:31:47.0456525Z Aug 05 18:31:47 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> 2022-08-05T18:31:47.0457512Z Aug 05 18:31:47 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
> 2022-08-05T18:31:47.0458637Z Aug 05 18:31:47 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
> 2022-08-05T18:31:47.0459742Z Aug 05 18:31:47 at
>
[jira] [Updated] (FLINK-28856) YARNHighAvailabilityITCase tests failed with NoSuchMethodError
[
https://issues.apache.org/jira/browse/FLINK-28856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-28856:
-
Issue Type: Technical Debt (was: Bug)
> YARNHighAvailabilityITCase tests failed with NoSuchMethodError
> --
>
> Key: FLINK-28856
> URL: https://issues.apache.org/jira/browse/FLINK-28856
> Project: Flink
> Issue Type: Technical Debt
> Components: Build System / Azure Pipelines, Tests
>Affects Versions: 1.16.0
>Reporter: Huang Xingbo
>Priority: Blocker
> Labels: test-stability
>
> {code:java}
> 2022-08-07T15:54:12.7203154Z Aug 07 15:54:12 [ERROR]
> org.apache.flink.yarn.YARNHighAvailabilityITCase Time elapsed: 4.606 s <<<
> ERROR!
> 2022-08-07T15:54:12.7203828Z Aug 07 15:54:12 java.lang.NoSuchMethodError:
> org.apache.curator.test.InstanceSpec.getHostname()Ljava/lang/String;
> 2022-08-07T15:54:12.7204675Z Aug 07 15:54:12 at
> org.apache.flink.runtime.testutils.ZooKeeperTestUtils.getZookeeperInstanceSpecWithIncreasedSessionTimeout(ZooKeeperTestUtils.java:71)
> 2022-08-07T15:54:12.7205582Z Aug 07 15:54:12 at
> org.apache.flink.runtime.testutils.ZooKeeperTestUtils.createAndStartZookeeperTestingServer(ZooKeeperTestUtils.java:49)
> 2022-08-07T15:54:12.7206508Z Aug 07 15:54:12 at
> org.apache.flink.yarn.YARNHighAvailabilityITCase.setup(YARNHighAvailabilityITCase.java:114)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=39502=logs=fc5181b0-e452-5c8f-68de-1097947f6483=995c650b-6573-581c-9ce6-7ad4cc038461
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28856) YARNHighAvailabilityITCase tests failed with NoSuchMethodError
[
https://issues.apache.org/jira/browse/FLINK-28856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-28856:
-
Fix Version/s: (was: 1.16.0)
> YARNHighAvailabilityITCase tests failed with NoSuchMethodError
> --
>
> Key: FLINK-28856
> URL: https://issues.apache.org/jira/browse/FLINK-28856
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines, Tests
>Affects Versions: 1.16.0
>Reporter: Huang Xingbo
>Priority: Blocker
> Labels: test-stability
>
> {code:java}
> 2022-08-07T15:54:12.7203154Z Aug 07 15:54:12 [ERROR]
> org.apache.flink.yarn.YARNHighAvailabilityITCase Time elapsed: 4.606 s <<<
> ERROR!
> 2022-08-07T15:54:12.7203828Z Aug 07 15:54:12 java.lang.NoSuchMethodError:
> org.apache.curator.test.InstanceSpec.getHostname()Ljava/lang/String;
> 2022-08-07T15:54:12.7204675Z Aug 07 15:54:12 at
> org.apache.flink.runtime.testutils.ZooKeeperTestUtils.getZookeeperInstanceSpecWithIncreasedSessionTimeout(ZooKeeperTestUtils.java:71)
> 2022-08-07T15:54:12.7205582Z Aug 07 15:54:12 at
> org.apache.flink.runtime.testutils.ZooKeeperTestUtils.createAndStartZookeeperTestingServer(ZooKeeperTestUtils.java:49)
> 2022-08-07T15:54:12.7206508Z Aug 07 15:54:12 at
> org.apache.flink.yarn.YARNHighAvailabilityITCase.setup(YARNHighAvailabilityITCase.java:114)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=39502=logs=fc5181b0-e452-5c8f-68de-1097947f6483=995c650b-6573-581c-9ce6-7ad4cc038461
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28931) BlockingPartitionBenchmark doesn't compile
[
https://issues.apache.org/jira/browse/FLINK-28931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-28931:
-
Issue Type: Technical Debt (was: Bug)
> BlockingPartitionBenchmark doesn't compile
> --
>
> Key: FLINK-28931
> URL: https://issues.apache.org/jira/browse/FLINK-28931
> Project: Flink
> Issue Type: Technical Debt
> Components: Benchmarks
>Affects Versions: 1.16.0
>Reporter: Roman Khachatryan
>Assignee: Roman Khachatryan
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.16.0
>
>
> {code}
> 10:15:12 [ERROR]
> /home/jenkins/workspace/flink-master-benchmarks-java8/flink-benchmarks/src/main/java/org/apache/flink/benchmark/BlockingPartitionBenchmark.java:117:50:
> error: cannot find symbol
> {code}
> Caused by
> https://github.com/apache/flink/commit/9f5d0c48f198ff69a175f630832687ba02cf4c3e#diff-f72e79ebd747b6fde91988d65de9121a5907c97e4630cb1e30ab65601b4d9753R79
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] yunfengzhou-hub commented on a diff in pull request #20454: [FLINK-28639][Runtime/Checkpointing] Preserve consistency of events from subtask to OC
yunfengzhou-hub commented on code in PR #20454:
URL: https://github.com/apache/flink/pull/20454#discussion_r944406293
##
flink-runtime/src/main/java/org/apache/flink/runtime/operators/coordination/SubtaskGatewayImpl.java:
##
@@ -220,12 +228,11 @@ void openGatewayAndUnmarkCheckpoint(long checkpointId) {
// Gateways should always be marked and closed for a specific
checkpoint before it can be
// reopened for that checkpoint. If a gateway is to be opened for an
unforeseen checkpoint,
-// exceptions should be thrown.
+// which might happen when the coordinator has been reset to a
previous checkpoint, warn
+// messages should be recorded.
if (lastCheckpointId < checkpointId) {
-throw new IllegalStateException(
-String.format(
-"Gateway closed for different checkpoint: closed
for = %d, expected = %d",
-currentCheckpointId, checkpointId));
+LOG.warn("Trying to open gateway for unknown checkpoint: " +
checkpointId);
Review Comment:
In the following case, we would have `lastCheckpointId < checkpointId`.
1. Suppose the coordinator has completed the checkpoint whose id is 10, and
checkpoint barriers are sent out to snapshot the operators.
2. A subtask of the coordinator completed the snapshot, and send back an
`AcknowledgeCloseGatewayEvent(checkpointId = 10)` (but the coordinator has not
received the event yet).
3. For some reason, checkpoint 10 is discarded and the job is reset to
checkpoint 5.
4. The coordinator receives the `AcknowledgeCloseGatewayEvent(checkpointId =
10)`.
This case happens because we introduced some operator events in the relevant
PRs, and we cannot guarantee that the events can arrive at their destinations
before the corresponding checkpoint finishes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] yunfengzhou-hub commented on pull request #20454: [FLINK-28639][Runtime/Checkpointing] Preserve consistency of events from subtask to OC
yunfengzhou-hub commented on PR #20454: URL: https://github.com/apache/flink/pull/20454#issuecomment-1213052159 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] yunfengzhou-hub commented on a diff in pull request #20454: [FLINK-28639][Runtime/Checkpointing] Preserve consistency of events from subtask to OC
yunfengzhou-hub commented on code in PR #20454:
URL: https://github.com/apache/flink/pull/20454#discussion_r944382861
##
flink-runtime/src/main/java/org/apache/flink/runtime/operators/coordination/OperatorCoordinatorHolder.java:
##
@@ -299,9 +364,78 @@ public void resetToCheckpoint(long checkpointId, @Nullable
byte[] checkpointData
}
private void checkpointCoordinatorInternal(
-final long checkpointId, final CompletableFuture result) {
+long checkpointId, CompletableFuture result) {
mainThreadExecutor.assertRunningInMainThread();
+try {
+if (currentPendingCheckpointId != NO_CHECKPOINT
+&& currentPendingCheckpointId != checkpointId) {
+throw new IllegalStateException(
+String.format(
+"Cannot checkpoint coordinator for checkpoint
%d, "
++ "since checkpoint %d has already
started.",
+checkpointId, currentPendingCheckpointId));
+}
+
+if (latestAttemptedCheckpointId >= checkpointId) {
+throw new IllegalStateException(
+String.format(
+"Regressing checkpoint IDs. Previous
checkpointId = %d, new checkpointId = %d",
+latestAttemptedCheckpointId, checkpointId));
+}
+
+subtaskGatewayMap.forEach(
+(subtask, gateway) ->
gateway.markForCheckpoint(checkpointId));
+
+
Preconditions.checkState(acknowledgeCloseGatewayFutureMap.isEmpty());
+} catch (Throwable t) {
+result.completeExceptionally(t);
+globalFailureHandler.handleGlobalFailure(t);
+return;
+}
+
+currentPendingCheckpointId = checkpointId;
+latestAttemptedCheckpointId = checkpointId;
+
+for (int subtask : subtaskGatewayMap.keySet()) {
+acknowledgeCloseGatewayFutureMap.put(subtask, new
CompletableFuture<>());
+final OperatorEvent closeGatewayEvent = new
CloseGatewayEvent(checkpointId, subtask);
+subtaskGatewayMap
+.get(subtask)
+.sendEventWithCallBackOnCompletion(
+closeGatewayEvent,
+(success, failure) -> {
+if (failure != null) {
+// If the close gateway event failed to
reach the subtask for
+// some reason, the coordinator would
trigger a fail-over on
+// the subtask if the subtask is still
running. This behavior
+// also guarantees that the coordinator
won't receive more
+// events from this subtask before the
current checkpoint
+// finishes, which is equivalent to
receiving ACK from this
+// subtask.
+if (!(failure instanceof
TaskNotRunningException)) {
+boolean isFailoverTriggered =
+subtaskGatewayMap
+.get(subtask)
+
.tryTriggerTaskFailover(
Review Comment:
`Runnables.assertNoException` has only been used around
`subtaskAccess.triggerTaskFailover`, instead of
`SubtaskGatewayImpl.tryTriggerTaskFailover`. The previous usage of this method
has been preserved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-web] SteNicholas commented on pull request #564: [FLINK-28930] Add roadmap of Flink Table Store to Apache Flink Roadmap
SteNicholas commented on PR #564: URL: https://github.com/apache/flink-web/pull/564#issuecomment-1213017632 @JingsongLi, PTAL. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] syhily commented on pull request #19972: [FLINK-27399][Connector/Pulsar] Change initial consuming position setting logic for better handle the checkpoint.
syhily commented on PR #19972: URL: https://github.com/apache/flink/pull/19972#issuecomment-1213014404 @tisonkun Finally, the ci turns green. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] fsk119 commented on pull request #20514: [FLINK-28855][hive] Fix 'Table.INDEX_TABLE' not found in the Hive3
fsk119 commented on PR #20514: URL: https://github.com/apache/flink/pull/20514#issuecomment-1212984028 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-27171) Add periodic kerberos delegation token obtain possibility to DelegationTokenManager
[ https://issues.apache.org/jira/browse/FLINK-27171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578920#comment-17578920 ] Gabor Somogyi commented on FLINK-27171: --- I see your point. OK, we can plan this feature when the basic functionality is rock stable. > Add periodic kerberos delegation token obtain possibility to > DelegationTokenManager > --- > > Key: FLINK-27171 > URL: https://issues.apache.org/jira/browse/FLINK-27171 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.16.0 >Reporter: Gabor Somogyi >Assignee: Gabor Somogyi >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] zentol commented on a diff in pull request #20555: [FLINK-28861][table] Make UID generation behavior configurable and plan-only by default
zentol commented on code in PR #20555:
URL: https://github.com/apache/flink/pull/20555#discussion_r944337888
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/ExecNodeBase.java:
##
@@ -49,6 +49,13 @@
@JsonIgnoreProperties(ignoreUnknown = true)
public abstract class ExecNodeBase implements ExecNode {
+/**
+ * The default value is chosen for the JSON deserialization case. Other
cases must set this flag
+ * accordingly via {@link #setCompiled(boolean)}. It is not exposed via a
constructor arg to
+ * avoid complex constructor overloading for all {@link ExecNode}s.
+ */
+private transient boolean isCompiled = true;
Review Comment:
Would it be useful to not have a general default and force all manual
constructions to set this, to make sure this isn't accidentally missed?
For the Json deserialization case you could try adding an `StdConverter`
with `@JsonDeserialize` that sets this to `true` post deserialization.
```
@JsonDeserialize(converter = ExecNodeBase.IsCompiledFlagDefaultSetter.class)
public abstract class ExecNodeBase implements ExecNode {
public static class IsCompiledFlagDefaultSetter extends
StdConverter, ExecNodeBase> {
@Override
public ExecNodeBase convert(ExecNodeBase execNodeBase) {
execNodeBase.setCompiled(true);
return execNodeBase;
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] SteNicholas commented on a diff in pull request #20530: [FLINK-28817] NullPointerException in HybridSource when restoring from checkpoint
SteNicholas commented on code in PR #20530:
URL: https://github.com/apache/flink/pull/20530#discussion_r944332749
##
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/hybrid/HybridSourceSplitEnumerator.java:
##
@@ -218,7 +218,8 @@ public void handleSourceEvent(int subtaskId, SourceEvent
sourceEvent) {
}
if (subtaskSourceIndex < currentSourceIndex) {
-subtaskSourceIndex++;
+// find initial or next index for the reader
+subtaskSourceIndex =
switchedSources.getNextSubtaskSourceIndex(subtaskSourceIndex);
Review Comment:
IMO, the `getNextSubtaskSourceIndex` method name isn't matched for
`SwitchedSources`. There is no any subtask concept in `SwitchedSources`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] SteNicholas commented on a diff in pull request #20530: [FLINK-28817] NullPointerException in HybridSource when restoring from checkpoint
SteNicholas commented on code in PR #20530:
URL: https://github.com/apache/flink/pull/20530#discussion_r944331765
##
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/hybrid/SwitchedSources.java:
##
@@ -45,4 +47,8 @@ public SimpleVersionedSerializer
serializerOf(int sourceIndex) {
public void put(int sourceIndex, Source source) {
sources.put(sourceIndex, Preconditions.checkNotNull(source));
}
+
+public Integer getNextSubtaskSourceIndex(int previousSubtaskSourceIndex) {
Review Comment:
```suggestion
public Integer getNextSourceIndex(int prevSourceIndex) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] xuyangzhong commented on pull request #20513: [FLINK-28858][docs] Add document to describe join hints for batch sql
xuyangzhong commented on PR #20513: URL: https://github.com/apache/flink/pull/20513#issuecomment-1212963681 Hi, thank for you contribution. I left some comments here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xuyangzhong commented on a diff in pull request #20513: [FLINK-28858][docs] Add document to describe join hints for batch sql
xuyangzhong commented on code in PR #20513:
URL: https://github.com/apache/flink/pull/20513#discussion_r944079451
##
docs/content/docs/dev/table/sql/queries/hints.md:
##
@@ -84,4 +84,222 @@ insert into kafka_table1 /*+
OPTIONS('sink.partitioner'='round-robin') */ select
```
+## Join Hints
+
+{{< label Batch >}}
+
+Join hints allow users to manually choose the join strategy in optimizer, and
manually optimize the execution plan to improve the performance of the query.
Now only support in batch mode.
+
+### Join Hints Strategies
+The following Join strategies are currently supported in Flink SQL for batch
job:
+
+
+**BroadCast Join**
+
+
+In this Join strategy, the data on the build side (usually a small table) will
be broadcast to each downstream operator,
+and the data on the probe side (usually a large table) will be sent directly
to the downstream operator with Forward.
+Then the data on the build side (small table) will be built into a Hash Table
for the probe side to query.
+
+**Hash Shuffle Join**
+
+
+In this Join strategy, the data on the Build side and the Probe side will be
shuffled with the join key, and the data
+with the same key will be distributed to the same downstream operator. Then
the data on the build side (smaller table)
Review Comment:
It seems 'the same downstream operator' should be replaced with 'the same
subtask of the downstream operator'
##
docs/content/docs/dev/table/sql/queries/hints.md:
##
@@ -84,4 +84,222 @@ insert into kafka_table1 /*+
OPTIONS('sink.partitioner'='round-robin') */ select
```
+## Join Hints
+
+{{< label Batch >}}
+
+Join hints allow users to manually choose the join strategy in optimizer, and
manually optimize the execution plan to improve the performance of the query.
Now only support in batch mode.
+
+### Join Hints Strategies
+The following Join strategies are currently supported in Flink SQL for batch
job:
+
+
+**BroadCast Join**
+
+
+In this Join strategy, the data on the build side (usually a small table) will
be broadcast to each downstream operator,
+and the data on the probe side (usually a large table) will be sent directly
to the downstream operator with Forward.
+Then the data on the build side (small table) will be built into a Hash Table
for the probe side to query.
+
+**Hash Shuffle Join**
+
+
+In this Join strategy, the data on the Build side and the Probe side will be
shuffled with the join key, and the data
+with the same key will be distributed to the same downstream operator. Then
the data on the build side (smaller table)
+will be built into a Hash Table for the probe side to query.
+
+
+Compared with the `Hash Shuffle Join` strategy, the `BroadCast Join` strategy
does not need to shuffle the probe side, which saves a lot of
+shuffle time. Therefore, when a table is tiny, the `BroadCast Join` strategy
is usually selected to avoid the shuffle
+cost and improve computing performance. However, when the scale of the small
table is large, the `BroadCast Join` strategy
+is not applicable, because the overhead of redundant data transmission will
exceed the overhead of shuffle.
+
+**Sort Merge Join**
+
+This Join strategy is aimed at the scenario of joining between two large
tables or the scenario that the data at both
+sides of the join is already in order. This strategy first shuffles the data
on both sides of the join to the downstream
+operator according to the Join Key. Then the downstream operator sorts the
data before joining, and finally joins the data at both ends.
+This strategy eliminates the need to load all data on one side into memory,
thus relieving the pressure on computational memory.
+
+**Nested Loop Join**
+
+In this Join strategy, the probe side is used as an Outer Loop, and the build
side is used as an Inner Loop, and the data is joined through two-layer loops.
+
+### Syntax
+The join hints syntax in Flink:
+```sql
+# query:
+SELECT /*+ hint_content[, hint_content] */ ...
+
+hint_content:
+hint_strategy_name(hint_item[, hint_item])
+
+hint_strategy_name:
+supported_join_hint_name
+
+hint_item:
+string_literal
+```
+
+### Examples
+
+**BroadCast Join**
+
+
+*NOTE:* `BroadCast Join` strategy only supports join with equivalence join
condition. And it not supports `Full Outer Join` join type.
+```sql
+CREATE TABLE t1 (id BIGINT, name STRING, age INT) WITH (...);
+CREATE TABLE t2 (id BIGINT, name STRING, age INT) WITH (...);
+CREATE TABLE t3 (id BIGINT, name STRING, age INT) WITH (...);
+
+-- Select broadcast join strategy with t1, t1 will be the broadcast table.
+SELECT /*+ BROADCAST(t1) */ * FROM t1 JOIN t2 ON t1.id = t2.id;
+
+-- Select broadcast join strategy with t1 and t3.
+-- For this case, when join between t1 and t2, t1 will be the broadcast table,
+-- after that, when join between the result after t1 joins t2 and t3, t3 will
be the broadcast table.
Review Comment:
", after that," -> ". And"
##
[GitHub] [flink] SteNicholas commented on pull request #20530: [FLINK-28817] NullPointerException in HybridSource when restoring from checkpoint
SteNicholas commented on PR #20530: URL: https://github.com/apache/flink/pull/20530#issuecomment-1212960450 @zhongqishang, it's better to add the test case for the situation described in [FLINK-26938](https://issues.apache.org/jira/browse/FLINK-26938). cc @tweise -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-28948) Add more test coverage for lookup table full caching
Qingsheng Ren created FLINK-28948: - Summary: Add more test coverage for lookup table full caching Key: FLINK-28948 URL: https://issues.apache.org/jira/browse/FLINK-28948 Project: Flink Issue Type: Improvement Components: Table SQL / Runtime Affects Versions: 1.16.0 Reporter: Qingsheng Ren Fix For: 1.16.0 Currently there's only IT case for lookup table full caching. We need to add more test cases to guard the correctness of it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] zhongqishang commented on pull request #20530: [FLINK-28817] NullPointerException in HybridSource when restoring from checkpoint
zhongqishang commented on PR #20530: URL: https://github.com/apache/flink/pull/20530#issuecomment-1212955847 @SteNicholas, In fact I encountered the situation described in FLINK-26938, I will try to add a test case as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] twalthr commented on pull request #20555: [FLINK-28861][table] Make UID generation behavior configurable and plan-only by default
twalthr commented on PR #20555: URL: https://github.com/apache/flink/pull/20555#issuecomment-1212955760 Thanks for the review @zentol. Next to the necessity to pass around the `isCompile` flag, I found another issue due to the fact that transformations can be caches in a transient variable. In the end, I found a way nicer solution by just moving the flag a call higher in the planning phase. The code looks much cleaner now, I also added an additional test. Please have another look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28890) Incorrect semantic of latestLoadTime in CachingLookupFunction and CachingAsyncLookupFunction
[ https://issues.apache.org/jira/browse/FLINK-28890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren updated FLINK-28890: -- Priority: Critical (was: Major) > Incorrect semantic of latestLoadTime in CachingLookupFunction and > CachingAsyncLookupFunction > > > Key: FLINK-28890 > URL: https://issues.apache.org/jira/browse/FLINK-28890 > Project: Flink > Issue Type: Bug > Components: Table SQL / Runtime >Affects Versions: 1.16.0 >Reporter: Qingsheng Ren >Assignee: Qingsheng Ren >Priority: Critical > Labels: pull-request-available > Fix For: 1.16.0 > > > The semantic of latestLoadTime in CachingLookupFunction and > CachingAsyncLookupFunction is not correct, which should be the time spent for > the latest load operation -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] SteNicholas commented on pull request #20530: [FLINK-28817] NullPointerException in HybridSource when restoring from checkpoint
SteNicholas commented on PR #20530: URL: https://github.com/apache/flink/pull/20530#issuecomment-1212938121 @zhongqishang, could you please add test which refers to [HybridSourceTest](https://issues.apache.org/jira/secure/attachment/13041815/HybridSourceTest.java) to verify whether to fix the bug of [FLINK-26938](https://issues.apache.org/jira/browse/FLINK-26938)? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-28817) NullPointerException in HybridSource when restoring from checkpoint
[
https://issues.apache.org/jira/browse/FLINK-28817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578893#comment-17578893
]
Nicholas Jiang commented on FLINK-28817:
[~thw], IMO, this pull request could also fixed the bug of FLINK-26938 , right?
cc [~zhongqishang]
> NullPointerException in HybridSource when restoring from checkpoint
> ---
>
> Key: FLINK-28817
> URL: https://issues.apache.org/jira/browse/FLINK-28817
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.14.4, 1.15.1
>Reporter: Michael
>Priority: Major
> Labels: pull-request-available
> Attachments: Preconditions-checkNotNull-error.zip,
> bf-29-JM-err-analysis.log
>
>
> Scenario:
> # CheckpointCoordinator - Completed checkpoint 14 for job
>
> # HybridSource successfully completed processing a few SourceFactories, that
> reads from s3
> # HybridSourceSplitEnumerator.switchEnumerator failed with
> com.amazonaws.SdkClientException: Unable to execute HTTP request: Read timed
> out. This is intermittent error, it is usually fixed, when Flink recover from
> checkpoint & repeat the operation.
> # Flink starts recovering from checkpoint,
> # HybridSourceSplitEnumerator receives
> SourceReaderFinishedEvent\{sourceIndex=-1}
> # Processing this event cause
> 2022/08/08 08:39:34.862 ERROR o.a.f.r.s.c.SourceCoordinator - Uncaught
> exception in the SplitEnumerator for Source Source: hybrid-source while
> handling operator event SourceEventWrapper[SourceReaderFinishedEvent
> {sourceIndex=-1}
> ] from subtask 6. Triggering job failover.
> java.lang.NullPointerException: Source for index=0 is not available from
> sources: \{788=org.apache.flink.connector.file.src.SppFileSource@5a3803f3}
> at org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:104)
> at
> org.apache.flink.connector.base.source.hybridspp.SwitchedSources.sourceOf(SwitchedSources.java:36)
> at
> org.apache.flink.connector.base.source.hybridspp.HybridSourceSplitEnumerator.sendSwitchSourceEvent(HybridSourceSplitEnumerator.java:152)
> at
> org.apache.flink.connector.base.source.hybridspp.HybridSourceSplitEnumerator.handleSourceEvent(HybridSourceSplitEnumerator.java:226)
> ...
> I'm running my version of the Hybrid Sources with additional logging, so line
> numbers & some names could be different from Flink Github.
> My Observation: the problem is intermittent, sometimes it works ok, i.e.
> SourceReaderFinishedEvent comes with correct sourceIndex. As I see from my
> log, it happens if my SourceFactory.create() is executed BEFORE
> HybridSourceSplitEnumerator - handleSourceEvent
> SourceReaderFinishedEvent\{sourceIndex=-1}.
> If HybridSourceSplitEnumerator - handleSourceEvent is executed before my
> SourceFactory.create(), then sourceIndex=-1 in SourceReaderFinishedEvent
> Preconditions-checkNotNull-error log from JobMgr is attached
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] wsry commented on pull request #20560: [FLINK-28942][network] Fix the deadlock bug when recycling readers for SortMergeResultPartition
wsry commented on PR #20560: URL: https://github.com/apache/flink/pull/20560#issuecomment-1212934966 @TanYuxin-tyx Thanks for the fix. LGTM. Merged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-28942) Deadlock may occurs when releasing readers for SortMergeResultPartition
[ https://issues.apache.org/jira/browse/FLINK-28942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao resolved FLINK-28942. - Resolution: Fixed Merged into master via f2fb6b20ec493a3af3f19a6f69f25e26ed226dda > Deadlock may occurs when releasing readers for SortMergeResultPartition > --- > > Key: FLINK-28942 > URL: https://issues.apache.org/jira/browse/FLINK-28942 > Project: Flink > Issue Type: Bug >Affects Versions: 1.16.0 >Reporter: Yuxin Tan >Assignee: Yuxin Tan >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0 > > > After adding the logic of recycling buffers in CompositeBuffer in > https://issues.apache.org/jira/browse/FLINK-28373, when reading data and > recycling buffers simultaneously, the deadlock between the lock of > SortMergeResultPartition and the lock of SingleInputGate may occur. > In short, the deadlock may occur as follows. > 1. SingleInputGate.getNextBufferOrEvent (SingleInputGate lock) > CompositeBuffer.getFullBufferData -> CompositeBuffer.recycleBuffer -> waiting > for > SortMergeResultPartition lock; > 2. ResultPartitionManager.releasePartition (SortMergeResultPartition lock) -> > SortMergeSubpartitionReader.notifyDataAvailable -> > SingleInputGate.notifyChannelNonEmpty -> waiting for SingleInputGate lock. > The possibility of this deadlock is very small, but we should fix the bug as > soon as possible. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-28942) Deadlock may occurs when releasing readers for SortMergeResultPartition
[ https://issues.apache.org/jira/browse/FLINK-28942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao reassigned FLINK-28942: --- Assignee: Yuxin Tan > Deadlock may occurs when releasing readers for SortMergeResultPartition > --- > > Key: FLINK-28942 > URL: https://issues.apache.org/jira/browse/FLINK-28942 > Project: Flink > Issue Type: Bug >Affects Versions: 1.16.0 >Reporter: Yuxin Tan >Assignee: Yuxin Tan >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0 > > > After adding the logic of recycling buffers in CompositeBuffer in > https://issues.apache.org/jira/browse/FLINK-28373, when reading data and > recycling buffers simultaneously, the deadlock between the lock of > SortMergeResultPartition and the lock of SingleInputGate may occur. > In short, the deadlock may occur as follows. > 1. SingleInputGate.getNextBufferOrEvent (SingleInputGate lock) > CompositeBuffer.getFullBufferData -> CompositeBuffer.recycleBuffer -> waiting > for > SortMergeResultPartition lock; > 2. ResultPartitionManager.releasePartition (SortMergeResultPartition lock) -> > SortMergeSubpartitionReader.notifyDataAvailable -> > SingleInputGate.notifyChannelNonEmpty -> waiting for SingleInputGate lock. > The possibility of this deadlock is very small, but we should fix the bug as > soon as possible. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28942) Deadlock may occurs when releasing readers for SortMergeResultPartition
[ https://issues.apache.org/jira/browse/FLINK-28942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao updated FLINK-28942: Fix Version/s: 1.16.0 > Deadlock may occurs when releasing readers for SortMergeResultPartition > --- > > Key: FLINK-28942 > URL: https://issues.apache.org/jira/browse/FLINK-28942 > Project: Flink > Issue Type: Bug >Affects Versions: 1.16.0 >Reporter: Yuxin Tan >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0 > > > After adding the logic of recycling buffers in CompositeBuffer in > https://issues.apache.org/jira/browse/FLINK-28373, when reading data and > recycling buffers simultaneously, the deadlock between the lock of > SortMergeResultPartition and the lock of SingleInputGate may occur. > In short, the deadlock may occur as follows. > 1. SingleInputGate.getNextBufferOrEvent (SingleInputGate lock) > CompositeBuffer.getFullBufferData -> CompositeBuffer.recycleBuffer -> waiting > for > SortMergeResultPartition lock; > 2. ResultPartitionManager.releasePartition (SortMergeResultPartition lock) -> > SortMergeSubpartitionReader.notifyDataAvailable -> > SingleInputGate.notifyChannelNonEmpty -> waiting for SingleInputGate lock. > The possibility of this deadlock is very small, but we should fix the bug as > soon as possible. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-28719) Mapping a data source before window aggregation causes Flink to stop handle late events correctly.
[ https://issues.apache.org/jira/browse/FLINK-28719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578890#comment-17578890 ] Chesnay Schepler edited comment on FLINK-28719 at 8/12/22 9:42 AM: --- ??As far as I understand, op1 and op2 should have watermark 1 and 2 respectively, because those subtasks don't have any events in them (besides 1 and 2, of course, which create those watermarks). Then, why do they get the maximum watermark of 7 after first step??? Watermarks are broadcasted to all downstream operators. So, if a source emits watermark 7, then all maps get it as an input. ??Also, why after op1 and op2, op4 and op5 get consumed? Is there any strategy that dictates in which order to process subtasks??? Whichever arrives first at the downstream operator. You have 7 map subtasks all sending data to the window operators at the same time, and the order in which they arrive is not deterministic. So they _may_ arrive in a perfect round-robin pattern, or sequentially, or in any other pattern. The only guarantee is that the elements from a particular map subtask arrive in the same order that they were sent in. ??Also, why op3 to op7 have such watermarks: W2, W5, W6, W6, W6, after first step? I thought those subtasks should have watermark of Long.MinValue, because there were no elements before??? The above 2 comments should answer it; all watermarks are sent to all map subtasks, and are consumed by the window operator in any order. was (Author: zentol): ??As far as I understand, op1 and op2 should have watermark 1 and 2 respectively, because those subtasks don't have any events in them (besides 1 and 2, of course, which create those watermarks). Then, why do they get the maximum watermark of 7 after first step??? Watermarks are broadcasted to all downstream operators. So, if a source emits watermark 7, then all maps get it as an input. ??Also, why after op1 and op2, op4 and op5 get consumed? Is there any strategy that dictates in which order to process subtasks? Whichever arrives first at the downstream operator. You have 7 map subtasks all sending data to the window operators at the same time, and the order in which they arrive is not deterministic. So they _may_ arrive in a perfect round-robin pattern, or sequentially, or in any other pattern. The only guarantee is that the elements from a particular map subtask arrive in the same order that they were sent in. ??Also, why op3 to op7 have such watermarks: W2, W5, W6, W6, W6, after first step? I thought those subtasks should have watermark of Long.MinValue, because there were no elements before? The above 2 comments should answer it; all watermarks are sent to all map subtasks, and are consumed by the window operator in any order. > Mapping a data source before window aggregation causes Flink to stop handle > late events correctly. > -- > > Key: FLINK-28719 > URL: https://issues.apache.org/jira/browse/FLINK-28719 > Project: Flink > Issue Type: Bug > Components: API / DataStream >Affects Versions: 1.15.1 >Reporter: Mykyta Mykhailenko >Priority: Major > > I have created a > [repository|https://github.com/mykytamykhailenko/flink-map-with-issue] where > I describe this issue in detail. > I have provided a few tests and source code so that you can reproduce the > issue on your own machine. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] wsry merged pull request #20560: [FLINK-28942][network] Fix the deadlock bug when recycling readers for SortMergeResultPartition
wsry merged PR #20560: URL: https://github.com/apache/flink/pull/20560 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-28719) Mapping a data source before window aggregation causes Flink to stop handle late events correctly.
[ https://issues.apache.org/jira/browse/FLINK-28719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578890#comment-17578890 ] Chesnay Schepler edited comment on FLINK-28719 at 8/12/22 9:43 AM: --- ??As far as I understand, op1 and op2 should have watermark 1 and 2 respectively, because those subtasks don't have any events in them (besides 1 and 2, of course, which create those watermarks). Then, why do they get the maximum watermark of 7 after first step??? Watermarks are broadcasted to all downstream operators. So, if a source emits watermark 7, then all map subtasks get it as an input. ??Also, why after op1 and op2, op4 and op5 get consumed? Is there any strategy that dictates in which order to process subtasks??? Whichever arrives first at the downstream operator. You have 7 map subtasks all sending data to the window operators at the same time, and the order in which they arrive is not deterministic. So they _may_ arrive in a perfect round-robin pattern, or sequentially, or in any other pattern. The only guarantee is that the elements from a particular map subtask arrive in the same order that they were sent in. ??Also, why op3 to op7 have such watermarks: W2, W5, W6, W6, W6, after first step? I thought those subtasks should have watermark of Long.MinValue, because there were no elements before??? The above 2 comments should answer it; all watermarks are sent to all map subtasks, and are consumed by the window operator in any order. was (Author: zentol): ??As far as I understand, op1 and op2 should have watermark 1 and 2 respectively, because those subtasks don't have any events in them (besides 1 and 2, of course, which create those watermarks). Then, why do they get the maximum watermark of 7 after first step??? Watermarks are broadcasted to all downstream operators. So, if a source emits watermark 7, then all maps get it as an input. ??Also, why after op1 and op2, op4 and op5 get consumed? Is there any strategy that dictates in which order to process subtasks??? Whichever arrives first at the downstream operator. You have 7 map subtasks all sending data to the window operators at the same time, and the order in which they arrive is not deterministic. So they _may_ arrive in a perfect round-robin pattern, or sequentially, or in any other pattern. The only guarantee is that the elements from a particular map subtask arrive in the same order that they were sent in. ??Also, why op3 to op7 have such watermarks: W2, W5, W6, W6, W6, after first step? I thought those subtasks should have watermark of Long.MinValue, because there were no elements before??? The above 2 comments should answer it; all watermarks are sent to all map subtasks, and are consumed by the window operator in any order. > Mapping a data source before window aggregation causes Flink to stop handle > late events correctly. > -- > > Key: FLINK-28719 > URL: https://issues.apache.org/jira/browse/FLINK-28719 > Project: Flink > Issue Type: Bug > Components: API / DataStream >Affects Versions: 1.15.1 >Reporter: Mykyta Mykhailenko >Priority: Major > > I have created a > [repository|https://github.com/mykytamykhailenko/flink-map-with-issue] where > I describe this issue in detail. > I have provided a few tests and source code so that you can reproduce the > issue on your own machine. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-28719) Mapping a data source before window aggregation causes Flink to stop handle late events correctly.
[ https://issues.apache.org/jira/browse/FLINK-28719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578890#comment-17578890 ] Chesnay Schepler edited comment on FLINK-28719 at 8/12/22 9:42 AM: --- ??As far as I understand, op1 and op2 should have watermark 1 and 2 respectively, because those subtasks don't have any events in them (besides 1 and 2, of course, which create those watermarks). Then, why do they get the maximum watermark of 7 after first step??? Watermarks are broadcasted to all downstream operators. So, if a source emits watermark 7, then all maps get it as an input. ??Also, why after op1 and op2, op4 and op5 get consumed? Is there any strategy that dictates in which order to process subtasks? Whichever arrives first at the downstream operator. You have 7 map subtasks all sending data to the window operators at the same time, and the order in which they arrive is not deterministic. So they _may_ arrive in a perfect round-robin pattern, or sequentially, or in any other pattern. The only guarantee is that the elements from a particular map subtask arrive in the same order that they were sent in. ??Also, why op3 to op7 have such watermarks: W2, W5, W6, W6, W6, after first step? I thought those subtasks should have watermark of Long.MinValue, because there were no elements before? The above 2 comments should answer it; all watermarks are sent to all map subtasks, and are consumed by the window operator in any order. was (Author: zentol): > As far as I understand, op1 and op2 should have watermark 1 and 2 > respectively, because those subtasks don't have any events in them (besides 1 > and 2, of course, which create those watermarks). Then, why do they get the > maximum watermark of 7 after first step? Watermarks are broadcasted to all downstream operators. So, if a source emits watermark 7, then all maps get it as an input. > Also, why after op1 and op2, op4 and op5 get consumed? Is there any strategy > that dictates in which order to process subtasks? Whichever arrives first at the downstream operator. You have 7 map subtasks all sending data to the window operators at the same time, and the order in which they arrive is not deterministic. So they _may_ arrive in a perfect round-robin pattern, or sequentially, or in any other pattern. The only guarantee is that the elements from a particular map subtask arrive in the same order that they were sent in. > Also, why op3 to op7 have such watermarks: W2, W5, W6, W6, W6, after first > step? I thought those subtasks should have watermark of Long.MinValue, > because there were no elements before? The above 2 comments should answer it; all watermarks are sent to all map subtasks, and are consumed by the window operator in any order. > Mapping a data source before window aggregation causes Flink to stop handle > late events correctly. > -- > > Key: FLINK-28719 > URL: https://issues.apache.org/jira/browse/FLINK-28719 > Project: Flink > Issue Type: Bug > Components: API / DataStream >Affects Versions: 1.15.1 >Reporter: Mykyta Mykhailenko >Priority: Major > > I have created a > [repository|https://github.com/mykytamykhailenko/flink-map-with-issue] where > I describe this issue in detail. > I have provided a few tests and source code so that you can reproduce the > issue on your own machine. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-28947) Curator framework fails with NullPointerException
Juha created FLINK-28947:
Summary: Curator framework fails with NullPointerException
Key: FLINK-28947
URL: https://issues.apache.org/jira/browse/FLINK-28947
Project: Flink
Issue Type: Bug
Affects Versions: 1.15.1
Reporter: Juha
I'm getting the following error in JobManager and as a result JobManager exits.
{code:java}
Aug 12 06:37:30 server_name java[173]: [2022-08-12 06:37:30,491] ERROR
Background exception was not retry-able or retry gave up
(org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.CuratorFrameworkImpl:733)
Aug 12 06:37:30 server_name java[173]: java.lang.NullPointerException: null
Aug 12 06:37:30 server_name java[173]: at
org.apache.flink.shaded.curator5.org.apache.curator.utils.Compatibility.getHostAddress(Compatibility.java:116)
~[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at
org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.EnsembleTracker.configToConnectionString(EnsembleTracker.java:185)
~[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at
org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.EnsembleTracker.processConfigData(EnsembleTracker.java:206)
~[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at
org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.EnsembleTracker.access$300(EnsembleTracker.java:50)
~[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at
org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.EnsembleTracker$2.processResult(EnsembleTracker.java:150)
~[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at
org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.CuratorFrameworkImpl.sendToBackgroundCallback(CuratorFrameworkImpl.java:926)
[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at
org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:683)
[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at
org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.WatcherRemovalFacade.processBackgroundOperation(WatcherRemovalFacade.java:152)
[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at
org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.GetConfigBuilderImpl$2.processResult(GetConfigBuilderImpl.java:222)
[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at
org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:598)
[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at
org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: [2022-08-12 06:37:30,493] ERROR
Unhandled error in curator framework, error message: Background exception was
not retry-able or retry gave up
(org.apache.flink.runtime.util.ZooKeeperUtils:292)
Aug 12 06:37:30 server_name java[173]: java.lang.NullPointerException: null
Aug 12 06:37:30 server_name java[173]: at
org.apache.flink.shaded.curator5.org.apache.curator.utils.Compatibility.getHostAddress(Compatibility.java:116)
~[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at
org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.EnsembleTracker.configToConnectionString(EnsembleTracker.java:185)
~[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at
org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.EnsembleTracker.processConfigData(EnsembleTracker.java:206)
~[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at
org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.EnsembleTracker.access$300(EnsembleTracker.java:50)
~[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at
org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.EnsembleTracker$2.processResult(EnsembleTracker.java:150)
~[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at
org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.CuratorFrameworkImpl.sendToBackgroundCallback(CuratorFrameworkImpl.java:926)
[flink-shaded-zookeeper-3.5.9.jar:3.5.9-15.0]
Aug 12 06:37:30 server_name java[173]: at
[jira] [Commented] (FLINK-28719) Mapping a data source before window aggregation causes Flink to stop handle late events correctly.
[ https://issues.apache.org/jira/browse/FLINK-28719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578890#comment-17578890 ] Chesnay Schepler commented on FLINK-28719: -- > As far as I understand, op1 and op2 should have watermark 1 and 2 > respectively, because those subtasks don't have any events in them (besides 1 > and 2, of course, which create those watermarks). Then, why do they get the > maximum watermark of 7 after first step? Watermarks are broadcasted to all downstream operators. So, if a source emits watermark 7, then all maps get it as an input. > Also, why after op1 and op2, op4 and op5 get consumed? Is there any strategy > that dictates in which order to process subtasks? Whichever arrives first at the downstream operator. You have 7 map subtasks all sending data to the window operators at the same time, and the order in which they arrive is not deterministic. So they _may_ arrive in a perfect round-robin pattern, or sequentially, or in any other pattern. The only guarantee is that the elements from a particular map subtask arrive in the same order that they were sent in. > Also, why op3 to op7 have such watermarks: W2, W5, W6, W6, W6, after first > step? I thought those subtasks should have watermark of Long.MinValue, > because there were no elements before? The above 2 comments should answer it; all watermarks are sent to all map subtasks, and are consumed by the window operator in any order. > Mapping a data source before window aggregation causes Flink to stop handle > late events correctly. > -- > > Key: FLINK-28719 > URL: https://issues.apache.org/jira/browse/FLINK-28719 > Project: Flink > Issue Type: Bug > Components: API / DataStream >Affects Versions: 1.15.1 >Reporter: Mykyta Mykhailenko >Priority: Major > > I have created a > [repository|https://github.com/mykytamykhailenko/flink-map-with-issue] where > I describe this issue in detail. > I have provided a few tests and source code so that you can reproduce the > issue on your own machine. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] zhuzhurk commented on pull request #20559: [FLINK-28878][tests] Increase slot request timeout of PipelinedRegionSchedulingITCase
zhuzhurk commented on PR #20559: URL: https://github.com/apache/flink/pull/20559#issuecomment-1212918280 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-28766) UnalignedCheckpointStressITCase.runStressTest failed with NoSuchFileException
[
https://issues.apache.org/jira/browse/FLINK-28766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578887#comment-17578887
]
Zhu Zhu commented on FLINK-28766:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=39908=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7
> UnalignedCheckpointStressITCase.runStressTest failed with NoSuchFileException
> -
>
> Key: FLINK-28766
> URL: https://issues.apache.org/jira/browse/FLINK-28766
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.16.0
>Reporter: Huang Xingbo
>Priority: Critical
> Labels: test-stability
> Fix For: 1.16.0
>
>
> {code:java}
> 2022-08-01T01:36:16.0563880Z Aug 01 01:36:16 [ERROR]
> org.apache.flink.test.checkpointing.UnalignedCheckpointStressITCase.runStressTest
> Time elapsed: 12.579 s <<< ERROR!
> 2022-08-01T01:36:16.0565407Z Aug 01 01:36:16 java.io.UncheckedIOException:
> java.nio.file.NoSuchFileException:
> /tmp/junit1058240190382532303/f0f99754a53d2c4633fed75011da58dd/chk-7/61092e4a-5b9a-4f56-83f7-d9960c53ed3e
> 2022-08-01T01:36:16.0566296Z Aug 01 01:36:16 at
> java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:88)
> 2022-08-01T01:36:16.0566972Z Aug 01 01:36:16 at
> java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:104)
> 2022-08-01T01:36:16.0567600Z Aug 01 01:36:16 at
> java.util.Iterator.forEachRemaining(Iterator.java:115)
> 2022-08-01T01:36:16.0568290Z Aug 01 01:36:16 at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> 2022-08-01T01:36:16.0569172Z Aug 01 01:36:16 at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> 2022-08-01T01:36:16.0569877Z Aug 01 01:36:16 at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> 2022-08-01T01:36:16.0570554Z Aug 01 01:36:16 at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> 2022-08-01T01:36:16.0571371Z Aug 01 01:36:16 at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> 2022-08-01T01:36:16.0572417Z Aug 01 01:36:16 at
> java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:546)
> 2022-08-01T01:36:16.0573618Z Aug 01 01:36:16 at
> org.apache.flink.test.checkpointing.UnalignedCheckpointStressITCase.discoverRetainedCheckpoint(UnalignedCheckpointStressITCase.java:289)
> 2022-08-01T01:36:16.0575187Z Aug 01 01:36:16 at
> org.apache.flink.test.checkpointing.UnalignedCheckpointStressITCase.runAndTakeExternalCheckpoint(UnalignedCheckpointStressITCase.java:262)
> 2022-08-01T01:36:16.0576540Z Aug 01 01:36:16 at
> org.apache.flink.test.checkpointing.UnalignedCheckpointStressITCase.runStressTest(UnalignedCheckpointStressITCase.java:158)
> 2022-08-01T01:36:16.0577684Z Aug 01 01:36:16 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2022-08-01T01:36:16.0578546Z Aug 01 01:36:16 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2022-08-01T01:36:16.0579374Z Aug 01 01:36:16 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-08-01T01:36:16.0580298Z Aug 01 01:36:16 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2022-08-01T01:36:16.0581243Z Aug 01 01:36:16 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 2022-08-01T01:36:16.0582029Z Aug 01 01:36:16 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2022-08-01T01:36:16.0582766Z Aug 01 01:36:16 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 2022-08-01T01:36:16.0583488Z Aug 01 01:36:16 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2022-08-01T01:36:16.0584203Z Aug 01 01:36:16 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2022-08-01T01:36:16.0585087Z Aug 01 01:36:16 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2022-08-01T01:36:16.0585778Z Aug 01 01:36:16 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2022-08-01T01:36:16.0586482Z Aug 01 01:36:16 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2022-08-01T01:36:16.0587155Z Aug 01 01:36:16 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2022-08-01T01:36:16.0587809Z Aug 01 01:36:16 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2022-08-01T01:36:16.0588434Z Aug 01 01:36:16 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2022-08-01T01:36:16.0589203Z Aug 01 01:36:16 at
[jira] [Commented] (FLINK-24318) Casting a number to boolean has different results between 'select' fields and 'where' condition
[
https://issues.apache.org/jira/browse/FLINK-24318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578884#comment-17578884
]
Yuan Kui commented on FLINK-24318:
--
In 1.13.4, correct commit is: ea53e4a6cd1d39592cf3a6ae911fed332234d00e
> Casting a number to boolean has different results between 'select' fields and
> 'where' condition
> ---
>
> Key: FLINK-24318
> URL: https://issues.apache.org/jira/browse/FLINK-24318
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: xuyang
>Assignee: xuyang
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.13.6, 1.14.3, 1.15.0
>
>
> The same cast in the following two sql:
> {code:java}
> // SQL 1
> SELECT cast(0.1 as boolean)
> // SQL 2
> SELECT * from test2 where cast(0.1 as boolean)
> {code}
> has different results.
> The cast result in SQL 1 is true and the cast in SQL 2 is false.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-28946) Remove RexSimplify together with calcite upgrade
Sergey Nuyanzin created FLINK-28946:
---
Summary: Remove RexSimplify together with calcite upgrade
Key: FLINK-28946
URL: https://issues.apache.org/jira/browse/FLINK-28946
Project: Flink
Issue Type: Sub-task
Components: Table SQL / API
Reporter: Sergey Nuyanzin
As mentioned in {{org.apache.calcite.rex.RexSimplify}} comment
{noformat}
* Copied to fix Calcite 1.26 bugs, should be removed for the next Calcite
upgrade.
*
* Changes (line numbers are from the original RexSimplify file):
*
*
* CALCITE-4364 & FLINK-19811: Line 1307, Line 1764, Line 2638 ~ Line
2656.
* CALCITE-4446 & FLINK-22015: Line 2542 ~ Line 2548, Line 2614 ~ Line
2619.
*
{noformta}
at
https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/java/org/apache/calcite/rex/RexSimplify.java#L73-L78
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28946) Remove RexSimplify together with calcite upgrade
[
https://issues.apache.org/jira/browse/FLINK-28946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Nuyanzin updated FLINK-28946:
Description:
As mentioned in {{org.apache.calcite.rex.RexSimplify}} comment
{noformat}
* Copied to fix Calcite 1.26 bugs, should be removed for the next Calcite
upgrade.
*
* Changes (line numbers are from the original RexSimplify file):
*
*
* CALCITE-4364 & FLINK-19811: Line 1307, Line 1764, Line 2638 ~ Line
2656.
* CALCITE-4446 & FLINK-22015: Line 2542 ~ Line 2548, Line 2614 ~ Line
2619.
*
{noformat}
at
https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/java/org/apache/calcite/rex/RexSimplify.java#L73-L78
both https://issues.apache.org/jira/browse/CALCITE-4364 and
https://issues.apache.org/jira/browse/CALCITE-4446 are in 1.27+
was:
As mentioned in {{org.apache.calcite.rex.RexSimplify}} comment
{noformat}
* Copied to fix Calcite 1.26 bugs, should be removed for the next Calcite
upgrade.
*
* Changes (line numbers are from the original RexSimplify file):
*
*
* CALCITE-4364 & FLINK-19811: Line 1307, Line 1764, Line 2638 ~ Line
2656.
* CALCITE-4446 & FLINK-22015: Line 2542 ~ Line 2548, Line 2614 ~ Line
2619.
*
{noformta}
at
https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/java/org/apache/calcite/rex/RexSimplify.java#L73-L78
> Remove RexSimplify together with calcite upgrade
>
>
> Key: FLINK-28946
> URL: https://issues.apache.org/jira/browse/FLINK-28946
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Sergey Nuyanzin
>Priority: Major
>
> As mentioned in {{org.apache.calcite.rex.RexSimplify}} comment
> {noformat}
> * Copied to fix Calcite 1.26 bugs, should be removed for the next Calcite
> upgrade.
> *
> * Changes (line numbers are from the original RexSimplify file):
> *
> *
> * CALCITE-4364 & FLINK-19811: Line 1307, Line 1764, Line 2638 ~ Line
> 2656.
> * CALCITE-4446 & FLINK-22015: Line 2542 ~ Line 2548, Line 2614 ~ Line
> 2619.
> *
> {noformat}
> at
> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/java/org/apache/calcite/rex/RexSimplify.java#L73-L78
> both https://issues.apache.org/jira/browse/CALCITE-4364 and
> https://issues.apache.org/jira/browse/CALCITE-4446 are in 1.27+
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] flinkbot commented on pull request #20561: [FLINK-28944][python][docs] Optimize the Python Execution Mode Documentation
flinkbot commented on PR #20561: URL: https://github.com/apache/flink/pull/20561#issuecomment-1212901358 ## CI report: * 974d52efffb1742d3d98b8b7cbae54b80c91c778 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28944) Optimize the Python Execution Mode Documentation
[ https://issues.apache.org/jira/browse/FLINK-28944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-28944: --- Labels: pull-request-available (was: ) > Optimize the Python Execution Mode Documentation > > > Key: FLINK-28944 > URL: https://issues.apache.org/jira/browse/FLINK-28944 > Project: Flink > Issue Type: Improvement > Components: API / Python, Documentation >Affects Versions: 1.16.0 >Reporter: Huang Xingbo >Assignee: Huang Xingbo >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0 > > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/python/python_execution_mode/ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] HuangXingBo opened a new pull request, #20561: [FLINK-28944][python][docs] Optimize the Python Execution Mode Documentation
HuangXingBo opened a new pull request, #20561: URL: https://github.com/apache/flink/pull/20561 ## What is the purpose of the change *This pull request will optimize the Python Execution Mode Documentation* ## Brief change log - *Optimize the Python Execution Mode Documentation* ## Verifying this change This change added tests and can be verified as follows: - *Execute `build_docs.sh`* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-benchmarks] rkhachatryan commented on pull request #58: [FLINK-28931] Fix BlockingPartitionBenchmark compile error
rkhachatryan commented on PR #58: URL: https://github.com/apache/flink-benchmarks/pull/58#issuecomment-1212877266 Thanks @reswqa and @fredia for the prompt review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhongqishang commented on pull request #20530: [FLINK-28817] NullPointerException in HybridSource when restoring from checkpoint
zhongqishang commented on PR #20530: URL: https://github.com/apache/flink/pull/20530#issuecomment-1212873257 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-27856) Adding pod template without spec crashes job manager
[
https://issues.apache.org/jira/browse/FLINK-27856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Wang closed FLINK-27856.
-
Resolution: Fixed
> Adding pod template without spec crashes job manager
>
>
> Key: FLINK-27856
> URL: https://issues.apache.org/jira/browse/FLINK-27856
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes
>Reporter: Jeesmon Jacob
>Assignee: Peng Yuan
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.16.0, 1.15.2
>
>
> While trying to add Pod annotation through pod template in FlinkDeployment,
> taskmanager was keep crashing.
> Pod template that I used:
> {code:java}
> taskManager:
> podTemplate:
> apiVersion: v1
> kind: Pod
> metadata:
> annotations:
> iam.amazonaws.com/role: fake-role-arn
> {code}
> It created below ConfigMap and mounted to the deployment:
> {code:java}
> apiVersion: v1
> data:
> taskmanager-pod-template.yaml: |
> ---
> apiVersion: "v1"
> kind: "Pod"
> metadata:
> annotations:
> iam.amazonaws.com/role: "fake-role-arn"
> kind: ConfigMap
> {code}
> Looks like missing "spec" stanza in pod template resulted in the crash and I
> couldn't find any documentation that "spec" is required for pod template even
> for just adding metadata annotations.
> Adding below worked fine
> {code:java}
> taskManager:
> podTemplate:
> apiVersion: v1
> kind: Pod
> metadata:
> annotations:
> iam.amazonaws.com/role: fake-role-arn
> spec: {}
> {code}
> Corresponding ConfigMap
> {code:java}
> apiVersion: v1
> data:
> taskmanager-pod-template.yaml: |
> ---
> apiVersion: "v1"
> kind: "Pod"
> metadata:
> annotations:
> iam.amazonaws.com/role: "fake-role-arn"
> spec:
> containers: []
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
