[GitHub] [flink] dawidwys commented on pull request #19439: [FLINK-26977] Remove no longer valid tests in JobMasterStopWithSavepointITCase
dawidwys commented on PR #19439: URL: https://github.com/apache/flink/pull/19439#issuecomment-1097626371 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-27215) JDBC sink transiently deleted a record because of -u message of that record
[
https://issues.apache.org/jira/browse/FLINK-27215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17521495#comment-17521495
]
tim yu commented on FLINK-27215:
Hi [~jark], Should JDBC sink skip -U message in upsert mode?
> JDBC sink transiently deleted a record because of -u message of that record
> ---
>
> Key: FLINK-27215
> URL: https://issues.apache.org/jira/browse/FLINK-27215
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / JDBC
>Affects Versions: 1.12.0
>Reporter: tim yu
>Priority: Major
>
> A record is deleted transiently when using JDBC sink in upsert mode.
> The -U message is processed as delete operation in class
> TableBufferReducedStatementExecutor.
> The following codes show how to process -U message:
> {code:java}
> /**
> * Returns true if the row kind is INSERT or UPDATE_AFTER, returns false
> if the row kind is
> * DELETE or UPDATE_BEFORE.
> */
> private boolean changeFlag(RowKind rowKind) {
> switch (rowKind) {
> case INSERT:
> case UPDATE_AFTER:
> return true;
> case DELETE:
> case UPDATE_BEFORE:
> return false;
> default:
> throw new UnsupportedOperationException(
> String.format(
> "Unknown row kind, the supported row kinds
> is: INSERT, UPDATE_BEFORE, UPDATE_AFTER,"
> + " DELETE, but get: %s.",
> rowKind));
> }
> }
> @Override
> public void executeBatch() throws SQLException {
> for (Map.Entry> entry :
> reduceBuffer.entrySet()) {
> if (entry.getValue().f0) {
> upsertExecutor.addToBatch(entry.getValue().f1);
> } else {
> // delete by key
> deleteExecutor.addToBatch(entry.getKey());
> }
> }
> upsertExecutor.executeBatch();
> deleteExecutor.executeBatch();
> reduceBuffer.clear();
> }
> {code}
> If -U and +U messages of one record are executed separately in different JDBC
> batches, that record will be deleted transiently in external database and
> then insert a new updated record to it. In fact, this record should be merely
> updated once in the external database.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Closed] (FLINK-8518) Support DOW, EPOCH, DECADE for EXTRACT
[ https://issues.apache.org/jira/browse/FLINK-8518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timo Walther closed FLINK-8518. --- Fix Version/s: 1.16.0 Resolution: Fixed Fixed in master: 7715392e511b75e1bb21ac1831bb278b4c66 > Support DOW, EPOCH, DECADE for EXTRACT > -- > > Key: FLINK-8518 > URL: https://issues.apache.org/jira/browse/FLINK-8518 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Reporter: Timo Walther >Assignee: Sergey Nuyanzin >Priority: Not a Priority > Labels: auto-deprioritized-major, auto-deprioritized-minor, > auto-unassigned, pull-request-available > Fix For: 1.16.0 > > > We upgraded Calcite to version 1.15 in FLINK-7934. The EXTRACT method > supports more conversion targets. The targets DOW, EPOCH, DECADE should be > implemented and tested for different datatypes. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] twalthr closed pull request #18179: [FLINK-8518][Table SQL / API] Add support for extract Epoch
twalthr closed pull request #18179: [FLINK-8518][Table SQL / API] Add support for extract Epoch URL: https://github.com/apache/flink/pull/18179 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20457) Fix the handling of timestamp in DataStream.from_collection
[ https://issues.apache.org/jira/browse/FLINK-20457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-20457: Fix Version/s: 1.16.0 > Fix the handling of timestamp in DataStream.from_collection > --- > > Key: FLINK-20457 > URL: https://issues.apache.org/jira/browse/FLINK-20457 > Project: Flink > Issue Type: Bug > Components: API / Python >Affects Versions: 1.12.0 >Reporter: Dian Fu >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned > Fix For: 1.15.0, 1.16.0 > > > Currently, DataStream.from_collection firstly converts date/time/dateTime > objects to int at Python side and then construct the corresponding > Date/Time/Timestamp object at Java side. It will lose the timezone > information. Pickle could handle date/time/datetime properly and the > conversion could be avoided. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25043) Allow calls to public @VisibleForTesting from the same package
[ https://issues.apache.org/jira/browse/FLINK-25043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-25043: Fix Version/s: 1.16.0 > Allow calls to public @VisibleForTesting from the same package > -- > > Key: FLINK-25043 > URL: https://issues.apache.org/jira/browse/FLINK-25043 > Project: Flink > Issue Type: Improvement > Components: Tests >Reporter: Chesnay Schepler >Priority: Major > Fix For: 1.15.0, 1.16.0 > > > Consider a class having some package-private method that is used by other > classes in said package. > If this method is then needed from outside the package, and thus made public > and annotated with VisibleForTesting, then the architecture tests currently > flag the original usage as well. > We could think about allowing package-private access if the method is public. > On the other hand, if the method was originally annotated with > VisibleForTesting, then marking it as public would remove a violation, which > would be incorrect. > Maybe we need to extend our VisibleForTesting annotation to provide this > information. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-18981) Support column comment for Hive tables
[ https://issues.apache.org/jira/browse/FLINK-18981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-18981: Fix Version/s: 1.16.0 > Support column comment for Hive tables > -- > > Key: FLINK-18981 > URL: https://issues.apache.org/jira/browse/FLINK-18981 > Project: Flink > Issue Type: New Feature > Components: Connectors / Hive >Reporter: Rui Li >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.15.0, 1.16.0 > > > Start working on this once FLINK-18958 is done -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-18564) Add Support for Azure Event Hub DataStream Connector
[ https://issues.apache.org/jira/browse/FLINK-18564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-18564: Fix Version/s: 1.16.0 > Add Support for Azure Event Hub DataStream Connector > > > Key: FLINK-18564 > URL: https://issues.apache.org/jira/browse/FLINK-18564 > Project: Flink > Issue Type: Improvement > Components: API / DataStream, Connectors / Common >Affects Versions: 1.12.0 >Reporter: Israel Ekpo >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.15.0, 1.16.0 > > > The objective of this improvement is to add Azure Event Hubs [2] as an input > source and output sink for the DataStream connectors [1] > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/connectors/#datastream-connectors > [2] https://docs.microsoft.com/en-us/azure/event-hubs/ -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25345) FLIP-196: Source API stability guarantees
[ https://issues.apache.org/jira/browse/FLINK-25345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-25345: Fix Version/s: 1.16.0 > FLIP-196: Source API stability guarantees > - > > Key: FLINK-25345 > URL: https://issues.apache.org/jira/browse/FLINK-25345 > Project: Flink > Issue Type: Improvement > Components: API / DataStream, Documentation >Affects Versions: 1.15.0 >Reporter: Till Rohrmann >Priority: Major > Fix For: 1.15.0, 1.16.0 > > > This ticket is an umbrella ticket for the work on > [FLIP-196|https://cwiki.apache.org/confluence/x/IJeqCw] which will properly > document and guard the agreed upon source API stability guarantees. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-15740) Remove Deadline#timeLeft()
[
https://issues.apache.org/jira/browse/FLINK-15740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-15740:
Fix Version/s: 1.16.0
> Remove Deadline#timeLeft()
> --
>
> Key: FLINK-15740
> URL: https://issues.apache.org/jira/browse/FLINK-15740
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream, Tests
>Reporter: Chesnay Schepler
>Priority: Minor
> Labels: auto-deprioritized-major, pull-request-available
> Fix For: 1.15.0, 1.16.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> As shown in FLINK-13662, {{Deadline#timeLeft()}} is conceptually broken since
> there is no reliable way to call said method while ensuring that
> a) the value is non-negative (desired since most time-based APIs reject
> negative values)
> b) the value sign (+,-) corresponds to preceding calls to {{#hasTimeLeft()}}
>
> As a result any usage of the following form is unreliable and obfuscating
> error messages.
> {code:java}
> while (deadline.hasTimeLeft()) {
> doSomething(deadline.timeLeft());
> } {code}
>
> All existing usage should be migrate to either
> {code:java}
> while (deadline.hasTimeLeft()) {
> doSomething();
> } {code}
> or
> {code:java}
> while (true) {
> doSomething(deadline.timeLeftIfAny());
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25250) Consider Java 11 as default version on CI
[ https://issues.apache.org/jira/browse/FLINK-25250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-25250: Fix Version/s: 1.16.0 > Consider Java 11 as default version on CI > - > > Key: FLINK-25250 > URL: https://issues.apache.org/jira/browse/FLINK-25250 > Project: Flink > Issue Type: Sub-task > Components: Build System / CI >Reporter: Chesnay Schepler >Priority: Major > Fix For: 1.15.0, 1.16.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-27215) JDBC sink transiently deleted a record because of -u message of that record
tim yu created FLINK-27215:
--
Summary: JDBC sink transiently deleted a record because of -u
message of that record
Key: FLINK-27215
URL: https://issues.apache.org/jira/browse/FLINK-27215
Project: Flink
Issue Type: Improvement
Components: Connectors / JDBC
Affects Versions: 1.12.0
Reporter: tim yu
A record is deleted transiently when using JDBC sink in upsert mode.
The -U message is processed as delete operation in class
TableBufferReducedStatementExecutor.
The following codes show how to process -U message:
{code:java}
/**
* Returns true if the row kind is INSERT or UPDATE_AFTER, returns false if
the row kind is
* DELETE or UPDATE_BEFORE.
*/
private boolean changeFlag(RowKind rowKind) {
switch (rowKind) {
case INSERT:
case UPDATE_AFTER:
return true;
case DELETE:
case UPDATE_BEFORE:
return false;
default:
throw new UnsupportedOperationException(
String.format(
"Unknown row kind, the supported row kinds is:
INSERT, UPDATE_BEFORE, UPDATE_AFTER,"
+ " DELETE, but get: %s.",
rowKind));
}
}
@Override
public void executeBatch() throws SQLException {
for (Map.Entry> entry :
reduceBuffer.entrySet()) {
if (entry.getValue().f0) {
upsertExecutor.addToBatch(entry.getValue().f1);
} else {
// delete by key
deleteExecutor.addToBatch(entry.getKey());
}
}
upsertExecutor.executeBatch();
deleteExecutor.executeBatch();
reduceBuffer.clear();
}
{code}
If -U and +U messages of one record are executed separately in different JDBC
batches, that record will be deleted transiently in external database and then
insert a new updated record to it. In fact, this record should be merely
updated once in the external database.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-22371) Chinese grammatical error
[
https://issues.apache.org/jira/browse/FLINK-22371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-22371:
Fix Version/s: 1.16.0
> Chinese grammatical error
> -
>
> Key: FLINK-22371
> URL: https://issues.apache.org/jira/browse/FLINK-22371
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Affects Versions: 1.12.2
>Reporter: simenliuxing
>Priority: Minor
> Labels: auto-deprioritized-major
> Fix For: 1.15.0, 1.16.0
>
>
> Chinese grammatical error in
> [https://ci.apache.org/projects/flink/flink-docs-release-1.12/zh/dev/table/common.html]
> . text content "下文讨论的 {{DataSet}} API 只与旧计划起有关。" Should be replaced with
> "下文讨论的 {{DataSet}} API 只与旧计划有关。"
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-21637) Wrong results when filtering by timestamp type
[
https://issues.apache.org/jira/browse/FLINK-21637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-21637:
Fix Version/s: 1.16.0
> Wrong results when filtering by timestamp type
> --
>
> Key: FLINK-21637
> URL: https://issues.apache.org/jira/browse/FLINK-21637
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Reporter: Rui Li
>Priority: Major
> Fix For: 1.15.0, 1.16.0
>
>
> The following test can reproduce the issue:
> {code}
> @Test
> def test(): Unit = {
> TestCollectionTableFactory.reset()
> TestCollectionTableFactory.initData(
> Collections.singletonList(Row.of("a",
> LocalDateTime.parse("2021-03-05T20:35:01.1"
> tEnv.executeSql("create table src(s string,ts timestamp) with('connector'
> = 'COLLECTION')")
> val results = CollectionUtil.iteratorToList(
> tEnv.executeSql("select s from src where timestamp '2021-03-05
> 20:35:01' = ts").collect())
> assertEquals(0, results.size())
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25027) Allow GC of a finished job's JobMaster before the slot timeout is reached
[
https://issues.apache.org/jira/browse/FLINK-25027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-25027:
Fix Version/s: 1.16.0
> Allow GC of a finished job's JobMaster before the slot timeout is reached
> -
>
> Key: FLINK-25027
> URL: https://issues.apache.org/jira/browse/FLINK-25027
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.14.0, 1.12.5, 1.13.3
>Reporter: Nico Kruber
>Assignee: Shammon
>Priority: Major
> Fix For: 1.15.0, 1.16.0
>
> Attachments: image-2021-11-23-20-32-20-479.png
>
>
> In a session cluster, after a (batch) job is finished, the JobMaster seems to
> stick around for another couple of minutes before being eligible for garbage
> collection.
> Looking into a heap dump, it seems to be tied to a
> {{PhysicalSlotRequestBulkCheckerImpl}} which is enqueued in the underlying
> Akka executor (and keeps the JM from being GC’d). Per default the action is
> scheduled for {{slot.request.timeout}} that defaults to 5 min (thanks
> [~trohrmann] for helping out here)
> !image-2021-11-23-20-32-20-479.png!
> With this setting, you will have to account for enough metaspace to cover 5
> minutes of time which may span a couple of jobs, needlessly!
> The problem seems to be that Flink is using the main thread executor for the
> scheduling that uses the {{ActorSystem}}'s scheduler and the future task
> scheduled with Akka can (probably) not be easily cancelled.
> One idea could be to use a dedicated thread pool per JM, that we shut down
> when the JM terminates. That way we would not keep the JM from being GC’d.
> (The concrete example we investigated was a DataSet job)
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-16078) Translate "Tuning Checkpoints and Large State" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-16078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-16078: Fix Version/s: 1.16.0 > Translate "Tuning Checkpoints and Large State" page into Chinese > > > Key: FLINK-16078 > URL: https://issues.apache.org/jira/browse/FLINK-16078 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation >Affects Versions: 1.11.0 >Reporter: Yu Li >Assignee: Zhuang Liu >Priority: Major > Labels: auto-unassigned, pull-request-available > Fix For: 1.15.0, 1.16.0 > > > Complete the translation in `docs/ops/state/large_state_tuning.zh.md` -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-20246) Add documentation on Python worker memory tuning in the memory tuning page
[ https://issues.apache.org/jira/browse/FLINK-20246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-20246: Fix Version/s: 1.16.0 > Add documentation on Python worker memory tuning in the memory tuning page > -- > > Key: FLINK-20246 > URL: https://issues.apache.org/jira/browse/FLINK-20246 > Project: Flink > Issue Type: Improvement > Components: API / Python, Documentation >Reporter: Dian Fu >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned > Fix For: 1.15.0, 1.16.0 > > > Per the discussion in FLINK-19177, we need to add some documentation on > Python worker memory tuning in the memory tuning page. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-18563) Add Support for Azure Cosmos DB DataStream Connector
[ https://issues.apache.org/jira/browse/FLINK-18563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-18563: Fix Version/s: 1.16.0 > Add Support for Azure Cosmos DB DataStream Connector > > > Key: FLINK-18563 > URL: https://issues.apache.org/jira/browse/FLINK-18563 > Project: Flink > Issue Type: Improvement > Components: API / DataStream, Connectors / Common >Affects Versions: 1.12.0 >Reporter: Israel Ekpo >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.15.0, 1.16.0 > > > Add Support for Azure Cosmos DB DataStream Connector > The objective of this improvement is to add Azure Cosmos DB [2] as an input > source and output sink for the DataStream connectors [1] > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/connectors/#datastream-connectors > [2] https://docs.microsoft.com/en-us/azure/cosmos-db/change-feed -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-21636) Setup a global timeout for all tests
[
https://issues.apache.org/jira/browse/FLINK-21636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-21636:
Fix Version/s: 1.16.0
> Setup a global timeout for all tests
>
>
> Key: FLINK-21636
> URL: https://issues.apache.org/jira/browse/FLINK-21636
> Project: Flink
> Issue Type: Improvement
> Components: Test Infrastructure
>Reporter: Chesnay Schepler
>Priority: Minor
> Labels: auto-deprioritized-major
> Fix For: 1.15.0, 1.16.0
>
>
> We should setup a timeout that is applied automatically to all tests. This
> can be even something like 10 minutes.
> Right now, a single test not terminating eats 4 hours of CI time, and with
> Azure's inability to cancel running stages can hurt productivity quite a bit.
> Implementation wise, we can just add this to the {{TestLogger}} class.
> {code}
> @Rule
> public Timeout globalTimeout = Timeout.minutes(10);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-11868) [filesystems] Introduce listStatusIterator API to file system
[
https://issues.apache.org/jira/browse/FLINK-11868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-11868:
Fix Version/s: 1.16.0
> [filesystems] Introduce listStatusIterator API to file system
> -
>
> Key: FLINK-11868
> URL: https://issues.apache.org/jira/browse/FLINK-11868
> Project: Flink
> Issue Type: New Feature
> Components: FileSystems
>Reporter: Yun Tang
>Assignee: Yun Tang
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned, stale-assigned
> Fix For: 1.15.0, 1.16.0
>
>
> From existed experience, we know {{listStatus}} is expensive for many
> distributed file systems especially when the folder contains too many files.
> This method would not only block the thread until result is return but also
> could cause OOM due to the returned array of {{FileStatus}} is really large.
> I think we should already learn it from FLINK-7266 and FLINK-8540.
> However, list file status under a path is really helpful in many situations.
> Thankfully, many distributed file system noticed that and provide API such as
> {{[listStatusIterator|https://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#listStatusIterator(org.apache.hadoop.fs.Path)]}}
> to call the file system on demand.
>
> We should also introduce this API and replace current implementation which
> used previous {{listStatus}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot commented on pull request #19449: Improvements to YAML parsing
flinkbot commented on PR #19449: URL: https://github.com/apache/flink/pull/19449#issuecomment-1097611420 ## CI report: * 62935b28efa7152eeaa8381d5743c4a668c597dd UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17535) Treat min/max as part of the hierarchy of config option
[ https://issues.apache.org/jira/browse/FLINK-17535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-17535: Fix Version/s: 1.16.0 > Treat min/max as part of the hierarchy of config option > --- > > Key: FLINK-17535 > URL: https://issues.apache.org/jira/browse/FLINK-17535 > Project: Flink > Issue Type: Improvement > Components: Runtime / Configuration >Reporter: Yangze Guo >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.15.0, 1.16.0 > > > As discussed in > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Should-max-min-be-part-of-the-hierarchy-of-config-option-td40578.html. > We decide to treat min/max as part of the hierarchy of config option. This > ticket is an umbrella of all tasks related to it. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-16155) Translate "Operator/Process Function" into Chinese
[ https://issues.apache.org/jira/browse/FLINK-16155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-16155: Fix Version/s: 1.16.0 > Translate "Operator/Process Function" into Chinese > -- > > Key: FLINK-16155 > URL: https://issues.apache.org/jira/browse/FLINK-16155 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation >Reporter: Yun Gao >Priority: Major > Labels: auto-unassigned, pull-request-available > Fix For: 1.15.0, 1.16.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > The page is located at _"docs/dev/stream/operators/process_function.zh.md"_ -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-24582) PrometheusReporterEndToEndITCase fails due to timeout
[
https://issues.apache.org/jira/browse/FLINK-24582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-24582:
Fix Version/s: 1.16.0
> PrometheusReporterEndToEndITCase fails due to timeout
> -
>
> Key: FLINK-24582
> URL: https://issues.apache.org/jira/browse/FLINK-24582
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Metrics
>Affects Versions: 1.12.5, 1.15.0
>Reporter: Xintong Song
>Priority: Major
> Labels: test-stability
> Fix For: 1.15.0, 1.16.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=25180&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee&t=070ff179-953e-5bda-71fa-d6599415701c&l=23451
> {code}
> Oct 18 18:17:44 [ERROR] Tests run: 6, Failures: 0, Errors: 6, Skipped: 0,
> Time elapsed: 9.138 s <<< FAILURE! - in
> org.apache.flink.metrics.prometheus.tests.PrometheusReporterEndToEndITCase
> Oct 18 18:17:44 [ERROR] testReporter[0: Jar in 'lib', instantiated via
> reflection] Time elapsed: 1.779 s <<< ERROR!
> Oct 18 18:17:44 java.io.IOException: Process ([wget, -q, -P,
> /home/vsts/work/1/e2e_cache/downloads/-1930370649, --timeout, 240,
> https://github.com/prometheus/prometheus/releases/download/v2.4.3/prometheus-2.4.3.linux-amd64.tar.gz])
> exceeded timeout (60) or number of retries (3).
> Oct 18 18:17:44 at
> org.apache.flink.tests.util.AutoClosableProcess$AutoClosableProcessBuilder.runBlockingWithRetry(AutoClosableProcess.java:168)
> Oct 18 18:17:44 at
> org.apache.flink.tests.util.cache.AbstractDownloadCache.getOrDownload(AbstractDownloadCache.java:135)
> Oct 18 18:17:44 at
> org.apache.flink.tests.util.cache.PersistingDownloadCache.getOrDownload(PersistingDownloadCache.java:36)
> Oct 18 18:17:44 at
> org.apache.flink.metrics.prometheus.tests.PrometheusReporterEndToEndITCase.testReporter(PrometheusReporterEndToEndITCase.java:209)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-16701) Elasticsearch sink support alias for indices.
[ https://issues.apache.org/jira/browse/FLINK-16701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-16701: Fix Version/s: 1.16.0 > Elasticsearch sink support alias for indices. > - > > Key: FLINK-16701 > URL: https://issues.apache.org/jira/browse/FLINK-16701 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: 1.11.0 >Reporter: Leonard Xu >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.15.0, 1.16.0 > > > This is related to > [FLINK-15400|https://issues.apache.org/jira/browse/FLINK-15400] FLINK-15400 > will only support dynamic index, and do not support the alias. Because > supporting alias both need in Streaming API and Table API, so I think split > the original design to two PRs make sense. > PR for FLINK-15400: > support dynamic index for ElasticsearchTableSink > PR for this issue: > support alias for Streaming API and Table API -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-22228) Random log exception when tasks finish after a checkpoint was started
[
https://issues.apache.org/jira/browse/FLINK-8?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-8:
Fix Version/s: 1.16.0
> Random log exception when tasks finish after a checkpoint was started
> -
>
> Key: FLINK-8
> URL: https://issues.apache.org/jira/browse/FLINK-8
> Project: Flink
> Issue Type: Sub-task
>Reporter: Chesnay Schepler
>Priority: Major
> Fix For: 1.15.0, 1.16.0
>
>
> ??StephanEwen: Random log exception when tasks finish after a checkpoint was
> started.??
>
> {code:java}
> 9407 [Checkpoint Timer] INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
> checkpoint 1 (type=CHECKPOINT) @ 1617983491976 for job
> d8225dac771bf607cf8dd869964c6265.
> 9501 [Source: numbers -> Map -> Sink: Data stream collect sink (1/1)#0] INFO
> org.apache.flink.runtime.taskmanager.Task [] - Source: numbers -> Map ->
> Sink: Data stream collect sink (1/1)#0 (3f61526eef13676a7d96010799c04e1c)
> switched from RUNNING to FINISHED.
> 9501 [Source: numbers -> Map -> Sink: Data stream collect sink (1/1)#0] INFO
> org.apache.flink.runtime.taskmanager.Task [] - Freeing task resources for
> Source: numbers -> Map -> Sink: Data stream collect sink (1/1)#0
> (3f61526eef13676a7d96010799c04e1c).
> 9501 [flink-akka.actor.default-dispatcher-2] INFO
> org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Un-registering task
> and sending final execution state FINISHED to JobManager for task Source:
> numbers -> Map -> Sink: Data stream collect sink (1/1)#0
> 3f61526eef13676a7d96010799c04e1c.
> 9501 [AsyncOperations-thread-1] INFO
> org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable [] - Source:
> numbers -> Map -> Sink: Data stream collect sink (1/1)#0 - asynchronous part
> of checkpoint 1 could not be completed.
> java.util.concurrent.CancellationException: null
> at java.util.concurrent.FutureTask.report(FutureTask.java:121) ~[?:?]
> at java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:?]
> at
> org.apache.flink.runtime.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:636)
> ~[classes/:?]
> at
> org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.(OperatorSnapshotFinalizer.java:60)
> ~[classes/:?]
> at
> org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:128)
> [classes/:?]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> [?:?]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> [?:?]
> at java.lang.Thread.run(Thread.java:834) [?:?]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-24680) Expose UserCodeClassLoader in OperatorCoordinator.Context for registering release hooks
[ https://issues.apache.org/jira/browse/FLINK-24680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-24680: Fix Version/s: 1.16.0 > Expose UserCodeClassLoader in OperatorCoordinator.Context for registering > release hooks > --- > > Key: FLINK-24680 > URL: https://issues.apache.org/jira/browse/FLINK-24680 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.14.0 >Reporter: Qingsheng Ren >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0, 1.16.0 > > > Currently `OperatorCoordinator.Context` only exposes `ClassLoader` for > accessing user code class loader, which doesn't support adding release hooks > in `OperatorCoordinator`, like sync releasing class loader and closing > operator coordinator. > We need to expose `UserCodeClassLoader` in the context fo registering hooks > in coordinator. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-24735) SQL client crashes with `Cannot add expression of different type to set`
[
https://issues.apache.org/jira/browse/FLINK-24735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-24735:
Fix Version/s: 1.16.0
> SQL client crashes with `Cannot add expression of different type to set`
>
>
> Key: FLINK-24735
> URL: https://issues.apache.org/jira/browse/FLINK-24735
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.14.0
>Reporter: Martijn Visser
>Assignee: Shengkai Fang
>Priority: Major
> Fix For: 1.15.0, 1.16.0, 1.14.5
>
>
> Reproductions steps:
> 1. Download airports.csv from https://www.kaggle.com/usdot/flight-delays
> 2. Start Flink SQL client and create table
> {code:sql}
> CREATE TABLE `airports` (
> `IATA_CODE` CHAR(3),
> `AIRPORT` STRING,
> `CITY` STRING,
> `STATE` CHAR(2),
> `COUNTRY` CHAR(3),
> `LATITUDE` DOUBLE NULL,
> `LONGITUDE` DOUBLE NULL,
> PRIMARY KEY (`IATA_CODE`) NOT ENFORCED
> ) WITH (
> 'connector' = 'filesystem',
> 'path' =
> 'file:///flink-sql-cookbook/other-builtin-functions/04_override_table_options/airports.csv',
> 'format' = 'csv'
> );
> {code}
> 3. Run the following SQL statement:
> {code:sql}
> SELECT * FROM `airports` /*+ OPTIONS('csv.ignore-parse-errors'='true') */
> WHERE COALESCE(`IATA_CODE`, `AIRPORT`) IS NULL;
> {code}
> Stacktrace:
> {code:bash}
> Exception in thread "main" org.apache.flink.table.client.SqlClientException:
> Unexpected exception. This is a bug. Please consider filing an issue.
> at
> org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:201)
> at org.apache.flink.table.client.SqlClient.main(SqlClient.java:161)
> Caused by: java.lang.AssertionError: Cannot add expression of different type
> to set:
> set type is RecordType(CHAR(3) CHARACTER SET "UTF-16LE" NOT NULL IATA_CODE,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" AIRPORT, VARCHAR(2147483647)
> CHARACTER SET "UTF-16LE" CITY, CHAR(2) CHARACTER SET "UTF-16LE" STATE,
> CHAR(3) CHARACTER SET "UTF-16LE" COUNTRY, DOUBLE LATITUDE, DOUBLE LONGITUDE)
> NOT NULL
> expression type is RecordType(CHAR(3) CHARACTER SET "UTF-16LE" IATA_CODE,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" AIRPORT, VARCHAR(2147483647)
> CHARACTER SET "UTF-16LE" CITY, CHAR(2) CHARACTER SET "UTF-16LE" STATE,
> CHAR(3) CHARACTER SET "UTF-16LE" COUNTRY, DOUBLE LATITUDE, DOUBLE LONGITUDE)
> NOT NULL
> set is rel#426:LogicalProject.NONE.any.None:
> 0.[NONE].[NONE](input=HepRelVertex#425,inputs=0..6)
> expression is LogicalProject(IATA_CODE=[null:CHAR(3) CHARACTER SET
> "UTF-16LE"], AIRPORT=[$1], CITY=[$2], STATE=[$3], COUNTRY=[$4],
> LATITUDE=[$5], LONGITUDE=[$6])
> LogicalFilter(condition=[IS NULL(CAST($0):VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE")])
> LogicalTableScan(table=[[default_catalog, default_database, airports]],
> hints=[[[OPTIONS inheritPath:[] options:{csv.ignore-parse-errors=true}]]])
> at
> org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:381)
> at
> org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:58)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:268)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:283)
> at
> org.apache.calcite.rel.rules.ReduceExpressionsRule$ProjectReduceExpressionsRule.onMatch(ReduceExpressionsRule.java:310)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:333)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:542)
> at
> org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:407)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:243)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:127)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:202)
> at
> org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:189)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkHepProgram.optimize(FlinkHepProgram.scala:69)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkHepRuleSetProgram.optimize(FlinkHepRuleSetProgram.scala:87)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram.$anonfun$optimize$1(FlinkChainedProgram.scala:62)
> at
> scala.collection.TraversableOnce.$anonfun$foldLeft$1(TraversableOnce.scala:156)
> at
> scala.collection.TraversableOnce.$anonfun$foldLeft$1$adapted(TraversableOnce.scala:156)
> at scala.collection.Iterator.foreach(Iterator.scala:937)
> at scala.collection.Iterator.foreach$(Iterator.scala:937)
> at scala.collection.Abst
[jira] [Updated] (FLINK-23955) submit flink sql job error when flink HA on yarn is configured
[
https://issues.apache.org/jira/browse/FLINK-23955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-23955:
Fix Version/s: 1.16.0
> submit flink sql job error when flink HA on yarn is configured
> --
>
> Key: FLINK-23955
> URL: https://issues.apache.org/jira/browse/FLINK-23955
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.13.0
>Reporter: Jun Zhang
>Priority: Major
> Fix For: 1.15.0, 1.16.0, 1.14.5
>
>
> 1.when I configured the flink HA ,like this
> {code:java}
> high-availability: zookeeper
> high-availability.storageDir: hdfs://xxx/flink/ha/
> high-availability.zookeeper.quorum: x:2181
> high-availability.zookeeper.path.root: /flink
> {code}
> 2. I start a flink session cluster
> 3. I submit a flink sql job by flink sql client and set the
> {code:java}
> set execution.target = yarn-per-job;
> {code}
> then I get the error
>
>
> {code:java}
> 2021-08-25 10:40:39,500 INFO org.apache.flink.yarn.YarnClusterDescriptor
> [] - Found Web Interface master3:38052 of application
> 'application_1629858010528_0002'.
> 2021-08-25 10:40:42,447 WARN
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher [] - An
> exception occurred when fetching query results
> java.util.concurrent.ExecutionException:
> org.apache.flink.runtime.rest.util.RestClientException: [Internal server
> error., org.apache.flink.runtime.messages.FlinkJobNotFoundException: Could not find
> Flink job (2cc726b9ae70c95f128f6c1e55cf874c)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.getJobMasterGateway(Dispatcher.java:878)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.performOperationOnJobMasterGateway(Dispatcher.java:892)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.deliverCoordinationRequestToCoordinator(Dispatcher.java:712)
> at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:305)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:212)
> at
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
> at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
> at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
> at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
> at akka.actor.Actor.aroundReceive(Actor.scala:517)
> at akka.actor.Actor.aroundReceive$(Actor.scala:515)
> at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
> at akka.actor.ActorCell.invoke(ActorCell.scala:561)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
> at akka.dispatch.Mailbox.run(Mailbox.scala:225)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
> at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at
> akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)End
> of exception on server side>]
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> ~[?:1.8.0_291]
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> ~[?:1.8.0_291]
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.sendRequest(CollectResultFetcher.java:163)
> ~[flink-dist_2.12-1.13.0.jar:1.13.0]
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:128)
> ~[flink-dist_2.12-1.13.0.jar:1.13.0]
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:106)
> ~[flink-dis
[jira] [Updated] (FLINK-21022) flink-connector-es add onSuccess handler after bulk process for sync success data to other third party system for data consistency checking
[ https://issues.apache.org/jira/browse/FLINK-21022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-21022: Fix Version/s: 1.16.0 > flink-connector-es add onSuccess handler after bulk process for sync success > data to other third party system for data consistency checking > --- > > Key: FLINK-21022 > URL: https://issues.apache.org/jira/browse/FLINK-21022 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Reporter: Zheng WEI >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available > Fix For: 1.15.0, 1.11.7, 1.16.0 > > > flink-connector-es add onSuccess handler after successful bulk process, in > order to sync success data to other third party system for data consistency > checking. Default the implementation of onSuccess function is empty logic, > user can set its own onSuccess handler when needed. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-15471) HA e2e check for empty .out files does not print specific error
[
https://issues.apache.org/jira/browse/FLINK-15471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-15471:
Fix Version/s: 1.16.0
> HA e2e check for empty .out files does not print specific error
> ---
>
> Key: FLINK-15471
> URL: https://issues.apache.org/jira/browse/FLINK-15471
> Project: Flink
> Issue Type: Improvement
> Components: Test Infrastructure
>Affects Versions: 1.9.0
>Reporter: Chesnay Schepler
>Priority: Minor
> Labels: auto-deprioritized-major
> Fix For: 1.10.4, 1.15.0, 1.11.7, 1.16.0
>
>
> {{common_ha.sh#verify_logs:}}
> {code}
> if ! check_logs_for_non_empty_out_files; then
> echo "FAILURE: Alerts found at the general purpose job."
> EXIT_CODE=1
> fi
> {code}
> Since check_logs_for_non_empty_out_files does only set EXIT_CODE without
> modifying the return value the check will never fail.
> While the test will still fail (since EXIT_CODE is later being evaluated), we
> may not actually print the error cause.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25252) Check whether Kafka E2E tests could be run on Java 11
[ https://issues.apache.org/jira/browse/FLINK-25252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-25252: Fix Version/s: 1.16.0 > Check whether Kafka E2E tests could be run on Java 11 > - > > Key: FLINK-25252 > URL: https://issues.apache.org/jira/browse/FLINK-25252 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Kafka, Tests >Reporter: Chesnay Schepler >Priority: Major > Fix For: 1.15.0, 1.16.0 > > > The Java Kafka E2E tests are currently not run on Java 11. We should check > what the actual issue is and whether it can be resolved (e.g., by a Kafka > server version bump): -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-23679) PyFlink end-to-end test fail due to test script contains errors
[ https://issues.apache.org/jira/browse/FLINK-23679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-23679: Fix Version/s: 1.16.0 > PyFlink end-to-end test fail due to test script contains errors > --- > > Key: FLINK-23679 > URL: https://issues.apache.org/jira/browse/FLINK-23679 > Project: Flink > Issue Type: Bug > Components: API / Python >Affects Versions: 1.14.0 >Reporter: Xintong Song >Priority: Major > Labels: test-stability > Fix For: 1.15.0, 1.16.0 > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=21728&view=logs&j=4dd4dbdd-1802-5eb7-a518-6acd9d24d0fc&t=7c4a8fb8--5a77-f518-4176bfae300b&l=10433 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-17683) Pojo with no-args constructor will be treated as ROW when returned by ScalarFunction， but not when it is a field of case class.
[
https://issues.apache.org/jira/browse/FLINK-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-17683:
Fix Version/s: 1.16.0
> Pojo with no-args constructor will be treated as ROW when returned by
> ScalarFunction， but not when it is a field of case class.
> ---
>
> Key: FLINK-17683
> URL: https://issues.apache.org/jira/browse/FLINK-17683
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.10.1
>Reporter: xiemeilong
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned
> Fix For: 1.15.0, 1.16.0
>
>
> Pojo.java
> {code:java}
> @Data
> @NoArgsConstructor
> public class Pojo {
> private String foo;
> }
> {code}
> {code:java}
> object Fun extends ScalarFunction {
>def eval(): Pojo = { new Pojo() }
> }
> case class CaseClass(pojo:Pojo)
> {code}
>
>
> {code:java}
> tableEnv.registerFunction("fun", Fun)
> val table = tableEnv.sqlQuery("select fun() as pojo")
> table.toRetractStream[CaseClass].print()
> {code}
>
> exception:
>
> {code:java}
> Exception in thread "main" org.apache.flink.table.api.ValidationException:
> Field types of query result and registered TableSink do not match.
> Query schema: [pojo: ROW<`foo` STRING>]
> Sink schema: [pojo: LEGACY('STRUCTURED_TYPE', 'POJO')]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-21765) Remove implementation-specific MetricGroup parents
[ https://issues.apache.org/jira/browse/FLINK-21765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-21765: Fix Version/s: 1.16.0 > Remove implementation-specific MetricGroup parents > -- > > Key: FLINK-21765 > URL: https://issues.apache.org/jira/browse/FLINK-21765 > Project: Flink > Issue Type: Technical Debt > Components: Runtime / Metrics >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned, > stale-assigned, starter > Fix For: 1.15.0, 1.16.0 > > > MetricGroups currently form a bi-directly graph, usually with explicit > requirements that type the parent must have. For example, an OperatorMG has a > hard requirement that the parent is a TaskMG. > As a result they are quite inflexible, which particular shows in tests, as > you can't just create one metric group, but have to build an entire tree. > The end goal of this ticket is to remove AbstractMetricGroup#parent, and > along the way we'll decouple the various MG implementations from each other. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-16531) Add full integration tests for "GROUPING SETS" for streaming mode
[
https://issues.apache.org/jira/browse/FLINK-16531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-16531:
Fix Version/s: 1.16.0
> Add full integration tests for "GROUPING SETS" for streaming mode
> -
>
> Key: FLINK-16531
> URL: https://issues.apache.org/jira/browse/FLINK-16531
> Project: Flink
> Issue Type: Technical Debt
> Components: Table SQL / Planner
>Reporter: Jark Wu
>Priority: Minor
> Labels: auto-deprioritized-major
> Fix For: 1.15.0, 1.16.0
>
>
> We have a plan test for GROUPING SETS for streaming mode, i.e.
> {{GroupingSetsTest}}. But we should also have a full IT coverage for it, just
> like batch's
> {{org.apache.flink.table.planner.runtime.batch.sql.agg.GroupingSetsITCase}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-22842) Streaming File Sink end-to-end test fail due to not finish after 900s
[ https://issues.apache.org/jira/browse/FLINK-22842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-22842: Fix Version/s: 1.16.0 > Streaming File Sink end-to-end test fail due to not finish after 900s > - > > Key: FLINK-22842 > URL: https://issues.apache.org/jira/browse/FLINK-22842 > Project: Flink > Issue Type: Bug > Components: Connectors / FileSystem >Affects Versions: 1.14.0 >Reporter: Xintong Song >Priority: Minor > Labels: auto-deprioritized-major, test-stability > Fix For: 1.15.0, 1.16.0 > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18525&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee&t=ff888d9b-cd34-53cc-d90f-3e446d355529&l=13152 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-23902) Support Hive version 2.3.8
[ https://issues.apache.org/jira/browse/FLINK-23902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-23902: Fix Version/s: 1.16.0 > Support Hive version 2.3.8 > -- > > Key: FLINK-23902 > URL: https://issues.apache.org/jira/browse/FLINK-23902 > Project: Flink > Issue Type: Technical Debt > Components: Connectors / Hive >Affects Versions: 1.14.0 >Reporter: Jeff Yang >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0, 1.16.0 > > > Make flink support Hive version 2.3.8 version. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-24386) JobMaster should guard against exceptions from OperatorCoordinator
[
https://issues.apache.org/jira/browse/FLINK-24386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-24386:
Fix Version/s: 1.16.0
> JobMaster should guard against exceptions from OperatorCoordinator
> --
>
> Key: FLINK-24386
> URL: https://issues.apache.org/jira/browse/FLINK-24386
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.14.0, 1.13.2
>Reporter: David Morávek
>Priority: Major
> Fix For: 1.15.0, 1.16.0
>
>
> Original report from [~sewen]:
> When the scheduler processes the call to trigger a _globalFailover_
> and something goes wrong in there, the _JobManager_ gets stuck. Concretely,
> I have an _OperatorCoordinator_ that throws an exception in
> _subtaskFailed()_, which gets called as part of processing the failover.
> While this is a bug in that coordinator, the whole thing seems a bit
> dangerous to me. If there is some bug in any part of the failover logic, we
> have no safety net. No "hard crash" and let the process be restarted. We only
> see a log line (below) and everything goes unresponsive.
> {code:java}
> ERROR org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor [] - Caught
> exception while executing runnable in main thread.
> {code}
> Shouldn't we have some safety nets in place here?
> * I am wondering if the place where that line is logged should actually
> invoke the fatal error handler. If an exception propagates out of a main
> thread action, we need to call off all bets and assume things have gotten
> inconsistent.
> * At the very least, the failover procedure itself should be guarded. If an
> error happens while processing the global failover, then we need to treat
> this as beyond redemption and declare a fatal error.
> The fatal error would give us a log line and the user a container restart,
> hopefully fixing things (unless it was a deterministic error).
> [~dmvk] notes:
> * OperatorCoordinator is part of a public API interface (part of JobGraph).
> ** Can be provided by implementing CoordinatedOperatorFactory
> ** This actually gives the issue higher priority than I initially thought.
> * We should guard against flaws in user code:
> ** There are two types of interfaces
> *** (CRITICAL) Public API for JobGraph construction / submission
> *** Semi-public interfaces such as custom HA Services, this is for power
> users, so I wouldn't be as concerned there.
> ** We already do good job guarding against failure on TM side
> ** Considering the critical parts on JM side, there two places where user
> can "hook"
> *** OperatorCoordinator
> *** InitializeOnMaster, FinalizeOnMaster (batch sinks only, legacy from the
> Hadoop world)
> ---
> We should audit all the calls to OperatorCoordinator and handle failures
> accordingly. We want to avoid unnecessary JVM terminations as much as
> possible (sometimes it's the only option though).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-19085) Remove deprecated methods for writing CSV and Text files from DataStream
[
https://issues.apache.org/jira/browse/FLINK-19085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-19085:
Fix Version/s: 1.16.0
> Remove deprecated methods for writing CSV and Text files from DataStream
>
>
> Key: FLINK-19085
> URL: https://issues.apache.org/jira/browse/FLINK-19085
> Project: Flink
> Issue Type: Sub-task
> Components: API / DataStream
>Reporter: Dawid Wysakowicz
>Priority: Major
> Labels: auto-unassigned
> Fix For: 1.15.0, 1.16.0
>
>
> We can remove long deprecated {{PublicEvolving}} methods:
> - DataStream#writeAsText
> - DataStream#writeAsCsv
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25290) Add table source and sink test suite in connector testing framework
[ https://issues.apache.org/jira/browse/FLINK-25290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-25290: Fix Version/s: 1.16.0 > Add table source and sink test suite in connector testing framework > --- > > Key: FLINK-25290 > URL: https://issues.apache.org/jira/browse/FLINK-25290 > Project: Flink > Issue Type: Improvement >Reporter: Qingsheng Ren >Assignee: Hang Ruan >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.15.0, 1.16.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-22418) The currently activated tab in the same page isn't consistent
[ https://issues.apache.org/jira/browse/FLINK-22418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-22418: Fix Version/s: 1.16.0 > The currently activated tab in the same page isn't consistent > - > > Key: FLINK-22418 > URL: https://issues.apache.org/jira/browse/FLINK-22418 > Project: Flink > Issue Type: Improvement > Components: Documentation >Affects Versions: 1.13.0 >Reporter: Dian Fu >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.15.0, 1.16.0 > > Attachments: image-2021-04-23-10-41-06-577.png > > > Currently, the activated tab isn't always the same in the same page after > click "Java/Scala/Python" a couple of times: > !image-2021-04-23-10-41-06-577.png! -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25469) FlinkKafkaProducerITCase.testScaleUpAfterScalingDown fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-25469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-25469:
Fix Version/s: 1.16.0
> FlinkKafkaProducerITCase.testScaleUpAfterScalingDown fails on AZP
> -
>
> Key: FLINK-25469
> URL: https://issues.apache.org/jira/browse/FLINK-25469
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.15.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0, 1.16.0
>
>
> The test {{FlinkKafkaProducerITCase.testScaleUpAfterScalingDown}} fails with
> {code}
> Dec 28 15:21:42 [ERROR]
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase.testScaleUpAfterScalingDown
> Time elapsed: 138.55 s <<< FAILURE!
> Dec 28 15:21:42 java.lang.AssertionError: Expected elements: <[0, 1, 2, 3, 4,
> 5, 6, 7, 8]>, but was: elements: <[0, 1, 2, 3, 4, 5, 6, 0, 1, 2, 3, 4, 5, 6,
> 7, 8, 0, 1, 2, 3, 4, 5, 6, 7, 8, 0, 1, 2, 3, 4, 5, 6, 7, 8]>
> Dec 28 15:21:42 at org.junit.Assert.fail(Assert.java:89)
> Dec 28 15:21:42 at
> org.apache.flink.streaming.connectors.kafka.KafkaTestBase.assertExactlyOnceForTopic(KafkaTestBase.java:331)
> Dec 28 15:21:42 at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase.testScaleUpAfterScalingDown(FlinkKafkaProducerITCase.java:460)
> Dec 28 15:21:42 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Dec 28 15:21:42 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Dec 28 15:21:42 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Dec 28 15:21:42 at java.lang.reflect.Method.invoke(Method.java:498)
> Dec 28 15:21:42 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Dec 28 15:21:42 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Dec 28 15:21:42 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> Dec 28 15:21:42 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Dec 28 15:21:42 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> Dec 28 15:21:42 at
> org.apache.flink.testutils.junit.RetryRule$RetryOnFailureStatement.evaluate(RetryRule.java:135)
> Dec 28 15:21:42 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> Dec 28 15:21:42 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> Dec 28 15:21:42 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Dec 28 15:21:42 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> Dec 28 15:21:42 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> Dec 28 15:21:42 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> Dec 28 15:21:42 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> Dec 28 15:21:42 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> Dec 28 15:21:42 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> Dec 28 15:21:42 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> Dec 28 15:21:42 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> Dec 28 15:21:42 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> Dec 28 15:21:42 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> Dec 28 15:21:42 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> Dec 28 15:21:42 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> Dec 28 15:21:42 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> Dec 28 15:21:42 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Dec 28 15:21:42 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> Dec 28 15:21:42 at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> Dec 28 15:21:42 at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> Dec 28 15:21:42 at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
> Dec 28 15:21:42 at
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
> Dec 28 15:21:42 at
> org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72)
> Dec 28 15:21:42 at
> org.junit.platform.launcher.core.EngineExecuti
[jira] [Updated] (FLINK-21634) ALTER TABLE statement enhancement
[
https://issues.apache.org/jira/browse/FLINK-21634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-21634:
Fix Version/s: 1.16.0
(was: 1.15.0)
> ALTER TABLE statement enhancement
> -
>
> Key: FLINK-21634
> URL: https://issues.apache.org/jira/browse/FLINK-21634
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API, Table SQL / Client
>Reporter: Jark Wu
>Assignee: Jark Wu
>Priority: Major
> Labels: auto-unassigned, stale-assigned
> Fix For: 1.16.0
>
>
> We already introduced ALTER TABLE statement in FLIP-69 [1], but only support
> to rename table name and change table options. One useful feature of ALTER
> TABLE statement is modifying schema. This is also heavily required by
> integration with data lakes (e.g. iceberg).
> Therefore, I propose to support following ALTER TABLE statements (except
> {{SET}} and {{{}RENAME TO{}}}, others are all new introduced syntax):
> {code:sql}
> ALTER TABLE table_name {
> ADD { | ( [, ...]) }
> | MODIFY { | ( [, ...]) }
> | DROP {column_name | (column_name, column_name, ) | PRIMARY KEY |
> CONSTRAINT constraint_name | WATERMARK}
> | RENAME old_column_name TO new_column_name
> | RENAME TO new_table_name
> | SET (key1=val1, ...)
> | RESET (key1, ...)
> }
> ::
> { | | }
> ::
> column_name [FIRST | AFTER column_name]
> ::
> [CONSTRAINT constraint_name] PRIMARY KEY (column_name, ...) NOT ENFORCED
> ::
> WATERMARK FOR rowtime_column_name AS watermark_strategy_expression
> ::
> { | |
> } [COMMENT column_comment]
> ::
> column_type
> ::
> column_type METADATA [ FROM metadata_key ] [ VIRTUAL ]
> ::
> AS computed_column_expression
> {code}
> And some examples:
> {code:sql}
> -- add a new column
> ALTER TABLE mytable ADD new_column STRING COMMENT 'new_column docs';
> -- add columns, constraint, and watermark
> ALTER TABLE mytable ADD (
> log_ts STRING COMMENT 'log timestamp string' FIRST,
> ts AS TO_TIMESTAMP(log_ts) AFTER log_ts,
> PRIMARY KEY (id) NOT ENFORCED,
> WATERMARK FOR ts AS ts - INTERVAL '3' SECOND
> );
> -- modify a column type
> ALTER TABLE prod.db.sample MODIFY measurement double COMMENT 'unit is bytes
> per second' AFTER `id`;
> -- modify definition of column log_ts and ts, primary key, watermark. they
> must exist in table schema
> ALTER TABLE mytable ADD (
> log_ts STRING COMMENT 'log timestamp string' AFTER `id`, -- reoder
> columns
> ts AS TO_TIMESTAMP(log_ts) AFTER log_ts,
> PRIMARY KEY (id) NOT ENFORCED,
> WATERMARK FOR ts AS ts - INTERVAL '3' SECOND
> );
> -- drop an old column
> ALTER TABLE prod.db.sample DROP measurement;
> -- drop columns
> ALTER TABLE prod.db.sample DROP (col1, col2, col3);
> -- drop a watermark
> ALTER TABLE prod.db.sample DROP WATERMARK;
> -- rename column name
> ALTER TABLE prod.db.sample RENAME `data` TO payload;
> -- rename table name
> ALTER TABLE mytable RENAME TO mytable2;
> -- set options
> ALTER TABLE kafka_table SET (
> 'scan.startup.mode' = 'specific-offsets',
> 'scan.startup.specific-offsets' = 'partition:0,offset:42'
> );
> -- reset options
> ALTER TABLE kafka_table RESET ('scan.startup.mode',
> 'scan.startup.specific-offsets');
> {code}
> Note: we don't need to introduce new interfaces, because all the alter table
> operation will be forward to catalog through
> {{{}Catalog#alterTable(tablePath, newTable, ignoreIfNotExists){}}}.
> [1]:
> [https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/table/sql/alter/#alter-table]
> [2]: [http://iceberg.apache.org/spark-ddl/#alter-table-alter-column]
> [3]: [https://trino.io/docs/current/sql/alter-table.html]
> [4]: [https://dev.mysql.com/doc/refman/8.0/en/alter-table.html]
> [5]: [https://www.postgresql.org/docs/9.1/sql-altertable.html]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-24242) KafkaSinkITCase.testAbortTransactionsOfPendingCheckpointsAfterFailure fails due to NoSuchFileException
[
https://issues.apache.org/jira/browse/FLINK-24242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-24242:
Fix Version/s: 1.16.0
> KafkaSinkITCase.testAbortTransactionsOfPendingCheckpointsAfterFailure fails
> due to NoSuchFileException
> --
>
> Key: FLINK-24242
> URL: https://issues.apache.org/jira/browse/FLINK-24242
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.15.0
>Reporter: Xintong Song
>Priority: Major
> Labels: test-stability
> Fix For: 1.15.0, 1.16.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=23876&view=logs&j=72d4811f-9f0d-5fd0-014a-0bc26b72b642&t=e424005a-b16e-540f-196d-da062cc19bdf&l=7335
> {code}
> Sep 10 00:06:17 [ERROR] Tests run: 8, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 79.401 s <<< FAILURE! - in
> org.apache.flink.connector.kafka.sink.KafkaSinkITCase
> Sep 10 00:06:17 [ERROR] testAbortTransactionsOfPendingCheckpointsAfterFailure
> Time elapsed: 2.358 s <<< ERROR!
> Sep 10 00:06:17 java.io.UncheckedIOException:
> java.nio.file.NoSuchFileException:
> /tmp/junit7927515964136749979/junit1325061264264479088/9554f057606834081bc37f12e8f298f5/chk-2
> Sep 10 00:06:17 at
> java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:88)
> Sep 10 00:06:17 at
> java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:104)
> Sep 10 00:06:17 at
> java.util.Iterator.forEachRemaining(Iterator.java:115)
> Sep 10 00:06:17 at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> Sep 10 00:06:17 at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> Sep 10 00:06:17 at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> Sep 10 00:06:17 at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> Sep 10 00:06:17 at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> Sep 10 00:06:17 at
> java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:546)
> Sep 10 00:06:17 at
> java.util.stream.ReferencePipeline.max(ReferencePipeline.java:582)
> Sep 10 00:06:17 at
> org.apache.flink.test.util.TestUtils.getMostRecentCompletedCheckpoint(TestUtils.java:99)
> Sep 10 00:06:17 at
> org.apache.flink.connector.kafka.sink.KafkaSinkITCase.testAbortTransactionsOfPendingCheckpointsAfterFailure(KafkaSinkITCase.java:247)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-19659) Array type supports equals and not_equals operator when element types are different but castable
[
https://issues.apache.org/jira/browse/FLINK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-19659:
Fix Version/s: 1.16.0
> Array type supports equals and not_equals operator when element types are
> different but castable
>
>
> Key: FLINK-19659
> URL: https://issues.apache.org/jira/browse/FLINK-19659
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API
>Affects Versions: 1.11.0
>Reporter: hailong wang
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned,
> pull-request-available
> Fix For: 1.15.0, 1.16.0
>
>
> Currently, Array type supports `equals` and `not_equals` when element types
> are the same or can not be cased. For example,
> {code:java}
> Array[1] <> Array[1] -> false{code}
> {code:java}
> Array[1] <> Array[cast(1 as bigint)] -> false
> {code}
> But for the element types which are castable, it will throw error,
> {code:java}
> org.apache.flink.table.planner.codegen.CodeGenException: Unsupported cast
> from 'ARRAY NOT NULL' to 'ARRAY NOT
> NULL'.org.apache.flink.table.planner.codegen.CodeGenException: Unsupported
> cast from 'ARRAY NOT NULL' to 'ARRAY NOT
> NULL'. at
> org.apache.flink.table.planner.codegen.calls.ScalarOperatorGens$.generateCast(ScalarOperatorGens.scala:1295)
> at
> org.apache.flink.table.planner.codegen.ExprCodeGenerator.generateCallExpression(ExprCodeGenerator.scala:703)
> at
> org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:498)
> at
> org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:55)
> at org.apache.calcite.rex.RexCall.accept(RexCall.java:288){code}
> But the result should be false or true, for example,
> {code:java}
> /Array[1] <> Array[cast(1 as bigint)] -> false
> {code}
>
> BTW, Map and MultiSet type are same as this, If it did, I am pleasure to open
> other issues to track those.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-22765) ExceptionUtilsITCase.testIsMetaspaceOutOfMemoryError is unstable
[
https://issues.apache.org/jira/browse/FLINK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-22765:
Fix Version/s: 1.16.0
> ExceptionUtilsITCase.testIsMetaspaceOutOfMemoryError is unstable
>
>
> Key: FLINK-22765
> URL: https://issues.apache.org/jira/browse/FLINK-22765
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.14.0, 1.15.0, 1.13.5
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>Priority: Major
> Labels: pull-request-available, stale-assigned, test-stability
> Fix For: 1.14.0, 1.15.0, 1.16.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18292&view=logs&j=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3&t=a99e99c7-21cd-5a1f-7274-585e62b72f56
> {code}
> May 25 00:56:38 java.lang.AssertionError:
> May 25 00:56:38
> May 25 00:56:38 Expected: is ""
> May 25 00:56:38 but: was "The system is out of resources.\nConsult the
> following stack trace for details."
> May 25 00:56:38 at
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> May 25 00:56:38 at org.junit.Assert.assertThat(Assert.java:956)
> May 25 00:56:38 at org.junit.Assert.assertThat(Assert.java:923)
> May 25 00:56:38 at
> org.apache.flink.runtime.util.ExceptionUtilsITCase.run(ExceptionUtilsITCase.java:94)
> May 25 00:56:38 at
> org.apache.flink.runtime.util.ExceptionUtilsITCase.testIsMetaspaceOutOfMemoryError(ExceptionUtilsITCase.java:70)
> May 25 00:56:38 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> May 25 00:56:38 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> May 25 00:56:38 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> May 25 00:56:38 at java.lang.reflect.Method.invoke(Method.java:498)
> May 25 00:56:38 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> May 25 00:56:38 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> May 25 00:56:38 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> May 25 00:56:38 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> May 25 00:56:38 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> May 25 00:56:38 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> May 25 00:56:38 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> May 25 00:56:38 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> May 25 00:56:38 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> May 25 00:56:38 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> May 25 00:56:38 at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> May 25 00:56:38 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> May 25 00:56:38 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> May 25 00:56:38 at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> May 25 00:56:38 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> May 25 00:56:38 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> May 25 00:56:38 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> May 25 00:56:38 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> May 25 00:56:38 at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> May 25 00:56:38 at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> May 25 00:56:38 at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> May 25 00:56:38 at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> May 25 00:56:38 at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> May 25 00:56:38 at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> May 25 00:56:38 at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> May 25 00:56:38 at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> May 25 00:56:38
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-15460) planner dependencies won't be necessary for JDBC connector
[ https://issues.apache.org/jira/browse/FLINK-15460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-15460: Fix Version/s: 1.16.0 > planner dependencies won't be necessary for JDBC connector > -- > > Key: FLINK-15460 > URL: https://issues.apache.org/jira/browse/FLINK-15460 > Project: Flink > Issue Type: Improvement > Components: Connectors / JDBC >Affects Versions: 1.10.0 >Reporter: Zhenghua Gao >Priority: Minor > Labels: pull-request-available > Fix For: 1.15.0, 1.16.0 > > Time Spent: 10m > Remaining Estimate: 0h > > remove planner dependencies from JDBC connector by changing the scope to test. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-14684) Add Pinterest to Chinese Powered By page
[ https://issues.apache.org/jira/browse/FLINK-14684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-14684: Fix Version/s: 1.16.0 > Add Pinterest to Chinese Powered By page > > > Key: FLINK-14684 > URL: https://issues.apache.org/jira/browse/FLINK-14684 > Project: Flink > Issue Type: New Feature > Components: chinese-translation >Reporter: Hequn Cheng >Priority: Minor > Labels: auto-unassigned, pull-request-available > Fix For: 1.15.0, 1.16.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Pinterest was added to the English Powered By page with commit: > [51f7e3ced85b94dcbe3c051069379d22c88fbc5c|https://github.com/apache/flink-web/pull/281] > It should be added to the Chinese Powered By (and index.html) page as well. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-15000) WebUI Metrics is very slow in large parallelism
[
https://issues.apache.org/jira/browse/FLINK-15000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-15000:
Fix Version/s: 1.16.0
> WebUI Metrics is very slow in large parallelism
> ---
>
> Key: FLINK-15000
> URL: https://issues.apache.org/jira/browse/FLINK-15000
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Web Frontend
>Affects Versions: 1.9.0, 1.9.1
>Reporter: fa zheng
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned,
> pull-request-available
> Fix For: 1.15.0, 1.16.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> metrics in web ui are very slow when parallelism is huge. It's hard to add
> metric and choose one metric. I run carTopSpeedWindowingExample with command
> {code:java}
> //代码占位符
> flink run -m yarn-cluster -p 1200 examples/streaming/TopSpeedWindowing.jar
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-19432) Whether to capture the updates which don't change any monitored columns
[ https://issues.apache.org/jira/browse/FLINK-19432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-19432: Fix Version/s: 1.16.0 > Whether to capture the updates which don't change any monitored columns > --- > > Key: FLINK-19432 > URL: https://issues.apache.org/jira/browse/FLINK-19432 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.11.1 >Reporter: shizhengchao >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.15.0, 1.16.0 > > > with `debezium-json` and `canal-json`: > Whether to capture the updates which don't change any monitored columns. This > may happen if the monitored columns (columns defined in Flink SQL DDL) is a > subset of the columns in database table. We can provide an optional option， > default 'true', which means all the updates will be captured. You can set to > 'false' to only capture changed updates -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-16800) TypeMappingUtils#checkIfCompatible didn't deal with nested types
[
https://issues.apache.org/jira/browse/FLINK-16800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-16800:
Fix Version/s: 1.16.0
> TypeMappingUtils#checkIfCompatible didn't deal with nested types
>
>
> Key: FLINK-16800
> URL: https://issues.apache.org/jira/browse/FLINK-16800
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.10.0
>Reporter: Zhenghua Gao
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned,
> pull-request-available
> Fix For: 1.15.0, 1.16.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> the planner uses TypeMappingUtils#checkIfCompatible to validate logical
> schema and physical schema are compatible when translate
> CatalogSinkModifyOperation to Calcite relational expression. The validation
> didn't deal with nested types well, which could throw the following
> ValidationException:
> {code:java}
> Exception in thread "main" org.apache.flink.table.api.ValidationException:
> Type ARRAY> of table field 'old'
> does not match with the physical type ARRAY LEGACY('DECIMAL', 'DECIMAL')>> of the 'old' field of the TableSource return
> type.
> at
> org.apache.flink.table.utils.TypeMappingUtils.lambda$checkPhysicalLogicalTypeCompatible$4(TypeMappingUtils.java:164)
> at
> org.apache.flink.table.utils.TypeMappingUtils$1.defaultMethod(TypeMappingUtils.java:277)
> at
> org.apache.flink.table.utils.TypeMappingUtils$1.defaultMethod(TypeMappingUtils.java:254)
> at
> org.apache.flink.table.types.logical.utils.LogicalTypeDefaultVisitor.visit(LogicalTypeDefaultVisitor.java:157)
> at org.apache.flink.table.types.logical.ArrayType.accept(ArrayType.java:110)
> at
> org.apache.flink.table.utils.TypeMappingUtils.checkIfCompatible(TypeMappingUtils.java:254)
> at
> org.apache.flink.table.utils.TypeMappingUtils.checkPhysicalLogicalTypeCompatible(TypeMappingUtils.java:160)
> at
> org.apache.flink.table.utils.TypeMappingUtils.lambda$computeInCompositeType$8(TypeMappingUtils.java:232)
> at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1321)
> at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
> at
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.flink.table.utils.TypeMappingUtils.computeInCompositeType(TypeMappingUtils.java:214)
> at
> org.apache.flink.table.utils.TypeMappingUtils.computePhysicalIndices(TypeMappingUtils.java:192)
> at
> org.apache.flink.table.utils.TypeMappingUtils.computePhysicalIndicesOrTimeAttributeMarkers(TypeMappingUtils.java:112)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecTableSourceScan.computeIndexMapping(StreamExecTableSourceScan.scala:212)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecTableSourceScan.translateToPlanInternal(StreamExecTableSourceScan.scala:107)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecTableSourceScan.translateToPlanInternal(StreamExecTableSourceScan.scala:62)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecNode$class.translateToPlan(ExecNode.scala:58)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecTableSourceScan.translateToPlan(StreamExecTableSourceScan.scala:62)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecExchange.translateToPlanInternal(StreamExecExchange.scala:84)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecExchange.translateToPlanInternal(StreamExecExchange.scala:44)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecNode$class.translateToPlan(ExecNode.scala:58)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecExchange.translateToPlan(StreamExecExchange.scala:44)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecLimit.translateToPlanInternal(StreamExecLimit.scala:161)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecLimit.translateToPlanInternal(StreamExecLimit.scala:51)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecNode$class.translateToPlan(ExecNode.scala:58)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecLimit.translateToPlan(StreamExecLimit.scala:51)
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecSink.translateToTransformation(Stream
[jira] [Updated] (FLINK-19895) Unify Life Cycle Management of ResultPartitionType Pipelined Family
[ https://issues.apache.org/jira/browse/FLINK-19895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-19895: Fix Version/s: 1.16.0 > Unify Life Cycle Management of ResultPartitionType Pipelined Family > --- > > Key: FLINK-19895 > URL: https://issues.apache.org/jira/browse/FLINK-19895 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Task >Reporter: Yuan Mei >Priority: Major > Fix For: 1.15.0, 1.16.0 > > > This ticket is to unify lifecycle management of > `ResultPartitionType.PIPELINED(_BOUNDED)` and > `ResultPartitionType.PIPELINED_APPOXIMATE`, so that we can get rid of the > hacky attribute `reconenctable` introduced in FLINK-19693 > > In short: > *The current behavior of PIPELINED(_BOUNDED) is* -> > Release partition as soon as consumer exits > Release partition as soon as producer fails/canceled > *Current behavior of PIPELINED_APPOXIMATE* -> > Release partition as soon as producer fails/canceled > Release partition when the job exists > *Unified Pipelined Family to* > Release partition when producer exits. > > One more question: > *whether we can unify Blocking + Pieliened Family to* > Producer release partition when producer fails/canceled > Release partition when the job exists -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-23683) KafkaSinkITCase hangs on azure
[ https://issues.apache.org/jira/browse/FLINK-23683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-23683: Fix Version/s: 1.16.0 > KafkaSinkITCase hangs on azure > -- > > Key: FLINK-23683 > URL: https://issues.apache.org/jira/browse/FLINK-23683 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.14.0 >Reporter: Xintong Song >Assignee: Fabian Paul >Priority: Major > Labels: stale-assigned, test-stability > Fix For: 1.15.0, 1.16.0 > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=21738&view=logs&j=4be4ed2b-549a-533d-aa33-09e28e360cc8&t=f7d83ad5-3324-5307-0eb3-819065cdcb38&l=7886 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-20920) Document how to connect to kerberized HMS
[ https://issues.apache.org/jira/browse/FLINK-20920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-20920: Fix Version/s: 1.16.0 > Document how to connect to kerberized HMS > - > > Key: FLINK-20920 > URL: https://issues.apache.org/jira/browse/FLINK-20920 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive, Documentation >Reporter: Rui Li >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned > Fix For: 1.15.0, 1.16.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-24707) Upgrade Curator framework dependecy.
[ https://issues.apache.org/jira/browse/FLINK-24707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-24707: Fix Version/s: 1.16.0 > Upgrade Curator framework dependecy. > > > Key: FLINK-24707 > URL: https://issues.apache.org/jira/browse/FLINK-24707 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Reporter: David Morávek >Priority: Major > Fix For: 1.15.0, 1.16.0 > > > We want to upgrade curator to latest version, to avoid working around > non-idempotent writes. > https://issues.apache.org/jira/browse/CURATOR-584 > Main reason for not doing this in patch release as part of FLINK-24543 is > that this effectively drops support for Zookeeper 3.4.x. > The very last past release two 3.4.x branch was more then two years ago (2 > April, 2019) and there are already three newer minor versions available > (3.5.x,3.6.x,3.7.x). -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-24330) HiveRunnerITCase JVM crash
[ https://issues.apache.org/jira/browse/FLINK-24330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-24330: Fix Version/s: 1.16.0 > HiveRunnerITCase JVM crash > -- > > Key: FLINK-24330 > URL: https://issues.apache.org/jira/browse/FLINK-24330 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Affects Versions: 1.15.0 >Reporter: Xintong Song >Priority: Major > Labels: test-stability > Fix For: 1.15.0, 1.16.0 > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24282&view=logs&j=fc5181b0-e452-5c8f-68de-1097947f6483&t=995c650b-6573-581c-9ce6-7ad4cc038461&l=24092 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25710) Multiple Kafka IT cases fail with "ContainerLaunch Container startup failed"
[
https://issues.apache.org/jira/browse/FLINK-25710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-25710:
Fix Version/s: 1.16.0
> Multiple Kafka IT cases fail with "ContainerLaunch Container startup failed"
>
>
> Key: FLINK-25710
> URL: https://issues.apache.org/jira/browse/FLINK-25710
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Tests
>Affects Versions: 1.15.0, 1.16.0
>Reporter: Roman Khachatryan
>Priority: Major
> Labels: test-stability
> Fix For: 1.15.0, 1.16.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=29731&view=logs&j=c5f0071e-1851-543e-9a45-9ac140befc32&t=15a22db7-8faa-5b34-3920-d33c9f0ca23c&l=35454
> {code}
> 2022-01-19T18:17:40.3503774Z Jan 19 18:17:40 [INFO]
> ---
> 2022-01-19T18:17:42.3992027Z Jan 19 18:17:42 [ERROR] Picked up
> JAVA_TOOL_OPTIONS: -XX:+HeapDumpOnOutOfMemoryError
> 2022-01-19T18:17:42.9262342Z Jan 19 18:17:42 [INFO] Running
> org.apache.flink.connector.kafka.sink.KafkaTransactionLogITCase
> 2022-01-19T18:18:47.9992530Z Jan 19 18:18:47 [ERROR] Tests run: 1, Failures:
> 0, Errors: 1, Skipped: 0, Time elapsed: 65.053 s <<< FAILURE! - in or
> g.apache.flink.connector.kafka.sink.KafkaTransactionLogITCase
> 2022-01-19T18:18:47.9993836Z Jan 19 18:18:47 [ERROR]
> org.apache.flink.connector.kafka.sink.KafkaTransactionLogITCase Time
> elapsed: 65.053 s <<< ERROR!
> 2022-01-19T18:18:47.9994507Z Jan 19 18:18:47
> org.testcontainers.containers.ContainerLaunchException: Container startup
> failed
> ...
> 2022-01-19T18:18:48.0038449Z Jan 19 18:18:47 Caused by:
> org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with
> exception
> 2022-01-19T18:18:48.0039451Z Jan 19 18:18:47at
> org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:88)
> 2022-01-19T18:18:48.0040449Z Jan 19 18:18:47at
> org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:329)
> 2022-01-19T18:18:48.0041204Z Jan 19 18:18:47... 27 more
> 2022-01-19T18:18:48.0041993Z Jan 19 18:18:47 Caused by:
> org.testcontainers.containers.ContainerLaunchException: Could not
> create/start container
> 2022-01-19T18:18:48.0043007Z Jan 19 18:18:47at
> org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:525)
> 2022-01-19T18:18:48.0044020Z Jan 19 18:18:47at
> org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:331)
> 2022-01-19T18:18:48.0045158Z Jan 19 18:18:47at
> org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
> 2022-01-19T18:18:48.0046043Z Jan 19 18:18:47... 28 more
> 2022-01-19T18:18:48.0047026Z Jan 19 18:18:47 Caused by:
> org.testcontainers.containers.ContainerLaunchException: Timed out waiting for
> container po*rt to open (172.17.0.1 ports: [56218, 56219] should be listening)
> 2022-01-19T18:18:48.0048320Z Jan 19 18:18:47at
> org.testcontainers.containers.wait.strategy.HostPortWaitStrategy.waitUntilReady(HostPortWaitStr
> ategy.java:90)
> 2022-01-19T18:18:48.0049465Z Jan 19 18:18:47at
> org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStr
> ategy.java:51)
> 2022-01-19T18:18:48.0050585Z Jan 19 18:18:47at
> org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:
> 929)
> 2022-01-19T18:18:48.0051628Z Jan 19 18:18:47at
> org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:468)
> 2022-01-19T18:18:48.0052380Z Jan 19 18:18:47... 30 more
> ...
> 2022-01-19T18:40:37.7197924Z Jan 19 18:40:37 [INFO] Results:
> 2022-01-19T18:40:37.7198526Z Jan 19 18:40:37 [INFO]
> 2022-01-19T18:40:37.7199093Z Jan 19 18:40:37 [ERROR] Errors:
> 2022-01-19T18:40:37.7200602Z Jan 19 18:40:37 [ERROR] KafkaSinkITCase »
> ContainerLaunch Container startup failed
> 2022-01-19T18:40:37.7201683Z Jan 19 18:40:37 [ERROR]
> KafkaTransactionLogITCase » ContainerLaunch Container startup failed
> 2022-01-19T18:40:37.7204632Z Jan 19 18:40:37 [ERROR]
> KafkaWriterITCase.beforeAll:99 » ContainerLaunch Container startup failed
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25612) Update the outdated illustration of ExecutionState in the documentation
[
https://issues.apache.org/jira/browse/FLINK-25612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-25612:
Fix Version/s: 1.16.0
> Update the outdated illustration of ExecutionState in the documentation
> ---
>
> Key: FLINK-25612
> URL: https://issues.apache.org/jira/browse/FLINK-25612
> Project: Flink
> Issue Type: Technical Debt
> Components: Documentation
>Affects Versions: 1.15.0, 1.13.5, 1.14.2
>Reporter: Zhilong Hong
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.15.0, 1.16.0
>
> Attachments: current-illustration-2.jpg, new-illustration-2.jpg
>
>
> Currently, the illustration of {{ExecutionState}} located in the page "Jobs
> and Scheduling"
> ([https://nightlies.apache.org/flink/flink-docs-master/docs/internals/job_scheduling/])
> is outdated. It doesn't involve the INITIALIZING state, which is introduced
> in FLINK-17102.
>
> Current illustration:
> !current-illustration-2.jpg|width=400!
> New illustration:
> !new-illustration-2.jpg|width=400!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-24688) yarn.application-attempt-failures-validity-interval link url is invalid
[ https://issues.apache.org/jira/browse/FLINK-24688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-24688: Fix Version/s: 1.16.0 > yarn.application-attempt-failures-validity-interval link url is invalid > --- > > Key: FLINK-24688 > URL: https://issues.apache.org/jira/browse/FLINK-24688 > Project: Flink > Issue Type: Bug > Components: Deployment / YARN >Affects Versions: 1.9.0, 1.10.0, 1.11.0, 1.12.0, 1.13.0, 1.14.0 >Reporter: baizhendong >Priority: Minor > Labels: pull-request-available > Fix For: 1.15.0, 1.16.0 > > > yarn.application-attempt-failures-validity-interval property was added in > YarnConfigOptions since FLINK-12472[yarn], but now the link > [https://hortonworks.com/blog/apache-hadoop-yarn-hdp-2-2-fault-tolerance-features-long-running-services/] > is not available. So we should update this link. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-22243) Reactive Mode parallelism changes are not shown in the job graph visualization in the UI
[ https://issues.apache.org/jira/browse/FLINK-22243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-22243: Fix Version/s: 1.16.0 > Reactive Mode parallelism changes are not shown in the job graph > visualization in the UI > > > Key: FLINK-22243 > URL: https://issues.apache.org/jira/browse/FLINK-22243 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Web Frontend >Affects Versions: 1.13.0, 1.14.0 >Reporter: Robert Metzger >Priority: Major > Fix For: 1.15.0, 1.16.0 > > Attachments: screenshot-1.png > > > As reported here FLINK-22134, the parallelism in the visual job graph on top > of the detail page is not in sync with the parallelism listed in the task > list below, when reactive mode causes a parallelism change. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-15745) KafkaITCase.testKeyValueSupport failure due to assertion error.
[
https://issues.apache.org/jira/browse/FLINK-15745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-15745:
Fix Version/s: 1.16.0
> KafkaITCase.testKeyValueSupport failure due to assertion error.
> ---
>
> Key: FLINK-15745
> URL: https://issues.apache.org/jira/browse/FLINK-15745
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Tests
>Affects Versions: 1.10.0
>Reporter: Jiangjie Qin
>Assignee: Fabian Paul
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned,
> stale-assigned, test-stability
> Fix For: 1.15.0, 1.16.0
>
>
> The failure cause was:
> {code:java}
> Caused by: java.lang.AssertionError: Wrong value 50
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaConsumerTestBase$15.flatMap(KafkaConsumerTestBase.java:1411)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaConsumerTestBase$15.flatMap(KafkaConsumerTestBase.java:1406)
> at
> org.apache.flink.streaming.api.operators.StreamFlatMap.processElement(StreamFlatMap.java:50)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:641)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:616)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:596)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:730)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:708)
> at
> org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:104)
> at
> org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collectWithTimestamp(StreamSourceContexts.java:111)
> at
> org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordWithTimestamp(AbstractFetcher.java:398)
> at
> org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.emitRecord(KafkaFetcher.java:185)
> at
> org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.runFetchLoop(KafkaFetcher.java:150)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:715)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63)
> at
> org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:196)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-23387) WindowRankITCase.testEventTimeTumbleWindowWithoutRankNumber fails on azure
[ https://issues.apache.org/jira/browse/FLINK-23387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-23387: Fix Version/s: 1.16.0 > WindowRankITCase.testEventTimeTumbleWindowWithoutRankNumber fails on azure > -- > > Key: FLINK-23387 > URL: https://issues.apache.org/jira/browse/FLINK-23387 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.14.0 >Reporter: Dawid Wysakowicz >Priority: Major > Labels: test-stability > Fix For: 1.15.0, 1.16.0 > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=20418&view=logs&j=0c940707-2659-5648-cbe6-a1ad63045f0a&t=8c0749ca-5030-5163-217b-f2e73cf2e271&l=230530 > stacktrace too long to copy -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-22605) Python UDF should only be created once regardless of how many times it is invoked
[ https://issues.apache.org/jira/browse/FLINK-22605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-22605: Fix Version/s: 1.16.0 > Python UDF should only be created once regardless of how many times it is > invoked > - > > Key: FLINK-22605 > URL: https://issues.apache.org/jira/browse/FLINK-22605 > Project: Flink > Issue Type: Improvement > Components: API / Python >Reporter: Yik San Chan >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.15.0, 1.16.0 > > > Follow up to > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/I-call-Pandas-UDF-N-times-do-I-have-to-initiate-the-UDF-N-times-tt43576.html. > Currently, when we call Python UDF N times, the Python UDF is constructed N > times. This may become a concern to performance, when we want to load large > resources in the open() method of the UDF, which is quite common in machine > learning use cases. > I propose we optimize in PyFlink framework level s.t. no matter how many > times a UDF is called in the execution environment, it is only initiated once. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25419) Support the metadata column to generate dynamic index
[ https://issues.apache.org/jira/browse/FLINK-25419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-25419: Fix Version/s: 1.16.0 > Support the metadata column to generate dynamic index > -- > > Key: FLINK-25419 > URL: https://issues.apache.org/jira/browse/FLINK-25419 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Reporter: jinfeng >Priority: Major > Fix For: 1.15.0, 1.16.0 > > > As mentioned in [https://github.com/apache/flink/pull/18058] We can > implement metadata column to increase the flexibility of using dynamic > indexes . > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-13809) The log directory of Flink Python API is unwritable if it is installed via "sudo"
[
https://issues.apache.org/jira/browse/FLINK-13809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-13809:
Fix Version/s: 1.16.0
> The log directory of Flink Python API is unwritable if it is installed via
> "sudo"
> --
>
> Key: FLINK-13809
> URL: https://issues.apache.org/jira/browse/FLINK-13809
> Project: Flink
> Issue Type: Improvement
> Components: API / Python
>Affects Versions: 1.9.0
>Reporter: Wei Zhong
>Priority: Minor
> Labels: auto-unassigned, pull-request-available
> Fix For: 1.15.0, 1.16.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, if the python apache-flink package is installed via "sudo", an
> exception will be thrown when starting the flink python shell:
> {code:java}
> log4j:ERROR setFile(null,false) call failed. java.io.FileNotFoundException:
> /Library/Python/2.7/site-packages/pyflink/log/flink-zhongwei-python-zhongweideMacBook-Pro.local.log
> (Permission denied) at java.io.FileOutputStream.open0(Native Method) at
> java.io.FileOutputStream.open(FileOutputStream.java:270) at
> java.io.FileOutputStream.(FileOutputStream.java:213) at
> java.io.FileOutputStream.(FileOutputStream.java:133) at
> org.apache.log4j.FileAppender.setFile(FileAppender.java:294) at
> org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165) at
> org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307) at
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
> at
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
> at
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
> at
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
> at
> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
> at
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
> at org.apache.log4j.LogManager.(LogManager.java:127) at
> org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:81) at
> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329) at
> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349) at
> org.apache.flink.api.java.ExecutionEnvironment.(ExecutionEnvironment.java:102)
> at java.lang.Class.forName0(Native Method) at
> java.lang.Class.forName(Class.java:348) at
> org.apache.flink.api.python.shaded.py4j.reflection.CurrentThreadClassLoadingStrategy.classForName(CurrentThreadClassLoadingStrategy.java:40)
> at
> org.apache.flink.api.python.shaded.py4j.reflection.ReflectionUtil.classForName(ReflectionUtil.java:51)
> at
> org.apache.flink.api.python.shaded.py4j.reflection.TypeUtil.forName(TypeUtil.java:243)
> at
> org.apache.flink.api.python.shaded.py4j.commands.ReflectionCommand.getUnknownMember(ReflectionCommand.java:175)
> at
> org.apache.flink.api.python.shaded.py4j.commands.ReflectionCommand.execute(ReflectionCommand.java:87)
> at
> org.apache.flink.api.python.shaded.py4j.GatewayConnection.run(GatewayConnection.java:238)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> It does not affect the running of flink python shell but it would be better
> if we can fix it.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-21871) Support watermark for Hive and Filesystem streaming source
[
https://issues.apache.org/jira/browse/FLINK-21871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-21871:
Fix Version/s: 1.16.0
> Support watermark for Hive and Filesystem streaming source
> --
>
> Key: FLINK-21871
> URL: https://issues.apache.org/jira/browse/FLINK-21871
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / FileSystem, Connectors / Hive, Table SQL /
> API
>Reporter: Jark Wu
>Priority: Major
> Labels: auto-unassigned, pull-request-available
> Fix For: 1.15.0, 1.16.0
>
>
> Hive and Filesystem already support streaming source. However, they doesn't
> support watermark on the source. That means users can't leverage the
> streaming source to perform the Flink powerful streaming analysis, e.g.
> window aggregate, interval join, and so on.
> In order to make more Hive users can leverage Flink to perform streaming
> analysis, and also cooperate with the new optimized window-TVF operations
> (FLIP-145), we need to support watermark for Hive and Filesystem.
> h2. How to emit watermark in Hive and Filesystem
> Factual data in Hive are usually partitioned by date time, e.g.
> {{pt_day=2021-03-19, pt_hour=10}}. In this case, when the data of partition
> {{pt_day=2021-03-19, pt_hour=10}} are emitted, we should be able to know all
> the data before {{2021-03-19 11:00:00}} have been arrived, so we can emit a
> watermark value of {{2021-03-19 11:00:00}}. We call this partition watermark.
> The partition watermark is much better than record watermark (extract
> watermark from record, e.g. {{ts - INTERVAL '1' MINUTE}}). Because in above
> example, if we are using partition watermark, the window of [10:00, 11:00)
> will be triggered when pt_hour=10 is finished. However, if we are using
> record watermark, the window of [10:00, 11:00) will be triggered when
> pt_hour=11 is arrived, that will make the pipeline have one more partition
> dely.
> Therefore, we firstly focus on support partition watermark for Hive and
> Filesystem.
> h2. Example
> In order to support such watermarks, we propose using the following DDL to
> define a Hive table with watermark defined:
> {code:sql}
> -- using hive dialect
> CREATE TABLE hive_table (
> x int,
> y string,
> z int,
> ts timestamp,
> WATERMARK FOR ts AS SOURCE_WATERMARK
> ) PARTITIONED BY (pt_day string, pt_hour string)
> TBLPROPERTIES (
> 'streaming-source.enable'='true',
> 'streaming-source.monitor-interval'='1s',
> 'partition.time-extractor.timestamp-pattern'='$pt_day $pt_hour:00:00',
> 'partition.time-interval'='1h'
> );
> -- window aggregate on the hive table
> SELECT window_start, window_end, COUNT(*), MAX(y), SUM(z)
> FROM TABLE(
>TUMBLE(TABLE hive_table, DESCRIPTOR(ts), INTERVAL '1' HOUR))
> GROUP BY window_start, window_end;
> {code}
> For filesystem connector, the DDL can be:
> {code:sql}
> CREATE TABLE fs_table (
> x int,
> y string,
> z int,
> ts TIMESTAMP(3),
> pt_day string,
> pt_hour string,
> WATERMARK FOR ts AS SOURCE_WATERMARK
> ) PARTITIONED BY (pt_day, pt_hour)
> WITH (
> 'connector' = 'filesystem',
> 'path' = '/path/to/file',
> 'format' = 'parquet',
> 'streaming-source.enable'='true',
> 'streaming-source.monitor-interval'='1s',
> 'partition.time-extractor.timestamp-pattern'='$pt_day $pt_hour:00:00',
> 'partition.time-interval'='1h'
> );
> {code}
> I will explain the new function/configuration.
> h2. SOURCE_WATERMARK built-in function
> FLIP-66[1] proposed {{SYSTEM_WATERMARK}} function for watermarks preserved in
> underlying source system.
> However, the SYSTEM prefix sounds like a Flink system generated value, but
> actually, this is a SOURCE system generated value.
> So I propose to use {{SOURCE_WATERMARK}} intead, this also keeps the concept
> align with the API of
> {{org.apache.flink.table.descriptors.Rowtime#watermarksFromSource}}.
> h2. Table Options for Watermark
> - {{partition.time-extractor.timestamp-pattern}}: this option already exists.
> This is used to extract/convert partition value to a timestamp value.
> - {{partition.time-interval}}: this is a new option. It indicates the minimal
> time interval of the partitions. It's used to calculate the correct watermark
> when a partition is finished. The watermark = partition-timestamp +
> time-inteval.
> h2. How to support watermark for existing Hive tables
> We all know that we can't create a new table for an existing Hive table. So
> we should support altering existing Hive table to add the watermark
> inforamtion.
> This can be supported by the new ALTER TABLE syntax proposed in FLINK-21634.
> Because watermark, computed column, table options are all encoded in Hive
> table parameters,
> so other systems (
[jira] [Updated] (FLINK-21693) TestStreamEnvironment does not implement executeAsync
[
https://issues.apache.org/jira/browse/FLINK-21693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-21693:
Fix Version/s: 1.16.0
> TestStreamEnvironment does not implement executeAsync
> -
>
> Key: FLINK-21693
> URL: https://issues.apache.org/jira/browse/FLINK-21693
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.11.3, 1.12.2, 1.13.0
>Reporter: Till Rohrmann
>Priority: Minor
> Labels: auto-deprioritized-major
> Fix For: 1.15.0, 1.16.0
>
>
> When implementing FLINK-14392 we forgot to implement
> {{TestStreamEnvironment.executeAsync}}. As a consequence, when using
> {{TestStreamEnvironment}} and calling {{executeAsync}} the system will always
> start a new local {{MiniCluster}}. The proper behaviour would be to use the
> {{MiniCluster}} specified in the {{TestStreamEnvironment}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-23412) Improve sourceSink description
[
https://issues.apache.org/jira/browse/FLINK-23412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-23412:
Fix Version/s: 1.16.0
> Improve sourceSink description
> --
>
> Key: FLINK-23412
> URL: https://issues.apache.org/jira/browse/FLINK-23412
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Affects Versions: 1.13.1
>Reporter: Mans Singh
>Priority: Minor
> Labels: docuentation, pull-request-available, sink, source, table
> Fix For: 1.15.0, 1.16.0
>
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> The table/sourcesink documentation indicates:
> {quote} the sink can solely accept insert-only rows and write out bounded
> streams.{quote}
> Perhaps can be:
> {quote} the sink can only accept insert-only rows and write out bounded
> streams.{quote}
> Also, improving full stack example bullet points.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-16762) Relocation Beam dependency of PyFlink
[ https://issues.apache.org/jira/browse/FLINK-16762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-16762: Fix Version/s: 1.16.0 > Relocation Beam dependency of PyFlink > - > > Key: FLINK-16762 > URL: https://issues.apache.org/jira/browse/FLINK-16762 > Project: Flink > Issue Type: Improvement > Components: API / Python >Affects Versions: 1.10.0 >Reporter: sunjincheng >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned, > pull-request-available > Fix For: 1.15.0, 1.16.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Some users may already use beam on their own cluster, which may cause the > conflict between the beam jar package carried by pyflink and the jar of the > user cluster beam to a certain extent. So, I would like to relocation the > Beam dependency of PyFlink. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-17784) Better detection for parquet and orc in hive
[ https://issues.apache.org/jira/browse/FLINK-17784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-17784: Fix Version/s: 1.16.0 > Better detection for parquet and orc in hive > > > Key: FLINK-17784 > URL: https://issues.apache.org/jira/browse/FLINK-17784 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive >Reporter: Jingsong Lee >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.15.0, 1.16.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-13876) Remove ExecutionConfig field from PojoSerializer
[
https://issues.apache.org/jira/browse/FLINK-13876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-13876:
Fix Version/s: 1.16.0
> Remove ExecutionConfig field from PojoSerializer
>
>
> Key: FLINK-13876
> URL: https://issues.apache.org/jira/browse/FLINK-13876
> Project: Flink
> Issue Type: Improvement
> Components: API / Type Serialization System
>Affects Versions: 1.7.2, 1.8.1, 1.9.0, 1.10.0
>Reporter: Dawid Wysakowicz
>Priority: Minor
> Labels: auto-deprioritized-major
> Fix For: 1.15.0, 1.16.0
>
>
> The PojoSerializers stores an instance of ExecutionConfig as internal field,
> even though that the only information it may ever need are the registered
> kryo serializers.
> This has a few drawbacks:
> * It blocks the evolution of {{ExecutionConfig}} as serializers where stored
> in a state. Therefore any change to ExecutionConfig must be backwards
> compatible in respect to java serialization
> * It probably already introduced a bug, as upon restore the Snapshot actually
> recreates the serializer with an empty ExecutionConfig (see
> org.apache.flink.api.java.typeutils.runtime.PojoSerializerSnapshot#restoreSerializer)
> I suggest to remove the field completely and adjust corresponding usages.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-20110) Support 'merge' method for first_value and last_value UDAF
[ https://issues.apache.org/jira/browse/FLINK-20110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-20110: Fix Version/s: 1.16.0 > Support 'merge' method for first_value and last_value UDAF > -- > > Key: FLINK-20110 > URL: https://issues.apache.org/jira/browse/FLINK-20110 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Runtime >Affects Versions: 1.12.0 >Reporter: hailong wang >Priority: Major > Labels: auto-unassigned, pull-request-available > Fix For: 1.15.0, 1.16.0 > > > From the user-zh email, when use first_value function in hop window, It > throws the exception because first_vaue does not implement the merge method. > We can support 'merge' method for first_value and last_value UDAF -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25181) KafkaSourceITCase.testValueOnlyDeserializer fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-25181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-25181:
Fix Version/s: 1.16.0
> KafkaSourceITCase.testValueOnlyDeserializer fails on AZP
>
>
> Key: FLINK-25181
> URL: https://issues.apache.org/jira/browse/FLINK-25181
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.14.0, 1.13.3, 1.15.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0, 1.16.0
>
>
> The test case {{KafkaSourceITCase.testValueOnlyDeserializer}} fails on AZP
> with
> {code}
> Dec 06 00:53:02 [ERROR] Tests run: 5, Failures: 1, Errors: 0, Skipped: 0,
> Time elapsed: 62.314 s <<< FAILURE! - in
> org.apache.flink.connector.kafka.source.KafkaSourceITCase
> Dec 06 00:53:02 [ERROR]
> testValueOnlyDeserializer(org.apache.flink.connector.kafka.source.KafkaSourceITCase)
> Time elapsed: 3.018 s <<< FAILURE!
> Dec 06 00:53:02 java.lang.AssertionError: expected:<660> but was:<456>
> Dec 06 00:53:02 at org.junit.Assert.fail(Assert.java:88)
> Dec 06 00:53:02 at org.junit.Assert.failNotEquals(Assert.java:834)
> Dec 06 00:53:02 at org.junit.Assert.assertEquals(Assert.java:645)
> Dec 06 00:53:02 at org.junit.Assert.assertEquals(Assert.java:631)
> Dec 06 00:53:02 at
> org.apache.flink.connector.kafka.source.KafkaSourceITCase.testValueOnlyDeserializer(KafkaSourceITCase.java:173)
> Dec 06 00:53:02 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Dec 06 00:53:02 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Dec 06 00:53:02 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Dec 06 00:53:02 at java.lang.reflect.Method.invoke(Method.java:498)
> Dec 06 00:53:02 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> Dec 06 00:53:02 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Dec 06 00:53:02 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> Dec 06 00:53:02 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Dec 06 00:53:02 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> Dec 06 00:53:02 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> Dec 06 00:53:02 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> Dec 06 00:53:02 at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> Dec 06 00:53:02 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> Dec 06 00:53:02 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> Dec 06 00:53:02 at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> Dec 06 00:53:02 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> Dec 06 00:53:02 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> Dec 06 00:53:02 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> Dec 06 00:53:02 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> Dec 06 00:53:02 at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> Dec 06 00:53:02 at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> Dec 06 00:53:02 at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> Dec 06 00:53:02 at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> Dec 06 00:53:02 at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> Dec 06 00:53:02 at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> Dec 06 00:53:02 at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> Dec 06 00:53:02 at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27568&view=logs&j=c5f0071e-1851-543e-9a45-9ac140befc32&t=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5&l=6558
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-14918) Add performance e2e test module and scripts
[ https://issues.apache.org/jira/browse/FLINK-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-14918: Fix Version/s: 1.16.0 > Add performance e2e test module and scripts > --- > > Key: FLINK-14918 > URL: https://issues.apache.org/jira/browse/FLINK-14918 > Project: Flink > Issue Type: Sub-task > Components: Test Infrastructure >Reporter: Yu Li >Priority: Major > Labels: auto-unassigned, pull-request-available > Fix For: 1.15.0, 1.16.0 > > Time Spent: 20m > Remaining Estimate: 0h > > As proposed in FLIP-83, create a separate directory/module in parallel with > flink-end-to-end-tests, with the name of flink-end-to-end-perf-tests, and add > necessary scripts to form the framework. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-26003) Use Jackson serialization for persisting TaskExecutor state to working directory
[
https://issues.apache.org/jira/browse/FLINK-26003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-26003:
Fix Version/s: 1.16.0
> Use Jackson serialization for persisting TaskExecutor state to working
> directory
>
>
> Key: FLINK-26003
> URL: https://issues.apache.org/jira/browse/FLINK-26003
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Affects Versions: 1.15.0
>Reporter: Till Rohrmann
>Priority: Major
> Fix For: 1.15.0, 1.16.0
>
>
> In order to avoid Java serialization, we should use a different serialization
> format for persisting {{TaskExecutor}} state in the working directory. One
> idea could be to use Jackson for the serialization.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-23932) KafkaTableITKafkaTableITCase.testKafkaSourceSinkWithKeyAndPartialValue hangs on AzureCase
[
https://issues.apache.org/jira/browse/FLINK-23932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-23932:
Fix Version/s: 1.16.0
> KafkaTableITKafkaTableITCase.testKafkaSourceSinkWithKeyAndPartialValue hangs
> on AzureCase
> -
>
> Key: FLINK-23932
> URL: https://issues.apache.org/jira/browse/FLINK-23932
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / Ecosystem
>Affects Versions: 1.14.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: stale-assigned, test-stability
> Fix For: 1.15.0, 1.16.0
>
>
> The test {{KafkaTableITCase.testKafkaSourceSinkWithKeyAndPartialValue}} hangs
> on Azure. Interestingly, the test case seems to spawn 400
> {{kafka-admin-client-thread | adminclient}} threads. I think there is
> something wrong with the test case setup.
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=22674&view=logs&j=c5f0071e-1851-543e-9a45-9ac140befc32&t=15a22db7-8faa-5b34-3920-d33c9f0ca23c&l=22875
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-22282) Move creation of SplitEnumerator to the SourceCoordinator thread
[ https://issues.apache.org/jira/browse/FLINK-22282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-22282: Fix Version/s: 1.16.0 > Move creation of SplitEnumerator to the SourceCoordinator thread > > > Key: FLINK-22282 > URL: https://issues.apache.org/jira/browse/FLINK-22282 > Project: Flink > Issue Type: Improvement > Components: Connectors / Common >Affects Versions: 1.12.2 >Reporter: Jiangjie Qin >Priority: Major > Labels: auto-deprioritized-critical, auto-unassigned > Fix For: 1.15.0, 1.12.8, 1.16.0 > > > Currently the creation of the SplitEnumerator is in the JM main thread. In > case the SplitEnumerator instantiation takes long, the job execution will > timeout. The fix is moving the SplitEnumerator creation to the coordinator > thread. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-18442) Move `testSessionWindowsWithContinuousEventTimeTrigger` to `ContinuousEventTimeTriggerTest`
[ https://issues.apache.org/jira/browse/FLINK-18442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-18442: Fix Version/s: 1.16.0 > Move `testSessionWindowsWithContinuousEventTimeTrigger` to > `ContinuousEventTimeTriggerTest` > --- > > Key: FLINK-18442 > URL: https://issues.apache.org/jira/browse/FLINK-18442 > Project: Flink > Issue Type: Improvement > Components: Tests >Reporter: Lijie Wang >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned > Fix For: 1.15.0, 1.16.0 > > > `testSessionWindowsWithContinuousEventTimeTrigger` in `WindowOperatorTest` is > introduced when fix > [FLINK-4862|https://issues.apache.org/jira/browse/FLINK-4862]. > But it's better to move `testSessionWindowsWithContinuousEventTimeTrigger` > into `ContinuousEventTimeTriggerTest`. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-15123) remove uniqueKeys from FlinkStatistic in blink planner
[
https://issues.apache.org/jira/browse/FLINK-15123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-15123:
Fix Version/s: 1.16.0
> remove uniqueKeys from FlinkStatistic in blink planner
> ---
>
> Key: FLINK-15123
> URL: https://issues.apache.org/jira/browse/FLINK-15123
> Project: Flink
> Issue Type: Technical Debt
> Components: Table SQL / Planner
>Reporter: godfrey he
>Priority: Major
> Labels: starter
> Fix For: 1.15.0, 1.16.0
>
> Attachments: b_5.txt
>
>
> {{uniqueKeys}} is a kind of constraint, it's unreasonable that {{uniqueKeys}}
> is a kind of statistic. so we should remove uniqueKeys from
> {{FlinkStatistic}} in blink planner. Some temporary solutions (e.g.
> {{RichTableSourceQueryOperation}}) should also be resolved after primaryKey
> is introduced in {{TableSchema}}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-21409) Add Avro to DataTypes & Serialization docs
[ https://issues.apache.org/jira/browse/FLINK-21409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-21409: Fix Version/s: 1.16.0 > Add Avro to DataTypes & Serialization docs > -- > > Key: FLINK-21409 > URL: https://issues.apache.org/jira/browse/FLINK-21409 > Project: Flink > Issue Type: Improvement > Components: API / Type Serialization System, Documentation, Formats > (JSON, Avro, Parquet, ORC, SequenceFile) >Reporter: Chesnay Schepler >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.15.0, 1.16.0 > > > The "Data Types & Serialization" barely mention Avro, which is surprising > given how common it is. > Even basic things like how to create a correct TypeInformation for > GenericRecords is missing, or special cases like FLINK-21386 which likely > just won't work. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-12491) Incorrect documentation for directory path separators of CoreOptions.TMP_DIRS
[
https://issues.apache.org/jira/browse/FLINK-12491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-12491:
Fix Version/s: 1.16.0
> Incorrect documentation for directory path separators of CoreOptions.TMP_DIRS
> -
>
> Key: FLINK-12491
> URL: https://issues.apache.org/jira/browse/FLINK-12491
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Configuration
>Affects Versions: 1.6.4, 1.7.2, 1.8.0, 1.9.0
>Reporter: Kezhu Wang
>Priority: Minor
> Labels: auto-unassigned, pull-request-available
> Fix For: 1.15.0, 1.16.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{CoreOptions.TMP_DIRS}} and {{ConfigConstants.TASK_MANAGER_TMP_DIR_KEY}}
> both say that:
> {quote}
> The config parameter defining the directories for temporary files, separated
> by
>* ",", "|", or the system's \{@link java.io.File#pathSeparator}.
> {quote}
> But the parsing phase uses {{String.split}} with argument {{",|" +
> File.pathSeparator}} eventually. However, in fact the sole parameter of
> {{String.split}} is a regular expression, so the directory path separators
> are "," or {{java.io.File#pathSeparator}}. After digging into history, I
> found that the documentation was introduced in commit
> {{a7c407ace4f6cbfbde3e247071cee5a755ae66db}} and inherited by
> {{76abcaa55d0d6ab704b7ab8164718e8e2dcae2c4}}. So, I think it is safe to drop
> "|" from documentation.
> {code:title=ConfigurationUtils.java}
> public class ConfigurationUtils {
> private static String[] splitPaths(@Nonnull String separatedPaths) {
> return separatedPaths.length() > 0 ? separatedPaths.split(",|"
> + File.pathSeparator) : EMPTY;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-24517) Streamline Flink releases
[ https://issues.apache.org/jira/browse/FLINK-24517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-24517: Fix Version/s: 1.16.0 > Streamline Flink releases > - > > Key: FLINK-24517 > URL: https://issues.apache.org/jira/browse/FLINK-24517 > Project: Flink > Issue Type: Technical Debt > Components: Release System >Reporter: Chesnay Schepler >Priority: Major > Fix For: 1.15.0, 1.16.0 > > > Collection of changes that I'd like to make based on recent experiences with > the 1.13.3 release. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-24666) The lenient option was not exposed to users on RetractableTopNFunction
[
https://issues.apache.org/jira/browse/FLINK-24666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-24666:
Fix Version/s: 1.16.0
> The lenient option was not exposed to users on RetractableTopNFunction
> --
>
> Key: FLINK-24666
> URL: https://issues.apache.org/jira/browse/FLINK-24666
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
>Affects Versions: 1.14.0, 1.13.3
>Reporter: lincoln lee
>Priority: Major
> Fix For: 1.15.0, 1.16.0
>
>
> Currently, the lenient option was not exposed to users on
> RetractableTopNFunction
> {quote}// flag to skip records with non-exist error instead to fail, true by
> default.
> private final boolean lenient = true
> {quote}
> So there's no chance to raise the exception when the record(s) unexpectedly
> cleared by state ttl. Commonly this happens because a too shorter ttl at Line
> 190 or inconstancy between the two internal state(dataState and treeMap) at
> other place.
> {quote}List inputs = dataState.get(key);
> if (inputs == null) {
> // Skip the data if it's state is cleared because of state ttl.
> if (lenient)
> Unknown macro: \{ LOG.warn(STATE_CLEARED_WARN_MSG); }
> else
> Unknown macro: \{ throw new RuntimeException(STATE_CLEARED_WARN_MSG); }
> }
> {quote}
> We'd better to expose it to users (default value can be true to keep
> consistent with previous version)
>
> And completely resolve the inconsistency problem (it's different from Line
> 190 which is uncontrollable by itself) between the two state as follows in
> another issue.
> {quote}// a map state stores mapping from sort key to records list
> private transient MapState> dataState;
> // a sorted map stores mapping from sort key to records count
> private transient ValueState> treeMap
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-23970) Split off the behaviour for finished StreamTask(s)
[ https://issues.apache.org/jira/browse/FLINK-23970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-23970: Fix Version/s: 1.16.0 > Split off the behaviour for finished StreamTask(s) > -- > > Key: FLINK-23970 > URL: https://issues.apache.org/jira/browse/FLINK-23970 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing >Reporter: Yun Gao >Priority: Major > Fix For: 1.15.0, 1.16.0 > > > We will have a double check on how we could better abstract the behavior of > the tasks marked as finished on recovery, the target is to make the behaviors > implemented as centralized as we can (like introducing specialized > _FinishedStreamTask_). -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-20036) Join Has NoUniqueKey when using mini-batch
[
https://issues.apache.org/jira/browse/FLINK-20036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-20036:
Fix Version/s: 1.16.0
> Join Has NoUniqueKey when using mini-batch
> --
>
> Key: FLINK-20036
> URL: https://issues.apache.org/jira/browse/FLINK-20036
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Affects Versions: 1.11.2
>Reporter: Rex Remind
>Priority: Minor
> Labels: auto-deprioritized-major
> Fix For: 1.15.0, 1.16.0
>
>
> Hello,
>
> We tried out mini-batch mode and our Join suddenly had NoUniqueKey.
> Join:
> {code:java}
> Table membershipsTable = tableEnv.from(SOURCE_MEMBERSHIPS)
> .renameColumns($("id").as("membership_id"))
> .select($("*")).join(usersTable, $("user_id").isEqual($("id")));
> {code}
> Mini-batch config:
> {code:java}
> configuration.setString("table.exec.mini-batch.enabled", "true"); // enable
> mini-batch optimization
> configuration.setString("table.exec.mini-batch.allow-latency", "5 s"); // use
> 5 seconds to buffer input records
> configuration.setString("table.exec.mini-batch.size", "5000"); // the maximum
> number of records can be buffered by each aggregate operator task
> {code}
>
> Join with mini-batch:
> {code:java}
> Join(joinType=[InnerJoin], where=[(user_id = id0)], select=[id,
> group_id, user_id, uuid, owner, id0, deleted_at],
> leftInputSpec=[NoUniqueKey], rightInputSpec=[NoUniqueKey])
> {code}
> Join without mini-batch:
> {code:java}
> Join(joinType=[InnerJoin], where=[(user_id = id0)], select=[id, group_id,
> user_id, uuid, owner, id0, deleted_at], leftInputSpec=[HasUniqueKey],
> rightInputSpec=[JoinKeyContainsUniqueKey])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-18777) Supports schema registry catalog
[ https://issues.apache.org/jira/browse/FLINK-18777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-18777: Fix Version/s: 1.16.0 > Supports schema registry catalog > > > Key: FLINK-18777 > URL: https://issues.apache.org/jira/browse/FLINK-18777 > Project: Flink > Issue Type: Bug > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.11.0 >Reporter: Danny Chen >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available > Fix For: 1.15.0, 1.16.0 > > > Design doc: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-125%3A+Confluent+Schema+Registry+Catalog -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-18199) Translate "Filesystem SQL Connector" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-18199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-18199: Fix Version/s: 1.16.0 > Translate "Filesystem SQL Connector" page into Chinese > -- > > Key: FLINK-18199 > URL: https://issues.apache.org/jira/browse/FLINK-18199 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Connectors / FileSystem, > Documentation, Table SQL / Ecosystem >Reporter: Jark Wu >Priority: Major > Labels: auto-unassigned, pull-request-available > Fix For: 1.15.0, 1.16.0 > > > The page url is > https://ci.apache.org/projects/flink/flink-docs-master/zh/dev/table/connectors/filesystem.html > The markdown file is located in > flink/docs/dev/table/connectors/filesystem.zh.md -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-23576) JoinITCase.testFullJoinWithEqualPkNonEqui fails due to Artificial Failure
[
https://issues.apache.org/jira/browse/FLINK-23576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-23576:
Fix Version/s: 1.16.0
> JoinITCase.testFullJoinWithEqualPkNonEqui fails due to Artificial Failure
> -
>
> Key: FLINK-23576
> URL: https://issues.apache.org/jira/browse/FLINK-23576
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.14.0
>Reporter: Xintong Song
>Priority: Major
> Labels: test-stability
> Fix For: 1.15.0, 1.16.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=21246&view=logs&j=f2c100be-250b-5e85-7bbe-176f68fcddc5&t=05efd11e-5400-54a4-0d27-a4663be008a9&l=9949
> {code}
> Jul 31 21:43:27 [ERROR] Tests run: 104, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 93.597 s <<< FAILURE! - in
> org.apache.flink.table.planner.runtime.stream.sql.JoinITCase
> Jul 31 21:43:27 [ERROR]
> testFullJoinWithEqualPkNonEqui[StateBackend=ROCKSDB](org.apache.flink.table.planner.runtime.stream.sql.JoinITCase)
> Time elapsed: 20.238 s <<< ERROR!
> Jul 31 21:43:27 org.apache.flink.runtime.client.JobExecutionException: Job
> execution failed.
> Jul 31 21:43:27 at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
> Jul 31 21:43:27 at
> org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:137)
> Jul 31 21:43:27 at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
> Jul 31 21:43:27 at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
> Jul 31 21:43:27 at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> Jul 31 21:43:27 at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> Jul 31 21:43:27 at
> org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:250)
> Jul 31 21:43:27 at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> Jul 31 21:43:27 at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> Jul 31 21:43:27 at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> Jul 31 21:43:27 at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> Jul 31 21:43:27 at
> org.apache.flink.util.concurrent.FutureUtils.doForward(FutureUtils.java:1389)
> Jul 31 21:43:27 at
> org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$null$1(ClassLoadingUtils.java:93)
> Jul 31 21:43:27 at
> org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
> Jul 31 21:43:27 at
> org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$2(ClassLoadingUtils.java:92)
> Jul 31 21:43:27 at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> Jul 31 21:43:27 at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> Jul 31 21:43:27 at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> Jul 31 21:43:27 at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> Jul 31 21:43:27 at
> org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$1.onComplete(AkkaFutureUtils.java:47)
> Jul 31 21:43:27 at akka.dispatch.OnComplete.internal(Future.scala:300)
> Jul 31 21:43:27 at akka.dispatch.OnComplete.internal(Future.scala:297)
> Jul 31 21:43:27 at
> akka.dispatch.japi$CallbackBridge.apply(Future.scala:224)
> Jul 31 21:43:27 at
> akka.dispatch.japi$CallbackBridge.apply(Future.scala:221)
> Jul 31 21:43:27 at
> scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
> Jul 31 21:43:27 at
> org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$DirectExecutionContext.execute(AkkaFutureUtils.java:65)
> Jul 31 21:43:27 at
> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
> Jul 31 21:43:27 at
> scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284)
> Jul 31 21:43:27 at
> scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284)
> Jul 31 21:43:27 at
> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284)
> Jul 31 21:43:27 at
> akka.pattern.PromiseActorRef.$bang(AskSupport.scala:621)
> Jul 31 21:43:27 at
> akka.pattern.PipeToSupport$PipeableFuture$$anonfun$
[jira] [Updated] (FLINK-20656) Update docs for new KafkaSource connector.
[ https://issues.apache.org/jira/browse/FLINK-20656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-20656: Fix Version/s: 1.16.0 > Update docs for new KafkaSource connector. > -- > > Key: FLINK-20656 > URL: https://issues.apache.org/jira/browse/FLINK-20656 > Project: Flink > Issue Type: New Feature > Components: Connectors / Kafka >Affects Versions: 1.12.0 >Reporter: Jiangjie Qin >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned > Fix For: 1.15.0, 1.12.8, 1.16.0 > > > We need to add docs for the KafkaSource connector. Namely the following page: > https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/connectors/kafka.html -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-20834) Add metrics for reporting the memory usage and CPU usage of the Python UDF workers
[ https://issues.apache.org/jira/browse/FLINK-20834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-20834: Fix Version/s: 1.16.0 > Add metrics for reporting the memory usage and CPU usage of the Python UDF > workers > -- > > Key: FLINK-20834 > URL: https://issues.apache.org/jira/browse/FLINK-20834 > Project: Flink > Issue Type: Improvement > Components: API / Python >Reporter: Wei Zhong >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.15.0, 1.16.0 > > > Currently these is no official approach to access the memory usage and CPU > usage of the Python UDF workers. We need to add these metrics to monitor the > running status of the Python processes. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-24949) KafkaITCase.testBigRecordJob fails on azure
[
https://issues.apache.org/jira/browse/FLINK-24949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-24949:
Fix Version/s: 1.16.0
> KafkaITCase.testBigRecordJob fails on azure
> ---
>
> Key: FLINK-24949
> URL: https://issues.apache.org/jira/browse/FLINK-24949
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.14.1
>Reporter: Yun Gao
>Assignee: Fabian Paul
>Priority: Major
> Labels: stale-assigned, test-stability
> Fix For: 1.15.0, 1.16.0
>
>
> {code:java}
> Nov 17 23:39:39 [ERROR] Tests run: 23, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 222.57 s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.kafka.KafkaITCase
> Nov 17 23:39:39 [ERROR] testBigRecordJob Time elapsed: 60.02 s <<< ERROR!
> Nov 17 23:39:39 org.junit.runners.model.TestTimedOutException: test timed out
> after 6 milliseconds
> Nov 17 23:39:39 at sun.misc.Unsafe.park(Native Method)
> Nov 17 23:39:39 at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> Nov 17 23:39:39 at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> Nov 17 23:39:39 at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> Nov 17 23:39:39 at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> Nov 17 23:39:39 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> Nov 17 23:39:39 at
> org.apache.flink.test.util.TestUtils.tryExecute(TestUtils.java:58)
> Nov 17 23:39:39 at
> org.apache.flink.streaming.connectors.kafka.KafkaConsumerTestBase.runBigRecordTestTopology(KafkaConsumerTestBase.java:1473)
> Nov 17 23:39:39 at
> org.apache.flink.streaming.connectors.kafka.KafkaITCase.testBigRecordJob(KafkaITCase.java:119)
> Nov 17 23:39:39 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Nov 17 23:39:39 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Nov 17 23:39:39 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Nov 17 23:39:39 at java.lang.reflect.Method.invoke(Method.java:498)
> Nov 17 23:39:39 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Nov 17 23:39:39 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Nov 17 23:39:39 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> Nov 17 23:39:39 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Nov 17 23:39:39 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> Nov 17 23:39:39 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> Nov 17 23:39:39 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> Nov 17 23:39:39 at java.lang.Thread.run(Thread.java:748)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26679&view=logs&j=c5f0071e-1851-543e-9a45-9ac140befc32&t=15a22db7-8faa-5b34-3920-d33c9f0ca23c&l=7161
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-19034) Remove deprecated StreamExecutionEnvironment#set/getNumberOfExecutionRetries
[
https://issues.apache.org/jira/browse/FLINK-19034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-19034:
Fix Version/s: 1.16.0
> Remove deprecated StreamExecutionEnvironment#set/getNumberOfExecutionRetries
>
>
> Key: FLINK-19034
> URL: https://issues.apache.org/jira/browse/FLINK-19034
> Project: Flink
> Issue Type: Sub-task
> Components: API / DataStream
>Reporter: Dawid Wysakowicz
>Assignee: Daisy Tsang
>Priority: Major
> Labels: auto-unassigned
> Fix For: 1.15.0, 1.16.0
>
>
> Remove deprecated
> {code}
> StreamExecutionEnvironment#setNumberOfExecutionRetries/getNumberOfExecutionRetries
> {code}
> The corresponding settings in {{ExecutionConfig}} will be removed in a
> separate issue, as they are {{Public}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-23254) Upgrade Test Framework to JUnit 5
[
https://issues.apache.org/jira/browse/FLINK-23254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-23254:
Fix Version/s: 1.16.0
> Upgrade Test Framework to JUnit 5
> -
>
> Key: FLINK-23254
> URL: https://issues.apache.org/jira/browse/FLINK-23254
> Project: Flink
> Issue Type: New Feature
> Components: Tests
>Affects Versions: 1.14.0
>Reporter: Qingsheng Ren
>Priority: Major
> Fix For: 1.15.0, 1.16.0
>
>
> Please see mailing list discussion about the background of this upgrade.
> [{color:#33}https://lists.apache.org/thread.html/r6c8047c7265b8a9f2cb3ef6d6153dd80b94d36ebb03daccf36ab4940%40%3Cdev.flink.apache.org%3E{color}]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-23849) Support react to the node decommissioning change state on yarn and do graceful restart
[ https://issues.apache.org/jira/browse/FLINK-23849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-23849: Fix Version/s: 1.16.0 > Support react to the node decommissioning change state on yarn and do > graceful restart > -- > > Key: FLINK-23849 > URL: https://issues.apache.org/jira/browse/FLINK-23849 > Project: Flink > Issue Type: New Feature > Components: Deployment / YARN >Reporter: zlzhang0122 >Priority: Major > Fix For: 1.15.0, 1.16.0 > > > Now we are not interested in node updates in > YarnContainerEventHandler.onNodesUpdated , but sometimes we want to evict the > running flink process on one node and graceful restart on the other node > because of some unexpected reason such as the physical machine need to be > recycle or the cloud computing cluster need to be migration. Thus, we can > react to the node decommissioning change state, and call the > stopWithSavepoint function and then restart it. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-17642) Exception while reading broken ORC file is hidden
[
https://issues.apache.org/jira/browse/FLINK-17642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-17642:
Fix Version/s: 1.16.0
> Exception while reading broken ORC file is hidden
> -
>
> Key: FLINK-17642
> URL: https://issues.apache.org/jira/browse/FLINK-17642
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / ORC
>Affects Versions: 1.8.3, 1.9.3, 1.10.1
>Reporter: Nikola
>Priority: Minor
> Labels: auto-deprioritized-major
> Fix For: 1.10.4, 1.15.0, 1.11.7, 1.16.0
>
>
> I have a simple setup of a batch job like this:
> {code:java}
> BatchTableEnvironment tableEnvFirst = BatchTableEnvironment.create(env);
> OrcTableSource orcTableSource = OrcTableSource.builder()
> .path("path", true)
> .forOrcSchema(ORC.getSchema())
> .withConfiguration(hdfsConfig)
> .build();
> tableEnvFirst.registerTableSource("table", orcTableSource);
> Table nnfTable = tableEnvFirst.sqlQuery(sqlString);
> return tableEnvFirst.toDataSet(nnfTable, Row.class);{code}
>
> And that works just fine to fetch ORC files from hdfs as a DataSet.
> However, there are some ORC files which are broken. "Broken" means that they
> are invalid in some way and cannot be processed / fetch normally. They throw
> exceptions. Examples of those are:
> {code:java}
> org.apache.orc.FileFormatException: Malformed ORC file /user/hdfs/orcfile-1
> Invalid postscript length 2
> at org.apache.orc.impl.ReaderImpl.ensureOrcFooter(ReaderImpl.java:258)
> at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:562)
> at org.apache.orc.impl.ReaderImpl.(ReaderImpl.java:370)
> at org.apache.orc.OrcFile.createReader(OrcFile.java:342)
> at org.apache.flink.orc.OrcRowInputFormat.open(OrcRowInputFormat.java:225)
> at org.apache.flink.orc.OrcRowInputFormat.open(OrcRowInputFormat.java:63)
> at
> org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:173)
>
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530)
> at java.lang.Thread.run(Thread.java:748){code}
>
> {code:java}
> com.google.protobuf.InvalidProtocolBufferException: Protocol message
> contained an invalid tag (zero).
> at
> com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:89)
>
> at com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:108)
> at org.apache.orc.OrcProto$PostScript.(OrcProto.java:18526)
> at org.apache.orc.OrcProto$PostScript.(OrcProto.java:18490)
> at org.apache.orc.OrcProto$PostScript$1.parsePartialFrom(OrcProto.java:18628)
> at org.apache.orc.OrcProto$PostScript$1.parsePartialFrom(OrcProto.java:18623)
> at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:89)
> at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:95)
> at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
> at org.apache.orc.OrcProto$PostScript.parseFrom(OrcProto.java:19022)
> at org.apache.orc.impl.ReaderImpl.extractPostScript(ReaderImpl.java:436)
> at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:564)
> at org.apache.orc.impl.ReaderImpl.(ReaderImpl.java:370)
> at org.apache.orc.OrcFile.createReader(OrcFile.java:342)
> at org.apache.flink.orc.OrcRowInputFormat.open(OrcRowInputFormat.java:225)
> at org.apache.flink.orc.OrcRowInputFormat.open(OrcRowInputFormat.java:63)
> at
> org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:173)
>
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530)
> at java.lang.Thread.run(Thread.java:748){code}
>
> Given that some specific files are broken, that's OK to throw exception.
> However, the issue is that I cannot catch those exceptions and they make my
> job to fail. I tried to wrap everything in a try-catch block just to see what
> I can catch and handle, but it seems that when flink runs it, it's not run
> from that place, but rather from DataSourceTask.invoke()
> I can digged a little bit to find out why don't I get an exception and I can
> see that {{OrcTableSource}} creates {{OrcRowInputFormat}} instance
> [here|#L157]] which then calls open() and open() has this signature:
> {code:java}
> public void open(FileInputSplit fileSplit) throws IOException {{code}
>
> So the open() throws the exception but I am not able to catch it.
> Is what I am doing correct or is there any other way to handle exception
> coming from DataSourceTask.invoke()? In general my goal would be to ignore
> all broken/corrupted ORC files but that does not seem to be possible
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-21613) Parse Compute Column with `IN` expression throws NPE
[
https://issues.apache.org/jira/browse/FLINK-21613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-21613:
Fix Version/s: 1.16.0
> Parse Compute Column with `IN` expression throws NPE
>
>
> Key: FLINK-21613
> URL: https://issues.apache.org/jira/browse/FLINK-21613
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.13.0
>Reporter: Shuo Cheng
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned
> Fix For: 1.15.0, 1.16.0
>
>
> Considering the following given sql:
> {code:sql}
> CREATE TABLE MyInputFormatTable (
> `a` INT,
> `b` BIGINT,
> `c` STRING,
> `d` as `c` IN ('Hi', 'Hello')
> ) WITH (
> 'connector' = 'values',
> 'data-id' = '$dataId',
> 'runtime-source' = 'InputFormat'
> )
> {code}
> NPE will be thrown during parsing the sql:
> `select * from MyInputFormatTable`
> It seems it's the commit "[hotfix][table-planner-blink] Simplify SQL
> expression to RexNode conversion" which introduces this problem. This hotfix
> uses a method `SqlToRelConverter#convertExpression` and this method does not
> has any tests and is not used in Calcite anywhere, which is unsafe. Maybe
> reverting the hotfix is a good choice.
> CC [~twalthr]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-19362) Remove confusing comment for `DOT` operator codegen
[
https://issues.apache.org/jira/browse/FLINK-19362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-19362:
Fix Version/s: 1.16.0
> Remove confusing comment for `DOT` operator codegen
> ---
>
> Key: FLINK-19362
> URL: https://issues.apache.org/jira/browse/FLINK-19362
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
>Affects Versions: 1.11.0
>Reporter: hailong wang
>Priority: Minor
> Labels: auto-deprioritized-major
> Fix For: 1.15.0, 1.16.0
>
>
> `DOT` operator codegen (ExprCodeGenerator#generateCallExpression) has comment
> as following:
> {code:java}
> // due to https://issues.apache.org/jira/browse/CALCITE-2162, expression such
> as
> // "array[1].a.b" won't work now.
> if (operands.size > 2) {
> throw new CodeGenException(
> "A DOT operator with more than 2 operands is not supported yet.")
> }
> {code}
> But `array[1].a.b` actually can work for flink job. `DOT` will be transform
> to `RexFieldAccess` for CALCITE-2542. And `generateDot` will never be invoked
> except suppporting ITEM for ROW types.
> Simply, I think we can only delete the comment which is confusing.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-24536) flink sql where condition is not supported != writing
[
https://issues.apache.org/jira/browse/FLINK-24536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-24536:
Fix Version/s: 1.16.0
> flink sql where condition is not supported != writing
> --
>
> Key: FLINK-24536
> URL: https://issues.apache.org/jira/browse/FLINK-24536
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Affects Versions: 1.14.0
>Reporter: simenliuxing
>Assignee: liwei li
>Priority: Minor
> Labels: pull-request-available, stale-assigned
> Fix For: 1.15.0, 1.16.0
>
>
> sql:
>
> {code:java}
> CREATE TABLE source
> (
> id INT,
> name STRING,
> money DECIMAL(32, 2),
> dateone timestamp,
> age bigint,
> datethree timestamp,
> datesix timestamp(6),
> datenigth timestamp(9),
> dtdate date,
> dttime time
> ) WITH (
> 'connector' = 'datagen'
> ,'rows-per-second' = '1'
> );
> CREATE TABLE sink
> (
> id bigint,
> name STRING
> ) WITH (
> 'connector' = 'print'
> );
> insert into sink
> select sum(id) as id, name
> from source
> where name != 'aa'
> group by name;
> {code}
>
> exception:
>
> {code:java}
> Caused by: org.apache.calcite.sql.parser.SqlParseException: Bang equal '!='
> is not allowed under the current SQL conformance levelCaused by:
> org.apache.calcite.sql.parser.SqlParseException: Bang equal '!=' is not
> allowed under the current SQL conformance level at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.convertException(FlinkSqlParserImpl.java:462)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.normalizeException(FlinkSqlParserImpl.java:225)
> at
> org.apache.calcite.sql.parser.SqlParser.handleException(SqlParser.java:140)
> at org.apache.calcite.sql.parser.SqlParser.parseQuery(SqlParser.java:155) at
> org.apache.calcite.sql.parser.SqlParser.parseStmt(SqlParser.java:180) at
> org.apache.flink.table.planner.parse.CalciteParser.parse(CalciteParser.java:54)
> ... 22 moreCaused by: org.apache.calcite.runtime.CalciteException: Bang
> equal '!=' is not allowed under the current SQL conformance level at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
> org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:467)
> at org.apache.calcite.runtime.Resources$ExInst.ex(Resources.java:560) at
> org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:883) at
> org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:868) at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.BinaryRowOperator(FlinkSqlParserImpl.java:31759)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.Expression2(FlinkSqlParserImpl.java:19802)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.Expression(FlinkSqlParserImpl.java:19553)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.WhereOpt(FlinkSqlParserImpl.java:14370)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.SqlSelect(FlinkSqlParserImpl.java:7836)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.LeafQuery(FlinkSqlParserImpl.java:704)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.LeafQueryOrExpr(FlinkSqlParserImpl.java:19536)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.QueryOrExpr(FlinkSqlParserImpl.java:18982)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.OrderedQueryOrExpr(FlinkSqlParserImpl.java:578)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.RichSqlInsert(FlinkSqlParserImpl.java:5596)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.SqlStmt(FlinkSqlParserImpl.java:3404)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.SqlStmtEof(FlinkSqlParserImpl.java:3980)
> at
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.parseSqlStmtEof(FlinkSqlParserImpl.java:273)
> at org.apache.calcite.sql.parser.SqlParser.parseQuery(SqlParser.java:153)
> ... 24 more{code}
>
> It is ok when I use the following syntax：
> where name <> 'aa'
> Why not support '!=' This kind of grammar, will it be supported later?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-21125) Sum or Sum0 overflow quietly
[
https://issues.apache.org/jira/browse/FLINK-21125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-21125:
Fix Version/s: 1.16.0
> Sum or Sum0 overflow quietly
>
>
> Key: FLINK-21125
> URL: https://issues.apache.org/jira/browse/FLINK-21125
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Reporter: Sebastian Liu
>Priority: Minor
> Labels: auto-deprioritized-major
> Fix For: 1.15.0, 1.16.0
>
>
> Overflow is not calculated correctly in the build-in sum function of Blink
> planner.
> For a aggregate calculation such as:
> {code:java}
> CREATE TABLE TestTable (
> amount INT
> );
> insert into TestTable (2147483647);
> insert into TestTable (1);
> SELECT sum(amount) FROM TestTable;
> The result will be: -2147483648, which is an overflowed value and no
> exception was thrown. {code}
> The overflow occurs quietly and is difficult to detect.
> Compare the processing semantics of other systems：
> * *mysql*: provide two SQL mode for handling overflow. If strict SQL mode is
> enabled, MySQL rejects the out-of-range value with an error, and the insert
> fails, in accordance with the SQL standard. If no restrictive modes are
> enabled, MySQL clips the value to the appropriate endpoint of the column data
> type range and stores the resulting value instead. FYI:
> [https://dev.mysql.com/doc/refman/8.0/en/out-of-range-and-overflow.html]
> * *presto*: all numeric types are automatically cast to Long type, and If
> the long is out of range, an exception is thrown to prompt user.
> IMO, exception hint is necessary, instead of quietly overflow.
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
