[jira] [Updated] (FLINK-32701) Potential Memory Leak in Flink CEP due to Persistent Starting States in NFAState
[
https://issues.apache.org/jira/browse/FLINK-32701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Puneet Duggal updated FLINK-32701:
--
Affects Version/s: 1.16.2
1.16.1
> Potential Memory Leak in Flink CEP due to Persistent Starting States in
> NFAState
>
>
> Key: FLINK-32701
> URL: https://issues.apache.org/jira/browse/FLINK-32701
> Project: Flink
> Issue Type: Bug
> Components: Library / CEP
>Affects Versions: 1.17.0, 1.16.1, 1.16.2, 1.17.1
>Reporter: Puneet Duggal
>Priority: Critical
> Attachments: Screenshot 2023-07-26 at 11.45.06 AM.png, Screenshot
> 2023-07-26 at 11.50.28 AM.png
>
>
> Our team has encountered a potential memory leak issue while working with the
> Complex Event Processing (CEP) library in Flink v1.17.
> h2. Context
> The CEP Operator maintains a keyed state called NFAState, which holds two
> queues: one for partial matches and one for completed matches. When a key is
> first encountered, the CEP creates a starting computation state and stores it
> in the partial matches queue. As more events occur that match the defined
> conditions (e.g., a TAKE condition), additional computation states get added
> to the queue, with their specific type (normal, pending, end) depending on
> the pattern sequence.
> However, I have noticed that the starting computation state remains in the
> partial matches queue even after the pattern sequence has been completely
> matched. This is also the case for keys that have already timed out. As a
> result, the state gets stored for all keys that the CEP ever encounters,
> leading to a continual increase in the checkpoint size.
> h2. How to reproduce this
> # Pattern Sequence - A not_followed_by B within 5 mins
> # Time Characteristic - EventTime
> # StateBackend - FsStateBackend
> On my local machine, I started this pipeline and started sending events at
> the rate of 10 events per second (only A) and as expected after 5 mins, CEP
> started sending pattern matched output with the same rate. But the issue was
> that after every 2 mins (checkpoint interval), checkpoint size kept on
> increasing. Expectation was that after 5 mins (2-3 checkpoints), checkpoint
> size will remain constant since any window of 5 mins will consist of the same
> number of unique keys (older ones will get matched or timed out hence removed
> from state). But as you can see below attached images, checkpoint size kept
> on increasing till 40 checkpoints (around 1.5hrs).
> P.S. - After 3 checkpoints (6 mins), the checkpoint size was around 1.78MB.
> Hence assumption is that ideal checkpoint size for a 5 min window should be
> less than 1.78MB.
> As you can see after 39 checkpoints, I triggered a savepoint for this
> pipeline. After that I used a savepoint reader to investigate what all is
> getting stored in CEP states. Below code investigates NFAState of CEPOperator
> for potential memory leak.
> {code:java}
> import lombok.AllArgsConstructor;
> import lombok.Data;
> import lombok.NoArgsConstructor;
> import org.apache.flink.api.common.state.ValueState;
> import org.apache.flink.api.common.state.ValueStateDescriptor;
> import org.apache.flink.cep.nfa.NFAState;
> import org.apache.flink.cep.nfa.NFAStateSerializer;
> import org.apache.flink.configuration.Configuration;
> import org.apache.flink.runtime.state.filesystem.FsStateBackend;
> import org.apache.flink.state.api.OperatorIdentifier;
> import org.apache.flink.state.api.SavepointReader;
> import org.apache.flink.state.api.functions.KeyedStateReaderFunction;
> import org.apache.flink.streaming.api.datastream.DataStream;
> import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
> import org.apache.flink.util.Collector;
> import org.junit.jupiter.api.Test;
> import java.io.Serializable;
> import java.util.Objects;
> public class NFAStateReaderTest {
> private static final String NFA_STATE_NAME = "nfaStateName";
> @Test
> public void testNfaStateReader() throws Exception {
> StreamExecutionEnvironment environment =
> StreamExecutionEnvironment.getExecutionEnvironment();
> SavepointReader savepointReader =
> SavepointReader.read(environment,
> "file:///opt/flink/savepoints/savepoint-093404-9bc0a38654df", new
> FsStateBackend("file:///abc"));
> DataStream stream =
> savepointReader.readKeyedState(OperatorIdentifier.forUid("select_pattern_events"),
> new NFAStateReaderTest.NFAStateReaderFunction());
> stream.print();
> environment.execute();
> }
> static class NFAStateReaderFunction extends
> KeyedStateReaderFunction {
> private ValueState computationStates;
> private static Long danglingNfaCount = 0L;
> private static
[jira] [Commented] (FLINK-32809) YarnClusterDescriptor.isarchiveonlyincludedinShipArchiveFiles dose not work as expected
[
https://issues.apache.org/jira/browse/FLINK-32809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17753044#comment-17753044
]
Rui Fan commented on FLINK-32809:
-
Thanks for the contribution! :)
Merged via ac85945947e11b2278c9fcdb24a6a6d695621ac4
> YarnClusterDescriptor.isarchiveonlyincludedinShipArchiveFiles dose not work
> as expected
> ---
>
> Key: FLINK-32809
> URL: https://issues.apache.org/jira/browse/FLINK-32809
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.18.0
>Reporter: junzhong qin
>Assignee: junzhong qin
>Priority: Major
> Labels: pull-request-available
>
> YarnClusterDescriptor.isarchiveonlyincludedinShipArchiveFiles(List
> shipFiles) check wether the shipFiles are all archive files, but it dose not
> work as expected.
> {code:java}
> public static boolean isArchiveOnlyIncludedInShipArchiveFiles(List
> shipFiles) {
> return shipFiles.stream()
> .filter(File::isFile)
> .map(File::getName)
> .map(String::toLowerCase)
> .allMatch(
> name ->
> name.endsWith(".tar.gz")
> || name.endsWith(".tar")
> || name.endsWith(".tgz")
> || name.endsWith(".dst")
> || name.endsWith(".jar")
> || name.endsWith(".zip"));
> } {code}
> When we pass a directory and an archive file it should return false but it
> returns true.
> {code:java}
> // dir1 is a directory and archive.zip is an archive file
> List files = Arrays.asList(new File("/tmp/dir1"), new
> File("/tmp/archive.zip"));
> boolean result = isArchiveOnlyIncludedInShipArchiveFiles(files);
> System.out.println(result); // Print true but is should print false{code}
> If flink user want to add directory as ship file, they should use
> configuration: yarn.ship-files .
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-32809) YarnClusterDescriptor.isarchiveonlyincludedinShipArchiveFiles dose not work as expected
[
https://issues.apache.org/jira/browse/FLINK-32809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Fan resolved FLINK-32809.
-
Fix Version/s: 1.18.0
Resolution: Fixed
> YarnClusterDescriptor.isarchiveonlyincludedinShipArchiveFiles dose not work
> as expected
> ---
>
> Key: FLINK-32809
> URL: https://issues.apache.org/jira/browse/FLINK-32809
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.18.0
>Reporter: junzhong qin
>Assignee: junzhong qin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.0
>
>
> YarnClusterDescriptor.isarchiveonlyincludedinShipArchiveFiles(List
> shipFiles) check wether the shipFiles are all archive files, but it dose not
> work as expected.
> {code:java}
> public static boolean isArchiveOnlyIncludedInShipArchiveFiles(List
> shipFiles) {
> return shipFiles.stream()
> .filter(File::isFile)
> .map(File::getName)
> .map(String::toLowerCase)
> .allMatch(
> name ->
> name.endsWith(".tar.gz")
> || name.endsWith(".tar")
> || name.endsWith(".tgz")
> || name.endsWith(".dst")
> || name.endsWith(".jar")
> || name.endsWith(".zip"));
> } {code}
> When we pass a directory and an archive file it should return false but it
> returns true.
> {code:java}
> // dir1 is a directory and archive.zip is an archive file
> List files = Arrays.asList(new File("/tmp/dir1"), new
> File("/tmp/archive.zip"));
> boolean result = isArchiveOnlyIncludedInShipArchiveFiles(files);
> System.out.println(result); // Print true but is should print false{code}
> If flink user want to add directory as ship file, they should use
> configuration: yarn.ship-files .
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] 1996fanrui merged pull request #23191: [FLINK-32809][yarn] Fixes YarnClusterDescriptor#isArchiveOnlyIncluded…
1996fanrui merged PR #23191: URL: https://github.com/apache/flink/pull/23191 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-32809) YarnClusterDescriptor.isarchiveonlyincludedinShipArchiveFiles dose not work as expected
[
https://issues.apache.org/jira/browse/FLINK-32809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Fan updated FLINK-32809:
Affects Version/s: (was: 1.16.2)
(was: 1.17.1)
> YarnClusterDescriptor.isarchiveonlyincludedinShipArchiveFiles dose not work
> as expected
> ---
>
> Key: FLINK-32809
> URL: https://issues.apache.org/jira/browse/FLINK-32809
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.18.0
>Reporter: junzhong qin
>Assignee: junzhong qin
>Priority: Major
> Labels: pull-request-available
>
> YarnClusterDescriptor.isarchiveonlyincludedinShipArchiveFiles(List
> shipFiles) check wether the shipFiles are all archive files, but it dose not
> work as expected.
> {code:java}
> public static boolean isArchiveOnlyIncludedInShipArchiveFiles(List
> shipFiles) {
> return shipFiles.stream()
> .filter(File::isFile)
> .map(File::getName)
> .map(String::toLowerCase)
> .allMatch(
> name ->
> name.endsWith(".tar.gz")
> || name.endsWith(".tar")
> || name.endsWith(".tgz")
> || name.endsWith(".dst")
> || name.endsWith(".jar")
> || name.endsWith(".zip"));
> } {code}
> When we pass a directory and an archive file it should return false but it
> returns true.
> {code:java}
> // dir1 is a directory and archive.zip is an archive file
> List files = Arrays.asList(new File("/tmp/dir1"), new
> File("/tmp/archive.zip"));
> boolean result = isArchiveOnlyIncludedInShipArchiveFiles(files);
> System.out.println(result); // Print true but is should print false{code}
> If flink user want to add directory as ship file, they should use
> configuration: yarn.ship-files .
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] 1996fanrui commented on pull request #23191: [FLINK-32809][yarn] Fixes YarnClusterDescriptor#isArchiveOnlyIncluded…
1996fanrui commented on PR #23191: URL: https://github.com/apache/flink/pull/23191#issuecomment-1674209284 Thanks @RocMarshal for the review, merging~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-32831) RuntimeFilterProgram should aware join type when looking for the build side
[
https://issues.apache.org/jira/browse/FLINK-32831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lijie Wang reassigned FLINK-32831:
--
Assignee: Lijie Wang
> RuntimeFilterProgram should aware join type when looking for the build side
> ---
>
> Key: FLINK-32831
> URL: https://issues.apache.org/jira/browse/FLINK-32831
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.18.0
>Reporter: Lijie Wang
>Assignee: Lijie Wang
>Priority: Major
>
> Currently, runtime filter program will try to look for an {{Exchange}} as
> build side to avoid affecting {{MultiInput}}. It will try to push down the
> runtime filter builder if the original build side is not {{Exchange}}.
> Currenlty, the builder-push-down does not aware the join type, which may lead
> to incorrect results(For example, push down the builder to the right input of
> left-join).
> We should only support following cases:
> 1. Inner join: builder can push to left + right input
> 2. semi join: builder can push to left + right input
> 3. left join: builder can only push to the left input
> 4. right join: builder can only push to the right input
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] qinf commented on a diff in pull request #23191: [FLINK-32809][yarn] Fixes YarnClusterDescriptor#isArchiveOnlyIncluded…
qinf commented on code in PR #23191:
URL: https://github.com/apache/flink/pull/23191#discussion_r1290869793
##
flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java:
##
@@ -305,23 +305,26 @@ public void addShipFiles(List shipFiles) {
private void addShipArchives(List shipArchives) {
checkArgument(
isArchiveOnlyIncludedInShipArchiveFiles(shipArchives),
-"Non-archive files are included.");
+"Directories or non-archive files are included.");
this.shipArchives.addAll(shipArchives);
}
private static boolean isArchiveOnlyIncludedInShipArchiveFiles(List
shipFiles) {
-return shipFiles.stream()
-.filter(File::isFile)
-.map(File::getName)
-.map(String::toLowerCase)
-.allMatch(
-name ->
-name.endsWith(".tar.gz")
-|| name.endsWith(".tar")
-|| name.endsWith(".tgz")
-|| name.endsWith(".dst")
-|| name.endsWith(".jar")
-|| name.endsWith(".zip"));
+long archivedFileCount =
+shipFiles.stream()
+.filter(File::isFile)
+.map(File::getName)
+.map(String::toLowerCase)
+.filter(
+name ->
+name.endsWith(".tar.gz")
+|| name.endsWith(".tar")
+|| name.endsWith(".tgz")
+|| name.endsWith(".dst")
+|| name.endsWith(".jar")
+|| name.endsWith(".zip"))
Review Comment:
@RocMarshal The archive types have not changed for over 2 years, maybe it is
a good time to do this when there is a new archive type need to add here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-32831) RuntimeFilterProgram should aware join type when looking for the build side
[
https://issues.apache.org/jira/browse/FLINK-32831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lijie Wang updated FLINK-32831:
---
Description:
Currently, runtime filter program will try to look for an {{Exchange}} as build
side to avoid affecting {{MultiInput}}. It will try to push down the runtime
filter builder if the original build side is not {{Exchange}}.
Currenlty, the builder-push-down does not aware the join type, which may lead
to incorrect results(For example, push down the builder to the right input of
left-join).
We should only support following cases:
1. Inner join: builder can push to left + right input
2. semi join: builder can push to left + right input
3. left join: builder can only push to the left input
4. right join: builder can only push to the right input
was:
Currently, runtime filter program will try to look for an {{Exchange}} as build
side to avoid affecting {{MultiInput}}. It will try to push down the runtime
filter builder if the original build side is not {{Exchange}}.
Currenlty, the builder-push-down does not aware the join type, which may lead
to incorrect results(For example, push down the builder to the right input of
left-join).
We should only support following cases:
1. Inner join: builder can push to left + right input
2. semi join: builder can push to left + right input
3. left join: builder can only push to the left input
4. right join: builder can only push to the right input
> RuntimeFilterProgram should aware join type when looking for the build side
> ---
>
> Key: FLINK-32831
> URL: https://issues.apache.org/jira/browse/FLINK-32831
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.18.0
>Reporter: Lijie Wang
>Priority: Major
>
> Currently, runtime filter program will try to look for an {{Exchange}} as
> build side to avoid affecting {{MultiInput}}. It will try to push down the
> runtime filter builder if the original build side is not {{Exchange}}.
> Currenlty, the builder-push-down does not aware the join type, which may lead
> to incorrect results(For example, push down the builder to the right input of
> left-join).
> We should only support following cases:
> 1. Inner join: builder can push to left + right input
> 2. semi join: builder can push to left + right input
> 3. left join: builder can only push to the left input
> 4. right join: builder can only push to the right input
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-32831) RuntimeFilterProgram should aware join type when looking for the build side
Lijie Wang created FLINK-32831:
--
Summary: RuntimeFilterProgram should aware join type when looking
for the build side
Key: FLINK-32831
URL: https://issues.apache.org/jira/browse/FLINK-32831
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Affects Versions: 1.18.0
Reporter: Lijie Wang
Currently, runtime filter program will try to look for an {{Exchange}} as build
side to avoid affecting {{MultiInput}}. It will try to push down the runtime
filter builder if the original build side is not {{Exchange}}.
Currenlty, the builder-push-down does not aware the join type, which may lead
to incorrect results(For example, push down the builder to the right input of
left-join).
We should only support following cases:
1. Inner join: builder can push to left + right input
2. semi join: builder can push to left + right input
3. left join: builder can only push to the left input
4. right join: builder can only push to the right input
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] lincoln-lil commented on a diff in pull request #23192: [FLINK-32730][table-planner] Fix the bug of Scan reuse after projection pushdown without considering the dpp pattern
lincoln-lil commented on code in PR #23192:
URL: https://github.com/apache/flink/pull/23192#discussion_r1290858921
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/reuse/ReplaceScanWithCalcShuttle.java:
##
@@ -57,4 +61,12 @@ public RelNode visit(RelNode rel) {
return super.visit(rel);
}
+
+private void visitDppSource(RelNode scan) {
+// If scan is BatchPhysicalDynamicFilteringTableSourceScan,the input
should be recursive
Review Comment:
nit: If scan is `BatchPhysicalDynamicFilteringTableSourceScan`, traverse
its input first
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/reuse/ReplaceScanWithCalcShuttle.java:
##
@@ -44,11 +44,15 @@ public RelNode visit(RelNode rel) {
// if there is already one Calc, we should merge it and new
projection node.
Calc calc = (Calc) rel;
RelNode input = calc.getInput();
+visitDppSource(input);
Review Comment:
Looks like this visit won't actually have an impact?
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/reuse/ReplaceScanWithCalcShuttle.java:
##
@@ -44,11 +44,15 @@ public RelNode visit(RelNode rel) {
// if there is already one Calc, we should merge it and new
projection node.
Calc calc = (Calc) rel;
RelNode input = calc.getInput();
+visitDppSource(input);
+
RelNode newNode = replaceMap.get(input);
if (newNode instanceof Calc && isMergeable(calc, (Calc) newNode)) {
return merge(calc, (Calc) newNode);
}
} else if (rel instanceof CommonPhysicalTableSourceScan) {
+visitDppSource(rel);
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] swuferhong commented on a diff in pull request #23192: [FLINK-32730][table-planner] Fix the bug of Scan reuse after projection pushdown without considering the dpp pattern
swuferhong commented on code in PR #23192:
URL: https://github.com/apache/flink/pull/23192#discussion_r1290855481
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/reuse/ScanReuserUtils.java:
##
@@ -329,6 +329,12 @@ public static String
getDigest(CommonPhysicalTableSourceScan scan, boolean witho
TableSourceTable table = scan.tableSourceTable();
List digest = new ArrayList<>();
digest.addAll(table.getNames());
+
+// input should be the first item
+if (!scan.getInputs().isEmpty()) {
+digest.add("input=[" + scan.getInputs() + "]");
Review Comment:
The length of `scan.getInputs()` will be very long? I think you can
simplify the output of the digest here
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/reuse/ReplaceScanWithCalcShuttle.java:
##
@@ -57,4 +61,12 @@ public RelNode visit(RelNode rel) {
return super.visit(rel);
}
+
+private void visitDppSource(RelNode scan) {
+// If scan is BatchPhysicalDynamicFilteringTableSourceScan,the input
should be recursive
+// first
Review Comment:
I think there needn't to strictly limit scan is dpp scan in the comments.
But let's talk about sources with input, such as dpp sources
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/reuse/ReusableScanVisitor.java:
##
@@ -42,6 +42,10 @@ public void visit(RelNode node, int ordinal, RelNode parent)
{
CommonPhysicalTableSourceScan scan =
(CommonPhysicalTableSourceScan) node;
String digest = getDigest(scan, true);
digestToReusableScans.computeIfAbsent(digest, k -> new
ArrayList<>()).add(scan);
+// BatchPhysicalDynamicFilteringTableSourceScan has one input, so
need to consider it
Review Comment:
ditto
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/optimize/program/DynamicPartitionPruningProgramTest.java:
##
@@ -658,4 +658,47 @@ public void testDPPWithJoinKeysNotIncludePartitionKeys() {
+ " and dim.price < 500 and dim.price > 300";
util.verifyRelPlan(query);
}
+
+@Test
+public void testDppFactSideCannotReuseWithSameCommonSource() {
+String query =
+"SELECT * FROM(\n"
++ " Select fact_part.id, fact_part.price,
fact_part.amount from fact_part join (Select * from dim) t1"
++ " on fact_part.fact_date_sk = dim_date_sk where
t1.price < 500\n"
++ " UNION ALL Select fact_part.id, fact_part.price,
fact_part.amount from fact_part)";
+util.verifyExecPlan(query);
+}
+
+@Test
+public void testDimSideReuseWithUnionAllTwoFactSide() {
+util.tableEnv()
+.executeSql(
Review Comment:
Can this test be simplified? I feel that the correlation between testing and
this pr is not strong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] FangYongs commented on pull request #23063: [FLINK-32660][table] Introduce external FS compatible FileCatalogStore implementation
FangYongs commented on PR #23063: URL: https://github.com/apache/flink/pull/23063#issuecomment-1674165313 Thanks @ferenc-csaky for updating this PR, it LGTM overall, please fix the failed ci -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-32712) Record with RowKind of type -U should not be sent downstream when the before field is empty in ogg-json which op_type is U .
[
https://issues.apache.org/jira/browse/FLINK-32712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17753033#comment-17753033
]
yuan commented on FLINK-32712:
--
[~lsy] Hi, thanks for the advice.
I have added a more detailed description for this issue.
> Record with RowKind of type -U should not be sent downstream when the before
> field is empty in ogg-json which op_type is U .
>
>
> Key: FLINK-32712
> URL: https://issues.apache.org/jira/browse/FLINK-32712
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.17.1
>Reporter: yuan
>Priority: Major
> Labels: pull-request-available
>
> The BEFORE field in ogg-json can be configured to contain no table fields, as
> in the ogg-json example below.
> For this case, the current version of ogg-json format, sends downstream a
> Record of type UPDATE_BEFORE with null fields, eg. `Record(-U, null, null,
> null,...)`, which obviously makes no sense.
> {code:java}
> {
> "table": "ZBZZZ",
> "op_type": "U",
> "op_ts": "2023-07-20 21:45:34.860817",
> "current_ts": "2023-07-21T05:45:36.615000",
> "pos": "2564940142073691",
> "before": {},
> "after": {
> "ID": 1461242,
> "PROPERTY_01": "tc",
> "PROPERTY_02": null,
> "PROPERTY_03": null,
> "PROPERTY_04": "K",
> "PROPERTY_05": "5",
> "PROPERTY_06": null,
> "PROPERTY_07": null,
> "PROPERTY_08": null,
> "PROPERTY_09": null,
> "PROPERTY_10": null
> }
> }{code}
> For this case, ogg-json format should not send a Record of type UPDATE_BEFORE
> downstream。 Worse, it can sometimes cause the flink job to crash out.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32712) Record with RowKind of type -U should not be sent downstream when the before field is empty in ogg-json which op_type is U .
[
https://issues.apache.org/jira/browse/FLINK-32712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yuan updated FLINK-32712:
-
Description:
The BEFORE field in ogg-json can be configured to contain no table fields, as
in the ogg-json example below.
For this case, the current version of ogg-json format, sends downstream a
Record of type UPDATE_BEFORE with null fields, eg. `Record(-U, null, null,
null,...)`, which obviously makes no sense.
{code:java}
{
"table": "ZBZZZ",
"op_type": "U",
"op_ts": "2023-07-20 21:45:34.860817",
"current_ts": "2023-07-21T05:45:36.615000",
"pos": "2564940142073691",
"before": {},
"after": {
"ID": 1461242,
"PROPERTY_01": "tc",
"PROPERTY_02": null,
"PROPERTY_03": null,
"PROPERTY_04": "K",
"PROPERTY_05": "5",
"PROPERTY_06": null,
"PROPERTY_07": null,
"PROPERTY_08": null,
"PROPERTY_09": null,
"PROPERTY_10": null
}
}{code}
For this case, ogg-json format should not send a Record of type UPDATE_BEFORE
downstream。 Worse, it can sometimes cause the flink job to crash out.
was:
The BEFORE field in ogg-json can be configured to contain no table fields, as
in the ogg-json example below.
For this case, the current version of ogg-json format, sends downstream a
Record of type UPDATE_BEFORE with null fields, eg. Record(-U, null, null,
null,...), which obviously makes no sense.
{code:java}
{
"table": "ZBZZZ",
"op_type": "U",
"op_ts": "2023-07-20 21:45:34.860817",
"current_ts": "2023-07-21T05:45:36.615000",
"pos": "2564940142073691",
"before": {},
"after": {
"ID": 1461242,
"PROPERTY_01": "tc",
"PROPERTY_02": null,
"PROPERTY_03": null,
"PROPERTY_04": "K",
"PROPERTY_05": "5",
"PROPERTY_06": null,
"PROPERTY_07": null,
"PROPERTY_08": null,
"PROPERTY_09": null,
"PROPERTY_10": null
}
}{code}
For this case, ogg-json format should not send a Record of type UPDATE_BEFORE
downstream。 Worse, it can sometimes cause the flink job to crash out.
> Record with RowKind of type -U should not be sent downstream when the before
> field is empty in ogg-json which op_type is U .
>
>
> Key: FLINK-32712
> URL: https://issues.apache.org/jira/browse/FLINK-32712
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.17.1
>Reporter: yuan
>Priority: Major
> Labels: pull-request-available
>
> The BEFORE field in ogg-json can be configured to contain no table fields, as
> in the ogg-json example below.
> For this case, the current version of ogg-json format, sends downstream a
> Record of type UPDATE_BEFORE with null fields, eg. `Record(-U, null, null,
> null,...)`, which obviously makes no sense.
> {code:java}
> {
> "table": "ZBZZZ",
> "op_type": "U",
> "op_ts": "2023-07-20 21:45:34.860817",
> "current_ts": "2023-07-21T05:45:36.615000",
> "pos": "2564940142073691",
> "before": {},
> "after": {
> "ID": 1461242,
> "PROPERTY_01": "tc",
> "PROPERTY_02": null,
> "PROPERTY_03": null,
> "PROPERTY_04": "K",
> "PROPERTY_05": "5",
> "PROPERTY_06": null,
> "PROPERTY_07": null,
> "PROPERTY_08": null,
> "PROPERTY_09": null,
> "PROPERTY_10": null
> }
> }{code}
> For this case, ogg-json format should not send a Record of type UPDATE_BEFORE
> downstream。 Worse, it can sometimes cause the flink job to crash out.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32712) Record with RowKind of type -U should not be sent downstream when the before field is empty in ogg-json which op_type is U .
[
https://issues.apache.org/jira/browse/FLINK-32712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yuan updated FLINK-32712:
-
Description:
The BEFORE field in ogg-json can be configured to contain no table fields, as
in the ogg-json example below.
For this case, the current version of ogg-json format, sends downstream a
Record of type UPDATE_BEFORE with null fields, eg. Record(-U, null, null,
null,...), which obviously makes no sense.
{code:java}
{
"table": "ZBZZZ",
"op_type": "U",
"op_ts": "2023-07-20 21:45:34.860817",
"current_ts": "2023-07-21T05:45:36.615000",
"pos": "2564940142073691",
"before": {},
"after": {
"ID": 1461242,
"PROPERTY_01": "tc",
"PROPERTY_02": null,
"PROPERTY_03": null,
"PROPERTY_04": "K",
"PROPERTY_05": "5",
"PROPERTY_06": null,
"PROPERTY_07": null,
"PROPERTY_08": null,
"PROPERTY_09": null,
"PROPERTY_10": null
}
}{code}
For this case, ogg-json format should not send a Record of type UPDATE_BEFORE
downstream。 Worse, it can sometimes cause the flink job to crash out.
was:
The BEFORE field in ogg-json can be configured to contain no table fields, as
in the ogg-json example below.
{code:java}
{
"table": "ZBZZZ",
"op_type": "U",
"op_ts": "2023-07-20 21:45:34.860817",
"current_ts": "2023-07-21T05:45:36.615000",
"pos": "2564940142073691",
"before": {},
"after": {
"ID": 1461242,
"PROPERTY_01": "tc",
"PROPERTY_02": null,
"PROPERTY_03": null,
"PROPERTY_04": "K",
"PROPERTY_05": "5",
"PROPERTY_06": null,
"PROPERTY_07": null,
"PROPERTY_08": null,
"PROPERTY_09": null,
"PROPERTY_10": null
}
}{code}
For this case, ogg-json format should not send a Record of type UPDATE_BEFORE
downstream。 Worse, it can sometimes cause the flink job to crash out.
> Record with RowKind of type -U should not be sent downstream when the before
> field is empty in ogg-json which op_type is U .
>
>
> Key: FLINK-32712
> URL: https://issues.apache.org/jira/browse/FLINK-32712
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.17.1
>Reporter: yuan
>Priority: Major
> Labels: pull-request-available
>
> The BEFORE field in ogg-json can be configured to contain no table fields, as
> in the ogg-json example below.
> For this case, the current version of ogg-json format, sends downstream a
> Record of type UPDATE_BEFORE with null fields, eg. Record(-U, null, null,
> null,...), which obviously makes no sense.
> {code:java}
> {
> "table": "ZBZZZ",
> "op_type": "U",
> "op_ts": "2023-07-20 21:45:34.860817",
> "current_ts": "2023-07-21T05:45:36.615000",
> "pos": "2564940142073691",
> "before": {},
> "after": {
> "ID": 1461242,
> "PROPERTY_01": "tc",
> "PROPERTY_02": null,
> "PROPERTY_03": null,
> "PROPERTY_04": "K",
> "PROPERTY_05": "5",
> "PROPERTY_06": null,
> "PROPERTY_07": null,
> "PROPERTY_08": null,
> "PROPERTY_09": null,
> "PROPERTY_10": null
> }
> }{code}
> For this case, ogg-json format should not send a Record of type UPDATE_BEFORE
> downstream。 Worse, it can sometimes cause the flink job to crash out.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32712) Record with RowKind of type -U should not be sent downstream when the before field is empty in ogg-json which op_type is U .
[
https://issues.apache.org/jira/browse/FLINK-32712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yuan updated FLINK-32712:
-
Summary: Record with RowKind of type -U should not be sent downstream when
the before field is empty in ogg-json which op_type is U . (was: Enhance Flink
ogg-json format)
> Record with RowKind of type -U should not be sent downstream when the before
> field is empty in ogg-json which op_type is U .
>
>
> Key: FLINK-32712
> URL: https://issues.apache.org/jira/browse/FLINK-32712
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.17.1
>Reporter: yuan
>Priority: Major
> Labels: pull-request-available
>
> The BEFORE field in ogg-json can be configured to contain no table fields, as
> in the ogg-json example below.
> {code:java}
> {
> "table": "ZBZZZ",
> "op_type": "U",
> "op_ts": "2023-07-20 21:45:34.860817",
> "current_ts": "2023-07-21T05:45:36.615000",
> "pos": "2564940142073691",
> "before": {},
> "after": {
> "ID": 1461242,
> "PROPERTY_01": "tc",
> "PROPERTY_02": null,
> "PROPERTY_03": null,
> "PROPERTY_04": "K",
> "PROPERTY_05": "5",
> "PROPERTY_06": null,
> "PROPERTY_07": null,
> "PROPERTY_08": null,
> "PROPERTY_09": null,
> "PROPERTY_10": null
> }
> }{code}
> For this case, ogg-json format should not send a Record of type UPDATE_BEFORE
> downstream。 Worse, it can sometimes cause the flink job to crash out.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-32709) Fix the bug of low memory utilization for Hybrid Shuffle
[ https://issues.apache.org/jira/browse/FLINK-32709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo closed FLINK-32709. -- Resolution: Fixed > Fix the bug of low memory utilization for Hybrid Shuffle > > > Key: FLINK-32709 > URL: https://issues.apache.org/jira/browse/FLINK-32709 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.18.0 >Reporter: Yuxin Tan >Assignee: Yuxin Tan >Priority: Major > Labels: pull-request-available > > Currently, each subpartition in Disk/Remote has a segment size of 8M. When > writing segments to the Disk tier with a parallelism of 1000, only shuffle > data exceeding 1000 * 8M can be written to the Memory tier again. However, > for most shuffles, the data volume size falls below this limit, significantly > impacting Memory tier utilization. > For better performance, it is necessary to address this issue to improve the > memory tier utilization. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32709) Fix the bug of low memory utilization for Hybrid Shuffle
[ https://issues.apache.org/jira/browse/FLINK-32709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo updated FLINK-32709: --- Fix Version/s: 1.18.0 > Fix the bug of low memory utilization for Hybrid Shuffle > > > Key: FLINK-32709 > URL: https://issues.apache.org/jira/browse/FLINK-32709 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.18.0 >Reporter: Yuxin Tan >Assignee: Yuxin Tan >Priority: Major > Labels: pull-request-available > Fix For: 1.18.0 > > > Currently, each subpartition in Disk/Remote has a segment size of 8M. When > writing segments to the Disk tier with a parallelism of 1000, only shuffle > data exceeding 1000 * 8M can be written to the Memory tier again. However, > for most shuffles, the data volume size falls below this limit, significantly > impacting Memory tier utilization. > For better performance, it is necessary to address this issue to improve the > memory tier utilization. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-32709) Fix the bug of low memory utilization for Hybrid Shuffle
[ https://issues.apache.org/jira/browse/FLINK-32709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17753025#comment-17753025 ] Weijie Guo commented on FLINK-32709: Master(1.18) via 0f841624a388be9ad551ec1ccbc15abb31044d6e. > Fix the bug of low memory utilization for Hybrid Shuffle > > > Key: FLINK-32709 > URL: https://issues.apache.org/jira/browse/FLINK-32709 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.18.0 >Reporter: Yuxin Tan >Assignee: Yuxin Tan >Priority: Major > Labels: pull-request-available > > Currently, each subpartition in Disk/Remote has a segment size of 8M. When > writing segments to the Disk tier with a parallelism of 1000, only shuffle > data exceeding 1000 * 8M can be written to the Memory tier again. However, > for most shuffles, the data volume size falls below this limit, significantly > impacting Memory tier utilization. > For better performance, it is necessary to address this issue to improve the > memory tier utilization. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] reswqa merged pull request #23095: [FLINK-32709][network] Fix the bug of low memory utilization for Hybrid Shuffle
reswqa merged PR #23095: URL: https://github.com/apache/flink/pull/23095 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] reswqa commented on a diff in pull request #23179: [FLINK-32770][network] Fix the inaccurate backlog number of Hybrid Shuffle
reswqa commented on code in PR #23179:
URL: https://github.com/apache/flink/pull/23179#discussion_r1290831536
##
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/netty/TieredStorageResultSubpartitionViewTest.java:
##
@@ -115,18 +113,18 @@ void testNotifyRequiredSegmentId() {
@Test
void testReleaseAllResources() throws IOException {
tieredStorageResultSubpartitionView.releaseAllResources();
-assertThat(nettyPayloadQueues.get(0)).hasSize(0);
-assertThat(nettyPayloadQueues.get(1)).hasSize(0);
+assertThat(nettyPayloadQueues.get(0).getBacklog()).isEqualTo(0);
+assertThat(nettyPayloadQueues.get(1).getBacklog()).isEqualTo(0);
Review Comment:
```suggestion
assertThat(nettyPayloadQueues.get(1).getBacklog()).isZero;
```
##
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/netty/TieredStorageResultSubpartitionView.java:
##
@@ -94,14 +93,14 @@ public BufferAndBacklog getNextBuffer() throws IOException {
@Override
public AvailabilityWithBacklog getAvailabilityAndBacklog(int
numCreditsAvailable) {
if (findCurrentNettyPayloadQueue()) {
-Queue currentQueue =
+NettyPayloadQueue currentQueue =
nettyPayloadQueues.get(queueIndexContainsCurrentSegment);
boolean availability = numCreditsAvailable > 0;
if (numCreditsAvailable <= 0
&& getNettyPayloadNextDataType(currentQueue) ==
Buffer.DataType.EVENT_BUFFER) {
Review Comment:
What should we do with error, it also does not have backlog.
##
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/netty/NettyPayloadQueue.java:
##
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition.hybrid.tiered.netty;
+
+import java.util.LinkedList;
+import java.util.Queue;
+
+/**
+ * The {@link NettyPayloadQueue} is the queue to contain different types of
{@link NettyPayload} and
+ * calculate the backlog number.
+ */
+public class NettyPayloadQueue {
Review Comment:
Basically, this should impl `Queue`.
##
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/netty/TieredStorageResultSubpartitionViewTest.java:
##
@@ -115,18 +113,18 @@ void testNotifyRequiredSegmentId() {
@Test
void testReleaseAllResources() throws IOException {
tieredStorageResultSubpartitionView.releaseAllResources();
-assertThat(nettyPayloadQueues.get(0)).hasSize(0);
-assertThat(nettyPayloadQueues.get(1)).hasSize(0);
+assertThat(nettyPayloadQueues.get(0).getBacklog()).isEqualTo(0);
Review Comment:
```suggestion
assertThat(nettyPayloadQueues.get(0).getBacklog()).isZero();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-32344) MongoDB connector support unbounded streaming read via ChangeStream feature
[ https://issues.apache.org/jira/browse/FLINK-32344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17753018#comment-17753018 ] Jiabao Sun commented on FLINK-32344: This PR has been open for a long time. Is there anyone have time to help with a review? Thanks a lot. :) > MongoDB connector support unbounded streaming read via ChangeStream feature > --- > > Key: FLINK-32344 > URL: https://issues.apache.org/jira/browse/FLINK-32344 > Project: Flink > Issue Type: New Feature > Components: Connectors / MongoDB >Affects Versions: mongodb-1.0.1 >Reporter: Jiabao Sun >Assignee: Jiabao Sun >Priority: Major > Labels: pull-request-available > > Change streams allow applications to access real-time data changes without > the complexity and risk of tailing the oplog. Applications can use change > streams to subscribe to all data changes on a single collection, a database, > or an entire deployment, and immediately react to them. Because change > streams use the aggregation framework, applications can also filter for > specific changes or transform the notifications at will. > We can use MongoDB change streams feature to support unbounded streaming read > for mongodb connector. > [Change > Streams|https://www.mongodb.com/docs/manual/changeStreams/#change-streams] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32344) MongoDB connector support unbounded streaming read via ChangeStream feature
[ https://issues.apache.org/jira/browse/FLINK-32344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiabao Sun updated FLINK-32344: --- Labels: pull-request-available (was: pull-request-available stale-assigned) > MongoDB connector support unbounded streaming read via ChangeStream feature > --- > > Key: FLINK-32344 > URL: https://issues.apache.org/jira/browse/FLINK-32344 > Project: Flink > Issue Type: New Feature > Components: Connectors / MongoDB >Affects Versions: mongodb-1.0.1 >Reporter: Jiabao Sun >Assignee: Jiabao Sun >Priority: Major > Labels: pull-request-available > > Change streams allow applications to access real-time data changes without > the complexity and risk of tailing the oplog. Applications can use change > streams to subscribe to all data changes on a single collection, a database, > or an entire deployment, and immediately react to them. Because change > streams use the aggregation framework, applications can also filter for > specific changes or transform the notifications at will. > We can use MongoDB change streams feature to support unbounded streaming read > for mongodb connector. > [Change > Streams|https://www.mongodb.com/docs/manual/changeStreams/#change-streams] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] RocMarshal commented on a diff in pull request #23191: [FLINK-32809][yarn] Fixes YarnClusterDescriptor#isArchiveOnlyIncluded…
RocMarshal commented on code in PR #23191:
URL: https://github.com/apache/flink/pull/23191#discussion_r1290829384
##
flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java:
##
@@ -305,23 +305,26 @@ public void addShipFiles(List shipFiles) {
private void addShipArchives(List shipArchives) {
checkArgument(
isArchiveOnlyIncludedInShipArchiveFiles(shipArchives),
-"Non-archive files are included.");
+"Directories or non-archive files are included.");
this.shipArchives.addAll(shipArchives);
}
private static boolean isArchiveOnlyIncludedInShipArchiveFiles(List
shipFiles) {
-return shipFiles.stream()
-.filter(File::isFile)
-.map(File::getName)
-.map(String::toLowerCase)
-.allMatch(
-name ->
-name.endsWith(".tar.gz")
-|| name.endsWith(".tar")
-|| name.endsWith(".tgz")
-|| name.endsWith(".dst")
-|| name.endsWith(".jar")
-|| name.endsWith(".zip"));
+long archivedFileCount =
+shipFiles.stream()
+.filter(File::isFile)
+.map(File::getName)
+.map(String::toLowerCase)
Review Comment:
Could these two lines be inlined?
Because there's no filters between the two `maps`
##
flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java:
##
@@ -305,23 +305,26 @@ public void addShipFiles(List shipFiles) {
private void addShipArchives(List shipArchives) {
checkArgument(
isArchiveOnlyIncludedInShipArchiveFiles(shipArchives),
-"Non-archive files are included.");
+"Directories or non-archive files are included.");
this.shipArchives.addAll(shipArchives);
}
private static boolean isArchiveOnlyIncludedInShipArchiveFiles(List
shipFiles) {
-return shipFiles.stream()
-.filter(File::isFile)
-.map(File::getName)
-.map(String::toLowerCase)
-.allMatch(
-name ->
-name.endsWith(".tar.gz")
-|| name.endsWith(".tar")
-|| name.endsWith(".tgz")
-|| name.endsWith(".dst")
-|| name.endsWith(".jar")
-|| name.endsWith(".zip"));
+long archivedFileCount =
+shipFiles.stream()
+.filter(File::isFile)
+.map(File::getName)
+.map(String::toLowerCase)
+.filter(
+name ->
+name.endsWith(".tar.gz")
+|| name.endsWith(".tar")
+|| name.endsWith(".tgz")
+|| name.endsWith(".dst")
+|| name.endsWith(".jar")
+|| name.endsWith(".zip"))
Review Comment:
Is there a possibility of frequent changes in the suffix format here?
If so, would you consider introducing a parameter to specify the suffix
format?
Please correct me if I'm wrong in my limited read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] reswqa merged pull request #23140: [hotfix][javadocs] Fix some typo in java doc and comments
reswqa merged PR #23140: URL: https://github.com/apache/flink/pull/23140 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-32821) Streaming examples failed to execute due to error in packaging
[ https://issues.apache.org/jira/browse/FLINK-32821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17753000#comment-17753000 ] Zhanghao Chen edited comment on FLINK-32821 at 8/11/23 12:43 AM: - Hi [~afedulov] , I execute them using the same procedure in [https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/resource-providers/standalone/overview/#starting-a-standalone-cluster-session-mode] using the latest master branch. was (Author: zhanghao chen): Hi [~afedulov] , I execute them using the same procedure in [Overview | Apache Flink|https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/resource-providers/standalone/overview/#starting-a-standalone-cluster-session-mode] using the latest master branch. > Streaming examples failed to execute due to error in packaging > -- > > Key: FLINK-32821 > URL: https://issues.apache.org/jira/browse/FLINK-32821 > Project: Flink > Issue Type: Bug > Components: Examples >Affects Versions: 1.18.0 >Reporter: Zhanghao Chen >Assignee: Zhanghao Chen >Priority: Major > > 5 out of the 7 streaming examples failed to run: > * Iteration & SessionWindowing & SocketWindowWordCount & WindowJoin failed > to run due to java.lang.NoClassDefFoundError: > org/apache/flink/streaming/examples/utils/ParameterTool > * TopSpeedWindowing failed to run due to: Caused by: > java.lang.ClassNotFoundException: > org.apache.flink.connector.datagen.source.GeneratorFunction > The NoClassDefFoundError with ParameterTool is introduced by [FLINK-32558] > Properly deprecate DataSet API - ASF JIRA (apache.org), and we'd better > resolve [FLINK-32820] ParameterTool is mistakenly marked as deprecated - ASF > JIRA (apache.org) first before we come to a fix for this problem. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-32821) Streaming examples failed to execute due to error in packaging
[ https://issues.apache.org/jira/browse/FLINK-32821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17753000#comment-17753000 ] Zhanghao Chen commented on FLINK-32821: --- Hi [~afedulov] , I execute them using the same procedure in [Overview | Apache Flink|https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/resource-providers/standalone/overview/#starting-a-standalone-cluster-session-mode] using the latest master branch. > Streaming examples failed to execute due to error in packaging > -- > > Key: FLINK-32821 > URL: https://issues.apache.org/jira/browse/FLINK-32821 > Project: Flink > Issue Type: Bug > Components: Examples >Affects Versions: 1.18.0 >Reporter: Zhanghao Chen >Assignee: Zhanghao Chen >Priority: Major > > 5 out of the 7 streaming examples failed to run: > * Iteration & SessionWindowing & SocketWindowWordCount & WindowJoin failed > to run due to java.lang.NoClassDefFoundError: > org/apache/flink/streaming/examples/utils/ParameterTool > * TopSpeedWindowing failed to run due to: Caused by: > java.lang.ClassNotFoundException: > org.apache.flink.connector.datagen.source.GeneratorFunction > The NoClassDefFoundError with ParameterTool is introduced by [FLINK-32558] > Properly deprecate DataSet API - ASF JIRA (apache.org), and we'd better > resolve [FLINK-32820] ParameterTool is mistakenly marked as deprecated - ASF > JIRA (apache.org) first before we come to a fix for this problem. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-32821) Streaming examples failed to execute due to error in packaging
[ https://issues.apache.org/jira/browse/FLINK-32821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17753000#comment-17753000 ] Zhanghao Chen edited comment on FLINK-32821 at 8/11/23 12:42 AM: - Hi [~afedulov] , I execute them using the same procedure in [Overview | Apache Flink|https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/resource-providers/standalone/overview/#starting-a-standalone-cluster-session-mode] using the latest master branch. was (Author: zhanghao chen): Hi [~afedulov] , I execute them using the same procedure in [Overview | Apache Flink|https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/resource-providers/standalone/overview/#starting-a-standalone-cluster-session-mode] using the latest master branch. > Streaming examples failed to execute due to error in packaging > -- > > Key: FLINK-32821 > URL: https://issues.apache.org/jira/browse/FLINK-32821 > Project: Flink > Issue Type: Bug > Components: Examples >Affects Versions: 1.18.0 >Reporter: Zhanghao Chen >Assignee: Zhanghao Chen >Priority: Major > > 5 out of the 7 streaming examples failed to run: > * Iteration & SessionWindowing & SocketWindowWordCount & WindowJoin failed > to run due to java.lang.NoClassDefFoundError: > org/apache/flink/streaming/examples/utils/ParameterTool > * TopSpeedWindowing failed to run due to: Caused by: > java.lang.ClassNotFoundException: > org.apache.flink.connector.datagen.source.GeneratorFunction > The NoClassDefFoundError with ParameterTool is introduced by [FLINK-32558] > Properly deprecate DataSet API - ASF JIRA (apache.org), and we'd better > resolve [FLINK-32820] ParameterTool is mistakenly marked as deprecated - ASF > JIRA (apache.org) first before we come to a fix for this problem. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32830) Support Histogram function
[
https://issues.apache.org/jira/browse/FLINK-32830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hanyu Zheng updated FLINK-32830:
Description:
This is an implementation of HISTOGRAM
Returns a map containing the distinct values of col1 mapped to the number of
times each one occurs for the given window. This version limits the number of
distinct values which can be counted to 1000, beyond which any additional
entries are ignored.
h3. Brief change log
HISTOGRAM for Table API and SQL
Syntax:
{code:java}
HISTOGRAM(col1){code}
Arguments:
col1: the data in col1
Examples:
{code:java}
Flink SQL> create temporary table orders (
> orderId INT,
> price DECIMAL(10,3)
> )with(
> 'connector' = 'datagen',
> 'rows-per-second' = '5',
> 'fields.orderId.min' = '1',
> 'fields.orderId.max' = '20',
> 'fields.price.min' = '1',
> 'fields.price.max' = '200'
> );
Flink SQL> select histogram(price) as map from orders;
res: {147.451 = 1， 65.765 = 1， 41.662 = 1 …}
{code}
see also:
KsqlDB:
[https://docs.ksqldb.io/en/latest/developer-guide/ksqldb-reference/aggregate-functions/#histogram]
was:
This is an implementation of HISTOGRAM
Returns a map containing the distinct values of col1 mapped to the number of
times each one occurs for the given window. This version limits the number of
distinct values which can be counted to 1000, beyond which any additional
entries are ignored.
h3. Brief change log
HISTOGRAM for Table API and SQL
Syntax:
`HISTOGRAM(col1)
`
Arguments:
col1: the data in col1
Examples:
```
Flink SQL> create temporary table orders (
> orderId INT,
> price DECIMAL(10,3)
> )with(
> 'connector' = 'datagen',
> 'rows-per-second' = '5',
> 'fields.orderId.min' = '1',
> 'fields.orderId.max' = '20',
> 'fields.price.min' = '1',
> 'fields.price.max' = '200'
> );
```
`Flink SQL> select histogram(price) as map from orders;`
`res: \{147.451 = 1， 65.765 = 1， 41.662 = 1 …}`
see also:
KsqlDB:
https://docs.ksqldb.io/en/latest/developer-guide/ksqldb-reference/aggregate-functions/#histogram
> Support Histogram function
> --
>
> Key: FLINK-32830
> URL: https://issues.apache.org/jira/browse/FLINK-32830
> Project: Flink
> Issue Type: Improvement
>Reporter: Hanyu Zheng
>Priority: Major
>
> This is an implementation of HISTOGRAM
> Returns a map containing the distinct values of col1 mapped to the number of
> times each one occurs for the given window. This version limits the number of
> distinct values which can be counted to 1000, beyond which any additional
> entries are ignored.
> h3. Brief change log
> HISTOGRAM for Table API and SQL
> Syntax:
> {code:java}
> HISTOGRAM(col1){code}
> Arguments:
> col1: the data in col1
> Examples:
> {code:java}
> Flink SQL> create temporary table orders (
> > orderId INT,
> > price DECIMAL(10,3)
> > )with(
> > 'connector' = 'datagen',
> > 'rows-per-second' = '5',
> > 'fields.orderId.min' = '1',
> > 'fields.orderId.max' = '20',
> > 'fields.price.min' = '1',
> > 'fields.price.max' = '200'
> > );
> Flink SQL> select histogram(price) as map from orders;
> res: {147.451 = 1， 65.765 = 1， 41.662 = 1 …}
> {code}
>
> see also:
> KsqlDB:
> [https://docs.ksqldb.io/en/latest/developer-guide/ksqldb-reference/aggregate-functions/#histogram]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] flinkbot commented on pull request #23194: [BP-1.17]historgram
flinkbot commented on PR #23194: URL: https://github.com/apache/flink/pull/23194#issuecomment-1674029593 ## CI report: * 238b57edaffbb47169864e94c718519a79af63c4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-32830) Support Histogram function
[
https://issues.apache.org/jira/browse/FLINK-32830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hanyu Zheng updated FLINK-32830:
Description:
This is an implementation of HISTOGRAM
Returns a map containing the distinct values of col1 mapped to the number of
times each one occurs for the given window. This version limits the number of
distinct values which can be counted to 1000, beyond which any additional
entries are ignored.
h3. Brief change log
HISTOGRAM for Table API and SQL
Syntax:
`HISTOGRAM(col1)
`
Arguments:
col1: the data in col1
Examples:
```
Flink SQL> create temporary table orders (
> orderId INT,
> price DECIMAL(10,3)
> )with(
> 'connector' = 'datagen',
> 'rows-per-second' = '5',
> 'fields.orderId.min' = '1',
> 'fields.orderId.max' = '20',
> 'fields.price.min' = '1',
> 'fields.price.max' = '200'
> );
```
`Flink SQL> select histogram(price) as map from orders;`
`res: \{147.451 = 1， 65.765 = 1， 41.662 = 1 …}`
see also:
KsqlDB:
https://docs.ksqldb.io/en/latest/developer-guide/ksqldb-reference/aggregate-functions/#histogram
was:
### What is the purpose of the change
This is an implementation of HISTOGRAM
Returns a map containing the distinct values of col1 mapped to the number of
times each one occurs for the given window. This version limits the number of
distinct values which can be counted to 1000, beyond which any additional
entries are ignored.
### Brief change log
HISTOGRAM for Table API and SQL
Syntax:
`HISTOGRAM(col1)
`
Arguments:
col1: the data in col1
Examples:
```
Flink SQL> create temporary table orders (
> orderId INT,
> price DECIMAL(10,3)
> )with(
> 'connector' = 'datagen',
> 'rows-per-second' = '5',
> 'fields.orderId.min' = '1',
> 'fields.orderId.max' = '20',
> 'fields.price.min' = '1',
> 'fields.price.max' = '200'
> );
```
`Flink SQL> select histogram(price) as map from orders;`
`res: \{147.451 = 1， 65.765 = 1， 41.662 = 1 …}`
see also:
KsqlDB:
https://docs.ksqldb.io/en/latest/developer-guide/ksqldb-reference/aggregate-functions/#histogram
> Support Histogram function
> --
>
> Key: FLINK-32830
> URL: https://issues.apache.org/jira/browse/FLINK-32830
> Project: Flink
> Issue Type: Improvement
>Reporter: Hanyu Zheng
>Priority: Major
>
> This is an implementation of HISTOGRAM
> Returns a map containing the distinct values of col1 mapped to the number of
> times each one occurs for the given window. This version limits the number of
> distinct values which can be counted to 1000, beyond which any additional
> entries are ignored.
> h3. Brief change log
> HISTOGRAM for Table API and SQL
> Syntax:
> `HISTOGRAM(col1)
> `
> Arguments:
> col1: the data in col1
> Examples:
> ```
> Flink SQL> create temporary table orders (
> > orderId INT,
> > price DECIMAL(10,3)
> > )with(
> > 'connector' = 'datagen',
> > 'rows-per-second' = '5',
> > 'fields.orderId.min' = '1',
> > 'fields.orderId.max' = '20',
> > 'fields.price.min' = '1',
> > 'fields.price.max' = '200'
> > );
> ```
> `Flink SQL> select histogram(price) as map from orders;`
> `res: \{147.451 = 1， 65.765 = 1， 41.662 = 1 …}`
> see also:
> KsqlDB:
> https://docs.ksqldb.io/en/latest/developer-guide/ksqldb-reference/aggregate-functions/#histogram
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32830) Support Histogram function
[
https://issues.apache.org/jira/browse/FLINK-32830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hanyu Zheng updated FLINK-32830:
Description:
### What is the purpose of the change
This is an implementation of HISTOGRAM
Returns a map containing the distinct values of col1 mapped to the number of
times each one occurs for the given window. This version limits the number of
distinct values which can be counted to 1000, beyond which any additional
entries are ignored.
### Brief change log
HISTOGRAM for Table API and SQL
Syntax:
`HISTOGRAM(col1)
`
Arguments:
col1: the data in col1
Examples:
```
Flink SQL> create temporary table orders (
> orderId INT,
> price DECIMAL(10,3)
> )with(
> 'connector' = 'datagen',
> 'rows-per-second' = '5',
> 'fields.orderId.min' = '1',
> 'fields.orderId.max' = '20',
> 'fields.price.min' = '1',
> 'fields.price.max' = '200'
> );
```
`Flink SQL> select histogram(price) as map from orders;`
`res: \{147.451 = 1， 65.765 = 1， 41.662 = 1 …}`
see also:
KsqlDB:
https://docs.ksqldb.io/en/latest/developer-guide/ksqldb-reference/aggregate-functions/#histogram
> Support Histogram function
> --
>
> Key: FLINK-32830
> URL: https://issues.apache.org/jira/browse/FLINK-32830
> Project: Flink
> Issue Type: Improvement
>Reporter: Hanyu Zheng
>Priority: Major
>
> ### What is the purpose of the change
> This is an implementation of HISTOGRAM
> Returns a map containing the distinct values of col1 mapped to the number of
> times each one occurs for the given window. This version limits the number of
> distinct values which can be counted to 1000, beyond which any additional
> entries are ignored.
> ### Brief change log
> HISTOGRAM for Table API and SQL
> Syntax:
> `HISTOGRAM(col1)
> `
> Arguments:
> col1: the data in col1
> Examples:
> ```
> Flink SQL> create temporary table orders (
> > orderId INT,
> > price DECIMAL(10,3)
> > )with(
> > 'connector' = 'datagen',
> > 'rows-per-second' = '5',
> > 'fields.orderId.min' = '1',
> > 'fields.orderId.max' = '20',
> > 'fields.price.min' = '1',
> > 'fields.price.max' = '200'
> > );
> ```
> `Flink SQL> select histogram(price) as map from orders;`
> `res: \{147.451 = 1， 65.765 = 1， 41.662 = 1 …}`
> see also:
> KsqlDB:
> https://docs.ksqldb.io/en/latest/developer-guide/ksqldb-reference/aggregate-functions/#histogram
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-32830) Support Histogram function
Hanyu Zheng created FLINK-32830: --- Summary: Support Histogram function Key: FLINK-32830 URL: https://issues.apache.org/jira/browse/FLINK-32830 Project: Flink Issue Type: Improvement Reporter: Hanyu Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] hanyuzheng7 opened a new pull request, #23194: [BP-1.17]historgram
hanyuzheng7 opened a new pull request, #23194:
URL: https://github.com/apache/flink/pull/23194
### What is the purpose of the change
This is an implementation of HISTOGRAM
Returns a map containing the distinct values of col1 mapped to the number of
times each one occurs for the given window. This version limits the number of
distinct values which can be counted to 1000, beyond which any additional
entries are ignored.
### Brief change log
HISTOGRAM for Table API and SQL
Syntax:
`HISTOGRAM(col1)
`
Arguments:
col1: the data in col1
Examples:
```
Flink SQL> create temporary table orders (
> orderId INT,
> price DECIMAL(10,3)
> )with(
> 'connector' = 'datagen',
> 'rows-per-second' = '5',
> 'fields.orderId.min' = '1',
> 'fields.orderId.max' = '20',
> 'fields.price.min' = '1',
> 'fields.price.max' = '200'
> );
```
`Flink SQL> select histogram(price) as map from orders;`
`res: {147.451 = 1， 65.765 = 1， 41.662 = 1 …}`
see also:
KsqlDB:
https://docs.ksqldb.io/en/latest/developer-guide/ksqldb-reference/aggregate-functions/#histogram
### Verifying this change
- This change added tests in CollectionFunctionsITCase.
### Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
@Public(Evolving): (yes)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
### Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (docs)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-29789) Fix flaky tests in CheckpointCoordinatorTest
[ https://issues.apache.org/jira/browse/FLINK-29789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-29789: --- Labels: pull-request-available stale-minor (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Fix flaky tests in CheckpointCoordinatorTest > > > Key: FLINK-29789 > URL: https://issues.apache.org/jira/browse/FLINK-29789 > Project: Flink > Issue Type: Bug >Reporter: Sopan Phaltankar >Priority: Minor > Labels: pull-request-available, stale-minor > > The test > org.apache.flink.runtime.checkpoint.CheckpointCoordinatorTest.testTriggerAndDeclineCheckpointComplex > is flaky and has the following failure: > Failures: > [ERROR] Failures: > [ERROR] > CheckpointCoordinatorTest.testTriggerAndDeclineCheckpointComplex:1054 > expected:<2> but was:<1> > I used the tool [NonDex|https://github.com/TestingResearchIllinois/NonDex] to > find this flaky test. > Command: mvn -pl flink-runtime edu.illinois:nondex-maven-plugun:1.1.2:nondex > -Dtest=org.apache.flink.runtime.checkpoint.CheckpointCoordinatorTest#testTriggerAndDeclineCheckpointComplex > I analyzed the assertion failure and found that checkpoint1Id and > checkpoint2Id are getting assigned by iterating over a HashMap. > As we know, iterator() returns elements in a random order > [(JavaDoc|https://docs.oracle.com/javase/8/docs/api/java/util/HashMap.html#entrySet--]) > and this might cause test failures for some orders. > Therefore, to remove this non-determinism, we would change HashMap to > LinkedHashMap. > On further analysis, it was found that the Map is getting initialized on line > 1894 of org.apache.flink.runtime.checkpoint.CheckpointCoordinator class. > After changing from HashMap to LinkedHashMap, the above test is passing > without any non-determinism. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32151) 'Run kubernetes pyflink application test' fails while pulling image
[
https://issues.apache.org/jira/browse/FLINK-32151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32151:
---
Labels: stale-major test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> 'Run kubernetes pyflink application test' fails while pulling image
> ---
>
> Key: FLINK-32151
> URL: https://issues.apache.org/jira/browse/FLINK-32151
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Deployment / Kubernetes
>Affects Versions: 1.16.2
>Reporter: Sergey Nuyanzin
>Priority: Major
> Labels: stale-major, test-stability
>
> {noformat}
> 2023-05-16T13:29:39.0614891Z May 16 13:29:39 Current logs for
> flink-native-k8s-pyflink-application-1-6f4c9bfc56-cstw7:
> 2023-05-16T13:29:39.1253736Z Error from server (BadRequest): container
> "flink-main-container" in pod
> "flink-native-k8s-pyflink-application-1-6f4c9bfc56-cstw7" is waiting to
> start: image can't be pulled
> 2023-05-16T13:29:39.2611218Z May 16 13:29:39 deployment.apps
> "flink-native-k8s-pyflink-application-1" deleted
> 2023-05-16T13:29:39.4214711Z May 16 13:29:39
> clusterrolebinding.rbac.authorization.k8s.io "flink-role-binding-default"
> deleted
> 2023-05-16T13:29:40.2644587Z May 16 13:29:40
> pod/flink-native-k8s-pyflink-application-1-6f4c9bfc56-cstw7 condition met
> 2023-05-16T13:29:40.2664618Z May 16 13:29:40 Stopping minikube ...
> 2023-05-16T13:29:40.3396336Z May 16 13:29:40 * Stopping node "minikube" ...
> 2023-05-16T13:29:50.7499872Z May 16 13:29:50 * 1 node stopped.
> {noformat}
> it's very similar to https://issues.apache.org/jira/browse/FLINK-28226
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32239) Unify TestJvmProcess and TestProcessBuilder
[ https://issues.apache.org/jira/browse/FLINK-32239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32239: --- Labels: stale-assigned starter (was: starter) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > Unify TestJvmProcess and TestProcessBuilder > --- > > Key: FLINK-32239 > URL: https://issues.apache.org/jira/browse/FLINK-32239 > Project: Flink > Issue Type: Technical Debt > Components: Test Infrastructure >Reporter: Chesnay Schepler >Assignee: Samrat Deb >Priority: Minor > Labels: stale-assigned, starter > Fix For: 1.18.0 > > > Both of these utility classes are used to spawn additional JVM processes > during tests, and contain a fair bit of duplicated logic. We can unify them > to ease maintenance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-27922) Flink SQL unique key lost when set table.exec.mini-batch.enabled=true
[
https://issues.apache.org/jira/browse/FLINK-27922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-27922:
---
Labels: auto-deprioritized-major pull-request-available (was:
pull-request-available stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Major, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> Flink SQL unique key lost when set table.exec.mini-batch.enabled=true
> -
>
> Key: FLINK-27922
> URL: https://issues.apache.org/jira/browse/FLINK-27922
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.12.2, 1.15.0
> Environment: Flink1.12.2
>Reporter: zhangbin
>Priority: Minor
> Labels: auto-deprioritized-major, pull-request-available
>
> Flink SQL table has primary keys, but when set table.exec.mini-batch.enabled
> =true, the unique key is lost.
> {code:java}
> @Test
> public void testJoinUniqueKey() throws Exception {
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env);
> Configuration configuration = tableEnv.getConfig().getConfiguration();
> configuration.setString("table.exec.mini-batch.enabled", "true");
> configuration.setString("table.exec.mini-batch.allow-latency", "5 s");
> configuration.setString("table.exec.mini-batch.size", "5000");
> StatementSet statementSet = tableEnv.createStatementSet();
> tableEnv.executeSql("CREATE TABLE `t_apply_sku_test`(`dt`
> BIGINT,`refund_apply_id` BIGINT,`base_sku_id` BIGINT,`order_id`
> BIGINT,`user_id` BIGINT,`poi_id` BIGINT,`refund_type`
> BIGINT,`apply_refund_reason_code` BIGINT,`apply_refund_reason_desc`
> VARCHAR,`apply_refund_review_status` BIGINT,`apply_refund_review_status_desc`
> VARCHAR,`apply_refund_reject_reason` VARCHAR,`apply_is_refunded`
> INTEGER,`apply_pic_url` VARCHAR,`remark` VARCHAR,`refund_apply_originator`
> BIGINT,`second_reason_code` BIGINT,`second_reason`
> VARCHAR,`refund_target_account` BIGINT,`after_service_id`
> BIGINT,`receipt_status` BIGINT,`group_header_goods_status`
> INTEGER,`apply_operator_mis_name` VARCHAR,`refund_apply_time`
> BIGINT,`update_time` BIGINT,`base_sku_name` VARCHAR,`apply_refund_num`
> BIGINT,`view_qty` DECIMAL(38,18),`refund_scale_type`
> BIGINT,`refund_scale_type_desc` VARCHAR,`refund_scale`
> DECIMAL(38,18),`apply_refund_amt` DECIMAL(38,18),`refund_scale_user_real_pay`
> DECIMAL(38,18),`refund_red_packet_price` DECIMAL(38,18),`load_time`
> VARCHAR,`take_rate_type` BIGINT,`platform_rate`
> DECIMAL(38,18),`order_sku_type` INTEGER,`second_reason_aggregated_code`
> INTEGER,`second_reason_aggregated` VARCHAR,`compensation_amount`
> DECIMAL(38,18),`aftersale_type` INTEGER,`group_header_parallel_status`
> INTEGER,`grid_parallel_status` INTEGER) WITH ('connector'='blackhole')");
> tableEnv.executeSql("CREATE TABLE `t_name`(`id` BIGINT,`after_service_id`
> BIGINT PRIMARY KEY NOT ENFORCED,`order_id` BIGINT,`user_id` BIGINT,`poi_id`
> BIGINT,`city_id` BIGINT,`refund_type` INTEGER,`first_reason_code`
> INTEGER,`first_reason` VARCHAR,`second_reason_code` INTEGER,`second_reason`
> VARCHAR,`pic_url` VARCHAR,`remark` VARCHAR,`refund_price`
> INTEGER,`refund_red_packet_price` INTEGER,`refund_total_price`
> INTEGER,`refund_promotion_price` INTEGER,`refund_coupon_price`
> INTEGER,`refund_other_price` INTEGER,`user_receipt_status`
> INTEGER,`collect_status` INTEGER,`refund_target_account` INTEGER,`status`
> INTEGER,`flow_instance_id` BIGINT,`create_time` BIGINT,`modify_time` BIGINT)
> WITH ('connector'='datagen')");
> tableEnv.executeSql("CREATE TABLE `t_item_name`(`id`
> BIGINT,`refund_fwd_item_id` BIGINT,`after_service_id` BIGINT,`order_id`
> BIGINT,`order_item_id` BIGINT,`stack_id` BIGINT,`sku_id` BIGINT,`sku_name`
> VARCHAR,`supplier_id` BIGINT,`refund_quantity` INTEGER,`item_sku_type`
> INTEGER,`refund_scale_type` INTEGER,`refund_scale` INTEGER,`accurate_refund`
> INTEGER,`refund_price` INTEGER,`refund_red_packet_price`
> INTEGER,`refund_price_info` VARCHAR,`refund_total_price`
> INTEGER,`refund_promotion_price` INTEGER,`refund_coupon_price`
> INTEGER,`refund_other_price` INTEGER,`extend_info` VARCHAR,`create_time`
> BIGINT,`modify_time` BIGINT) WITH ('connector'='datagen')");
> tableEnv.executeSql("CREATE TABLE `t_progress_name`(`id`
> BIGINT,`after_service_id` BIGINT PRIMARY KEY NOT ENFORCED,`order_id`
> BIGINT,`progress_node` VARCHAR,`progress_node_status` INTEGER,`operator`
> VARCHAR,`parallel` INTEGER,`flow_element_id`
[jira] [Updated] (FLINK-32188) Support to "where" query with a fixed-value array and simplify condition for an array-type filed.
[ https://issues.apache.org/jira/browse/FLINK-32188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32188: --- Labels: pull-request-available stale-major (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 60 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Support to "where" query with a fixed-value array and simplify condition for > an array-type filed. > - > > Key: FLINK-32188 > URL: https://issues.apache.org/jira/browse/FLINK-32188 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.12.2, 1.17.0, 1.16.1 >Reporter: Xin Chen >Priority: Major > Labels: pull-request-available, stale-major > Attachments: image-2023-05-25-17-16-02-288.png, > image-2023-05-25-20-44-08-834.png, image-2023-05-25-20-44-47-581.png, > image-2023-06-06-16-50-10-805.png, image-2023-06-06-16-50-54-467.png, > screenshot-1.png, screenshot-10.png, screenshot-11.png, screenshot-12.png, > screenshot-2.png, screenshot-3.png, screenshot-4.png, screenshot-5.png, > screenshot-6.png, screenshot-7.png, screenshot-8.png, screenshot-9.png > > > When I customized a data source connector which assumed as image-connector, I > met issues while creating a table with ddl to specify a field "URL" as an > array type. When submitting an SQL task with Flink, I specified query of this > field with a fixed array. For example, "select * from image source where > URL=ARRAY ['/flink. jpg', '/flink_1. jpg']", but it couldn't obtain the > corresponding predicate filters at all. > Does the custom connector not support to query fields of array type with > "where"？ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31755) ROW function can not work with RewriteIntersectAllRule
[
https://issues.apache.org/jira/browse/FLINK-31755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-31755:
---
Labels: pull-request-available stale-major (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> ROW function can not work with RewriteIntersectAllRule
> --
>
> Key: FLINK-31755
> URL: https://issues.apache.org/jira/browse/FLINK-31755
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: Aitozi
>Priority: Major
> Labels: pull-request-available, stale-major
>
> Reproduce case:
> {code:java}
> create table row_sink (
> `b` ROW
> ) with (
> 'connector' = 'values'
> )
> util.verifyRelPlanInsert(
> "INSERT INTO row_sink " +
> "SELECT ROW(a, b) FROM complex_type_src intersect all " +
> "SELECT ROW(c, d) FROM complex_type_src ")
> {code}
> It will fails with
> {code:java}
> Caused by: java.lang.IllegalArgumentException: Type mismatch:
> rel rowtype: RecordType(RecordType:peek_no_expand(VARCHAR(2147483647)
> CHARACTER SET "UTF-16LE" EXPR$0, INTEGER EXPR$1) NOT NULL EXPR$0) NOT NULL
> equiv rowtype: RecordType(RecordType(VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" EXPR$0, INTEGER EXPR$1) NOT NULL EXPR$0) NOT NULL
> Difference:
> EXPR$0: RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" EXPR$0, INTEGER EXPR$1) NOT NULL -> RecordType(VARCHAR(2147483647)
> CHARACTER SET "UTF-16LE" EXPR$0, INTEGER EXPR$1) NOT NULL
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:592)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:613)
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:144)
> ... 68 more
> {code}
> The reason is:
> ROW function will generates the {{FULLY_QUALIFIED}} type. But after the
> {{RewriteIntersectAllRule}} optimization, it will produce the
> {{PEEK_FIELDS_NO_EXPAND}}. So the volcano planner complains with type
> mismatch.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31862) KafkaSinkITCase.testStartFromSavepoint is unstable
[
https://issues.apache.org/jira/browse/FLINK-31862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-31862:
---
Labels: stale-major test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> KafkaSinkITCase.testStartFromSavepoint is unstable
> --
>
> Key: FLINK-31862
> URL: https://issues.apache.org/jira/browse/FLINK-31862
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Tests
>Affects Versions: 1.17.0, 1.18.0
>Reporter: Sergey Nuyanzin
>Priority: Major
> Labels: stale-major, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=48243=logs=9c5a5fe6-2f39-545e-1630-feb3d8d0a1ba=99b23320-1d05-5741-d63f-9e78473da39e=36611
> {noformat}
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.TimeoutException: The request timed out.
> Apr 19 01:42:20 at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> Apr 19 01:42:20 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> Apr 19 01:42:20 at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
> Apr 19 01:42:20 at
> org.apache.flink.connector.kafka.sink.testutils.KafkaSinkExternalContext.createTopic(KafkaSinkExternalContext.java:101)
> Apr 19 01:42:20 ... 111 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31493) helm upgrade does not work, because repo path does not follow helm standards
[ https://issues.apache.org/jira/browse/FLINK-31493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-31493: --- Labels: helm stale-major (was: helm) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 60 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > helm upgrade does not work, because repo path does not follow helm standards > > > Key: FLINK-31493 > URL: https://issues.apache.org/jira/browse/FLINK-31493 > Project: Flink > Issue Type: Bug > Components: Kubernetes Operator >Reporter: Emmanuel Leroy >Priority: Major > Labels: helm, stale-major > > the helm repo for flink-operator is a folder that includes the version, which > is not following the helm chart repo standards. > In a standard helm repo, the repo URL is the name of the product (without > version) and then the folder includes the different versions of the chart. > This is an issue because the repo itself needs to be installed every time the > version is upgraded, as opposed to adding the repo once and then upgrading > the version. > When attempting to add the latest repo, helm will complain that the repo > already exists. It is necessary to first remove the repo, and then add the > updated one. > When trying to upgrade the chart, it doesn't work, because helm expects the > chart of the previous version to be in the same repo, but it cannot be found > in the newly added repo. > So the chart needs to be uninstalled, then the new one installed. > The solution is to use a common path for all versions of the chart, and > maintain a manifest with the various versions (instead of different folders > with different manifests) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32156) Int2AdaptiveHashJoinOperatorTest produced no output for 900s on AZP
[
https://issues.apache.org/jira/browse/FLINK-32156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32156:
---
Labels: stale-critical test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Critical but is unassigned and neither itself nor its Sub-Tasks have been
updated for 14 days. I have gone ahead and marked it "stale-critical". If this
ticket is critical, please either assign yourself or give an update.
Afterwards, please remove the label or in 7 days the issue will be
deprioritized.
> Int2AdaptiveHashJoinOperatorTest produced no output for 900s on AZP
> ---
>
> Key: FLINK-32156
> URL: https://issues.apache.org/jira/browse/FLINK-32156
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.17.2
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: stale-critical, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=48892=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=10930
> {noformat}
> May 11 06:25:13
> ==
> May 11 06:25:13 Process produced no output for 900 seconds.
> May 11 06:25:13
> ==
> ...
> May 11 06:25:14 "main" #1 prio=5 os_prio=0 tid=0x7f672c00b800 nid=0x4b8
> waiting on condition [0x7f6735dbd000]
> May 11 06:25:14java.lang.Thread.State: RUNNABLE
> May 11 06:25:14 at
> org.apache.flink.table.runtime.util.UniformBinaryRowGenerator.next(UniformBinaryRowGenerator.java:90)
> May 11 06:25:14 at
> org.apache.flink.table.runtime.util.UniformBinaryRowGenerator.next(UniformBinaryRowGenerator.java:27)
> May 11 06:25:14 at
> org.apache.flink.runtime.operators.testutils.UnionIterator.next(UnionIterator.java:61)
> May 11 06:25:14 at
> org.apache.flink.table.runtime.operators.join.Int2HashJoinOperatorTestBase.joinAndAssert(Int2HashJoinOperatorTestBase.java:271)
> May 11 06:25:14 at
> org.apache.flink.table.runtime.operators.join.Int2HashJoinOperatorTestBase.buildJoin(Int2HashJoinOperatorTestBase.java:77)
> May 11 06:25:14 at
> org.apache.flink.table.runtime.operators.join.Int2AdaptiveHashJoinOperatorTest.testBuildFirstHashLeftOutJoinFallbackToSMJ(Int2AdaptiveHashJoinOperatorTest.java:114)
> May 11 06:25:14 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> May 11 06:25:14 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> May 11 06:25:14 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> May 11 06:25:14 at java.lang.reflect.Method.invoke(Method.java:498)
> May 11 06:25:14 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> May 11 06:25:14 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> May 11 06:25:14 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31567) Promote release 1.17
[
https://issues.apache.org/jira/browse/FLINK-31567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-31567:
---
Labels: pull-request-available stale-major (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Promote release 1.17
>
>
> Key: FLINK-31567
> URL: https://issues.apache.org/jira/browse/FLINK-31567
> Project: Flink
> Issue Type: New Feature
>Reporter: Matthias Pohl
>Priority: Major
> Labels: pull-request-available, stale-major
>
> Once the release has been finalized (FLINK-31562), the last step of the
> process is to promote the release within the project and beyond. Please wait
> for 24h after finalizing the release in accordance with the [ASF release
> policy|http://www.apache.org/legal/release-policy.html#release-announcements].
> *Final checklist to declare this issue resolved:*
> # Website pull request to [list the
> release|http://flink.apache.org/downloads.html] merged
> # Release announced on the user@ mailing list.
> # Blog post published, if applicable.
> # Release recorded in
> [reporter.apache.org|https://reporter.apache.org/addrelease.html?flink].
> # Release announced on social media.
> # Completion declared on the dev@ mailing list.
> # Update Homebrew:
> [https://docs.brew.sh/How-To-Open-a-Homebrew-Pull-Request] (seems to be done
> automatically - at least for minor releases for both minor and major
> releases)
> # Update quickstart scripts in {{{}flink-web{}}}, under the {{q/}} directory
> # Updated the japicmp configuration
> ** corresponding SNAPSHOT branch japicmp reference version set to the just
> released version, and API compatibiltity checks for {{@PublicEvolving}} was
> enabled
> ** (minor version release only) master branch japicmp reference version set
> to the just released version
> ** (minor version release only) master branch japicmp exclusions have been
> cleared
> # Update the list of previous version in {{docs/config.toml}} on the master
> branch.
> # Set {{show_outdated_warning: true}} in {{docs/config.toml}} in the branch
> of the _now deprecated_ Flink version (i.e. 1.15 if 1.17.0 is released)
> # Update stable and master alias in
> [https://github.com/apache/flink/blob/master/.github/workflows/docs.yml]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32036) TableEnvironmentTest.test_explain is unstable on azure ci
[
https://issues.apache.org/jira/browse/FLINK-32036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32036:
---
Labels: stale-critical test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Critical but is unassigned and neither itself nor its Sub-Tasks have been
updated for 14 days. I have gone ahead and marked it "stale-critical". If this
ticket is critical, please either assign yourself or give an update.
Afterwards, please remove the label or in 7 days the issue will be
deprioritized.
> TableEnvironmentTest.test_explain is unstable on azure ci
> -
>
> Key: FLINK-32036
> URL: https://issues.apache.org/jira/browse/FLINK-32036
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.17.1
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: stale-critical, test-stability
>
> it's failed on ci (1.17 branch so far)
> {noformat}
> May 07 01:51:35 === FAILURES
> ===
> May 07 01:51:35 __ TableEnvironmentTest.test_explain
> ___
> May 07 01:51:35
> May 07 01:51:35 self =
> testMethod=test_explain>
> May 07 01:51:35
> May 07 01:51:35 def test_explain(self):
> May 07 01:51:35 schema = RowType() \
> May 07 01:51:35 .add('a', DataTypes.INT()) \
> May 07 01:51:35 .add('b', DataTypes.STRING()) \
> May 07 01:51:35 .add('c', DataTypes.STRING())
> May 07 01:51:35 t_env = self.t_env
> May 07 01:51:35 t = t_env.from_elements([], schema)
> May 07 01:51:35 result = t.select(t.a + 1, t.b, t.c)
> May 07 01:51:35
> May 07 01:51:35 > actual = result.explain()
> May 07 01:51:35
> May 07 01:51:35 pyflink/table/tests/test_table_environment_api.py:66
> {noformat}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=48766=logs=bf5e383b-9fd3-5f02-ca1c-8f788e2e76d3=85189c57-d8a0-5c9c-b61d-fc05cfac62cf=25029
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32159) Hudi Source throws NPE
[ https://issues.apache.org/jira/browse/FLINK-32159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32159: --- Labels: pull-request-available stale-major (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 60 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Hudi Source throws NPE > -- > > Key: FLINK-32159 > URL: https://issues.apache.org/jira/browse/FLINK-32159 > Project: Flink > Issue Type: Bug > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.15.0, 1.16.0, 1.17.0, 1.18.0 >Reporter: Bo Cui >Priority: Major > Labels: pull-request-available, stale-major > Attachments: image-2023-05-23-14-45-29-151.png > > > spark/hive write hudi, and flink read hudi and job failed. because > !image-2023-05-23-14-45-29-151.png! > > The null judgment logic should be added to AbstractColumnReader#readToVector > https://github.com/apache/flink/blob/119b8c584dc865ee8a40a5c6410dddf8b36bac5a/flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/vector/reader/AbstractColumnReader.java#LL155C19-L155C20 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-26514) YARNSessionFIFOITCase.testDetachedMode failed on azure
[
https://issues.apache.org/jira/browse/FLINK-26514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-26514:
---
Labels: auto-deprioritized-major test-stability (was: stale-major
test-stability)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Major, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> YARNSessionFIFOITCase.testDetachedMode failed on azure
> --
>
> Key: FLINK-26514
> URL: https://issues.apache.org/jira/browse/FLINK-26514
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.15.0, 1.16.0, 1.17.0
>Reporter: Yun Gao
>Priority: Minor
> Labels: auto-deprioritized-major, test-stability
>
> {code:java}
> /tmp/.yarn-properties-agent07_azpcontainer
> 2022-03-07T03:32:11.1761848Z Mar 07 03:32:11 03:32:10,835 [ Time-limited
> test] INFO org.apache.flink.yarn.YARNSessionFIFOITCase [] -
> Finished testDetachedMode()
> 2022-03-07T03:32:35.2261406Z Mar 07 03:32:35 [INFO] Tests run: 3, Failures:
> 0, Errors: 0, Skipped: 0, Time elapsed: 70.987 s - in
> org.apache.flink.yarn.YARNSessionFIFOSecuredITCase
> 2022-03-07T03:32:35.6496081Z Mar 07 03:32:35 [INFO]
> 2022-03-07T03:32:35.6497443Z Mar 07 03:32:35 [INFO] Results:
> 2022-03-07T03:32:35.6498560Z Mar 07 03:32:35 [INFO]
> 2022-03-07T03:32:35.6499136Z Mar 07 03:32:35 [ERROR] Failures:
> 2022-03-07T03:32:35.6501226Z Mar 07 03:32:35 [ERROR]
> YARNSessionFIFOITCase.checkForProhibitedLogContents:82->YarnTestBase.ensureNoProhibitedStringInLogFiles:591
> Found a file
> /__w/1/s/flink-yarn-tests/target/flink-yarn-tests-fifo/flink-yarn-tests-fifo-logDir-nm-0_0/application_1646623837899_0001/container_1646623837899_0001_01_01/jobmanager.log
> with a prohibited string (one of [Exception, Started
> SelectChannelConnector@0.0.0.0:8081]). Excerpts:
> 2022-03-07T03:32:35.6502705Z Mar 07 03:32:35 [
> 2022-03-07T03:32:35.6503734Z Mar 07 03:32:35 2022-03-07 03:30:59,734 INFO
> org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent
> [] - Closing components.
> 2022-03-07T03:32:35.6505105Z Mar 07 03:32:35 2022-03-07 03:30:59,735 INFO
> org.apache.flink.runtime.dispatcher.runner.DefaultDispatcherRunner [] -
> DefaultDispatcherRunner was revoked the leadership with leader id
> ----. Stopping the DispatcherLeaderProcess.
> 2022-03-07T03:32:35.6507058Z Mar 07 03:32:35 2022-03-07 03:30:59,735 INFO
> org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess []
> - Stopping SessionDispatcherLeaderProcess.
> 2022-03-07T03:32:35.6508245Z Mar 07 03:32:35 2022-03-07 03:30:59,735 INFO
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Stopping
> dispatcher akka.tcp://flink@87ed88cbeaa9:44509/user/rpc/dispatcher_0.
> 2022-03-07T03:32:35.6509555Z Mar 07 03:32:35 2022-03-07 03:30:59,735 INFO
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Stopping
> all currently running jobs of dispatcher
> akka.tcp://flink@87ed88cbeaa9:44509/user/rpc/dispatcher_0.
> 2022-03-07T03:32:35.6511062Z Mar 07 03:32:35 2022-03-07 03:30:59,736 INFO
> org.apache.flink.runtime.resourcemanager.ResourceManagerServiceImpl [] -
> Stopping resource manager service.
> 2022-03-07T03:32:35.6512171Z Mar 07 03:32:35 2022-03-07 03:30:59,736 INFO
> org.apache.flink.runtime.resourcemanager.ResourceManagerServiceImpl [] -
> Resource manager service is not running. Ignore revoking leadership.
> 2022-03-07T03:32:35.6513468Z Mar 07 03:32:35 2022-03-07 03:30:59,737 INFO
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Stopped
> dispatcher akka.tcp://flink@87ed88cbeaa9:44509/user/rpc/dispatcher_0.
> 2022-03-07T03:32:35.6515017Z Mar 07 03:32:35 2022-03-07 03:30:59,739 ERROR
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl [] -
> Exception on heartbeat
> 2022-03-07T03:32:35.6515805Z Mar 07 03:32:35 java.io.InterruptedIOException:
> Interrupted waiting to send RPC request to server
> 2022-03-07T03:32:35.6516376Z Mar 07 03:32:35 java.io.InterruptedIOException:
> Interrupted waiting to send RPC request to server
> 2022-03-07T03:32:35.6517169Z Mar 07 03:32:35 at
> org.apache.hadoop.ipc.Client.call(Client.java:1446)
> ~[hadoop-common-3.1.3.jar:?]
> 2022-03-07T03:32:35.6517979Z Mar 07 03:32:35 at
> org.apache.hadoop.ipc.Client.call(Client.java:1388)
> ~[hadoop-common-3.1.3.jar:?]
> 2022-03-07T03:32:35.6518850Z Mar 07 03:32:35 at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
> ~[hadoop-common-3.1.3.jar:?]
>
[jira] [Updated] (FLINK-28358) when debug in local ,throw out "The system time period specification expects Timestamp type but is 'TIMESTAMP_WITH_LOCAL_TIME_ZONE' " exception
[
https://issues.apache.org/jira/browse/FLINK-28358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-28358:
---
Labels: debug pull-request-available stale-minor (was: debug
pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> when debug in local ,throw out "The system time period specification expects
> Timestamp type but is 'TIMESTAMP_WITH_LOCAL_TIME_ZONE' " exception
> ---
>
> Key: FLINK-28358
> URL: https://issues.apache.org/jira/browse/FLINK-28358
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.14.4
> Environment: maven:3.2.5 maven:3.6.1 maven:3.3.9
> openjdk:1.8.0_333
> idea:IntelliJ IDEA 2021.3 (Ultimate Edition)
>Reporter: PengfeiChang
>Priority: Minor
> Labels: debug, pull-request-available, stale-minor
> Attachments: image-2022-07-06-20-31-29-743.png
>
>
> h1. subject
> when i debug in local to see the jdbcconnector lookup mechanism and run
> org.apache.flink.connector.jdbc.table.JdbcLookupTableITCase.testLookup,throw
> out a exception ,detail as follow:
> {code:java}
> org.apache.flink.table.api.ValidationException: SQL validation failed. From
> line 1, column 106 to line 1, column 120: The system time period
> specification expects Timestamp type but is 'TIMESTAMP_WITH_LOCAL_TIME_ZONE'
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:164)
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:107)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:215)
> at
> org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:101)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:736)
> at
> org.apache.flink.connector.jdbc.table.JdbcLookupTableITCase.useDynamicTableFactory(JdbcLookupTableITCase.java:195)
> at
> org.apache.flink.connector.jdbc.table.JdbcLookupTableITCase.testLookup(JdbcLookupTableITCase.java:81)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at
[jira] [Updated] (FLINK-28750) Whether to add field comment for hive table
[
https://issues.apache.org/jira/browse/FLINK-28750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-28750:
---
Labels: pull-request-available stale-minor (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Whether to add field comment for hive table
> ---
>
> Key: FLINK-28750
> URL: https://issues.apache.org/jira/browse/FLINK-28750
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Hive
>Affects Versions: 1.14.5
>Reporter: hehuiyuan
>Priority: Minor
> Labels: pull-request-available, stale-minor
> Attachments: image-2022-07-30-15-53-03-754.png,
> image-2022-07-30-16-36-37-032.png
>
>
> Currently, I have a hive ddl，as follows
> {code:java}
> "set table.sql-dialect=hive;\n" +
> "CREATE TABLE IF NOT EXISTS myhive.dev.shipu3_test_1125 (\n" +
> " `id` int COMMENT 'ia',\n" +
> " `cartdid` bigint COMMENT 'aaa',\n" +
> " `customer` string COMMENT '',\n" +
> " `product` string COMMENT '',\n" +
> " `price` double COMMENT '',\n" +
> " `dt` STRING COMMENT ''\n" +
> ") PARTITIONED BY (dt STRING) STORED AS TEXTFILE TBLPROPERTIES (\n" +
> " 'streaming-source.enable' = 'false',\n" +
> " 'streaming-source.partition.include' = 'all',\n" +
> " 'lookup.join.cache.ttl' = '12 h'\n" +
> ")"; {code}
> It is parsed as SqlCreateHiveTable by hive dialect parser. But the field
> commet is lost.
>
>
> !image-2022-07-30-16-36-37-032.png|width=777,height=526!
>
>
>
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32393) NettyClientServerSslTest.testSslPinningForInvalidFingerprint fails with Address already in use
[
https://issues.apache.org/jira/browse/FLINK-32393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32393:
---
Labels: stale-critical test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Critical but is unassigned and neither itself nor its Sub-Tasks have been
updated for 14 days. I have gone ahead and marked it "stale-critical". If this
ticket is critical, please either assign yourself or give an update.
Afterwards, please remove the label or in 7 days the issue will be
deprioritized.
> NettyClientServerSslTest.testSslPinningForInvalidFingerprint fails with
> Address already in use
> --
>
> Key: FLINK-32393
> URL: https://issues.apache.org/jira/browse/FLINK-32393
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.17.2
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: stale-critical, test-stability
>
> This build
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=50162=logs=77a9d8e1-d610-59b3-fc2a-4766541e0e33=125e07e7-8de0-5c6c-a541-a567415af3ef=7794
> fails with
> {noformat}
> Jun 19 05:40:33 [ERROR] Tests run: 14, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 10.095 s <<< FAILURE! - in
> org.apache.flink.runtime.io.network.netty.NettyClientServerSslTest
> Jun 19 05:40:33 [ERROR]
> NettyClientServerSslTest.testSslPinningForInvalidFingerprint Time elapsed:
> 1.236 s <<< ERROR!
> Jun 19 05:40:33
> org.apache.flink.shaded.netty4.io.netty.channel.unix.Errors$NativeIoException:
> bind(..) failed: Address already in use
> Jun 19 05:40:33
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32353) Update Cassandra connector archunit violations with Flink 1.18 rules
[
https://issues.apache.org/jira/browse/FLINK-32353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32353:
---
Labels: pull-request-available stale-assigned (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 30 days, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it.
> Update Cassandra connector archunit violations with Flink 1.18 rules
>
>
> Key: FLINK-32353
> URL: https://issues.apache.org/jira/browse/FLINK-32353
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Cassandra
>Reporter: Martijn Visser
>Assignee: Etienne Chauchot
>Priority: Major
> Labels: pull-request-available, stale-assigned
>
> The current Cassandra connector in {{main}} fails when testing against Flink
> 1.18-SNAPSHOT
> {code:java}
> Error: Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 8.1 s
> <<< FAILURE! - in org.apache.flink.architecture.rules.ITCaseRules
> Error: ITCaseRules.ITCASE_USE_MINICLUSTER Time elapsed: 0.025 s <<<
> FAILURE!
> java.lang.AssertionError:
> Architecture Violation [Priority: MEDIUM] - Rule 'ITCASE tests should use a
> MiniCluster resource or extension' was violated (1 times):
> org.apache.flink.streaming.connectors.cassandra.CassandraConnectorITCase does
> not satisfy: only one of the following predicates match:
> * reside in a package 'org.apache.flink.runtime.*' and contain any fields
> that are static, final, and of type InternalMiniClusterExtension and
> annotated with @RegisterExtension
> * reside outside of package 'org.apache.flink.runtime.*' and contain any
> fields that are static, final, and of type MiniClusterExtension and annotated
> with @RegisterExtension or are , and of type MiniClusterTestEnvironment and
> annotated with @TestEnv
> * reside in a package 'org.apache.flink.runtime.*' and is annotated with
> @ExtendWith with class InternalMiniClusterExtension
> * reside outside of package 'org.apache.flink.runtime.*' and is annotated
> with @ExtendWith with class MiniClusterExtension
> or contain any fields that are public, static, and of type
> MiniClusterWithClientResource and final and annotated with @ClassRule or
> contain any fields that is of type MiniClusterWithClientResource and public
> and final and not static and annotated with @Rule
> {code}
> https://github.com/apache/flink-connector-cassandra/actions/runs/5276835802/jobs/9544092571#step:13:811
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31877) StreamExecutionEnvironmentTests.test_from_collection_with_data_types is unstable
[
https://issues.apache.org/jira/browse/FLINK-31877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-31877:
---
Labels: stale-critical test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Critical but is unassigned and neither itself nor its Sub-Tasks have been
updated for 14 days. I have gone ahead and marked it "stale-critical". If this
ticket is critical, please either assign yourself or give an update.
Afterwards, please remove the label or in 7 days the issue will be
deprioritized.
> StreamExecutionEnvironmentTests.test_from_collection_with_data_types is
> unstable
>
>
> Key: FLINK-31877
> URL: https://issues.apache.org/jira/browse/FLINK-31877
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.16.1
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: stale-critical, test-stability
>
> {noformat}
> Apr 21 05:11:45 === FAILURES
> ===
> Apr 21 05:11:45 _
> StreamExecutionEnvironmentTests.test_from_collection_with_data_types _
> Apr 21 05:11:45
> Apr 21 05:11:45 self =
> testMethod=test_from_collection_with_data_types>
> Apr 21 05:11:45
> Apr 21 05:11:45 def test_from_collection_with_data_types(self):
> Apr 21 05:11:45 # verify from_collection for the collection with
> single object.
> Apr 21 05:11:45 ds = self.env.from_collection(['Hi', 'Hello'],
> type_info=Types.STRING())
> Apr 21 05:11:45 ds.add_sink(self.test_sink)
> Apr 21 05:11:45 > self.env.execute("test from collection with single
> object")
> Apr 21 05:11:45
> Apr 21 05:11:45
> pyflink/datastream/tests/test_stream_execution_environment.py:257:
> Apr 21 05:11:45 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _ _ _ _
> Apr 21 05:11:45 pyflink/datastream/stream_execution_environment.py:764: in
> execute
> Apr 21 05:11:45 return
> JobExecutionResult(self._j_stream_execution_environment.execute(j_stream_graph))
> Apr 21 05:11:45
> .tox/py38-cython/lib/python3.8/site-packages/py4j/java_gateway.py:1321: in
> __call__
> Apr 21 05:11:45 return_value = get_return_value(
> Apr 21 05:11:45 pyflink/util/exceptions.py:146: in deco
> Apr 21 05:11:45 return f(*a, **kw)
> Apr 21 05:11:45 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _ _ _ _
> {noformat}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=48320=logs=821b528f-1eed-5598-a3b4-7f748b13f261=6bb545dd-772d-5d8c-f258-f5085fba3295=31864
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31595) MiniBatchLocalGroupAggFunction produces wrong aggregate results with state clean up
[ https://issues.apache.org/jira/browse/FLINK-31595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-31595: --- Labels: pull-request-available stale-critical (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Critical but is unassigned and neither itself nor its Sub-Tasks have been updated for 14 days. I have gone ahead and marked it "stale-critical". If this ticket is critical, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > MiniBatchLocalGroupAggFunction produces wrong aggregate results with state > clean up > --- > > Key: FLINK-31595 > URL: https://issues.apache.org/jira/browse/FLINK-31595 > Project: Flink > Issue Type: Bug >Reporter: Bo Cui >Priority: Critical > Labels: pull-request-available, stale-critical > > If the upstream operator supports retract data, and the first data in a batch > may be retract data, and the retract data should be ignored. > https://github.com/apache/flink/blob/a64781b1ef8f129021bdcddd3b07548e6caa4a72/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/aggregate/MiniBatchLocalGroupAggFunction.java#L68 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31756) KafkaTableITCase.testStartFromGroupOffsetsNone fails due to UnknownTopicOrPartitionException
[
https://issues.apache.org/jira/browse/FLINK-31756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-31756:
---
Labels: stale-critical test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Critical but is unassigned and neither itself nor its Sub-Tasks have been
updated for 14 days. I have gone ahead and marked it "stale-critical". If this
ticket is critical, please either assign yourself or give an update.
Afterwards, please remove the label or in 7 days the issue will be
deprioritized.
> KafkaTableITCase.testStartFromGroupOffsetsNone fails due to
> UnknownTopicOrPartitionException
>
>
> Key: FLINK-31756
> URL: https://issues.apache.org/jira/browse/FLINK-31756
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.17.0
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: stale-critical, test-stability
>
> The following build fails with {{UnknownTopicOrPartitionException}}
> {noformat}
> Dec 03 01:10:59 Multiple Failures (1 failure)
> Dec 03 01:10:59 -- failure 1 --
> Dec 03 01:10:59 [Any cause is instance of class 'class
> org.apache.kafka.clients.consumer.NoOffsetForPartitionException']
> Dec 03 01:10:59 Expecting any element of:
> Dec 03 01:10:59 [java.lang.IllegalStateException: Fail to create topic
> [groupOffset_json_dc640086-d1f1-48b8-ad7a-f83d33b6a03c partitions: 4
> replication factor: 1].
> Dec 03 01:10:59 at
> org.apache.flink.streaming.connectors.kafka.table.KafkaTableTestBase.createTestTopic(KafkaTableTestBase.java:143)
> Dec 03 01:10:59 at
> org.apache.flink.streaming.connectors.kafka.table.KafkaTableITCase.startFromGroupOffset(KafkaTableITCase.java:881)
> Dec 03 01:10:59 at
> org.apache.flink.streaming.connectors.kafka.table.KafkaTableITCase.testStartFromGroupOffsetsWithNoneResetStrategy(KafkaTableITCase.java:981)
> Dec 03 01:10:59 ...(64 remaining lines not displayed - this can be
> changed with Assertions.setMaxStackTraceElementsDisplayed),
> Dec 03 01:10:59 java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.TimeoutException: The request timed out.
> Dec 03 01:10:59 at
> java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
> Dec 03 01:10:59 at
> java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
> Dec 03 01:10:59 at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
> Dec 03 01:10:59 ...(67 remaining lines not displayed - this can be
> changed with Assertions.setMaxStackTraceElementsDisplayed),
> Dec 03 01:10:59 org.apache.kafka.common.errors.TimeoutException: The
> request timed out.
> Dec 03 01:10:59 ]
> Dec 03 01:10:59 to satisfy the given assertions requirements but none did:
> Dec 03 01:10:59
> Dec 03 01:10:59 java.lang.IllegalStateException: Fail to create topic
> [groupOffset_json_dc640086-d1f1-48b8-ad7a-f83d33b6a03c partitions: 4
> replication factor: 1].
> Dec 03 01:10:59 at
> org.apache.flink.streaming.connectors.kafka.table.KafkaTableTestBase.createTestTopic(KafkaTableTestBase.java:143)
> Dec 03 01:10:59 at
> org.apache.flink.streaming.connectors.kafka.table.KafkaTableITCase.startFromGroupOffset(KafkaTableITCase.java:881)
> Dec 03 01:10:59 at
> org.apache.flink.streaming.connectors.kafka.table.KafkaTableITCase.testStartFromGroupOffsetsWithNoneResetStrategy(KafkaTableITCase.java:981)
> Dec 03 01:10:59 ...(64 remaining lines not displayed - this can be
> changed with Assertions.setMaxStackTraceElementsDisplayed)
> Dec 03 01:10:59 error:
> Dec 03 01:10:59 Expecting actual throwable to be an instance of:
> Dec 03 01:10:59
> org.apache.kafka.clients.consumer.NoOffsetForPartitionException
> Dec 03 01:10:59 but was:
> Dec 03 01:10:59 java.lang.IllegalStateException: Fail to create topic
> [groupOffset_json_dc640086-d1f1-48b8-ad7a-f83d33b6a03c partitions: 4
> replication factor: 1].
> [...]
> {noformat}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=47892=logs=aa18c3f6-13b8-5f58-86bb-c1cffb239496=502fb6c0-30a2-5e49-c5c2-a00fa3acb203=36657
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28839) Incorrect english sentence in docs/content/docs/dev/dataset/iterations.md
[ https://issues.apache.org/jira/browse/FLINK-28839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-28839: --- Labels: easy-fix stale-minor starter (was: easy-fix starter) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Incorrect english sentence in docs/content/docs/dev/dataset/iterations.md > - > > Key: FLINK-28839 > URL: https://issues.apache.org/jira/browse/FLINK-28839 > Project: Flink > Issue Type: Bug > Components: Documentation >Reporter: Bisvarup Mukherjee >Priority: Minor > Labels: easy-fix, stale-minor, starter > Attachments: Screenshot 2022-08-05 at 5.56.52 PM.png > > > There is this line in the[ iteration > documentation|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/dataset/iterations/#example-propagate-minimum-in-graph] > that reads like > _"it can be skipped it in the next workset",_ > This line does not look like properly formatted English, it should have been > _"it can be skipped in the next workset"_ > > !Screenshot 2022-08-05 at 5.56.52 PM.png|width=878,height=320! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-30743) Improve Kubernetes HA Services docs
[
https://issues.apache.org/jira/browse/FLINK-30743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-30743:
---
Labels: pull-request-available stale-minor (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Improve Kubernetes HA Services docs
> ---
>
> Key: FLINK-30743
> URL: https://issues.apache.org/jira/browse/FLINK-30743
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Wolfgang Buchner
>Priority: Minor
> Labels: pull-request-available, stale-minor
>
> i recently tried to setup a flink standalone session cluster with kubernetes
> HA and needed to adjust configmap RBACs settings via try & error because the
> documentation of:
> *
> [https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/ha/kubernetes_ha/]
> *
> [https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/resource-providers/standalone/kubernetes/#high-availability-with-standalone-kubernetes]
> didn't state everything which was needed. E.g. for configmap resource the
> verb "watch" is essential.
> With
> {color:#569cd6}apiVersion{color}{color:#d4d4d4}:
> {color}{color:#ce9178}v1{color}
> {color:#569cd6}kind{color}{color:#d4d4d4}:
> {color}{color:#ce9178}ServiceAccount{color}
> {color:#569cd6}metadata{color}{color:#d4d4d4}:{color}
> {color:#d4d4d4} {color}{color:#569cd6}labels{color}{color:#d4d4d4}:{color}
> {color:#d4d4d4} {color}{color:#569cd6}app{color}{color:#d4d4d4}:
> {color}{color:#ce9178}flink{color}
> {color:#d4d4d4} {color}{color:#569cd6}name{color}{color:#d4d4d4}:
> {color}{color:#ce9178}flink-service-account{color}
> {color:#d4d4d4} {color}{color:#569cd6}namespace{color}{color:#d4d4d4}:
> {color}{color:#ce9178}lakehouse{color}
> {color:#d4d4d4}---{color}
> {color:#569cd6}apiVersion{color}{color:#d4d4d4}:
> {color}{color:#ce9178}rbac.authorization.k8s.io/v1{color}
> {color:#569cd6}kind{color}{color:#d4d4d4}: {color}{color:#ce9178}Role{color}
> {color:#569cd6}metadata{color}{color:#d4d4d4}:{color}
> {color:#d4d4d4} {color}{color:#569cd6}labels{color}{color:#d4d4d4}:{color}
> {color:#d4d4d4} {color}{color:#569cd6}app{color}{color:#d4d4d4}:
> {color}{color:#ce9178}flink{color}
> {color:#d4d4d4} {color}{color:#569cd6}name{color}{color:#d4d4d4}:
> {color}{color:#ce9178}flink-role-binding-flink{color}
> {color:#d4d4d4} {color}{color:#569cd6}namespace{color}{color:#d4d4d4}:
> {color}{color:#ce9178}lakehouse{color}
> {color:#569cd6}rules{color}{color:#d4d4d4}:{color}
> {color:#d4d4d4}- {color}{color:#569cd6}apiGroups{color}{color:#d4d4d4}:{color}
> {color:#d4d4d4} - {color}{color:#ce9178}""{color}
> {color:#d4d4d4} {color}{color:#569cd6}resources{color}{color:#d4d4d4}:{color}
> {color:#d4d4d4} - {color}{color:#ce9178}configmaps{color}
> {color:#d4d4d4} {color}{color:#569cd6}verbs{color}{color:#d4d4d4}:{color}
> {color:#d4d4d4} - {color}{color:#ce9178}get{color}
> {color:#d4d4d4} - {color}{color:#ce9178}create{color}
> {color:#d4d4d4} - {color}{color:#ce9178}delete{color}
> {color:#d4d4d4} - {color}{color:#ce9178}update{color}
> {color:#d4d4d4} - {color}{color:#ce9178}list{color}
> {color:#d4d4d4} - {color}{color:#ce9178}watch{color}
> {color:#d4d4d4}---{color}
> {color:#569cd6}apiVersion{color}{color:#d4d4d4}:
> {color}{color:#ce9178}rbac.authorization.k8s.io/v1{color}
> {color:#569cd6}kind{color}{color:#d4d4d4}:
> {color}{color:#ce9178}RoleBinding{color}
> {color:#569cd6}metadata{color}{color:#d4d4d4}:{color}
> {color:#d4d4d4} {color}{color:#569cd6}labels{color}{color:#d4d4d4}:{color}
> {color:#d4d4d4} {color}{color:#569cd6}app{color}{color:#d4d4d4}:
> {color}{color:#ce9178}flink{color}
> {color:#d4d4d4} {color}{color:#569cd6}name{color}{color:#d4d4d4}:
> {color}{color:#ce9178}flink-role-binding-default{color}
> {color:#d4d4d4} {color}{color:#569cd6}namespace{color}{color:#d4d4d4}:
> {color}{color:#ce9178}lakehouse{color}
> {color:#569cd6}roleRef{color}{color:#d4d4d4}:{color}
> {color:#d4d4d4} {color}{color:#569cd6}apiGroup{color}{color:#d4d4d4}:
> {color}{color:#ce9178}rbac.authorization.k8s.io{color}
> {color:#d4d4d4} {color}{color:#569cd6}kind{color}{color:#d4d4d4}:
> {color}{color:#ce9178}Role{color}
> {color:#d4d4d4} {color}{color:#569cd6}name{color}{color:#d4d4d4}:
> {color}{color:#ce9178}flink-role-binding-flink{color}
>
[jira] [Updated] (FLINK-31525) JdbcExactlyOnceSinkE2eTest.testInsert times out
[
https://issues.apache.org/jira/browse/FLINK-31525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-31525:
---
Labels: pull-request-available stale-critical test-stability (was:
pull-request-available test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Critical but is unassigned and neither itself nor its Sub-Tasks have been
updated for 14 days. I have gone ahead and marked it "stale-critical". If this
ticket is critical, please either assign yourself or give an update.
Afterwards, please remove the label or in 7 days the issue will be
deprioritized.
> JdbcExactlyOnceSinkE2eTest.testInsert times out
> ---
>
> Key: FLINK-31525
> URL: https://issues.apache.org/jira/browse/FLINK-31525
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.16.1, jdbc-3.0.0
>Reporter: Matthias Pohl
>Priority: Critical
> Labels: pull-request-available, stale-critical, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=47317=logs=075127ba-54d5-54b0-cccf-6a36778b332d=c35a13eb-0df9-505f-29ac-8097029d4d79=16612
> {code}
> "main" #1 prio=5 os_prio=0 tid=0x7fa64400b800 nid=0x79e6 waiting on
> condition [0x7fa64bbe2000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xd2736728> (a
> java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:2081)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:2049)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:2027)
> at
> org.apache.flink.connector.jdbc.xa.JdbcExactlyOnceSinkE2eTest.testInsert(JdbcExactlyOnceSinkE2eTest.java:224)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [...]
> {code}
> There's another issue with this test in FLINK-28424 which has different
> stacktraces.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28291) Add kerberos delegation token renewer feature instead of logged from keytab individually
[
https://issues.apache.org/jira/browse/FLINK-28291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-28291:
---
Labels: PatchAvailable patch-available stale-minor (was: PatchAvailable
patch-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Add kerberos delegation token renewer feature instead of logged from keytab
> individually
>
>
> Key: FLINK-28291
> URL: https://issues.apache.org/jira/browse/FLINK-28291
> Project: Flink
> Issue Type: New Feature
> Components: Deployment / YARN
>Affects Versions: 1.13.5
>Reporter: jiulong.zhu
>Priority: Minor
> Labels: PatchAvailable, patch-available, stale-minor
> Attachments: FLINK-28291.0001.patch
>
>
> h2. 1. Design
> LifeCycle of delegation token in RM:
> # Container starts with DT given by client.
> # Enable delegation token renewer by:
> ## set {{security.kerberos.token.renew.enabled}} true, default false. And
> ## specify {{security.kerberos.login.keytab}} and
> {{security.kerberos.login.principal}}
> # When enabled delegation token renewer, the renewer thread will re-obtain
> tokens from DelegationTokenProvider(only HadoopFSDelegationTokenProvider
> now). Then the renewer thread will broadcast new tokens to RM locally, all
> JMs and all TMs by RPCGateway.
> # RM process adds new tokens in context by UserGroupInformation.
> LifeCycle of delegation token in JM / TM:
> # TaskManager starts with keytab stored in remote hdfs.
> # When registered successfully, JM / TM get the current tokens of RM boxed
> by {{JobMasterRegistrationSuccess}} / {{{}TaskExecutorRegistrationSuccess{}}}.
> # JM / TM process add new tokens in context by UserGroupInformation.
> It’s too heavy and unnecessary to retrieval leader of ResourceManager by
> HAService, so DelegationTokenManager is instanced by ResourceManager. So
> DelegationToken can hold the reference of ResourceManager, instead of RM
> RPCGateway or self gateway.
> h2. 2. Test
> # No local junit test. It’s too heavy to build junit environments including
> KDC and local hadoop.
> # Cluster test
> step 1: Specify krb5.conf with short token lifetime(ticket_lifetime,
> renew_lifetime) when submitting flink application.
> ```
> {{flink run -yD security.kerberos.token.renew.enabled=true -yD
> security.kerberos.krb5-conf.path= /home/work/krb5.conf -yD
> security.kerberos.login.use-ticket-cache=false ...}}
> ```
> step 2: Watch token identifier changelog and synchronizer between rm and
> worker.
> >>
> In RM / JM log,
> 2022-06-28 15:13:03,509 INFO org.apache.flink.runtime.util.HadoopUtils [] -
> New token (HDFS_DELEGATION_TOKEN token 52101 for work on ha-hdfs:newfyyy)
> created in KerberosDelegationToken, and next schedule delay is 64799880 ms.
> 2022-06-28 15:13:03,529 INFO org.apache.flink.runtime.util.HadoopUtils [] -
> Updating delegation tokens for current user. 2022-06-28 15:13:04,729 INFO
> org.apache.flink.runtime.util.HadoopUtils [] - JobMaster receives new token
> (HDFS_DELEGATION_TOKEN token 52101 for work on ha-hdfs:newfyyy) from RM.
> …
> 2022-06-29 09:13:03,732 INFO org.apache.flink.runtime.util.HadoopUtils [] -
> New token (HDFS_DELEGATION_TOKEN token 52310 for work on ha-hdfs:newfyyy)
> created in KerberosDelegationToken, and next schedule delay is 64800045 ms.
> 2022-06-29 09:13:03,805 INFO org.apache.flink.runtime.util.HadoopUtils [] -
> Updating delegation tokens for current user.
> 2022-06-29 09:13:03,806 INFO org.apache.flink.runtime.util.HadoopUtils [] -
> JobMaster receives new token (HDFS_DELEGATION_TOKEN token 52310 for work on
> ha-hdfs:newfyyy) from RM.
> >>
> In TM log,
> 2022-06-28 15:13:17,983 INFO org.apache.flink.runtime.util.HadoopUtils [] -
> TaskManager receives new token (HDFS_DELEGATION_TOKEN token 52101 for work on
> ha-hdfs:newfyyy) from RM.
> 2022-06-28 15:13:18,016 INFO org.apache.flink.runtime.util.HadoopUtils [] -
> Updating delegation tokens for current user.
> …
> 2022-06-29 09:13:03,809 INFO org.apache.flink.runtime.util.HadoopUtils [] -
> TaskManager receives new token (HDFS_DELEGATION_TOKEN token 52310 for work on
> ha-hdfs:newfyyy) from RM.
> 2022-06-29 09:13:03,836 INFO org.apache.flink.runtime.util.HadoopUtils [] -
> Updating delegation tokens for current user.
--
This message was sent by Atlassian Jira
[jira] [Updated] (FLINK-32269) CreateTableAsITCase.testCreateTableAsInStatementSet fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-32269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32269:
---
Labels: stale-critical test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Critical but is unassigned and neither itself nor its Sub-Tasks have been
updated for 14 days. I have gone ahead and marked it "stale-critical". If this
ticket is critical, please either assign yourself or give an update.
Afterwards, please remove the label or in 7 days the issue will be
deprioritized.
> CreateTableAsITCase.testCreateTableAsInStatementSet fails on AZP
>
>
> Key: FLINK-32269
> URL: https://issues.apache.org/jira/browse/FLINK-32269
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: stale-critical, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=49532=logs=af184cdd-c6d8-5084-0b69-7e9c67b35f7a=0f3adb59-eefa-51c6-2858-3654d9e0749d=15797
> {noformat}
> Jun 01 03:40:51 03:40:51.881 [ERROR] Tests run: 4, Failures: 1, Errors: 0,
> Skipped: 0, Time elapsed: 104.874 s <<< FAILURE! - in
> org.apache.flink.table.sql.codegen.CreateTableAsITCase
> Jun 01 03:40:51 03:40:51.881 [ERROR]
> CreateTableAsITCase.testCreateTableAsInStatementSet Time elapsed: 40.729 s
> <<< FAILURE!
> Jun 01 03:40:51 org.opentest4j.AssertionFailedError: Did not get expected
> results before timeout, actual result:
> [{"before":null,"after":{"user_name":"Bob","order_cnt":1},"op":"c"},
> {"before":null,"after":{"user_name":"Alice","order_cnt":1},"op":"c"},
> {"before":{"user_name":"Bob","order_cnt":1},"after":null,"op":"d"},
> {"before":null,"after":{"user_name":"Bob","order_cnt":2},"op":"c"}]. ==>
> expected: but was:
> Jun 01 03:40:51 at
> org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
> Jun 01 03:40:51 at
> org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
> Jun 01 03:40:51 at
> org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
> Jun 01 03:40:51 at
> org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
> Jun 01 03:40:51 at
> org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:211)
> Jun 01 03:40:51 at
> org.apache.flink.table.sql.codegen.SqlITCaseBase.checkJsonResultFile(SqlITCaseBase.java:168)
> Jun 01 03:40:51 at
> org.apache.flink.table.sql.codegen.SqlITCaseBase.runAndCheckSQL(SqlITCaseBase.java:111)
> Jun 01 03:40:51 at
> org.apache.flink.table.sql.codegen.CreateTableAsITCase.testCreateTableAsInStatementSet(CreateTableAsITCase.java:50)
> Jun 01 03:40:51 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Jun 01 03:40:51 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Jun 01 03:40:51 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Jun 01 03:40:51 at java.lang.reflect.Method.invoke(Method.java:498)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32113) TtlMapStateAllEntriesTestContext failure in generic types
[
https://issues.apache.org/jira/browse/FLINK-32113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32113:
---
Labels: stale-major test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> TtlMapStateAllEntriesTestContext failure in generic types
> -
>
> Key: FLINK-32113
> URL: https://issues.apache.org/jira/browse/FLINK-32113
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.18.0
>Reporter: Matthias Pohl
>Priority: Major
> Labels: stale-major, test-stability
>
> I have the same test failure in both e2e test runs:
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=49076=logs=af184cdd-c6d8-5084-0b69-7e9c67b35f7a=160c9ae5-96fd-516e-1c91-deb81f59292a=2924]
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=49076=logs=bea52777-eaf8-5663-8482-18fbc3630e81=b2642e3a-5b86-574d-4c8a-f7e2842bfb14=2924]
> {code:java}
> 16:36:27.471 [ERROR]
> /home/vsts/work/1/s/flink-runtime/src/test/java/org/apache/flink/runtime/state/ttl/TtlMapStateAllEntriesTestContext.java:[49,30]
> incompatible types: inference variable T0 has incompatible bounds
> equality constraints: java.lang.String,java.lang.Integer,UK,T0,T0
> lower bounds: java.lang.Integer
> [...]
> 16:36:27.495 [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.8.0:testCompile
> (default-testCompile) on project flink-runtime: Compilation failure
> 16:36:27.495 [ERROR]
> /home/vsts/work/1/s/flink-runtime/src/test/java/org/apache/flink/runtime/state/ttl/TtlMapStateAllEntriesTestContext.java:[49,30]
> incompatible types: inference variable T0 has incompatible bounds
> 16:36:27.496 [ERROR] equality constraints:
> java.lang.String,java.lang.Integer,UK,T0,T0
> 16:36:27.496 [ERROR] lower bounds: java.lang.Integer
> 16:36:27.496 [ERROR] -> [Help 1] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-27986) Refactor the name of finish method for JdbcOutputFormatBuilder
[ https://issues.apache.org/jira/browse/FLINK-27986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-27986: --- Labels: pull-request-available stale-minor (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Refactor the name of finish method for JdbcOutputFormatBuilder > -- > > Key: FLINK-27986 > URL: https://issues.apache.org/jira/browse/FLINK-27986 > Project: Flink > Issue Type: Improvement >Reporter: Lei Xie >Priority: Minor > Labels: pull-request-available, stale-minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32553) ClusterEntrypointTest.testCloseAsyncShouldNotDeregisterApp failed on AZP
[
https://issues.apache.org/jira/browse/FLINK-32553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32553:
---
Labels: stale-critical test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Critical but is unassigned and neither itself nor its Sub-Tasks have been
updated for 14 days. I have gone ahead and marked it "stale-critical". If this
ticket is critical, please either assign yourself or give an update.
Afterwards, please remove the label or in 7 days the issue will be
deprioritized.
> ClusterEntrypointTest.testCloseAsyncShouldNotDeregisterApp failed on AZP
>
>
> Key: FLINK-32553
> URL: https://issues.apache.org/jira/browse/FLINK-32553
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.17.2
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: stale-critical, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=51013=logs=77a9d8e1-d610-59b3-fc2a-4766541e0e33=125e07e7-8de0-5c6c-a541-a567415af3ef=7961
> {noformat}
> Jul 06 05:38:37 [ERROR] Tests run: 9, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 71.304 s <<< FAILURE! - in
> org.apache.flink.runtime.entrypoint.ClusterEntrypointTest
> Jul 06 05:38:37 [ERROR]
> org.apache.flink.runtime.entrypoint.ClusterEntrypointTest.testCloseAsyncShouldNotDeregisterApp
> Time elapsed: 22.51 s <<< ERROR!
> Jul 06 05:38:37
> org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to
> initialize the cluster entrypoint TestingEntryPoint.
> Jul 06 05:38:37 at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:255)
> Jul 06 05:38:37 at
> org.apache.flink.runtime.entrypoint.ClusterEntrypointTest.startClusterEntrypoint(ClusterEntrypointTest.java:347)
> Jul 06 05:38:37 at
> org.apache.flink.runtime.entrypoint.ClusterEntrypointTest.testCloseAsyncShouldNotDeregisterApp(ClusterEntrypointTest.java:175)
> Jul 06 05:38:37 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Jul 06 05:38:37 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Jul 06 05:38:37 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Jul 06 05:38:37 at java.lang.reflect.Method.invoke(Method.java:498)
> Jul 06 05:38:37 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Jul 06 05:38:37 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Jul 06 05:38:37 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32483) RescaleCheckpointManuallyITCase.testCheckpointRescalingOutKeyedState fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-32483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32483:
---
Labels: stale-critical test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Critical but is unassigned and neither itself nor its Sub-Tasks have been
updated for 14 days. I have gone ahead and marked it "stale-critical". If this
ticket is critical, please either assign yourself or give an update.
Afterwards, please remove the label or in 7 days the issue will be
deprioritized.
> RescaleCheckpointManuallyITCase.testCheckpointRescalingOutKeyedState fails on
> AZP
> -
>
> Key: FLINK-32483
> URL: https://issues.apache.org/jira/browse/FLINK-32483
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Tests
>Affects Versions: 1.17.2
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: stale-critical, test-stability
>
> This build
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=50397=logs=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3=0c010d0c-3dec-5bf1-d408-7b18988b1b2b=7495
> fails with
> {noformat}
> Jun 26 06:08:57 [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0,
> Time elapsed: 21.041 s <<< FAILURE! - in
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase
> Jun 26 06:08:57 [ERROR]
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase.testCheckpointRescalingOutKeyedState
> Time elapsed: 6.435 s <<< FAILURE!
> Jun 26 06:08:57 java.lang.AssertionError: expected:<[(0,24000), (2,58500),
> (0,34500), (0,45000), (3,43500), (2,18000), (1,6000), (1,16500), (0,28500),
> (0,52500), (3,27000), (1,51000), (2,25500), (0,1500), (0,49500), (3,0),
> (3,48000), (0,36000), (2,22500), (1,10500), (0,46500), (2,33000), (1,21000),
> (0,9000), (0,57000), (3,31500), (2,19500), (1,7500), (1,55500), (3,42000),
> (2,3), (0,54000), (2,40500), (1,4500), (3,15000), (2,3000), (1,39000),
> (2,13500), (0,37500), (0,61500), (3,12000), (3,6)]> but was:<[(2,58500),
> (0,34500), (0,45000), (3,43500), (2,18000), (1,16500), (0,52500), (3,27000),
> (2,25500), (0,49500), (3,0), (3,48000), (0,36000), (2,22500), (1,21000),
> (0,9000), (0,57000), (3,31500), (1,7500), (2,3), (0,54000), (2,40500),
> (1,4500), (2,3000), (1,39000), (2,13500), (0,61500), (3,12000)]>
> Jun 26 06:08:57 at org.junit.Assert.fail(Assert.java:89)
> Jun 26 06:08:57 at org.junit.Assert.failNotEquals(Assert.java:835)
> Jun 26 06:08:57 at org.junit.Assert.assertEquals(Assert.java:120)
> Jun 26 06:08:57 at org.junit.Assert.assertEquals(Assert.java:146)
> Jun 26 06:08:57 at
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase.restoreAndAssert(RescaleCheckpointManuallyITCase.java:219)
> Jun 26 06:08:57 at
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase.testCheckpointRescalingKeyedState(RescaleCheckpointManuallyITCase.java:138)
> Jun 26 06:08:57 at
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase.testCheckpointRescalingOutKeyedState(RescaleCheckpointManuallyITCase.java:116)
> Jun 26 06:08:57 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32661) OperationRelatedITCase.testOperationRelatedApis fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-32661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32661:
---
Labels: stale-critical test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Critical but is unassigned and neither itself nor its Sub-Tasks have been
updated for 14 days. I have gone ahead and marked it "stale-critical". If this
ticket is critical, please either assign yourself or give an update.
Afterwards, please remove the label or in 7 days the issue will be
deprioritized.
> OperationRelatedITCase.testOperationRelatedApis fails on AZP
>
>
> Key: FLINK-32661
> URL: https://issues.apache.org/jira/browse/FLINK-32661
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Gateway
>Affects Versions: 1.18.0
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: stale-critical, test-stability
>
> This build
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=51452=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=12114
> fails as
> {noformat}
> Jul 20 04:23:49 org.opentest4j.AssertionFailedError:
> Jul 20 04:23:49
> Jul 20 04:23:49 Expecting actual's toString() to return:
> Jul 20 04:23:49 "PENDING"
> Jul 20 04:23:49 but was:
> Jul 20 04:23:49 "RUNNING"
> Jul 20 04:23:49 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> Jul 20 04:23:49 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> Jul 20 04:23:49 at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> Jul 20 04:23:49 at
> org.apache.flink.table.gateway.rest.OperationRelatedITCase.testOperationRelatedApis(OperationRelatedITCase.java:91)
> Jul 20 04:23:49 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Jul 20 04:23:49 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Jul 20 04:23:49 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Jul 20 04:23:49 at java.lang.reflect.Method.invoke(Method.java:498)
> Jul 20 04:23:49 at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
> Jul 20 04:23:49 at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> Jul 20 04:23:49 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> Jul 20 04:23:49 at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
> Jul 20 04:23:49 at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
> Jul 20 04:23:49 at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
> Jul 20 04:23:49 at
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
> Jul 20 04:23:49 at
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
> Jul 20 04:23:49 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> Jul 20 04:23:49 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> Jul 20 04:23:49 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> Jul 20 04:23:49 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> Jul 20 04:23:49 at
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
> Jul 20 04:23:49 at
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
> Jul 20 04:23:49 at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:21
> ...
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31379) ZooKeeperMultipleComponentLeaderElectionDriverTest.testLeaderElectionWithMultipleDrivers runs into timeout
[
https://issues.apache.org/jira/browse/FLINK-31379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-31379:
---
Labels: stale-major test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> ZooKeeperMultipleComponentLeaderElectionDriverTest.testLeaderElectionWithMultipleDrivers
> runs into timeout
> --
>
> Key: FLINK-31379
> URL: https://issues.apache.org/jira/browse/FLINK-31379
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.16.1
>Reporter: Matthias Pohl
>Priority: Major
> Labels: stale-major, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46843=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7=9655
> {code}
> Mar 06 12:16:45 "ForkJoinPool-51-worker-25" #645 daemon prio=5 os_prio=0
> tid=0x7fe20f633000 nid=0xdd4 waiting on condition [0x7fe0342c5000]
> Mar 06 12:16:45java.lang.Thread.State: WAITING (parking)
> Mar 06 12:16:45 at sun.misc.Unsafe.park(Native Method)
> Mar 06 12:16:45 - parking to wait for <0xd213d1f8> (a
> java.util.concurrent.CompletableFuture$Signaller)
> Mar 06 12:16:45 at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> Mar 06 12:16:45 at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> Mar 06 12:16:45 at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3313)
> Mar 06 12:16:45 at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> Mar 06 12:16:45 at
> java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1947)
> Mar 06 12:16:45 at
> org.apache.flink.runtime.leaderelection.ZooKeeperMultipleComponentLeaderElectionDriverTest.testLeaderElectionWithMultipleDrivers(ZooKeeperMultipleComponentLeaderElectionDriverTest.java:256)
> Mar 06 12:16:45 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31526) Various test failures in PyFlink
[
https://issues.apache.org/jira/browse/FLINK-31526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-31526:
---
Labels: stale-major test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Various test failures in PyFlink
>
>
> Key: FLINK-31526
> URL: https://issues.apache.org/jira/browse/FLINK-31526
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.16.1
>Reporter: Matthias Pohl
>Priority: Major
> Labels: stale-major, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=47328=logs=821b528f-1eed-5598-a3b4-7f748b13f261=6bb545dd-772d-5d8c-f258-f5085fba3295=37186
> {code}
> Mar 19 04:08:57 ERROR
> pyflink/table/tests/test_udf.py::PyFlinkBatchUserDefinedFunctionTests::test_udf_in_join_condition_2
> Mar 19 04:08:57 ERROR
> pyflink/table/tests/test_udf.py::PyFlinkBatchUserDefinedFunctionTests::test_udf_with_constant_params
> Mar 19 04:08:57 ERROR
> pyflink/table/tests/test_udf.py::PyFlinkBatchUserDefinedFunctionTests::test_udf_without_arguments
> Mar 19 04:08:57 ERROR
> pyflink/table/tests/test_udf.py::PyFlinkEmbeddedThreadTests::test_all_data_types
> Mar 19 04:08:57 ERROR
> pyflink/table/tests/test_udf.py::PyFlinkEmbeddedThreadTests::test_all_data_types_expression
> Mar 19 04:08:57 ERROR
> pyflink/table/tests/test_udf.py::PyFlinkEmbeddedThreadTests::test_chaining_scalar_function
> Mar 19 04:08:57 ERROR
> pyflink/table/tests/test_udf.py::PyFlinkEmbeddedThreadTests::test_create_and_drop_function
> Mar 19 04:08:57 ERROR
> pyflink/table/tests/test_udf.py::PyFlinkEmbeddedThreadTests::test_open
> Mar 19 04:08:57 ERROR
> pyflink/table/tests/test_udf.py::PyFlinkEmbeddedThreadTests::test_overwrite_builtin_function
> Mar 19 04:08:57 ERROR
> pyflink/table/tests/test_udf.py::PyFlinkEmbeddedThreadTests::test_scalar_function
> Mar 19 04:08:57 ERROR
> pyflink/table/tests/test_udf.py::PyFlinkEmbeddedThreadTests::test_udf_in_join_condition
> Mar 19 04:08:57 ERROR
> pyflink/table/tests/test_udf.py::PyFlinkEmbeddedThreadTests::test_udf_in_join_condition_2
> Mar 19 04:08:57 ERROR
> pyflink/table/tests/test_udf.py::PyFlinkEmbeddedThreadTests::test_udf_with_constant_params
> Mar 19 04:08:57 ERROR
> pyflink/table/tests/test_udf.py::PyFlinkEmbeddedThreadTests::test_udf_without_arguments
> Mar 19 04:08:57 ERROR
> pyflink/table/tests/test_udtf.py::PyFlinkStreamUserDefinedFunctionTests::test_execute_from_json_plan
> Mar 19 04:08:57 ERROR
> pyflink/table/tests/test_udtf.py::PyFlinkStreamUserDefinedFunctionTests::test_table_function
> Mar 19 04:08:57 ERROR
> pyflink/table/tests/test_udtf.py::PyFlinkStreamUserDefinedFunctionTests::test_table_function_with_sql_query
> Mar 19 04:08:57 ERROR
> pyflink/table/tests/test_udtf.py::PyFlinkBatchUserDefinedFunctionTests::test_table_function
> Mar 19 04:08:57 ERROR
> pyflink/table/tests/test_udtf.py::PyFlinkBatchUserDefinedFunctionTests::test_table_function_with_sql_query
> Mar 19 04:08:57 ERROR
> pyflink/table/tests/test_udtf.py::PyFlinkEmbeddedThreadTests::test_table_function
> Mar 19 04:08:57 ERROR
> pyflink/table/tests/test_udtf.py::PyFlinkEmbeddedThreadTests::test_table_function_with_sql_query
> Mar 19 04:08:57 ERROR
> pyflink/table/tests/test_window.py::StreamTableWindowTests::test_over_window
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31928) flink-kubernetes works not properly in k8s with IPv6 stack
[ https://issues.apache.org/jira/browse/FLINK-31928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-31928: --- Labels: pull-request-available stale-major (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 60 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > flink-kubernetes works not properly in k8s with IPv6 stack > -- > > Key: FLINK-31928 > URL: https://issues.apache.org/jira/browse/FLINK-31928 > Project: Flink > Issue Type: Bug > Components: Deployment / Kubernetes > Environment: Kubernetes of IPv6 stack. >Reporter: Yi Cai >Priority: Major > Labels: pull-request-available, stale-major > > As > [https://github.com/square/okhttp/issues/7368|https://github.com/square/okhttp/issues/7368,] > ,okhttp3 shaded in flink-kubernetes works not properly in IPv6 stack in k8s, > need to upgrade okhttp3 to version 4.10.0 and shade dependency of > okhttp3:4.10.0 > org.jetbrains.kotlin:kotlin-stdlib in flink-kubernetes or just upgrade > kubernetes-client to latest version, and release a new version of > flink-kubernetes-operator. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31828) List field in a POJO data stream results in table program compilation failure
[
https://issues.apache.org/jira/browse/FLINK-31828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-31828:
---
Labels: pull-request-available stale-major (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> List field in a POJO data stream results in table program compilation failure
> -
>
> Key: FLINK-31828
> URL: https://issues.apache.org/jira/browse/FLINK-31828
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.16.1
> Environment: Java 11
> Flink 1.16.1
>Reporter: Vladimir Matveev
>Priority: Major
> Labels: pull-request-available, stale-major
> Attachments: MainPojo.java, generated-code.txt, stacktrace.txt
>
>
> Suppose I have a POJO class like this:
> {code:java}
> public class Example {
> private String key;
> private List> values;
> // getters, setters, equals+hashCode omitted
> }
> {code}
> When a DataStream with this class is converted to a table, and some
> operations are performed on it, it results in an exception which explicitly
> says that I should file a ticket:
> {noformat}
> Caused by: org.apache.flink.api.common.InvalidProgramException: Table program
> cannot be compiled. This is a bug. Please file an issue.
> {noformat}
> Please find the example Java code and the full stack trace attached.
> From the exception and generated code it seems that Flink is upset with the
> list field being treated as an array - but I cannot have an array type there
> in the real code.
> Also note that if I _don't_ specify the schema explicitly, it then maps the
> {{values}} field to a `RAW('java.util.List', '...')` type, which also does
> not work correctly and fails the job in case of even simplest operations like
> printing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32155) Multiple CIs jobs failed due to "Could not connect to azure.archive.ubuntu.com"
[ https://issues.apache.org/jira/browse/FLINK-32155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32155: --- Labels: stale-major test-stability (was: test-stability) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 60 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Multiple CIs jobs failed due to "Could not connect to > azure.archive.ubuntu.com" > --- > > Key: FLINK-32155 > URL: https://issues.apache.org/jira/browse/FLINK-32155 > Project: Flink > Issue Type: Bug > Components: Test Infrastructure >Affects Versions: 1.18.0 >Reporter: Sergey Nuyanzin >Priority: Major > Labels: stale-major, test-stability > > The issue is very similar to https://issues.apache.org/jira/browse/FLINK-30921 > the difference is that https://issues.apache.org/jira/browse/FLINK-30921 is > for e2e jobs while this one is not > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=49065=logs=0da23115-68bb-5dcd-192c-bd4c8adebde1=24c3384f-1bcb-57b3-224f-51bf973bbee8=37 > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=49065=logs=4eda0b4a-bd0d-521a-0916-8285b9be9bb5=2ff6d5fa-53a6-53ac-bff7-fa524ea361a9=37 > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=49065=logs=aa18c3f6-13b8-5f58-86bb-c1cffb239496=502fb6c0-30a2-5e49-c5c2-a00fa3acb203=37 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32138) SQLClientSchemaRegistryITCase fails with timeout on AZP
[
https://issues.apache.org/jira/browse/FLINK-32138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32138:
---
Labels: stale-major test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> SQLClientSchemaRegistryITCase fails with timeout on AZP
> ---
>
> Key: FLINK-32138
> URL: https://issues.apache.org/jira/browse/FLINK-32138
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.16.2
>Reporter: Sergey Nuyanzin
>Priority: Major
> Labels: stale-major, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=49174=logs=6e8542d7-de38-5a33-4aca-458d6c87066d=10d6732b-d79a-5c68-62a5-668516de5313=15753
> {{SQLClientSchemaRegistryITCase}} fails on AZP as
> {noformat}
> May 20 03:41:34 [ERROR]
> org.apache.flink.tests.util.kafka.SQLClientSchemaRegistryITCase Time
> elapsed: 600.05 s <<< ERROR!
> May 20 03:41:34 org.junit.runners.model.TestTimedOutException: test timed out
> after 10 minutes
> May 20 03:41:34 at
> java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
> May 20 03:41:34 at
> java.base@11.0.19/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
> May 20 03:41:34 at
> java.base@11.0.19/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
> May 20 03:41:34 at
> java.base@11.0.19/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
> May 20 03:41:34 at
> java.base@11.0.19/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
> May 20 03:41:34 at
> java.base@11.0.19/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
> May 20 03:41:34 at
> app//com.github.dockerjava.api.async.ResultCallbackTemplate.awaitCompletion(ResultCallbackTemplate.java:91)
> May 20 03:41:34 at
> app//org.testcontainers.images.TimeLimitedLoggedPullImageResultCallback.awaitCompletion(TimeLimitedLoggedPullImageResultCallback.java:52)
> May 20 03:41:34 at
> app//org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:89)
> May 20 03:41:34 at
> app//org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:28)
> May 20 03:41:34 at
> app//org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:17)
> May 20 03:41:34 at
> app//org.testcontainers.utility.LazyFuture.get(LazyFuture.java:39)
> May 20 03:41:34 at
> app//org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1330)
> May 20 03:41:34 at
> app//org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:640)
> May 20 03:41:34 at
> app//org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:335)
> May 20 03:41:34 at
> app//org.testcontainers.containers.GenericContainer.start(GenericContainer.java:326)
> May 20 03:41:34 at
> app//org.testcontainers.containers.GenericContainer.starting(GenericContainer.java:1063)
> May 20 03:41:34 at
> app//org.testcontainers.containers.FailureDetectingExternalResource$1.evaluate(FailureDetectingExternalResource.java:29)
> May 20 03:41:34 at
> app//org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> May 20 03:41:34 at
> app//org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> May 20 03:41:34 at
> java.base@11.0.19/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> May 20 03:41:34 at
> java.base@11.0.19/java.lang.Thread.run(Thread.java:829)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31992) FlinkKafkaConsumer API is suggested to use as part of documentation, when that API is deprecated for flink version 1.14
[ https://issues.apache.org/jira/browse/FLINK-31992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-31992: --- Labels: documentation documentation-update good-first-issue newbie stale-major (was: documentation documentation-update good-first-issue newbie) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 60 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > FlinkKafkaConsumer API is suggested to use as part of documentation, when > that API is deprecated for flink version 1.14 > --- > > Key: FLINK-31992 > URL: https://issues.apache.org/jira/browse/FLINK-31992 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Affects Versions: 1.14.2 >Reporter: Sandesh Mendan >Priority: Major > Labels: documentation, documentation-update, good-first-issue, > newbie, stale-major > > In Flink version 1.14, even though the API class FlinkKafkaConsumer had been > [deprecated|https://nightlies.apache.org/flink/flink-docs-release-1.14/api/java/], > the official > [documentation|https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/event-time/generating_watermarks/#watermark-strategies-and-the-kafka-connector] > suggests that API to use. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32101) FlinkKafkaInternalProducerITCase.testInitTransactionId test failed
[
https://issues.apache.org/jira/browse/FLINK-32101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32101:
---
Labels: stale-major test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> FlinkKafkaInternalProducerITCase.testInitTransactionId test failed
> --
>
> Key: FLINK-32101
> URL: https://issues.apache.org/jira/browse/FLINK-32101
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.18.0
>Reporter: Yuxin Tan
>Priority: Major
> Labels: stale-major, test-stability
>
> FlinkKafkaInternalProducerITCase.testInitTransactionId test failed.
> Caused by: org.apache.kafka.common.KafkaException: Unexpected error in
> InitProducerIdResponse; The request timed out.
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=48990=logs=aa18c3f6-13b8-5f58-86bb-c1cffb239496=502fb6c0-30a2-5e49-c5c2-a00fa3acb203=22973
> {code:java}
> Caused by: org.apache.kafka.common.KafkaException:
> org.apache.kafka.common.KafkaException: Unexpected error in
> InitProducerIdResponse; The request timed out.
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:593)
> at
> java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:677)
> at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:735)
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
> at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
> at
> java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.abortTransactions(FlinkKafkaProducer.java:1290)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.initializeState(FlinkKafkaProducer.java:1216)
> at
> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:189)
> at
> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:171)
> at
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:95)
> at
> org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:122)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:274)
> at
> org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:106)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:747)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.call(StreamTaskActionExecutor.java:100)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:722)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:688)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952)
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:921)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.kafka.common.KafkaException: Unexpected error in
> InitProducerIdResponse; The request timed out.
> at
>
[jira] [Updated] (FLINK-28837) Translate "Hybrid Source" page of "DataStream Connectors" into Chinese
[ https://issues.apache.org/jira/browse/FLINK-28837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-28837: --- Labels: pull-request-available stale-minor (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Translate "Hybrid Source" page of "DataStream Connectors" into Chinese > -- > > Key: FLINK-28837 > URL: https://issues.apache.org/jira/browse/FLINK-28837 > Project: Flink > Issue Type: Improvement > Components: chinese-translation >Reporter: JasonLee >Priority: Minor > Labels: pull-request-available, stale-minor > > The page url is > [https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/datastream/hybridsource/] > The markdown file is located in > docs/content.zh/docs/connectors/datastream/hybridsource.md -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28847) Typo in FileSinkProgram.java in file-sink-file-test module
[ https://issues.apache.org/jira/browse/FLINK-28847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-28847: --- Labels: easyfix stale-minor (was: easyfix) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Typo in FileSinkProgram.java in file-sink-file-test module > -- > > Key: FLINK-28847 > URL: https://issues.apache.org/jira/browse/FLINK-28847 > Project: Flink > Issue Type: Improvement > Components: Tests >Affects Versions: 1.15.1 >Reporter: Xin Wen >Priority: Minor > Labels: easyfix, stale-minor > Attachments: image-2022-08-07-11-42-25-221.png > > Original Estimate: 1h > Remaining Estimate: 1h > > There is a redundant semicolon in > [FileSinkProgram.java|https://github.com/apache/flink/blob/b0859789e7733c73a21e600ec0d595ead730c59d/flink-end-to-end-tests/flink-file-sink-test/src/main/java/org/apache/flink/connector/file/sink/FileSinkProgram.java#L57] > which will confuse the users. > !image-2022-08-07-11-42-25-221.png|width=700,height=290! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-30782) Use https for schemaLocations
[
https://issues.apache.org/jira/browse/FLINK-30782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-30782:
---
Labels: pull-request-available stale-minor (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Use https for schemaLocations
> -
>
> Key: FLINK-30782
> URL: https://issues.apache.org/jira/browse/FLINK-30782
> Project: Flink
> Issue Type: Technical Debt
>Reporter: Sergey Nuyanzin
>Priority: Minor
> Labels: pull-request-available, stale-minor
>
> In poms
> {code:xml}
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/maven-v4_0_0.xsd;>
> {code}
> use https for xsd like https://maven.apache.org/xsd/maven-4.0.0.xsd
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32422) EmbeddedLeaderService doesn't handle the leader events properly in edge cases
[ https://issues.apache.org/jira/browse/FLINK-32422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32422: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > EmbeddedLeaderService doesn't handle the leader events properly in edge cases > - > > Key: FLINK-32422 > URL: https://issues.apache.org/jira/browse/FLINK-32422 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Affects Versions: 1.18.0 >Reporter: Matthias Pohl >Assignee: Matthias Pohl >Priority: Minor > Labels: pull-request-available, stale-assigned > > The leadership is granted when registering the first contender. This sets the > leadership flag within the EmbeddedLeaderService (see > [EmbeddedLeaderService:312ff|https://github.com/apache/flink/blob/033aca7566a0a561410b3c0e1ae8dca856cd26ce/flink-runtime/src/main/java/org/apache/flink/runtime/highavailability/nonha/embedded/EmbeddedLeaderService.java#L312]: > the grantLeadershipCall is triggered afterwards informing the contender > about its leadership). In the meantime, close can be called on the contender > which deregisters the contender again calling revoke on the contender without > having been able to gain the leadership. > This issue was introduced by FLINK-30765. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-25413) Use append dfs.nameservices hadoop config to replace override
[
https://issues.apache.org/jira/browse/FLINK-25413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-25413:
---
Labels: auto-deprioritized-major auto-deprioritized-minor
pull-request-available (was: auto-deprioritized-major pull-request-available
stale-minor)
Priority: Not a Priority (was: Minor)
This issue was labeled "stale-minor" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Minor, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> Use append dfs.nameservices hadoop config to replace override
> --
>
> Key: FLINK-25413
> URL: https://issues.apache.org/jira/browse/FLINK-25413
> Project: Flink
> Issue Type: Improvement
>Reporter: qiunan
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor,
> pull-request-available
>
> In FLINK-16005[flink-yarn] Support yarn and hadoop config override.
> In flink-conf.yaml
> flink.hadoop.dfs.namenode.rpc-address.nameservice1.nn1: bigdata1:8020
> flink.hadoop.dfs.namenode.rpc-address.nameservice1.nn2: bigdata2:8020
> flink.hadoop.dfs.client.failover.proxy.provider.nameservice1:
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
> flink.hadoop.dfs.ha.namenodes.nameservice1: nn1,nn2
> flink.hadoop.dfs.nameservices: nameservice1
> code：
> {code:java}
> // Approach 4: Flink configuration
> // add all configuration key with prefix 'flink.hadoop.' in flink conf to
> hadoop conf
> for (String key : flinkConfiguration.keySet()) {
> for (String prefix : FLINK_CONFIG_PREFIXES) {
> if (key.startsWith(prefix)) {
> String newKey = key.substring(prefix.length());
> String value = flinkConfiguration.getString(key, null);
> result.set(newKey, value);
> LOG.debug(
> "Adding Flink config entry for {} as {}={} to Hadoop
> config",
> key,
> newKey,
> value);
> foundHadoopConfiguration = true;
> }
> }
> } {code}
> If my HADOOP_CONF_DIR hdfs-site.xml have dfs.nameservices: nameservice2, see
> the code logic this config will be override. I think this config should not
> be override append will be better. if override we should add all config, but
> we have many clusters in production, it is impossible to configure all
> configurations in flink-conf.yaml.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32435) Merge testing implementations of LeaderContender
[
https://issues.apache.org/jira/browse/FLINK-32435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32435:
---
Labels: pull-request-available stale-assigned (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 30 days, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it.
> Merge testing implementations of LeaderContender
>
>
> Key: FLINK-32435
> URL: https://issues.apache.org/jira/browse/FLINK-32435
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination, Tests
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
> Labels: pull-request-available, stale-assigned
>
> We have several testing implementations of the {{LeaderContender}} interface.
> We could merge all of them into a single {{TestingLeaderContender}}
> implementation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32440) Introduce file merging configuration
[ https://issues.apache.org/jira/browse/FLINK-32440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32440: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > Introduce file merging configuration > > > Key: FLINK-32440 > URL: https://issues.apache.org/jira/browse/FLINK-32440 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Yanfei Lei >Assignee: Yanfei Lei >Priority: Major > Labels: pull-request-available, stale-assigned > > Introduce file merging configuration and config FileMergingSnapshotManager. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32253) Blocklist unblockResources does not update the pending resource request
[ https://issues.apache.org/jira/browse/FLINK-32253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32253: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > Blocklist unblockResources does not update the pending resource request > --- > > Key: FLINK-32253 > URL: https://issues.apache.org/jira/browse/FLINK-32253 > Project: Flink > Issue Type: Improvement > Components: Deployment / Kubernetes, Deployment / YARN >Affects Versions: 1.17.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Labels: pull-request-available, stale-assigned > > Blocklist unblockResources does not update the existing pending resource > request from YARN/K8S. It updates only for the new resource requests. The > existing pending resource requests are not scheduled on the nodes which are > unblocked. > ResourceManager#unblockResources has to notify > YarnResourceManagerDriver/KubernetesResourceManagerDriver so that the driver > updates the pending resource request. > YarnResourceManagerDriver#tryUpdateApplicationBlockList could be called > during unblockResources. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-25448) TaskManagerRunnerTest.testShouldShutdownIfRegistrationWithJobManagerFails failed on azure
[
https://issues.apache.org/jira/browse/FLINK-25448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-25448:
---
Labels: auto-deprioritized-major auto-deprioritized-minor test-stability
(was: auto-deprioritized-major stale-minor test-stability)
Priority: Not a Priority (was: Minor)
This issue was labeled "stale-minor" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Minor, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> TaskManagerRunnerTest.testShouldShutdownIfRegistrationWithJobManagerFails
> failed on azure
> -
>
> Key: FLINK-25448
> URL: https://issues.apache.org/jira/browse/FLINK-25448
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.14.2
>Reporter: Yun Gao
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor,
> test-stability
>
> {code:java}
> Dec 25 01:36:58 [ERROR] Failures:
> Dec 25 01:36:58 [ERROR]
> TaskManagerRunnerTest.testShouldShutdownIfRegistrationWithJobManagerFails
> Multiple Failures (2 failures)
> Dec 25 01:36:58 java.util.concurrent.CompletionException:
> java.util.concurrent.TimeoutException
> Dec 25 01:36:58 java.util.concurrent.TimeoutException:
> Dec 25 01:36:58 [INFO]
> Dec 25 01:36:58 [ERROR] Tests run: 6035, Failures: 1, Errors: 0, Skipped: 97
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=28588=logs=6bfdaf55-0c08-5e3f-a2d2-2a0285fd41cf=cb073eeb-41fa-5f93-7035-c175e0e49392=8009
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-26470) [Java][TypeExtractor] Missing type information in POJO types of some types (List, Map, UUID)
[
https://issues.apache.org/jira/browse/FLINK-26470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-26470:
---
Labels: auto-deprioritized-minor pojo (was: pojo stale-minor)
Priority: Not a Priority (was: Minor)
This issue was labeled "stale-minor" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Minor, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> [Java][TypeExtractor] Missing type information in POJO types of some types
> (List, Map, UUID)
>
>
> Key: FLINK-26470
> URL: https://issues.apache.org/jira/browse/FLINK-26470
> Project: Flink
> Issue Type: Bug
> Components: API / Core
>Affects Versions: 1.13.2
>Reporter: Antoine Michaud
>Priority: Not a Priority
> Labels: auto-deprioritized-minor, pojo
>
> h2. Problem:
> h4. Basic collections (List, Map) and custom types are not compatible with
> flink pojo serialization.
> Here the exception:
>
> {code:java}
> Generic types have been disabled in the ExecutionConfig and type
> java.util.List is treated as a generic type.
> java.lang.UnsupportedOperationException: Generic types have been disabled in
> the ExecutionConfig and type java.util.List is treated as a generic type.
> at
> org.apache.flink.api.java.typeutils.GenericTypeInfo.createSerializer(GenericTypeInfo.java:87)
> at
> org.apache.flink.api.java.typeutils.PojoTypeInfo.createPojoSerializer(PojoTypeInfo.java:346)
> [... nothing interesting ...]{code}
>
> h2. Explanation:
> Like docs said, we should not use kryo in production since it's not
> performant at all.
> To stop using kryo, and use the native pojos serialization, we do this:
> {code:java}
> env.getConfig().disableGenericTypes(){code}
>
> But pojos have to meet [some
> requirements|https://nightlies.apache.org/flink/flink-docs-release-1.10/dev/types_serialization.html#rules-for-pojo-types].
> Regarding the following code coming from flink-core v1.13.2 (and looks the
> same in v1.14.4):
> {code:java}
> private TypeInformation privateGetForClass(
> Class clazz,
> List typeHierarchy,
> ParameterizedType parameterizedType,
> TypeInformation in1Type,
> TypeInformation in2Type) {
> checkNotNull(clazz);
> // check if type information can be produced using a factory
> final TypeInformation typeFromFactory =
> createTypeInfoFromFactory(clazz, typeHierarchy, in1Type, in2Type);
> if (typeFromFactory != null) {
> return typeFromFactory;
> }
> // Object is handled as generic type info
> if (clazz.equals(Object.class)) {
> return new GenericTypeInfo<>(clazz);
> }
> // Class is handled as generic type info
> if (clazz.equals(Class.class)) {
> return new GenericTypeInfo<>(clazz);
> }
> // recursive types are handled as generic type info
> if (countTypeInHierarchy(typeHierarchy, clazz) > 1) {
> return new GenericTypeInfo<>(clazz);
> }
> // check for arrays
> if (clazz.isArray()) {
> // primitive arrays: int[], byte[], ...
> PrimitiveArrayTypeInfo primitiveArrayInfo =
> PrimitiveArrayTypeInfo.getInfoFor(clazz);
> if (primitiveArrayInfo != null) {
> return primitiveArrayInfo;
> }
> // basic type arrays: String[], Integer[], Double[]
> BasicArrayTypeInfo basicArrayInfo =
> BasicArrayTypeInfo.getInfoFor(clazz);
> if (basicArrayInfo != null) {
> return basicArrayInfo;
> }
> // object arrays
> else {
> TypeInformation componentTypeInfo =
> createTypeInfoWithTypeHierarchy(
> typeHierarchy, clazz.getComponentType(), in1Type,
> in2Type);
> return ObjectArrayTypeInfo.getInfoFor(clazz, componentTypeInfo);
> }
> }
> // check for writable types
> if (isHadoopWritable(clazz)) {
> return createHadoopWritableTypeInfo(clazz);
> }
> // check for basic types
> TypeInformation basicTypeInfo = BasicTypeInfo.getInfoFor(clazz);
> if (basicTypeInfo != null) {
> return basicTypeInfo;
> }
> // check for SQL time types
> TypeInformation timeTypeInfo = SqlTimeTypeInfo.getInfoFor(clazz);
> if (timeTypeInfo != null) {
> return timeTypeInfo;
> }
> // check for subclasses of Value
> if (Value.class.isAssignableFrom(clazz)) {
> Class valueClass = clazz.asSubclass(Value.class);
> return (TypeInformation)
> ValueTypeInfo.getValueTypeInfo(valueClass);
> }
> // check for subclasses of Tuple
> if
[jira] [Updated] (FLINK-6757) Investigate Apache Atlas integration
[ https://issues.apache.org/jira/browse/FLINK-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-6757: -- Labels: auto-deprioritized-major auto-deprioritized-minor auto-unassigned (was: auto-deprioritized-major auto-unassigned stale-minor) Priority: Not a Priority (was: Minor) This issue was labeled "stale-minor" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Minor, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Investigate Apache Atlas integration > > > Key: FLINK-6757 > URL: https://issues.apache.org/jira/browse/FLINK-6757 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Reporter: Till Rohrmann >Priority: Not a Priority > Labels: auto-deprioritized-major, auto-deprioritized-minor, > auto-unassigned > > Users asked for an integration of Apache Flink with Apache Atlas. It might be > worthwhile to investigate what is necessary to achieve this task. > References: > http://atlas.incubator.apache.org/StormAtlasHook.html -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32344) MongoDB connector support unbounded streaming read via ChangeStream feature
[ https://issues.apache.org/jira/browse/FLINK-32344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32344: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > MongoDB connector support unbounded streaming read via ChangeStream feature > --- > > Key: FLINK-32344 > URL: https://issues.apache.org/jira/browse/FLINK-32344 > Project: Flink > Issue Type: New Feature > Components: Connectors / MongoDB >Affects Versions: mongodb-1.0.1 >Reporter: Jiabao Sun >Assignee: Jiabao Sun >Priority: Major > Labels: pull-request-available, stale-assigned > > Change streams allow applications to access real-time data changes without > the complexity and risk of tailing the oplog. Applications can use change > streams to subscribe to all data changes on a single collection, a database, > or an entire deployment, and immediately react to them. Because change > streams use the aggregation framework, applications can also filter for > specific changes or transform the notifications at will. > We can use MongoDB change streams feature to support unbounded streaming read > for mongodb connector. > [Change > Streams|https://www.mongodb.com/docs/manual/changeStreams/#change-streams] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-30817) ClassCastException in TestValuesTableFactory.TestValuesScanTableSourceWithoutProjectionPushDown
[
https://issues.apache.org/jira/browse/FLINK-30817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-30817:
---
Labels: pull-request-available stale-minor (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> ClassCastException in
> TestValuesTableFactory.TestValuesScanTableSourceWithoutProjectionPushDown
> ---
>
> Key: FLINK-30817
> URL: https://issues.apache.org/jira/browse/FLINK-30817
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.16.0, 1.17.0
>Reporter: Shuiqiang Chen
>Priority: Minor
> Labels: pull-request-available, stale-minor
>
> When applying partitions in
> TestValuesScanTableSourceWithoutProjectionPushDown with no partition
> provided, the following code will cause ClassCastException
> {code:java}
> remainingPartitions = (List>) Collections.emptyMap();
> this.data.put(Collections.emptyMap(), Collections.emptyList());
> {code}
> {code:java}
> java.lang.ClassCastException: java.util.Collections$EmptyMap cannot be cast
> to java.util.List
> at
> org.apache.flink.table.planner.factories.TestValuesTableFactory$TestValuesScanTableSourceWithoutProjectionPushDown.applyPartitions(TestValuesTableFactory.java:1222)
> at
> org.apache.flink.table.planner.plan.abilities.source.PartitionPushDownSpec.apply(PartitionPushDownSpec.java:57)
> at
> org.apache.flink.table.planner.plan.rules.logical.PushPartitionIntoTableSourceScanRule.onMatch(PushPartitionIntoTableSourceScanRule.java:183)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:343)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32309) Shared classpaths and jars manager for jobs in sql gateway cause confliction
[ https://issues.apache.org/jira/browse/FLINK-32309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32309: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > Shared classpaths and jars manager for jobs in sql gateway cause confliction > > > Key: FLINK-32309 > URL: https://issues.apache.org/jira/browse/FLINK-32309 > Project: Flink > Issue Type: Bug > Components: Table SQL / Gateway >Affects Versions: 1.18.0 >Reporter: Fang Yong >Assignee: FangYong >Priority: Major > Labels: pull-request-available, stale-assigned > > Current all jobs in the same session of sql gateway will share the resource > manager which provide the classpath for jobs. After a job is performed, it's > classpath and jars will be in the shared resource manager which are used by > the next jobs. It may cause too many unnecessary jars in a job or even cause > confliction -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32256) Add ARRAY_MIN support in SQL & Table API
[
https://issues.apache.org/jira/browse/FLINK-32256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32256:
---
Labels: pull-request-available stale-assigned (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 30 days, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it.
> Add ARRAY_MIN support in SQL & Table API
>
>
> Key: FLINK-32256
> URL: https://issues.apache.org/jira/browse/FLINK-32256
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / Planner
>Affects Versions: 1.18.0
>Reporter: Bonnie Varghese
>Assignee: Hanyu Zheng
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Fix For: 1.18.0
>
>
> Find the minimum among all elements in the array for which ordering is
> supported.
> Syntax:
> array_min(array)
> Arguments:
> array: An ARRAY to be handled.
> Returns:
> The result matches the type of the elements. NULL elements are skipped. If
> array is empty, or contains only NULL elements, NULL is returned.
> Examples:
> {code:sql}
> SELECT array_min(array(1, 20, NULL, 3));
> -- 1
> {code}
> See also
> spark [https://spark.apache.org/docs/latest/api/sql/index.html#array_min]
> presto [https://prestodb.io/docs/current/functions/array.html]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32410) Allocate hash-based collections with sufficient capacity for expected size
[ https://issues.apache.org/jira/browse/FLINK-32410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32410: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > Allocate hash-based collections with sufficient capacity for expected size > -- > > Key: FLINK-32410 > URL: https://issues.apache.org/jira/browse/FLINK-32410 > Project: Flink > Issue Type: Improvement >Reporter: Stefan Richter >Assignee: Stefan Richter >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.18.0 > > > The JDK API to create hash-based collections for a certain capacity is > arguable misleading because it doesn't size the collections to "hold a > specific number of items" like you'd expect it would. Instead it sizes it to > hold load-factor% of the specified number. > For the common pattern to allocate a hash-based collection with the size of > expected elements to avoid rehashes, this means that a rehash is essentially > guaranteed. > We should introduce helper methods (similar to Guava's > `Maps.newHashMapWithExpectedSize(int)`) for allocations for expected size and > replace the direct constructor calls with those. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32437) Determine and set correct maxParallelism for operator chains
[
https://issues.apache.org/jira/browse/FLINK-32437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32437:
---
Labels: pull-request-available stale-assigned (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 30 days, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it.
> Determine and set correct maxParallelism for operator chains
>
>
> Key: FLINK-32437
> URL: https://issues.apache.org/jira/browse/FLINK-32437
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Reporter: Stefan Richter
>Assignee: Stefan Richter
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Fix For: 2.0.0
>
>
> Current code in {{StreamingJobGraphGenerator}} does not properly determine
> and set the correct maxParallelism of operator chains. We should set the
> maxParallelism of the chain as the minimum of all the maxParallelism values
> among operators in the chain.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32821) Streaming examples failed to execute due to error in packaging
[ https://issues.apache.org/jira/browse/FLINK-32821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752918#comment-17752918 ] Alexander Fedulov commented on FLINK-32821: --- [~Zhanghao Chen] I cannot reproduce the listed issues. How do you execute the examples? Do you face the issue also when running in IDE? > Streaming examples failed to execute due to error in packaging > -- > > Key: FLINK-32821 > URL: https://issues.apache.org/jira/browse/FLINK-32821 > Project: Flink > Issue Type: Bug > Components: Examples >Affects Versions: 1.18.0 >Reporter: Zhanghao Chen >Assignee: Zhanghao Chen >Priority: Major > > 5 out of the 7 streaming examples failed to run: > * Iteration & SessionWindowing & SocketWindowWordCount & WindowJoin failed > to run due to java.lang.NoClassDefFoundError: > org/apache/flink/streaming/examples/utils/ParameterTool > * TopSpeedWindowing failed to run due to: Caused by: > java.lang.ClassNotFoundException: > org.apache.flink.connector.datagen.source.GeneratorFunction > The NoClassDefFoundError with ParameterTool is introduced by [FLINK-32558] > Properly deprecate DataSet API - ASF JIRA (apache.org), and we'd better > resolve [FLINK-32820] ParameterTool is mistakenly marked as deprecated - ASF > JIRA (apache.org) first before we come to a fix for this problem. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] architgyl commented on a diff in pull request #23164: [FLINK- 32775]Add parent dir of files to classpath using yarn.provided.lib.dirs
architgyl commented on code in PR #23164:
URL: https://github.com/apache/flink/pull/23164#discussion_r1290630407
##
flink-yarn/src/test/java/org/apache/flink/yarn/YarnApplicationFileUploaderTest.java:
##
@@ -73,6 +74,34 @@ void testRegisterProvidedLocalResources(@TempDir File
flinkLibDir) throws IOExce
}
}
+@Test
+void testRegisterProvidedLocalResourcesWithParentDir(@TempDir File
flinkLibDir)
+throws IOException {
+final Map filesWithParentDir = getFilesWithParentDir();
+
+generateFilesInDirectory(flinkLibDir, filesWithParentDir);
+
+try (final YarnApplicationFileUploader yarnApplicationFileUploader =
+YarnApplicationFileUploader.from(
+FileSystem.get(new YarnConfiguration()),
+new Path(flinkLibDir.toURI()),
+Collections.singletonList(new
Path(flinkLibDir.toURI())),
+ApplicationId.newInstance(0, 0),
+DFSConfigKeys.DFS_REPLICATION_DEFAULT)) {
+
+List classPath =
yarnApplicationFileUploader.registerProvidedLocalResources();
+
+Set expectedClassPathEntries = new HashSet<>();
+for (String filePath : filesWithParentDir.keySet()) {
+String parentDir = new Path(filePath).getParent().toString();
+expectedClassPathEntries.add(parentDir);
+expectedClassPathEntries.add(filePath);
+}
+
+
assertThat(classPath).containsExactlyInAnyOrderElementsOf(expectedClassPathEntries);
Review Comment:
Updated the test case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] architgyl commented on a diff in pull request #23164: [FLINK- 32775]Add parent dir of files to classpath using yarn.provided.lib.dirs
architgyl commented on code in PR #23164:
URL: https://github.com/apache/flink/pull/23164#discussion_r1290630117
##
flink-yarn/src/main/java/org/apache/flink/yarn/YarnApplicationFileUploader.java:
##
@@ -360,6 +361,17 @@ List registerProvidedLocalResources() {
envShipResourceList.add(descriptor);
if (!isFlinkDistJar(filePath.getName()) &&
!isPlugin(filePath)) {
+URI parentDirectoryUri = new
Path(fileName).getParent().toUri();
+String relativeParentDirectory =
+new Path(filePath.getName())
+.toUri()
+.relativize(parentDirectoryUri)
+.toString();
+
+if
(!addedParentDirectories.contains(relativeParentDirectory)) {
+classPaths.add(relativeParentDirectory);
Review Comment:
Definitely, addressed the comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-32828) Kafka partition aware watermark not handled correctly shortly after job start up from checkpoint or savepoint
[
https://issues.apache.org/jira/browse/FLINK-32828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752905#comment-17752905
]
Grzegorz Liter commented on FLINK-32828:
[~mason6345] its not about idle partitions, its about a quite opposite
situation. During normal operation all partitions are active and more or less
aligned with timestamp. Now imagine a situation during startup when one thread
starts to consume some partitions faster than others. If it will progress
through events from that single partition fast enough to consume events with
timestamps that are bigger than some events in other partitions + watermark
time, it will start dropping those events as late events.
To mitigate this situation Kafka partition aware watermarks were implemented.
There is a clear bug here where in case of running fresh job or after having
job settled and having traffic on all partition works correctly. That means the
watermark emitted by source is a minimal watermark of all partitions. But in in
short window just after startup from checkpoint/savepoint watermark incorrectly
progresses just based on traffic from single partitions, where it should wait
until traffic on all partitions to take a minimal watermark of all partitions.
Please not that the logs are from the minimal example I have created (attached
to the tickets) and events are send by hand.
In production restarting job causes around 1 - 10% of events be dropped at the
very start of the job. Even thou in scope of single partition the out of
ordness is well below allowed one.
> Kafka partition aware watermark not handled correctly shortly after job start
> up from checkpoint or savepoint
> -
>
> Key: FLINK-32828
> URL: https://issues.apache.org/jira/browse/FLINK-32828
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Connectors / Kafka
>Affects Versions: 1.17.1
> Environment: Affected environments:
> * Local MiniCluster + Confluent Kafka run in docker
> ** See attached files
> * Flink job run in Kubernetes using Flink Operator 1.15 + 3 node Kafka
> cluster run in Kubernetes cluster
>Reporter: Grzegorz Liter
>Priority: Major
> Attachments: docker-compose.yml, test-job.java
>
>
> When using KafkaSource with partition aware watermarks. Watermarks are being
> emitted even when only one partition has some events just after job startup
> from savepoint/checkpoint. After it has some events on other partitions the
> watermark behaviour is correct and watermark is emited as a minimum watarmark
> from each partition.
>
> Steps to reproduce:
> # Setup a Kafka cluster with a topic that has 2 or more partitions. (see
> attached docker-compose.yml)
> ## {{./kafka-topics.sh --bootstrap-server localhost:9092 --create --topic
> test-2 --partitions 4}}
> # Create a job that (see attached `test-job.java`):
> ## uses a KafkaSource with
> `WatermarkStrategy.forBoundedOutOfOrderness(Duration.ofSeconds(10L)`
> ## has parallelism lower than number of partitions
> ## stores checkpoint/savepoint
> # Start job
> # Send events only on single partition
> ## {{./kafka-console-producer.sh --bootstrap-server localhost:9092 --topic
> test-2 --property "parse.key=true" --property "key.separator=:"}}
>
> {{14:51:19,883 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:51:18.849Z, watermark
> -292275055-05-16T16:47:04.192Z}}
> {{14:51:32,484 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:51:31.475Z, watermark
> -292275055-05-16T16:47:04.192Z}}
> {{14:51:35,914 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:51:34.909Z, watermark
> -292275055-05-16T16:47:04.192Z}}
> Expected: Watermark does not progress. Actual: Watermark does not progress.
> 5. Stop the job
> 6. Startup job from last checkpoint/savepoint
> 7. Send events only on single partitions
> {{14:53:41,693 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:53:40.662Z, watermark
> -292275055-05-16T16:47:04.192Z}}
> {{14:53:46,088 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:53:45.078Z, watermark
> 2023-08-10T12:53:30.661Z}}
> {{14:53:49,520 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:53:48.511Z, watermark
> 2023-08-10T12:53:35.077Z}}
> Expected: Watermark does not progress. {color:#ff}*Actual: Watermark has
> progress*{color}
>
> {color:#172b4d}To add bit more of
[jira] [Comment Edited] (FLINK-32828) Kafka partition aware watermark not handled correctly shortly after job start up from checkpoint or savepoint
[
https://issues.apache.org/jira/browse/FLINK-32828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752901#comment-17752901
]
Mason Chen edited comment on FLINK-32828 at 8/10/23 6:45 PM:
-
>From the logs, it looks like you have a key by and watermark is progressing
>because that one active partition is moving data to all other operators. I
>would start by setting and tuning the
>https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#dealing-with-idle-sources
was (Author: mason6345):
>From the logs, it looks like you have a key by and watermark is progressing
>because that one active partition is moving data to all other operators. I
>would start by increasing the `forBoundedOutOfOrderness` to account for the
>"late" arriving data
> Kafka partition aware watermark not handled correctly shortly after job start
> up from checkpoint or savepoint
> -
>
> Key: FLINK-32828
> URL: https://issues.apache.org/jira/browse/FLINK-32828
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Connectors / Kafka
>Affects Versions: 1.17.1
> Environment: Affected environments:
> * Local MiniCluster + Confluent Kafka run in docker
> ** See attached files
> * Flink job run in Kubernetes using Flink Operator 1.15 + 3 node Kafka
> cluster run in Kubernetes cluster
>Reporter: Grzegorz Liter
>Priority: Major
> Attachments: docker-compose.yml, test-job.java
>
>
> When using KafkaSource with partition aware watermarks. Watermarks are being
> emitted even when only one partition has some events just after job startup
> from savepoint/checkpoint. After it has some events on other partitions the
> watermark behaviour is correct and watermark is emited as a minimum watarmark
> from each partition.
>
> Steps to reproduce:
> # Setup a Kafka cluster with a topic that has 2 or more partitions. (see
> attached docker-compose.yml)
> ## {{./kafka-topics.sh --bootstrap-server localhost:9092 --create --topic
> test-2 --partitions 4}}
> # Create a job that (see attached `test-job.java`):
> ## uses a KafkaSource with
> `WatermarkStrategy.forBoundedOutOfOrderness(Duration.ofSeconds(10L)`
> ## has parallelism lower than number of partitions
> ## stores checkpoint/savepoint
> # Start job
> # Send events only on single partition
> ## {{./kafka-console-producer.sh --bootstrap-server localhost:9092 --topic
> test-2 --property "parse.key=true" --property "key.separator=:"}}
>
> {{14:51:19,883 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:51:18.849Z, watermark
> -292275055-05-16T16:47:04.192Z}}
> {{14:51:32,484 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:51:31.475Z, watermark
> -292275055-05-16T16:47:04.192Z}}
> {{14:51:35,914 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:51:34.909Z, watermark
> -292275055-05-16T16:47:04.192Z}}
> Expected: Watermark does not progress. Actual: Watermark does not progress.
> 5. Stop the job
> 6. Startup job from last checkpoint/savepoint
> 7. Send events only on single partitions
> {{14:53:41,693 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:53:40.662Z, watermark
> -292275055-05-16T16:47:04.192Z}}
> {{14:53:46,088 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:53:45.078Z, watermark
> 2023-08-10T12:53:30.661Z}}
> {{14:53:49,520 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:53:48.511Z, watermark
> 2023-08-10T12:53:35.077Z}}
> Expected: Watermark does not progress. {color:#ff}*Actual: Watermark has
> progress*{color}
>
> {color:#172b4d}To add bit more of context:{color}
> {color:#172b4d}8. Send events on other partitions and then send events only
> on single partitions{color}
> {{{color:#172b4d}14:54:55,112 WARN com.example.TestJob6$InputSink2
> [] - == Received: test-2/0: 2 -> a, timestamp
> 2023-08-10T12:54:54.104Z, watermark 2023-08-10T12:53:38.510Z
> 14:54:57,673 WARN com.example.TestJob6$InputSink2 [] - ==
> Received: test-2/1: 4 -> a, timestamp 2023-08-10T12:54:56.665Z, watermark
> 2023-08-10T12:53:38.510Z
> 14:54:57,673 WARN com.example.TestJob6$InputSink2 [] - ==
> Received: test-2/2: 5 -> a, timestamp 2023-08-10T12:54:57.554Z, watermark
> 2023-08-10T12:53:38.510Z
> 14:55:12,821 WARN
[jira] [Comment Edited] (FLINK-32828) Kafka partition aware watermark not handled correctly shortly after job start up from checkpoint or savepoint
[
https://issues.apache.org/jira/browse/FLINK-32828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752901#comment-17752901
]
Mason Chen edited comment on FLINK-32828 at 8/10/23 6:45 PM:
-
>From the logs, it looks like you have a key by and watermark is progressing
>because that one active partition is moving data to all other operators. I
>would start by setting and tuning the
>[https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#dealing-with-idle-sources]
> to account for the idle partitions
was (Author: mason6345):
>From the logs, it looks like you have a key by and watermark is progressing
>because that one active partition is moving data to all other operators. I
>would start by setting and tuning the
>https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#dealing-with-idle-sources
> Kafka partition aware watermark not handled correctly shortly after job start
> up from checkpoint or savepoint
> -
>
> Key: FLINK-32828
> URL: https://issues.apache.org/jira/browse/FLINK-32828
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Connectors / Kafka
>Affects Versions: 1.17.1
> Environment: Affected environments:
> * Local MiniCluster + Confluent Kafka run in docker
> ** See attached files
> * Flink job run in Kubernetes using Flink Operator 1.15 + 3 node Kafka
> cluster run in Kubernetes cluster
>Reporter: Grzegorz Liter
>Priority: Major
> Attachments: docker-compose.yml, test-job.java
>
>
> When using KafkaSource with partition aware watermarks. Watermarks are being
> emitted even when only one partition has some events just after job startup
> from savepoint/checkpoint. After it has some events on other partitions the
> watermark behaviour is correct and watermark is emited as a minimum watarmark
> from each partition.
>
> Steps to reproduce:
> # Setup a Kafka cluster with a topic that has 2 or more partitions. (see
> attached docker-compose.yml)
> ## {{./kafka-topics.sh --bootstrap-server localhost:9092 --create --topic
> test-2 --partitions 4}}
> # Create a job that (see attached `test-job.java`):
> ## uses a KafkaSource with
> `WatermarkStrategy.forBoundedOutOfOrderness(Duration.ofSeconds(10L)`
> ## has parallelism lower than number of partitions
> ## stores checkpoint/savepoint
> # Start job
> # Send events only on single partition
> ## {{./kafka-console-producer.sh --bootstrap-server localhost:9092 --topic
> test-2 --property "parse.key=true" --property "key.separator=:"}}
>
> {{14:51:19,883 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:51:18.849Z, watermark
> -292275055-05-16T16:47:04.192Z}}
> {{14:51:32,484 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:51:31.475Z, watermark
> -292275055-05-16T16:47:04.192Z}}
> {{14:51:35,914 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:51:34.909Z, watermark
> -292275055-05-16T16:47:04.192Z}}
> Expected: Watermark does not progress. Actual: Watermark does not progress.
> 5. Stop the job
> 6. Startup job from last checkpoint/savepoint
> 7. Send events only on single partitions
> {{14:53:41,693 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:53:40.662Z, watermark
> -292275055-05-16T16:47:04.192Z}}
> {{14:53:46,088 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:53:45.078Z, watermark
> 2023-08-10T12:53:30.661Z}}
> {{14:53:49,520 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:53:48.511Z, watermark
> 2023-08-10T12:53:35.077Z}}
> Expected: Watermark does not progress. {color:#ff}*Actual: Watermark has
> progress*{color}
>
> {color:#172b4d}To add bit more of context:{color}
> {color:#172b4d}8. Send events on other partitions and then send events only
> on single partitions{color}
> {{{color:#172b4d}14:54:55,112 WARN com.example.TestJob6$InputSink2
> [] - == Received: test-2/0: 2 -> a, timestamp
> 2023-08-10T12:54:54.104Z, watermark 2023-08-10T12:53:38.510Z
> 14:54:57,673 WARN com.example.TestJob6$InputSink2 [] - ==
> Received: test-2/1: 4 -> a, timestamp 2023-08-10T12:54:56.665Z, watermark
> 2023-08-10T12:53:38.510Z
> 14:54:57,673 WARN com.example.TestJob6$InputSink2 [] - ==
> Received: test-2/2: 5 -> a, timestamp
[jira] [Comment Edited] (FLINK-32828) Kafka partition aware watermark not handled correctly shortly after job start up from checkpoint or savepoint
[
https://issues.apache.org/jira/browse/FLINK-32828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752901#comment-17752901
]
Mason Chen edited comment on FLINK-32828 at 8/10/23 6:45 PM:
-
>From the logs, it looks like you have a key by and watermark is progressing
>because that one active partition is moving data to all other operators. I
>would start by setting and tuning the idleness
>([https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#dealing-with-idle-sources)|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#dealing-with-idle-sources]
> to account for the idle partitions
was (Author: mason6345):
>From the logs, it looks like you have a key by and watermark is progressing
>because that one active partition is moving data to all other operators. I
>would start by setting and tuning the
>[https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#dealing-with-idle-sources]
> to account for the idle partitions
> Kafka partition aware watermark not handled correctly shortly after job start
> up from checkpoint or savepoint
> -
>
> Key: FLINK-32828
> URL: https://issues.apache.org/jira/browse/FLINK-32828
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Connectors / Kafka
>Affects Versions: 1.17.1
> Environment: Affected environments:
> * Local MiniCluster + Confluent Kafka run in docker
> ** See attached files
> * Flink job run in Kubernetes using Flink Operator 1.15 + 3 node Kafka
> cluster run in Kubernetes cluster
>Reporter: Grzegorz Liter
>Priority: Major
> Attachments: docker-compose.yml, test-job.java
>
>
> When using KafkaSource with partition aware watermarks. Watermarks are being
> emitted even when only one partition has some events just after job startup
> from savepoint/checkpoint. After it has some events on other partitions the
> watermark behaviour is correct and watermark is emited as a minimum watarmark
> from each partition.
>
> Steps to reproduce:
> # Setup a Kafka cluster with a topic that has 2 or more partitions. (see
> attached docker-compose.yml)
> ## {{./kafka-topics.sh --bootstrap-server localhost:9092 --create --topic
> test-2 --partitions 4}}
> # Create a job that (see attached `test-job.java`):
> ## uses a KafkaSource with
> `WatermarkStrategy.forBoundedOutOfOrderness(Duration.ofSeconds(10L)`
> ## has parallelism lower than number of partitions
> ## stores checkpoint/savepoint
> # Start job
> # Send events only on single partition
> ## {{./kafka-console-producer.sh --bootstrap-server localhost:9092 --topic
> test-2 --property "parse.key=true" --property "key.separator=:"}}
>
> {{14:51:19,883 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:51:18.849Z, watermark
> -292275055-05-16T16:47:04.192Z}}
> {{14:51:32,484 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:51:31.475Z, watermark
> -292275055-05-16T16:47:04.192Z}}
> {{14:51:35,914 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:51:34.909Z, watermark
> -292275055-05-16T16:47:04.192Z}}
> Expected: Watermark does not progress. Actual: Watermark does not progress.
> 5. Stop the job
> 6. Startup job from last checkpoint/savepoint
> 7. Send events only on single partitions
> {{14:53:41,693 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:53:40.662Z, watermark
> -292275055-05-16T16:47:04.192Z}}
> {{14:53:46,088 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:53:45.078Z, watermark
> 2023-08-10T12:53:30.661Z}}
> {{14:53:49,520 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:53:48.511Z, watermark
> 2023-08-10T12:53:35.077Z}}
> Expected: Watermark does not progress. {color:#ff}*Actual: Watermark has
> progress*{color}
>
> {color:#172b4d}To add bit more of context:{color}
> {color:#172b4d}8. Send events on other partitions and then send events only
> on single partitions{color}
> {{{color:#172b4d}14:54:55,112 WARN com.example.TestJob6$InputSink2
> [] - == Received: test-2/0: 2 -> a, timestamp
> 2023-08-10T12:54:54.104Z, watermark 2023-08-10T12:53:38.510Z
> 14:54:57,673 WARN com.example.TestJob6$InputSink2 [] - ==
> Received: test-2/1: 4 -> a, timestamp
[jira] [Commented] (FLINK-32828) Kafka partition aware watermark not handled correctly shortly after job start up from checkpoint or savepoint
[
https://issues.apache.org/jira/browse/FLINK-32828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752901#comment-17752901
]
Mason Chen commented on FLINK-32828:
>From the logs, it looks like you have a key by and watermark is progressing
>because that one active partition is moving data to all other operators. I
>would start by increasing the `forBoundedOutOfOrderness` to account for the
>"late" arriving data
> Kafka partition aware watermark not handled correctly shortly after job start
> up from checkpoint or savepoint
> -
>
> Key: FLINK-32828
> URL: https://issues.apache.org/jira/browse/FLINK-32828
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Connectors / Kafka
>Affects Versions: 1.17.1
> Environment: Affected environments:
> * Local MiniCluster + Confluent Kafka run in docker
> ** See attached files
> * Flink job run in Kubernetes using Flink Operator 1.15 + 3 node Kafka
> cluster run in Kubernetes cluster
>Reporter: Grzegorz Liter
>Priority: Major
> Attachments: docker-compose.yml, test-job.java
>
>
> When using KafkaSource with partition aware watermarks. Watermarks are being
> emitted even when only one partition has some events just after job startup
> from savepoint/checkpoint. After it has some events on other partitions the
> watermark behaviour is correct and watermark is emited as a minimum watarmark
> from each partition.
>
> Steps to reproduce:
> # Setup a Kafka cluster with a topic that has 2 or more partitions. (see
> attached docker-compose.yml)
> ## {{./kafka-topics.sh --bootstrap-server localhost:9092 --create --topic
> test-2 --partitions 4}}
> # Create a job that (see attached `test-job.java`):
> ## uses a KafkaSource with
> `WatermarkStrategy.forBoundedOutOfOrderness(Duration.ofSeconds(10L)`
> ## has parallelism lower than number of partitions
> ## stores checkpoint/savepoint
> # Start job
> # Send events only on single partition
> ## {{./kafka-console-producer.sh --bootstrap-server localhost:9092 --topic
> test-2 --property "parse.key=true" --property "key.separator=:"}}
>
> {{14:51:19,883 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:51:18.849Z, watermark
> -292275055-05-16T16:47:04.192Z}}
> {{14:51:32,484 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:51:31.475Z, watermark
> -292275055-05-16T16:47:04.192Z}}
> {{14:51:35,914 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:51:34.909Z, watermark
> -292275055-05-16T16:47:04.192Z}}
> Expected: Watermark does not progress. Actual: Watermark does not progress.
> 5. Stop the job
> 6. Startup job from last checkpoint/savepoint
> 7. Send events only on single partitions
> {{14:53:41,693 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:53:40.662Z, watermark
> -292275055-05-16T16:47:04.192Z}}
> {{14:53:46,088 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:53:45.078Z, watermark
> 2023-08-10T12:53:30.661Z}}
> {{14:53:49,520 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:53:48.511Z, watermark
> 2023-08-10T12:53:35.077Z}}
> Expected: Watermark does not progress. {color:#ff}*Actual: Watermark has
> progress*{color}
>
> {color:#172b4d}To add bit more of context:{color}
> {color:#172b4d}8. Send events on other partitions and then send events only
> on single partitions{color}
> {{{color:#172b4d}14:54:55,112 WARN com.example.TestJob6$InputSink2
> [] - == Received: test-2/0: 2 -> a, timestamp
> 2023-08-10T12:54:54.104Z, watermark 2023-08-10T12:53:38.510Z
> 14:54:57,673 WARN com.example.TestJob6$InputSink2 [] - ==
> Received: test-2/1: 4 -> a, timestamp 2023-08-10T12:54:56.665Z, watermark
> 2023-08-10T12:53:38.510Z
> 14:54:57,673 WARN com.example.TestJob6$InputSink2 [] - ==
> Received: test-2/2: 5 -> a, timestamp 2023-08-10T12:54:57.554Z, watermark
> 2023-08-10T12:53:38.510Z
> 14:55:12,821 WARN com.example.TestJob6$InputSink2 [] - ==
> Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:55:11.814Z, watermark
> 2023-08-10T12:54:44.103Z
> 14:55:16,099 WARN com.example.TestJob6$InputSink2 [] - ==
> Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:55:15.091Z, watermark
> 2023-08-10T12:54:44.103Z
> 14:55:19,122 WARN com.example.TestJob6$InputSink2 [] - ==
>
[GitHub] [flink] venkata91 commented on pull request #23164: [FLINK- 32775]Add parent dir of files to classpath using yarn.provided.lib.dirs
venkata91 commented on PR #23164: URL: https://github.com/apache/flink/pull/23164#issuecomment-1673681802 Could you please add the before and after classpath output? Please add it to description if it is small if not a gist probably would be better. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] venkata91 commented on a diff in pull request #23164: [FLINK- 32775]Add parent dir of files to classpath using yarn.provided.lib.dirs
venkata91 commented on code in PR #23164:
URL: https://github.com/apache/flink/pull/23164#discussion_r1290512478
##
flink-yarn/src/test/java/org/apache/flink/yarn/YarnApplicationFileUploaderTest.java:
##
@@ -143,6 +172,16 @@ private static Map getLibJars() {
return libJars;
}
+private static Map getFilesWithParentDir() {
+final HashMap filesWithParentDir = new HashMap<>(2);
Review Comment:
super nit: `HashMap` -> `Map`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
