[GitHub] [flink] shuiqiangchen commented on pull request #19958: [FLINK-27159][table-api] Support first_value/last_value in the Table API
shuiqiangchen commented on PR #19958: URL: https://github.com/apache/flink/pull/19958#issuecomment-1157258316 @dianfu Thanks, I will refer to those commits in the ticket you mentioned. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dusukang commented on pull request #19741: [FLINK-27794][connectors/jdbc]Fix the bug of wrong primary key in MysqlCatalog
dusukang commented on PR #19741: URL: https://github.com/apache/flink/pull/19741#issuecomment-1157248524 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28087) Add validation for the meta.name of FlinkDeployment CR
[
https://issues.apache.org/jira/browse/FLINK-28087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fred k updated FLINK-28087:
---
Description:
The meta.name of FlinkDeployment CR meta.name must be match the regex:
{code:java}
'[a-z]([-a-z0-9]*[a-z0-9])?'{code}
By RRC-1035, a DNS-1035 label must consist of lower case alphanumeric
characters or '-', start with an alphabetic character, and end with an
alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for validation
is
'[a-z|https://github.com/apache/flink-kubernetes-operator/pull/%5B-a-z0-9%5D*%5Ba-z0-9%5D]?').
The rest service name of Flink Cluster uses the meta.name of FlinkDeployment
CR, So the value of meta.name must follow the convention of DNS-1035 label.
To avoid operator repeatedly creating and destroying flink clusters due to
invalid service name, So I suggest add validation for the meta.name of
FlinkDeployment CR.
was:
The meta.name of FlinkDeployment CR meta.name must be match the regex:
{{}}
{code:java}
'[a-z]([-a-z0-9]*[a-z0-9])?'{code}
{{}}
By RRC-1035, a DNS-1035 label must consist of lower case alphanumeric
characters or '-', start with an alphabetic character, and end with an
alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for validation
is
'[a-z|https://github.com/apache/flink-kubernetes-operator/pull/%5B-a-z0-9%5D*%5Ba-z0-9%5D]?').
The rest service name of Flink Cluster uses the meta.name of FlinkDeployment
CR, So the value of meta.name must follow the convention of DNS-1035 label.
To avoid operator repeatedly creating and destroying flink clusters due to
invalid service name, So I suggest add validation for the meta.name of
FlinkDeployment CR.
> Add validation for the meta.name of FlinkDeployment CR
> --
>
> Key: FLINK-28087
> URL: https://issues.apache.org/jira/browse/FLINK-28087
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Fred k
>Priority: Major
> Labels: pull-request-available
> Fix For: kubernetes-operator-1.1.0
>
> Attachments: operator-flink.jpg
>
>
> The meta.name of FlinkDeployment CR meta.name must be match the regex:
> {code:java}
> '[a-z]([-a-z0-9]*[a-z0-9])?'{code}
> By RRC-1035, a DNS-1035 label must consist of lower case alphanumeric
> characters or '-', start with an alphabetic character, and end with an
> alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for
> validation is
> '[a-z|https://github.com/apache/flink-kubernetes-operator/pull/%5B-a-z0-9%5D*%5Ba-z0-9%5D]?').
> The rest service name of Flink Cluster uses the meta.name of FlinkDeployment
> CR, So the value of meta.name must follow the convention of DNS-1035 label.
> To avoid operator repeatedly creating and destroying flink clusters due to
> invalid service name, So I suggest add validation for the meta.name of
> FlinkDeployment CR.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (FLINK-28087) Add validation for meta.name of FlinkDeployment CR
[
https://issues.apache.org/jira/browse/FLINK-28087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fred k updated FLINK-28087:
---
Description:
The meta.name of FlinkDeployment CR meta.name must be match the regex:
{{}}
{code:java}
'[a-z]([-a-z0-9]*[a-z0-9])?'{code}
{{}}
By RRC-1035, a DNS-1035 label must consist of lower case alphanumeric
characters or '-', start with an alphabetic character, and end with an
alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for validation
is
'[a-z|https://github.com/apache/flink-kubernetes-operator/pull/%5B-a-z0-9%5D*%5Ba-z0-9%5D]?').
The rest service name of Flink Cluster uses the meta.name of FlinkDeployment
CR, So the value of meta.name must follow the convention of DNS-1035 label.
To avoid operator repeatedly creating and destroying flink clusters due to
invalid service name, So I suggest add validation for the meta.name of
FlinkDeployment CR.
was:
The FlinkDeployment CR meta.name must be match the regex:
{code:java}
'[a-z]([-a-z0-9]*[a-z0-9])?'{code}
By RRC-1035, a DNS-1035 label must consist of lower case alphanumeric
characters or '-', start with an alphabetic character, and end with an
alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for
validation is '[a-z]([-a-z0-9]*[a-z0-9])?').
The rest service name of Flink Cluster uses the meta.name of FlinkDeployment
CR, So meta.name value must follow the convention of DNS-1035 label.
To avoid operator repeatedly creating and destroying flink clusters due to
invalid service name, So I suggest add validation for meta.name of
FlinkDeployment CR.
> Add validation for meta.name of FlinkDeployment CR
> --
>
> Key: FLINK-28087
> URL: https://issues.apache.org/jira/browse/FLINK-28087
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Fred k
>Priority: Major
> Labels: pull-request-available
> Fix For: kubernetes-operator-1.1.0
>
> Attachments: operator-flink.jpg
>
>
> The meta.name of FlinkDeployment CR meta.name must be match the regex:
> {{}}
> {code:java}
> '[a-z]([-a-z0-9]*[a-z0-9])?'{code}
> {{}}
> By RRC-1035, a DNS-1035 label must consist of lower case alphanumeric
> characters or '-', start with an alphabetic character, and end with an
> alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for
> validation is
> '[a-z|https://github.com/apache/flink-kubernetes-operator/pull/%5B-a-z0-9%5D*%5Ba-z0-9%5D]?').
> The rest service name of Flink Cluster uses the meta.name of FlinkDeployment
> CR, So the value of meta.name must follow the convention of DNS-1035 label.
> To avoid operator repeatedly creating and destroying flink clusters due to
> invalid service name, So I suggest add validation for the meta.name of
> FlinkDeployment CR.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (FLINK-28087) Add validation for the meta.name of FlinkDeployment CR
[
https://issues.apache.org/jira/browse/FLINK-28087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fred k updated FLINK-28087:
---
Summary: Add validation for the meta.name of FlinkDeployment CR (was: Add
validation for meta.name of FlinkDeployment CR)
> Add validation for the meta.name of FlinkDeployment CR
> --
>
> Key: FLINK-28087
> URL: https://issues.apache.org/jira/browse/FLINK-28087
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Fred k
>Priority: Major
> Labels: pull-request-available
> Fix For: kubernetes-operator-1.1.0
>
> Attachments: operator-flink.jpg
>
>
> The meta.name of FlinkDeployment CR meta.name must be match the regex:
> {{}}
> {code:java}
> '[a-z]([-a-z0-9]*[a-z0-9])?'{code}
> {{}}
> By RRC-1035, a DNS-1035 label must consist of lower case alphanumeric
> characters or '-', start with an alphabetic character, and end with an
> alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for
> validation is
> '[a-z|https://github.com/apache/flink-kubernetes-operator/pull/%5B-a-z0-9%5D*%5Ba-z0-9%5D]?').
> The rest service name of Flink Cluster uses the meta.name of FlinkDeployment
> CR, So the value of meta.name must follow the convention of DNS-1035 label.
> To avoid operator repeatedly creating and destroying flink clusters due to
> invalid service name, So I suggest add validation for the meta.name of
> FlinkDeployment CR.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (FLINK-28087) Add validation for meta.name of FlinkDeployment CR
[
https://issues.apache.org/jira/browse/FLINK-28087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-28087:
---
Labels: pull-request-available (was: )
> Add validation for meta.name of FlinkDeployment CR
> --
>
> Key: FLINK-28087
> URL: https://issues.apache.org/jira/browse/FLINK-28087
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Fred k
>Priority: Major
> Labels: pull-request-available
> Fix For: kubernetes-operator-1.1.0
>
> Attachments: operator-flink.jpg
>
>
> The FlinkDeployment CR meta.name must be match the regex:
> {code:java}
> '[a-z]([-a-z0-9]*[a-z0-9])?'{code}
> By RRC-1035, a DNS-1035 label must consist of lower case alphanumeric
> characters or '-', start with an alphabetic character, and end with an
> alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for
> validation is '[a-z]([-a-z0-9]*[a-z0-9])?').
> The rest service name of Flink Cluster uses the meta.name of FlinkDeployment
> CR, So meta.name value must follow the convention of DNS-1035 label.
> To avoid operator repeatedly creating and destroying flink clusters due to
> invalid service name, So I suggest add validation for meta.name of
> FlinkDeployment CR.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[GitHub] [flink-kubernetes-operator] Grypse opened a new pull request, #270: [FLINK-28087]add validation for the meta.name of FlinkDeployment CR.
Grypse opened a new pull request, #270: URL: https://github.com/apache/flink-kubernetes-operator/pull/270 The meta.name of FlinkDeployment CR meta.name must be match the regex: ` '[a-z]([-a-z0-9]*[a-z0-9])?'` By RRC-1035, a DNS-1035 label must consist of lower case alphanumeric characters or '-', start with an alphabetic character, and end with an alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for validation is '[a-z]([-a-z0-9]*[a-z0-9])?'). The rest service name of Flink Cluster uses the meta.name of FlinkDeployment CR, So the value of meta.name must follow the convention of DNS-1035 label. To avoid operator repeatedly creating and destroying flink clusters due to invalid service name, So I suggest add validation for the meta.name of FlinkDeployment CR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dianfu commented on a diff in pull request #19295: [FLINK-26941][cep] Support Pattern end with notFollowedBy with window
dianfu commented on code in PR #19295:
URL: https://github.com/apache/flink/pull/19295#discussion_r898692689
##
docs/content/docs/libs/cep.md:
##
@@ -718,6 +718,35 @@ next.within(Time.seconds(10))
{{< /tab >}}
{{< /tabs >}}
+Notice that A pattern sequence can end in notFollowedBy() with temporal
constraint
+E.g. a pattern like:
Review Comment:
need also update the Chinese documentation which is located under directory
*content.zh*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] dianfu commented on a diff in pull request #19295: [FLINK-26941][cep] Support Pattern end with notFollowedBy with window
dianfu commented on code in PR #19295:
URL: https://github.com/apache/flink/pull/19295#discussion_r898652853
##
flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/nfa/compiler/NFACompiler.java:
##
@@ -157,11 +157,8 @@ public static boolean canProduceEmptyMatches(final
Pattern pattern) {
* create multiple NFAs.
*/
void compileFactory() {
-if (currentPattern.getQuantifier().getConsumingStrategy()
-== Quantifier.ConsumingStrategy.NOT_FOLLOW) {
-throw new MalformedPatternException(
-"NotFollowedBy is not supported as a last part of a
Pattern!");
-}
+
+Pattern lastPattern = currentPattern;
Review Comment:
```suggestion
Pattern lastPattern = currentPattern;
```
Nit
##
flink-libraries/flink-cep/src/test/java/org/apache/flink/cep/CEPITCase.java:
##
@@ -630,6 +630,226 @@ public String select(Map> pattern) {
assertEquals(expected, resultList);
}
+@Test
+public void testNotFollowedByWithIn() throws Exception {
Review Comment:
Move these test cases to NotPatternITCase?
##
docs/content/docs/libs/cep.md:
##
@@ -718,6 +718,35 @@ next.within(Time.seconds(10))
{{< /tab >}}
{{< /tabs >}}
+Notice that A pattern sequence can end in notFollowedBy() with temporal
constraint
Review Comment:
```suggestion
Notice that a pattern sequence can end in notFollowedBy() with temporal
constraint
```
##
docs/content/docs/libs/cep.md:
##
@@ -718,6 +718,35 @@ next.within(Time.seconds(10))
{{< /tab >}}
{{< /tabs >}}
+Notice that A pattern sequence can end in notFollowedBy() with temporal
constraint
+E.g. a pattern like:
Review Comment:
need also update the chinese documentation which is located under directory
*content.zh*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] liuzhuang2017 commented on pull request #19966: [hotfix][docs] Fix the Intellij key nouns.
liuzhuang2017 commented on PR #19966: URL: https://github.com/apache/flink/pull/19966#issuecomment-1157228755 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] luoyuxia commented on a diff in pull request #19851: [FLINK-20765][table-planner]The nullability of result for boolean expression in ScalarOperatorGens should be compatible with the r
luoyuxia commented on code in PR #19851:
URL: https://github.com/apache/flink/pull/19851#discussion_r898680314
##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/calls/ScalarOperatorGens.scala:
##
@@ -351,9 +351,11 @@ object ScalarOperatorGens {
checkImplicitConversionValidity(left, right)
val canEqual = isInteroperable(left.resultType, right.resultType)
if (isCharacterString(left.resultType) &&
isCharacterString(right.resultType)) {
- generateOperatorIfNotNull(ctx, new BooleanType(), left, right) {
-(leftTerm, rightTerm) => s"$leftTerm.equals($rightTerm)"
- }
+ generateOperatorIfNotNull(
+ctx,
+new BooleanType(left.resultType.isNullable ||
right.resultType.isNullable),
Review Comment:
I have fix all the expressions. Hope I haven't missed
any expression.
##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/calls/ScalarOperatorGens.scala:
##
@@ -351,9 +351,11 @@ object ScalarOperatorGens {
checkImplicitConversionValidity(left, right)
val canEqual = isInteroperable(left.resultType, right.resultType)
if (isCharacterString(left.resultType) &&
isCharacterString(right.resultType)) {
- generateOperatorIfNotNull(ctx, new BooleanType(), left, right) {
-(leftTerm, rightTerm) => s"$leftTerm.equals($rightTerm)"
- }
+ generateOperatorIfNotNull(
+ctx,
+new BooleanType(left.resultType.isNullable ||
right.resultType.isNullable),
Review Comment:
I have fix all the expressions. Hope I haven't missed any expression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] luoyuxia commented on a diff in pull request #19851: [FLINK-20765][table-planner]The nullability of result for boolean expression in ScalarOperatorGens should be compatible with the r
luoyuxia commented on code in PR #19851:
URL: https://github.com/apache/flink/pull/19851#discussion_r898680313
##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/calls/ScalarOperatorGens.scala:
##
@@ -351,9 +351,11 @@ object ScalarOperatorGens {
checkImplicitConversionValidity(left, right)
val canEqual = isInteroperable(left.resultType, right.resultType)
if (isCharacterString(left.resultType) &&
isCharacterString(right.resultType)) {
- generateOperatorIfNotNull(ctx, new BooleanType(), left, right) {
-(leftTerm, rightTerm) => s"$leftTerm.equals($rightTerm)"
- }
+ generateOperatorIfNotNull(
+ctx,
+new BooleanType(left.resultType.isNullable ||
right.resultType.isNullable),
Review Comment:
I have fix all the expressions. Hope I haven't miss any expression.
##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/calls/ScalarOperatorGens.scala:
##
@@ -351,9 +351,11 @@ object ScalarOperatorGens {
checkImplicitConversionValidity(left, right)
val canEqual = isInteroperable(left.resultType, right.resultType)
if (isCharacterString(left.resultType) &&
isCharacterString(right.resultType)) {
- generateOperatorIfNotNull(ctx, new BooleanType(), left, right) {
-(leftTerm, rightTerm) => s"$leftTerm.equals($rightTerm)"
- }
+ generateOperatorIfNotNull(
+ctx,
+new BooleanType(left.resultType.isNullable ||
right.resultType.isNullable),
Review Comment:
I have fix all the expressions. Hope I haven't miss any expression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] luoyuxia commented on a diff in pull request #19851: [FLINK-20765][table-planner]The nullability of result for boolean expression in ScalarOperatorGens should be compatible with the r
luoyuxia commented on code in PR #19851:
URL: https://github.com/apache/flink/pull/19851#discussion_r898679763
##
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/expressions/utils/ScalarOperatorsTestBase.scala:
##
@@ -67,8 +67,8 @@ abstract class ScalarOperatorsTestBase extends
ExpressionTestBase {
DataTypes.FIELD("f8", DataTypes.INT()),
DataTypes.FIELD("f9", DataTypes.INT()),
DataTypes.FIELD("f10", DataTypes.STRING()),
- DataTypes.FIELD("f11", DataTypes.BOOLEAN()),
- DataTypes.FIELD("f12", DataTypes.BOOLEAN()),
+ DataTypes.FIELD("f11", DataTypes.BOOLEAN().notNull()),
+ DataTypes.FIELD("f12", DataTypes.BOOLEAN().notNull()),
Review Comment:
haven't noticed `f12=null`, by declaring it null, I want to test the case
`and(boolean not null, boolean not null)`. I use another way to test it, so
revert this changes.
##
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/expressions/utils/ScalarOperatorsTestBase.scala:
##
@@ -67,8 +67,8 @@ abstract class ScalarOperatorsTestBase extends
ExpressionTestBase {
DataTypes.FIELD("f8", DataTypes.INT()),
DataTypes.FIELD("f9", DataTypes.INT()),
DataTypes.FIELD("f10", DataTypes.STRING()),
- DataTypes.FIELD("f11", DataTypes.BOOLEAN()),
- DataTypes.FIELD("f12", DataTypes.BOOLEAN()),
+ DataTypes.FIELD("f11", DataTypes.BOOLEAN().notNull()),
+ DataTypes.FIELD("f12", DataTypes.BOOLEAN().notNull()),
Review Comment:
haven't noticed `f12=null`, by declaring it null, I want to test the case
`and(boolean not null, boolean not null)`. I use another way to test it, so
revert this changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-28087) Add validation for meta.name of FlinkDeployment CR
[
https://issues.apache.org/jira/browse/FLINK-28087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fred k updated FLINK-28087:
---
Description:
The FlinkDeployment CR meta.name must be match the regex:
{code:java}
'[a-z]([-a-z0-9]*[a-z0-9])?'{code}
By RRC-1035, a DNS-1035 label must consist of lower case alphanumeric
characters or '-', start with an alphabetic character, and end with an
alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for
validation is '[a-z]([-a-z0-9]*[a-z0-9])?').
The rest service name of Flink Cluster uses the meta.name of FlinkDeployment
CR, So meta.name value must follow the convention of DNS-1035 label.
To avoid operator repeatedly creating and destroying flink clusters due to
invalid service name, So I suggest add validation for meta.name of
FlinkDeployment CR.
was:
The FlinkDeployment CR meta.name must be match the regex:
{code:java}
'[a-z]([-a-z0-9]*[a-z0-9])?'{code}
By RRC-1035, a DNS-1035 label must consist of lower case alphanumeric
characters or '-', start with an alphabetic character, and end with an
alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for
validation is '[a-z]([-a-z0-9]*[a-z0-9])?').
The rest service name of Flink Cluster uses the meta.name of FlinkDeployment,
So meta.name value must follow the convention of DNS-1035 label.
To avoid operator repeatedly creating and destroying flink clusters due to
invalid service name, So I suggest add validation for meta.name of
FlinkDeployment CR.
> Add validation for meta.name of FlinkDeployment CR
> --
>
> Key: FLINK-28087
> URL: https://issues.apache.org/jira/browse/FLINK-28087
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Fred k
>Priority: Major
> Fix For: kubernetes-operator-1.1.0
>
> Attachments: operator-flink.jpg
>
>
> The FlinkDeployment CR meta.name must be match the regex:
> {code:java}
> '[a-z]([-a-z0-9]*[a-z0-9])?'{code}
> By RRC-1035, a DNS-1035 label must consist of lower case alphanumeric
> characters or '-', start with an alphabetic character, and end with an
> alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for
> validation is '[a-z]([-a-z0-9]*[a-z0-9])?').
> The rest service name of Flink Cluster uses the meta.name of FlinkDeployment
> CR, So meta.name value must follow the convention of DNS-1035 label.
> To avoid operator repeatedly creating and destroying flink clusters due to
> invalid service name, So I suggest add validation for meta.name of
> FlinkDeployment CR.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Comment Edited] (FLINK-27953) using the original order to add the primary key in PushProjectIntoTableSourceScanRule
[
https://issues.apache.org/jira/browse/FLINK-27953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17554894#comment-17554894
]
zoucao edited comment on FLINK-27953 at 6/16/22 3:57 AM:
-
Hi [~godfreyhe], could you have time to take a look?
was (Author: zoucao):
Hi [~godfrey], could you have time to take a look?
> using the original order to add the primary key in
> PushProjectIntoTableSourceScanRule
> -
>
> Key: FLINK-27953
> URL: https://issues.apache.org/jira/browse/FLINK-27953
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Affects Versions: 1.14.4
>Reporter: zoucao
>Priority: Major
>
> In PushProjectIntoTableSourceScanRule, if the source produces a changelog
> stream, the primary key will be added to the end of projected fields, see the
> following SQL:
> {code:java}
> StreamTableTestUtil util = streamTestUtil(TableConfig.getDefault());
> TableEnvironment tEnv = util.getTableEnv();
> String srcTableDdl =
> "CREATE TABLE fs (\n"
> + " a bigint,\n"
> + " b int,\n"
> + " c varchar,\n"
> + " d int,\n"
> + " e int,\n "
> + " primary key (a,b) not enforced \n"
> + ") with (\n"
> + " 'connector' = 'values',\n"
> + " 'disable-lookup'='true',\n"
> + " 'changelog-mode' = 'I,UB,UA,D')";
> tEnv.executeSql(srcTableDdl);
> tEnv.getConfig().set("table.exec.source.cdc-events-duplicate",
> "true");
> {code}
> {code:java}
> System.out.println(tEnv.explainSql("select a, c from fs where c > 0 and b =
> 0"));
> projected list:
> [[0],[1],[2]]
> == Optimized Execution Plan ==
> Calc(select=[a, c], where=[(CAST(c AS BIGINT) > 0)])
> +- ChangelogNormalize(key=[a, b])
>+- Exchange(distribution=[hash[a, b]])
> +- Calc(select=[a, b, c], where=[(b = 0)])
> +- DropUpdateBefore
> +- TableSourceScan(table=[[default_catalog, default_database, fs,
> filter=[], project=[a, b, c], metadata=[]]], fields=[a, b, c])
> {code}
> {code:java}
> System.out.println(tEnv.explainSql("select a, c from fs where c > 0"));
> projected list:
> [[0],[2],[1]]
> == Optimized Execution Plan ==
> Calc(select=[a, c], where=[(CAST(c AS BIGINT) > 0)])
> +- ChangelogNormalize(key=[a, b])
>+- Exchange(distribution=[hash[a, b]])
> +- DropUpdateBefore
> +- TableSourceScan(table=[[default_catalog, default_database, fs,
> filter=[], project=[a, c, b], metadata=[]]], fields=[a, c, b])
> {code}
> Field b is not involved in
> {code:sql}
> select a, c from fs where c > 0{code}
> , but it is a primary key, so we add it to the end of projected list, If
> 'table.exec.source.cdc-events-duplicate' is enabled. The condition about
> field b will change output type, that says the duplicate node will get the
> different input type, and the state serializer will also be changed, leading
> to state incompatibility.
> I think we can use the original order from the source table to add the
> primary key to projected list.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (FLINK-27953) using the original order to add the primary key in PushProjectIntoTableSourceScanRule
[
https://issues.apache.org/jira/browse/FLINK-27953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17554894#comment-17554894
]
zoucao commented on FLINK-27953:
Hi [~godfrey], could you have time to take a look?
> using the original order to add the primary key in
> PushProjectIntoTableSourceScanRule
> -
>
> Key: FLINK-27953
> URL: https://issues.apache.org/jira/browse/FLINK-27953
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Affects Versions: 1.14.4
>Reporter: zoucao
>Priority: Major
>
> In PushProjectIntoTableSourceScanRule, if the source produces a changelog
> stream, the primary key will be added to the end of projected fields, see the
> following SQL:
> {code:java}
> StreamTableTestUtil util = streamTestUtil(TableConfig.getDefault());
> TableEnvironment tEnv = util.getTableEnv();
> String srcTableDdl =
> "CREATE TABLE fs (\n"
> + " a bigint,\n"
> + " b int,\n"
> + " c varchar,\n"
> + " d int,\n"
> + " e int,\n "
> + " primary key (a,b) not enforced \n"
> + ") with (\n"
> + " 'connector' = 'values',\n"
> + " 'disable-lookup'='true',\n"
> + " 'changelog-mode' = 'I,UB,UA,D')";
> tEnv.executeSql(srcTableDdl);
> tEnv.getConfig().set("table.exec.source.cdc-events-duplicate",
> "true");
> {code}
> {code:java}
> System.out.println(tEnv.explainSql("select a, c from fs where c > 0 and b =
> 0"));
> projected list:
> [[0],[1],[2]]
> == Optimized Execution Plan ==
> Calc(select=[a, c], where=[(CAST(c AS BIGINT) > 0)])
> +- ChangelogNormalize(key=[a, b])
>+- Exchange(distribution=[hash[a, b]])
> +- Calc(select=[a, b, c], where=[(b = 0)])
> +- DropUpdateBefore
> +- TableSourceScan(table=[[default_catalog, default_database, fs,
> filter=[], project=[a, b, c], metadata=[]]], fields=[a, b, c])
> {code}
> {code:java}
> System.out.println(tEnv.explainSql("select a, c from fs where c > 0"));
> projected list:
> [[0],[2],[1]]
> == Optimized Execution Plan ==
> Calc(select=[a, c], where=[(CAST(c AS BIGINT) > 0)])
> +- ChangelogNormalize(key=[a, b])
>+- Exchange(distribution=[hash[a, b]])
> +- DropUpdateBefore
> +- TableSourceScan(table=[[default_catalog, default_database, fs,
> filter=[], project=[a, c, b], metadata=[]]], fields=[a, c, b])
> {code}
> Field b is not involved in
> {code:sql}
> select a, c from fs where c > 0{code}
> , but it is a primary key, so we add it to the end of projected list, If
> 'table.exec.source.cdc-events-duplicate' is enabled. The condition about
> field b will change output type, that says the duplicate node will get the
> different input type, and the state serializer will also be changed, leading
> to state incompatibility.
> I think we can use the original order from the source table to add the
> primary key to projected list.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (FLINK-28087) Add validation for meta.name of FlinkDeployment CR
[
https://issues.apache.org/jira/browse/FLINK-28087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fred k updated FLINK-28087:
---
Summary: Add validation for meta.name of FlinkDeployment CR (was:
FlinkDeployment meta.name validation)
> Add validation for meta.name of FlinkDeployment CR
> --
>
> Key: FLINK-28087
> URL: https://issues.apache.org/jira/browse/FLINK-28087
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Fred k
>Priority: Major
> Fix For: kubernetes-operator-1.1.0
>
> Attachments: operator-flink.jpg
>
>
> The FlinkDeployment CR meta.name must be match the regex:
> {code:java}
> '[a-z]([-a-z0-9]*[a-z0-9])?'{code}
> By RRC-1035, a DNS-1035 label must consist of lower case alphanumeric
> characters or '-', start with an alphabetic character, and end with an
> alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for
> validation is '[a-z]([-a-z0-9]*[a-z0-9])?').
> The rest service name of Flink Cluster uses the meta.name of FlinkDeployment,
> So meta.name value must follow the convention of DNS-1035 label.
> To avoid operator repeatedly creating and destroying flink clusters due to
> invalid service name, So I suggest add validation for meta.name of
> FlinkDeployment CR.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[GitHub] [flink] lincoln-lil commented on pull request #19942: [FLINK-28019][table] fix error when retract a staled record if state ttl enabled in RetractableTopNFunction
lincoln-lil commented on PR #19942: URL: https://github.com/apache/flink/pull/19942#issuecomment-1157199426 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28087) FlinkDeployment meta.name validation
[
https://issues.apache.org/jira/browse/FLINK-28087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fred k updated FLINK-28087:
---
Summary: FlinkDeployment meta.name validation (was: FlinkDeployment name
validation)
> FlinkDeployment meta.name validation
>
>
> Key: FLINK-28087
> URL: https://issues.apache.org/jira/browse/FLINK-28087
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Fred k
>Priority: Major
> Fix For: kubernetes-operator-1.1.0
>
> Attachments: operator-flink.jpg
>
>
> The FlinkDeployment CR meta.name must be match the regex:
> {code:java}
> '[a-z]([-a-z0-9]*[a-z0-9])?'{code}
> By RRC-1035, a DNS-1035 label must consist of lower case alphanumeric
> characters or '-', start with an alphabetic character, and end with an
> alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for
> validation is '[a-z]([-a-z0-9]*[a-z0-9])?').
> The rest service name of Flink Cluster uses the meta.name of FlinkDeployment,
> So meta.name value must follow the convention of DNS-1035 label.
> To avoid operator repeatedly creating and destroying flink clusters due to
> invalid service name, So I suggest add validation for meta.name of
> FlinkDeployment CR.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Assigned] (FLINK-27987) Let FileSystemTableSource extend from SupportsStatisticReport
[ https://issues.apache.org/jira/browse/FLINK-27987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] godfrey he reassigned FLINK-27987: -- Assignee: godfrey he (was: Yunhong Zheng) > Let FileSystemTableSource extend from SupportsStatisticReport > - > > Key: FLINK-27987 > URL: https://issues.apache.org/jira/browse/FLINK-27987 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: godfrey he >Assignee: godfrey he >Priority: Major > Fix For: 1.16.0 > > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (FLINK-28087) FlinkDeployment name validation
[
https://issues.apache.org/jira/browse/FLINK-28087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fred k updated FLINK-28087:
---
Description:
The FlinkDeployment CR meta.name must be match the regex:
{code:java}
'[a-z]([-a-z0-9]*[a-z0-9])?'{code}
By RRC-1035, a DNS-1035 label must consist of lower case alphanumeric
characters or '-', start with an alphabetic character, and end with an
alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for
validation is '[a-z]([-a-z0-9]*[a-z0-9])?').
The rest service name of Flink Cluster uses the meta.name of FlinkDeployment,
So meta.name value must follow the convention of DNS-1035 label.
To avoid operator repeatedly creating and destroying flink clusters due to
invalid service name, So I suggest add validation for meta.name of
FlinkDeployment CR.
was:
The FlinkDeployment CR meta.name must be match the regex:
{code:java}
'[a-z]([-a-z0-9]*[a-z0-9])?'{code}
By RRC-1035, a DNS-1035 label must consist of lower case alphanumeric
characters or '-', start with an alphabetic character, and end with an
alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for
validation is '[a-z]([-a-z0-9]*[a-z0-9])?').
The rest service name of Flink Cluster uses the meta.name of FlinkDeployment,
So meta.name value must follow the convention of DNS-1035 label.
> FlinkDeployment name validation
> ---
>
> Key: FLINK-28087
> URL: https://issues.apache.org/jira/browse/FLINK-28087
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Fred k
>Priority: Major
> Fix For: kubernetes-operator-1.1.0
>
> Attachments: operator-flink.jpg
>
>
> The FlinkDeployment CR meta.name must be match the regex:
> {code:java}
> '[a-z]([-a-z0-9]*[a-z0-9])?'{code}
> By RRC-1035, a DNS-1035 label must consist of lower case alphanumeric
> characters or '-', start with an alphabetic character, and end with an
> alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for
> validation is '[a-z]([-a-z0-9]*[a-z0-9])?').
> The rest service name of Flink Cluster uses the meta.name of FlinkDeployment,
> So meta.name value must follow the convention of DNS-1035 label.
> To avoid operator repeatedly creating and destroying flink clusters due to
> invalid service name, So I suggest add validation for meta.name of
> FlinkDeployment CR.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (FLINK-28074) show statistics details for DESCRIBE EXTENDED
[ https://issues.apache.org/jira/browse/FLINK-28074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] godfrey he updated FLINK-28074: --- Description: Currently, DESCRIBE command only show the schema of a given table, EXTENDED does not work. so for EXTENDED mode, the statistics details can also be shown. (was: Currently, ) > show statistics details for DESCRIBE EXTENDED > - > > Key: FLINK-28074 > URL: https://issues.apache.org/jira/browse/FLINK-28074 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Planner >Reporter: godfrey he >Assignee: Yunhong Zheng >Priority: Major > Fix For: 1.16.0 > > > Currently, DESCRIBE command only show the schema of a given table, EXTENDED > does not work. so for EXTENDED mode, the statistics details can also be shown. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (FLINK-28087) FlinkDeployment name validation
[
https://issues.apache.org/jira/browse/FLINK-28087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fred k updated FLINK-28087:
---
Attachment: operator-flink.jpg
> FlinkDeployment name validation
> ---
>
> Key: FLINK-28087
> URL: https://issues.apache.org/jira/browse/FLINK-28087
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Fred k
>Priority: Major
> Fix For: kubernetes-operator-1.1.0
>
> Attachments: operator-flink.jpg
>
>
> The FlinkDeployment CR meta.name must be match the regex:
> {code:java}
> '[a-z]([-a-z0-9]*[a-z0-9])?'{code}
> By RRC-1035, a DNS-1035 label must consist of lower case alphanumeric
> characters or '-', start with an alphabetic character, and end with an
> alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for
> validation is '[a-z]([-a-z0-9]*[a-z0-9])?').
> The rest service name of Flink Cluster uses the meta.name of FlinkDeployment,
> So meta.name value must follow the convention of DNS-1035 label.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (FLINK-28087) FlinkDeployment name validation
Fred k created FLINK-28087:
--
Summary: FlinkDeployment name validation
Key: FLINK-28087
URL: https://issues.apache.org/jira/browse/FLINK-28087
Project: Flink
Issue Type: Improvement
Components: Kubernetes Operator
Reporter: Fred k
Fix For: kubernetes-operator-1.1.0
The FlinkDeployment CR meta.name must be match the regex:
{code:java}
'[a-z]([-a-z0-9]*[a-z0-9])?'{code}
By RRC-1035, a DNS-1035 label must consist of lower case alphanumeric
characters or '-', start with an alphabetic character, and end with an
alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for
validation is '[a-z]([-a-z0-9]*[a-z0-9])?').
The rest service name of Flink Cluster uses the meta.name of FlinkDeployment,
So meta.name value must follow the convention of DNS-1035 label.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[GitHub] [flink] flinkbot commented on pull request #19984: [hotfix][tests] test the serialized object in GlobFilePathFilterTest#testGlobFilterSerializable
flinkbot commented on PR #19984: URL: https://github.com/apache/flink/pull/19984#issuecomment-1157186972 ## CI report: * 6726e9d27b74ebe57e9e986ebc4a8fe8a3afc628 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhoulii opened a new pull request, #19984: [hotfix][tests] test the serialized object in GlobFilePathFilterTest#testGlobFilterSerializable
zhoulii opened a new pull request, #19984: URL: https://github.com/apache/flink/pull/19984 ## What is the purpose of the change - test the serialized object in GlobFilePathFilterTest#testGlobFilterSerializable - Rework GlobFilePathFilterTest to be based on AssertJ ## Verifying this change - covered by existed ut ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] gaoyunhaii commented on pull request #545: Add blogs for FLIP-147 support checkpoints after tasks finished
gaoyunhaii commented on PR #545: URL: https://github.com/apache/flink-web/pull/545#issuecomment-1157179944 Thanks @zhuzhurk @wanglijie95 for the review! I have updated the blogs according to the comments and also rewrite some part. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25655) FlinkKafkaInternalProducerITCase.testInitTransactionId failed on azure
[
https://issues.apache.org/jira/browse/FLINK-25655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17554883#comment-17554883
]
lincoln lee commented on FLINK-25655:
-
Any new progress here? I encountered two in a row, recent one:
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=36657=ms.vss-test-web.build-test-results-tab=2822502=12=debug]
{code}
org.opentest4j.AssertionFailedError: expected: 5 but was: 4
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducerITCase.testInitTransactionId(FlinkKafkaInternalProducerITCase.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
at
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
at
org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
at
org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
at
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at
org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
at ...
{code}
> FlinkKafkaInternalProducerITCase.testInitTransactionId failed on azure
> --
>
> Key: FLINK-25655
> URL: https://issues.apache.org/jira/browse/FLINK-25655
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.14.2
>Reporter: Yun Gao
>Priority: Major
>
> {code:java}
> 2022-01-14T02:15:13.6453364Z Jan 14 02:15:13
> 2022-01-14T02:15:13.6453706Z Jan 14 02:15:13 Expected: <4>
> 2022-01-14T02:15:13.6454039Z Jan 14 02:15:13 but: was <0>
> 2022-01-14T02:15:13.6454489Z
[GitHub] [flink-web] gaoyunhaii commented on pull request #546: Add blogs for FLIP-187 adaptive batch scheduler
gaoyunhaii commented on PR #546: URL: https://github.com/apache/flink-web/pull/546#issuecomment-1157174790 Thanks @wanglijie95 for the update! LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-28086) Table Store Catalog supports partition methods
Jingsong Lee created FLINK-28086: Summary: Table Store Catalog supports partition methods Key: FLINK-28086 URL: https://issues.apache.org/jira/browse/FLINK-28086 Project: Flink Issue Type: New Feature Components: Table Store Reporter: Jingsong Lee Fix For: table-store-0.2.0 Table Store Catalog can support: * listPartitions * listPartitionsByFilter * getPartition * partitionExists * dropPartition -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [flink] imaffe commented on a diff in pull request #19973: [FLINK-28082][Connector/Pulsar] Add end-to-end encryption support for Pulsar connector.
imaffe commented on code in PR #19973:
URL: https://github.com/apache/flink/pull/19973#discussion_r898643192
##
flink-connectors/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/source/PulsarSourceBuilder.java:
##
@@ -369,6 +373,18 @@ public PulsarSourceBuilder
setDeserializationSchema(
return self;
}
+/**
+ * Sets a {@link CryptoKeyReader}. Configure the key reader to be used to
decrypt the message
+ * payloads.
+ *
+ * @param cryptoKeyReader CryptoKeyReader object
+ * @return this PulsarSourceBuilder.
+ */
+public PulsarSourceBuilder setCryptoKeyReader(CryptoKeyReader
cryptoKeyReader) {
+this.cryptoKeyReader = checkNotNull(cryptoKeyReader);
Review Comment:
Do we need to add e2e encryption in the documentation as well ? The
cryptoKeyReader contains private key, I think we need to let users know what is
the recommended/safe way to create a cryptoKeyReader~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #19983: [FLINK-27878][datastream] Add Retry Support For Async I/O In DataStream API
flinkbot commented on PR #19983: URL: https://github.com/apache/flink/pull/19983#issuecomment-1157166737 ## CI report: * 7cac33be1271b73b8f28069ee17db53a790bef29 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] imaffe commented on pull request #19974: [FLINK-28083][Connector/Pulsar] Object-reusing for Pulsar source
imaffe commented on PR #19974: URL: https://github.com/apache/flink/pull/19974#issuecomment-1157166125 I'm not very familiar with the object-reuse deserialization schema here. Could you describe briefly why we introduce this change ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] godfreyhe commented on pull request #19939: [FLINK-27983][FLINK-27984][FLINK-27985] Introduce SupportsStatisticsReport, FileBasedStatisticsReportableDecodingFormat, FlinkRecomputeStat
godfreyhe commented on PR #19939: URL: https://github.com/apache/flink/pull/19939#issuecomment-1157166185 Thanks for the review, I will merge it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-27878) Add Retry Support For Async I/O In DataStream API
[ https://issues.apache.org/jira/browse/FLINK-27878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-27878: --- Labels: pull-request-available (was: ) > Add Retry Support For Async I/O In DataStream API > - > > Key: FLINK-27878 > URL: https://issues.apache.org/jira/browse/FLINK-27878 > Project: Flink > Issue Type: New Feature > Components: API / DataStream >Reporter: lincoln lee >Assignee: lincoln lee >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0 > > > FLIP-232: Add Retry Support For Async I/O In DataStream API > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211883963 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [flink] lincoln-lil opened a new pull request, #19983: [FLINK-27878][datastream] Add Retry Support For Async I/O In DataStream API
lincoln-lil opened a new pull request, #19983: URL: https://github.com/apache/flink/pull/19983 ## What is the purpose of the change This is the implementation of [FLIP-232](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211883963) ## Brief change log * add new interface and api proposed in the flip to support retry * add a RetryableResultHandlerDelegator to AsyncWaitOperator to support retry, if retry disabled, all the behaviors keep the same as previous version which does not support retry * add tests to ensure retry logic is covered under both ordered and unordered mode ## Verifying this change AsyncWaitOperatorTest & AsyncDataStreamITCase ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with @Public(Evolving): (no) - The serializers: (no ) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (docs) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] imaffe commented on pull request #19977: [FLINK-28084][Connector/Pulsar] Remove the reconsume later logic for Pulsar unordered reader.
imaffe commented on PR #19977: URL: https://github.com/apache/flink/pull/19977#issuecomment-1157163612 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] imaffe commented on pull request #19978: [FLINK-27881][Connector/Pulsar] Never return null for PulsarMessageBuilder
imaffe commented on PR #19978: URL: https://github.com/apache/flink/pull/19978#issuecomment-1157163110 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #19982: [BP-1.14][FLINK-27762][connector/kafka] Catch WakeupException and retry KafkaConsumer invocations in split assignment
flinkbot commented on PR #19982: URL: https://github.com/apache/flink/pull/19982#issuecomment-1157162626 ## CI report: * 698c594f054ff7ef72dc5982dc71e7fcd2d6d359 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Aitozi commented on pull request #19840: [FLINK-24713][Runtime/Coordination] Support the initial delay for SlotManager to wait fo…
Aitozi commented on PR #19840: URL: https://github.com/apache/flink/pull/19840#issuecomment-1157161267 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Aitozi commented on pull request #19840: [FLINK-24713][Runtime/Coordination] Support the initial delay for SlotManager to wait fo…
Aitozi commented on PR #19840: URL: https://github.com/apache/flink/pull/19840#issuecomment-1157161191 The failed test with unfinished test case `org.apache.flink.test.checkpointing.StreamFaultToleranceTestBase.runCheckpointedProgram` can finish locally -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #19981: [BP-1.15][FLINK-27762][connector/kafka] Catch WakeupException and retry KafkaConsumer invocations in split assignment
flinkbot commented on PR #19981: URL: https://github.com/apache/flink/pull/19981#issuecomment-1157160602 ## CI report: * 7544a90ee0babb2b5b087d4cab91b4f6442b968b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #19980: [BP-1.15][FLINK-27041][connector/kafka] Catch IllegalStateException in KafkaPartitionSplitReader.fetch() to handle no valid partition case
flinkbot commented on PR #19980: URL: https://github.com/apache/flink/pull/19980#issuecomment-1157160564 ## CI report: * 6e88bc91064cdec9d581add9928c3d9f05e835af UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] PatrickRen opened a new pull request, #19982: [FLINK-27762][connector/kafka] Catch WakeupException and retry KafkaConsumer invocations in split assignment
PatrickRen opened a new pull request, #19982: URL: https://github.com/apache/flink/pull/19982 Unchanged back port of #19828 on release-1.14 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] imaffe commented on a diff in pull request #19976: [FLINK-27400][Connector/Pulsar] Never subscribe the Pulsar internal topics by using regex
imaffe commented on code in PR #19976:
URL: https://github.com/apache/flink/pull/19976#discussion_r898633161
##
flink-connectors/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/source/enumerator/topic/TopicNameUtils.java:
##
@@ -30,13 +31,35 @@
import java.util.List;
import java.util.Map;
import java.util.Set;
+import java.util.regex.Pattern;
import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.pulsar.common.naming.NamespaceName.SYSTEM_NAMESPACE;
+import static org.apache.pulsar.common.naming.TopicDomain.persistent;
/** util for topic name. */
@Internal
public final class TopicNameUtils {
+private static final Pattern HEARTBEAT_NAMESPACE_PATTERN =
Review Comment:
This change is cool, we can use this filter system topics in PulsarCatalog
as well ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] PatrickRen opened a new pull request, #19981: [BP-1.15][FLINK-27762][connector/kafka] Catch WakeupException and retry KafkaConsumer invocations in split assignment
PatrickRen opened a new pull request, #19981: URL: https://github.com/apache/flink/pull/19981 Unchanged back port of #19828 on release-1.15 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] PatrickRen opened a new pull request, #19980: [BP-1.15][FLINK-27041][connector/kafka] Catch IllegalStateException in KafkaPartitionSplitReader.fetch() to handle no valid partition cas
PatrickRen opened a new pull request, #19980: URL: https://github.com/apache/flink/pull/19980 Unchanged back port of #19456 on release-1.15 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #19979: [BP-1.14][FLINK-27041][connector/kafka] Catch IllegalStateException in KafkaPartitionSplitReader.fetch() to handle no valid partition case
flinkbot commented on PR #19979: URL: https://github.com/apache/flink/pull/19979#issuecomment-1157156391 ## CI report: * 79f25b1182a1f059b30603b78e95f9ce47c4f61b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] imaffe commented on a diff in pull request #19099: [FLINK-26182][Connector/pulsar] Create a e2e tests for Pulsar sink.
imaffe commented on code in PR #19099:
URL: https://github.com/apache/flink/pull/19099#discussion_r898626758
##
flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/testutils/PulsarTestContext.java:
##
@@ -19,44 +19,61 @@
package org.apache.flink.connector.pulsar.testutils;
import
org.apache.flink.connector.pulsar.testutils.runtime.PulsarRuntimeOperator;
-import
org.apache.flink.connector.testframe.external.source.DataStreamSourceExternalContext;
+import org.apache.flink.connector.testframe.external.ExternalContext;
+
+import org.apache.pulsar.client.api.Schema;
import java.net.URL;
-import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
import java.util.List;
+import java.util.Set;
+
+import static
org.apache.flink.connector.pulsar.source.enumerator.topic.TopicNameUtils.topicName;
+
+/**
+ * The implementation for Flink connector test tools. Providing the common
test case writing
+ * constraint for both source, sink and table API.
+ */
+public abstract class PulsarTestContext implements ExternalContext {
Review Comment:
How will change to `ExternalContext` affect the tests for Table API ?
##
flink-connectors/flink-connector-pulsar/archunit-violations/f4d91193-72ba-4ce4-ad83-98f780dce581:
##
@@ -1,3 +1,9 @@
+org.apache.flink.connector.pulsar.sink.PulsarSinkITCase does not satisfy: only
one of the following predicates match:\
+* reside in a package 'org.apache.flink.runtime.*' and contain any fields that
are static, final, and of type InternalMiniClusterExtension and annotated with
@RegisterExtension\
Review Comment:
Is it possible we can fix this arch violation ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] PatrickRen opened a new pull request, #19979: [BP-1.14][FLINK-27041][connector/kafka] Catch IllegalStateException in KafkaPartitionSplitReader.fetch() to handle no valid partition cas
PatrickRen opened a new pull request, #19979: URL: https://github.com/apache/flink/pull/19979 Unchanged back port of #19456 on release-1.14 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-28077) KeyedStateCheckpointingITCase.testWithMemoryBackendSync runs into timeout
[
https://issues.apache.org/jira/browse/FLINK-28077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17554862#comment-17554862
]
fanrui commented on FLINK-28077:
Hi [~mapohl] [~chesnay] , thanks for this information, I will take a look this
week.:)
> KeyedStateCheckpointingITCase.testWithMemoryBackendSync runs into timeout
> -
>
> Key: FLINK-28077
> URL: https://issues.apache.org/jira/browse/FLINK-28077
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Tests
>Affects Versions: 1.16.0
>Reporter: Matthias Pohl
>Priority: Major
> Labels: test-stability
>
> [Build
> #36209|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=36209=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7=9370]
> got stuck in {{KeyedStateCheckpointingITCase.testWithMemoryBackendSync}}:
> {code}
> "main" #1 prio=5 os_prio=0 tid=0x7f849c00b800 nid=0x19c3 waiting on
> condition [0x7f84a45b7000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x80074870> (a
> java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1989)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1951)
> at
> org.apache.flink.test.checkpointing.KeyedStateCheckpointingITCase.testProgramWithBackend(KeyedStateCheckpointingITCase.java:175)
> at
> org.apache.flink.test.checkpointing.KeyedStateCheckpointingITCase.testWithMemoryBackendSync(KeyedStateCheckpointingITCase.java:104)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[GitHub] [flink-table-store] tsreaper commented on a diff in pull request #158: [FLINK-28079] Check Hive DDL against table store schema when creating table
tsreaper commented on code in PR #158:
URL: https://github.com/apache/flink-table-store/pull/158#discussion_r898626398
##
flink-table-store-hive/src/main/java/org/apache/flink/table/store/hive/HiveSchema.java:
##
@@ -73,11 +90,134 @@ public static HiveSchema extract(Properties properties) {
String columnTypes =
properties.getProperty(serdeConstants.LIST_COLUMN_TYPES);
List typeInfos =
TypeInfoUtils.getTypeInfosFromTypeString(columnTypes);
+List partitionKeys = new ArrayList<>();
+if
(properties.containsKey(hive_metastoreConstants.META_TABLE_PARTITION_COLUMNS)) {
Review Comment:
Why not? We can support partitions now that we have the real schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-table-store] tsreaper commented on a diff in pull request #158: [FLINK-28079] Check Hive DDL against table store schema when creating table
tsreaper commented on code in PR #158:
URL: https://github.com/apache/flink-table-store/pull/158#discussion_r898626041
##
flink-table-store-hive/src/main/java/org/apache/flink/table/store/hive/HiveSchema.java:
##
@@ -18,50 +18,67 @@
package org.apache.flink.table.store.hive;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.table.store.file.FileStoreOptions;
+import org.apache.flink.table.store.file.schema.Schema;
+import org.apache.flink.table.store.file.schema.SchemaManager;
+import org.apache.flink.table.types.logical.LogicalType;
import org.apache.flink.util.Preconditions;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
import org.apache.hadoop.hive.serde.serdeConstants;
import org.apache.hadoop.hive.serde2.SerDeUtils;
import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils;
+import java.util.ArrayList;
import java.util.Arrays;
+import java.util.HashMap;
import java.util.List;
+import java.util.Map;
+import java.util.Objects;
import java.util.Properties;
+import java.util.stream.Collectors;
/** Column names, types and comments of a Hive table. */
public class HiveSchema {
-private final List fieldNames;
-private final List fieldTypeInfos;
+private static final String TBLPROPERTIES_PREFIX = "table-store.";
+private static final String TBLPROPERTIES_PRIMARY_KEYS =
TBLPROPERTIES_PREFIX + "primary-keys";
Review Comment:
We need to check that the primary keys in Hive DDL match the primary keys in
schema. As Hive does not support primary keys we have to implement this in
table properties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] deadwind4 commented on a diff in pull request #19777: [hotfix][connector/pulsar][docs] Update atEventTime to atPublishTime in JavaDoc
deadwind4 commented on code in PR #19777:
URL: https://github.com/apache/flink/pull/19777#discussion_r898621744
##
flink-connectors/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/source/PulsarSourceBuilder.java:
##
@@ -107,7 +107,7 @@
* .setSubscriptionName("flink-source-1")
* .setTopics(Arrays.asList(TOPIC1, TOPIC2))
* .setDeserializationSchema(PulsarDeserializationSchema.flinkSchema(new
SimpleStringSchema()))
- *
.setUnboundedStopCursor(StopCursor.atEventTime(System.currentTimeMillis()))
+ *
.setUnboundedStopCursor(StopCursor.atPublishTime(System.currentTimeMillis()))
Review Comment:
@syhily What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] deadwind4 commented on a diff in pull request #19777: [hotfix][connector/pulsar][docs] Update atEventTime to atPublishTime in JavaDoc
deadwind4 commented on code in PR #19777:
URL: https://github.com/apache/flink/pull/19777#discussion_r898621580
##
flink-connectors/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/source/PulsarSourceBuilder.java:
##
@@ -107,7 +107,7 @@
* .setSubscriptionName("flink-source-1")
* .setTopics(Arrays.asList(TOPIC1, TOPIC2))
* .setDeserializationSchema(PulsarDeserializationSchema.flinkSchema(new
SimpleStringSchema()))
- *
.setUnboundedStopCursor(StopCursor.atEventTime(System.currentTimeMillis()))
+ *
.setUnboundedStopCursor(StopCursor.atPublishTime(System.currentTimeMillis()))
Review Comment:
@imaffe What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] liujiawinds commented on pull request #19965: [FLINK-22920][formats] Fix guava version conflict in `flink-orc` and `flink-parquet` module
liujiawinds commented on PR #19965: URL: https://github.com/apache/flink/pull/19965#issuecomment-1157128850 @JingsongLi Could you review this pr? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on pull request #109: [hotfix][flink-ml-lib][syntax]fix the forgotten static import method compareResultCollections from TestBaseUtils
yunfengzhou-hub commented on PR #109: URL: https://github.com/apache/flink-ml/pull/109#issuecomment-1157125976 Got it. From the information provided in [FLINK-27454](https://issues.apache.org/jira/browse/FLINK-27454) it seems that AbstractTestBase would still extend TestBaseUtils until flink 1.16.0. I think we could temporarily close this PR for now and re-visit it when Flink ML depends on newer versions of flink. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #19978: [FLINK-27881][Connector/Pulsar] Never return null for PulsarMessageBuilder
flinkbot commented on PR #19978: URL: https://github.com/apache/flink/pull/19978#issuecomment-1157016182 ## CI report: * a0384dee5b76b645ec0947dd208ce027cbce9717 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-27736) Pulsar sink catch watermark error
[
https://issues.apache.org/jira/browse/FLINK-27736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17554812#comment-17554812
]
Yufan Sheng edited comment on FLINK-27736 at 6/15/22 10:33 PM:
---
Tks for your report. This is a known bug and gets fixed in FLINK-28083. The
source shouldn't set the timestamp to zero.
was (Author: syhily):
Tks for your report This is a known bug and fixed in FLINK-28083.
> Pulsar sink catch watermark error
> -
>
> Key: FLINK-27736
> URL: https://issues.apache.org/jira/browse/FLINK-27736
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Connectors / Pulsar
>Affects Versions: 1.15.0
>Reporter: LuNing Wang
>Priority: Critical
>
> The following is my demo code.
> {code:java}
> public class WatermarkDemo {
> private final static String SERVICE_URL = "pulsar://localhost:6650";
> private final static String ADMIN_URL = "http://localhost:8080;;
> public static void main(String[] args) throws Exception {
> final StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> PulsarSource source = PulsarSource.builder()
> .setServiceUrl(SERVICE_URL)
> .setAdminUrl(ADMIN_URL)
> .setStartCursor(StartCursor.earliest())
> .setTopics("ada")
>
> .setDeserializationSchema(PulsarDeserializationSchema.flinkSchema(new
> SimpleStringSchema()))
> .setSubscriptionName("my-subscription")
> .setSubscriptionType(SubscriptionType.Exclusive)
> .build();
> PulsarSink sink = PulsarSink.builder()
> .setServiceUrl(SERVICE_URL)
> .setAdminUrl(ADMIN_URL)
> .setTopics("beta")
>
> .setSerializationSchema(PulsarSerializationSchema.flinkSchema(new
> SimpleStringSchema()))
> .setDeliveryGuarantee(DeliveryGuarantee.AT_LEAST_ONCE)
> .build();
> DataStream stream = env.fromSource(source,
> WatermarkStrategy.forMonotonousTimestamps(), "Pulsar Source");
> stream.sinkTo(sink);
> env.execute();
> }
> } {code}
> It will throw the following error.
> {code:java}
> Caused by: java.lang.IllegalArgumentException: Invalid timestamp : '0'
> at
> org.apache.pulsar.shade.com.google.common.base.Preconditions.checkArgument(Preconditions.java:203)
> at
> org.apache.pulsar.client.impl.TypedMessageBuilderImpl.eventTime(TypedMessageBuilderImpl.java:204)
> at
> org.apache.flink.connector.pulsar.sink.writer.PulsarWriter.createMessageBuilder(PulsarWriter.java:216)
> at
> org.apache.flink.connector.pulsar.sink.writer.PulsarWriter.write(PulsarWriter.java:141)
> at
> org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator.processElement(SinkWriterOperator.java:158)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:82)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
> at
> org.apache.flink.streaming.runtime.tasks.SourceOperatorStreamTask$AsyncDataOutputToOutput.emitRecord(SourceOperatorStreamTask.java:313)
> at
> org.apache.flink.streaming.api.operators.source.SourceOutputWithWatermarks.collect(SourceOutputWithWatermarks.java:110)
> at
> org.apache.flink.connector.pulsar.source.reader.emitter.PulsarRecordEmitter.emitRecord(PulsarRecordEmitter.java:41)
> at
> org.apache.flink.connector.pulsar.source.reader.emitter.PulsarRecordEmitter.emitRecord(PulsarRecordEmitter.java:33)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:143)
> at
> org.apache.flink.connector.pulsar.source.reader.source.PulsarOrderedSourceReader.pollNext(PulsarOrderedSourceReader.java:106)
> at
> org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:385)
> at
> org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:519)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:203)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:804)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:753)
> at
>
[jira] [Commented] (FLINK-27736) Pulsar sink catch watermark error
[
https://issues.apache.org/jira/browse/FLINK-27736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17554812#comment-17554812
]
Yufan Sheng commented on FLINK-27736:
-
Tks for your report This is a known bug and fixed in FLINK-28083.
> Pulsar sink catch watermark error
> -
>
> Key: FLINK-27736
> URL: https://issues.apache.org/jira/browse/FLINK-27736
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Connectors / Pulsar
>Affects Versions: 1.15.0
>Reporter: LuNing Wang
>Priority: Critical
>
> The following is my demo code.
> {code:java}
> public class WatermarkDemo {
> private final static String SERVICE_URL = "pulsar://localhost:6650";
> private final static String ADMIN_URL = "http://localhost:8080;;
> public static void main(String[] args) throws Exception {
> final StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> PulsarSource source = PulsarSource.builder()
> .setServiceUrl(SERVICE_URL)
> .setAdminUrl(ADMIN_URL)
> .setStartCursor(StartCursor.earliest())
> .setTopics("ada")
>
> .setDeserializationSchema(PulsarDeserializationSchema.flinkSchema(new
> SimpleStringSchema()))
> .setSubscriptionName("my-subscription")
> .setSubscriptionType(SubscriptionType.Exclusive)
> .build();
> PulsarSink sink = PulsarSink.builder()
> .setServiceUrl(SERVICE_URL)
> .setAdminUrl(ADMIN_URL)
> .setTopics("beta")
>
> .setSerializationSchema(PulsarSerializationSchema.flinkSchema(new
> SimpleStringSchema()))
> .setDeliveryGuarantee(DeliveryGuarantee.AT_LEAST_ONCE)
> .build();
> DataStream stream = env.fromSource(source,
> WatermarkStrategy.forMonotonousTimestamps(), "Pulsar Source");
> stream.sinkTo(sink);
> env.execute();
> }
> } {code}
> It will throw the following error.
> {code:java}
> Caused by: java.lang.IllegalArgumentException: Invalid timestamp : '0'
> at
> org.apache.pulsar.shade.com.google.common.base.Preconditions.checkArgument(Preconditions.java:203)
> at
> org.apache.pulsar.client.impl.TypedMessageBuilderImpl.eventTime(TypedMessageBuilderImpl.java:204)
> at
> org.apache.flink.connector.pulsar.sink.writer.PulsarWriter.createMessageBuilder(PulsarWriter.java:216)
> at
> org.apache.flink.connector.pulsar.sink.writer.PulsarWriter.write(PulsarWriter.java:141)
> at
> org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator.processElement(SinkWriterOperator.java:158)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:82)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
> at
> org.apache.flink.streaming.runtime.tasks.SourceOperatorStreamTask$AsyncDataOutputToOutput.emitRecord(SourceOperatorStreamTask.java:313)
> at
> org.apache.flink.streaming.api.operators.source.SourceOutputWithWatermarks.collect(SourceOutputWithWatermarks.java:110)
> at
> org.apache.flink.connector.pulsar.source.reader.emitter.PulsarRecordEmitter.emitRecord(PulsarRecordEmitter.java:41)
> at
> org.apache.flink.connector.pulsar.source.reader.emitter.PulsarRecordEmitter.emitRecord(PulsarRecordEmitter.java:33)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:143)
> at
> org.apache.flink.connector.pulsar.source.reader.source.PulsarOrderedSourceReader.pollNext(PulsarOrderedSourceReader.java:106)
> at
> org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:385)
> at
> org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:519)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:203)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:804)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:753)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:948)
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927)
> at
[jira] [Updated] (FLINK-27881) The key(String) in PulsarMessageBuilder returns null
[ https://issues.apache.org/jira/browse/FLINK-27881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-27881: --- Labels: pull-request-available (was: ) > The key(String) in PulsarMessageBuilder returns null > > > Key: FLINK-27881 > URL: https://issues.apache.org/jira/browse/FLINK-27881 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.15.0 >Reporter: Shuiqiang Chen >Priority: Major > Labels: pull-request-available > > The PulsarMessageBuild.key(String) always return null, which might cause NPE > in later call. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [flink] syhily opened a new pull request, #19978: [FLINK-27881][Connector/Pulsar] Never return null for PulsarMessageBuilder
syhily opened a new pull request, #19978: URL: https://github.com/apache/flink/pull/19978 ## What is the purpose of the change The PulsarMessageBuild.key(String) always return null, which cause NPE. ## Brief change log `return this` instead of `return null` ## Verifying this change This change is a trivial rework without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-27881) The key(String) in PulsarMessageBuilder returns null
[ https://issues.apache.org/jira/browse/FLINK-27881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17554811#comment-17554811 ] Yufan Sheng commented on FLINK-27881: - I'll submit a PR today. This is a known bug. > The key(String) in PulsarMessageBuilder returns null > > > Key: FLINK-27881 > URL: https://issues.apache.org/jira/browse/FLINK-27881 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.15.0 >Reporter: Shuiqiang Chen >Priority: Major > > The PulsarMessageBuild.key(String) always return null, which might cause NPE > in later call. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (FLINK-28085) Close all the pending Pulsar transactions when flink shutdown the pipeline.
Yufan Sheng created FLINK-28085: --- Summary: Close all the pending Pulsar transactions when flink shutdown the pipeline. Key: FLINK-28085 URL: https://issues.apache.org/jira/browse/FLINK-28085 Project: Flink Issue Type: Bug Components: Connectors / Pulsar Affects Versions: 1.14.4, 1.15.0 Reporter: Yufan Sheng Fix For: 1.16.0 Currently transactionId is not persisted. After a job restart we lose handle to the transaction which is still not aborted in Pulsar broker. Pulsar broker will abort these hanging transactions after a timeout but this is not desirable. We need to close all the pending transactionId. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [flink] flinkbot commented on pull request #19977: [FLINK-28084][Connector/Pulsar] Remove the reconsume later logic for Pulsar unordered reader.
flinkbot commented on PR #19977: URL: https://github.com/apache/flink/pull/19977#issuecomment-1156981879 ## CI report: * 9756d018cd49ff6b7f3f6f1a46e5dd1b6cc4aaa4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28084) Pulsar unordered reader should disable retry and delete reconsume logic.
[ https://issues.apache.org/jira/browse/FLINK-28084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-28084: --- Labels: pull-request-available (was: ) > Pulsar unordered reader should disable retry and delete reconsume logic. > > > Key: FLINK-28084 > URL: https://issues.apache.org/jira/browse/FLINK-28084 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.15.0, 1.14.4 >Reporter: Yufan Sheng >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0 > > > UnroderdPulsarSourceReader currently calls reconsume, but this feature relys > on retry topic. But if retry topic is enabled the initial search will only > support earliest and lates (because it will be a multiconsumer impl). We plan > to delete the reconsume logic to get rid of dependency on retry topic and > should disable retry. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [flink] syhily opened a new pull request, #19977: [FLINK-28084][Connector/Pulsar] Remove the reconsume later logic for Pulsar unordered reader.
syhily opened a new pull request, #19977: URL: https://github.com/apache/flink/pull/19977 ## What is the purpose of the change Pulsar unordered reader is used for Shared and Key_Shared subscriptions. It would reconsume a message if it didn't acknowledge in a transaction. But this didn't works now. Because we don't use Pulsar's Multiple consumer. The best way to solve this problem is just delete this reconsumeLater logic. ## Brief change log Delete the reconsumeLater when meeting exceptions in acknowledge. ## Verifying this change This change is a trivial rework without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28084) Pulsar unordered reader should disable retry and delete reconsume logic.
[ https://issues.apache.org/jira/browse/FLINK-28084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufan Sheng updated FLINK-28084: Description: UnroderdPulsarSourceReader currently calls reconsume, but this feature relys on retry topic. But if retry topic is enabled the initial search will only support earliest and lates (because it will be a multiconsumer impl). We plan to delete the reconsume logic to get rid of dependency on retry topic and should disable retry. (was: Pulsar unordered reader is used for Shared and Key_Shared subscriptions. It would reconsume a message if it didn't acknowledge in a transaction. But this didn't works now. Because we don't use Pulsar's Multiple consumer. The best way to solve this problem is just delete this reconsumeLater logic.) > Pulsar unordered reader should disable retry and delete reconsume logic. > > > Key: FLINK-28084 > URL: https://issues.apache.org/jira/browse/FLINK-28084 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.15.0, 1.14.4 >Reporter: Yufan Sheng >Priority: Major > Fix For: 1.16.0 > > > UnroderdPulsarSourceReader currently calls reconsume, but this feature relys > on retry topic. But if retry topic is enabled the initial search will only > support earliest and lates (because it will be a multiconsumer impl). We plan > to delete the reconsume logic to get rid of dependency on retry topic and > should disable retry. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (FLINK-27400) Pulsar connector subscribed the system topic when using the regex
[ https://issues.apache.org/jira/browse/FLINK-27400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufan Sheng updated FLINK-27400: Description: UnroderdPulsarSourceReader currently calls reconsume, but this feature relys on retry topic. But if retry topic is enabled the initial search will only support earliest and lates (because it will be a multiconsumer impl). We plan to delete the reconsume logic to get rid of dependency on retry topic and should disable retry. (was: Pulsar has a lot of internal topics which is used for metadata. It couldn't be consumed directly by the user. We accidentally exposed these topics to end-users when using the regex topics.) > Pulsar connector subscribed the system topic when using the regex > - > > Key: FLINK-27400 > URL: https://issues.apache.org/jira/browse/FLINK-27400 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.15.0, 1.14.4, 1.16.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0, 1.15.2, 1.14.6 > > > UnroderdPulsarSourceReader currently calls reconsume, but this feature relys > on retry topic. But if retry topic is enabled the initial search will only > support earliest and lates (because it will be a multiconsumer impl). We plan > to delete the reconsume logic to get rid of dependency on retry topic and > should disable retry. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (FLINK-27400) Pulsar connector subscribed the system topic when using the regex
[ https://issues.apache.org/jira/browse/FLINK-27400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufan Sheng updated FLINK-27400: Description: Pulsar has a lot of internal topics which is used for metadata. It couldn't be consumed directly by the user. We accidentally exposed these topics to end-users when using the regex topics. (was: UnroderdPulsarSourceReader currently calls reconsume, but this feature relys on retry topic. But if retry topic is enabled the initial search will only support earliest and lates (because it will be a multiconsumer impl). We plan to delete the reconsume logic to get rid of dependency on retry topic and should disable retry.) > Pulsar connector subscribed the system topic when using the regex > - > > Key: FLINK-27400 > URL: https://issues.apache.org/jira/browse/FLINK-27400 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.15.0, 1.14.4, 1.16.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0, 1.15.2, 1.14.6 > > > Pulsar has a lot of internal topics which is used for metadata. It couldn't > be consumed directly by the user. We accidentally exposed these topics to > end-users when using the regex topics. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (FLINK-28084) Pulsar unordered reader should disable retry and delete reconsume logic.
Yufan Sheng created FLINK-28084: --- Summary: Pulsar unordered reader should disable retry and delete reconsume logic. Key: FLINK-28084 URL: https://issues.apache.org/jira/browse/FLINK-28084 Project: Flink Issue Type: Bug Components: Connectors / Pulsar Affects Versions: 1.14.4, 1.15.0 Reporter: Yufan Sheng Fix For: 1.16.0 Pulsar unordered reader is used for Shared and Key_Shared subscriptions. It would reconsume a message if it didn't acknowledge in a transaction. But this didn't works now. Because we don't use Pulsar's Multiple consumer. The best way to solve this problem is just delete this reconsumeLater logic. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [flink] flinkbot commented on pull request #19976: [FLINK-27400][Connector/Pulsar] Never subscribe the Pulsar internal topics by using regex
flinkbot commented on PR #19976: URL: https://github.com/apache/flink/pull/19976#issuecomment-1156917077 ## CI report: * 67a21b87b36684c7dd551e29e77663be2487075e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-27400) Pulsar connector subscribed the system topic when using the regex
[ https://issues.apache.org/jira/browse/FLINK-27400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-27400: --- Labels: pull-request-available (was: ) > Pulsar connector subscribed the system topic when using the regex > - > > Key: FLINK-27400 > URL: https://issues.apache.org/jira/browse/FLINK-27400 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.15.0, 1.14.4, 1.16.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0, 1.15.2, 1.14.6 > > > Pulsar has a lot of internal topics which is used for metadata. It couldn't > be consumed directly by the user. We accidentally exposed these topics to > end-users when using the regex topics. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [flink] syhily opened a new pull request, #19976: [FLINK-27400][Connector/Pulsar] Never subscribe the Pulsar internal topics by using regex
syhily opened a new pull request, #19976: URL: https://github.com/apache/flink/pull/19976 ## What is the purpose of the change The current Pulsar connector would subscribe the Pulsar internal system topics, which is not allowed. ## Brief change log Add a method for filtering the system topics in Pulsar. ## Verifying this change This change is a trivial rework without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #19975: [FLINK-27611][Connector/Pulsar] Fix the checkpoint issue on shared reader.
flinkbot commented on PR #19975: URL: https://github.com/apache/flink/pull/19975#issuecomment-1156883644 ## CI report: * d11c0d8f91702f7c7d83819edd30916aed2940ec UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-27611) ConcurrentModificationException during Flink-Pulsar checkpoint notification
[ https://issues.apache.org/jira/browse/FLINK-27611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-27611: --- Labels: pull-request-available (was: ) > ConcurrentModificationException during Flink-Pulsar checkpoint notification > --- > > Key: FLINK-27611 > URL: https://issues.apache.org/jira/browse/FLINK-27611 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.15.0 >Reporter: Jason Kania >Priority: Major > Labels: pull-request-available > > When attempting to run a job that was working in 1.12.7, but upgraded to > 1.15.0, the following exception is occurring outside of the control of my own > code: > > java.util.ConcurrentModificationException > at > java.base/java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1208) > at java.base/java.util.TreeMap$EntryIterator.next(TreeMap.java:1244) > at java.base/java.util.TreeMap$EntryIterator.next(TreeMap.java:1239) > at > org.apache.flink.connector.pulsar.source.reader.source.PulsarUnorderedSourceReader.notifyCheckpointComplete(PulsarUnorderedSourceReader.java:129) > at > org.apache.flink.streaming.api.operators.SourceOperator.notifyCheckpointComplete(SourceOperator.java:511) > at > org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.notifyCheckpointComplete(StreamOperatorWrapper.java:104) > at > org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.notifyCheckpointComplete(RegularOperatorChain.java:145) > at > org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.notifyCheckpoint(SubtaskCheckpointCoordinatorImpl.java:409) > at > org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.notifyCheckpointComplete(SubtaskCheckpointCoordinatorImpl.java:343) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.notifyCheckpointComplete(StreamTask.java:1384) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointCompleteAsync$14(StreamTask.java:1325) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointOperation$17(StreamTask.java:1364) > at > org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) > at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) > at > org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:338) > at > org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:324) > at > org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:201) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:804) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:753) > at > org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:948) > at > org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927) > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:741) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:563) > at java.base/java.lang.Thread.run(Thread.java:829) -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [flink] syhily opened a new pull request, #19975: [FLINK-27611][Connector/Pulsar] Fix the checkpoint issue on shared reader.
syhily opened a new pull request, #19975: URL: https://github.com/apache/flink/pull/19975 ## What is the purpose of the change Pulsar entered a ConcurrentModificationException because I don't property used the Iterator. This PR fixes this issue. ## Brief change log Filter the keys which need to delete from map first, them delete it one by one which fixes the ConcurrentModificationException ## Verifying this change This change is a trivial rework without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #19974: [FLINK-28083][Connector/Pulsar] Object-reusing for Pulsar source
flinkbot commented on PR #19974: URL: https://github.com/apache/flink/pull/19974#issuecomment-1156861952 ## CI report: * 2dc70b01fc3387cab385936f03069075471aecac UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28083) PulsarSource cannot work with object-reusing DeserializationSchema.
[ https://issues.apache.org/jira/browse/FLINK-28083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-28083: --- Labels: pull-request-available (was: ) > PulsarSource cannot work with object-reusing DeserializationSchema. > --- > > Key: FLINK-28083 > URL: https://issues.apache.org/jira/browse/FLINK-28083 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.15.0, 1.14.4 >Reporter: Yufan Sheng >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0 > > > This issue is the same as Kafka's > https://issues.apache.org/jira/browse/FLINK-25132 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [flink] syhily opened a new pull request, #19974: [FLINK-28083][Connector/Pulsar] Object-reusing for Pulsar source
syhily opened a new pull request, #19974: URL: https://github.com/apache/flink/pull/19974 ## What is the purpose of the change This pull request moves record deserializing in PulsarSource from SplitFetcher to RecordEmitter to support object-reusing deserializer. ## Brief change log Move record deserializing from PulsarPartitionSplitFetcher to PulsarRecordEmitter ## Verifying this change This change is already covered by existing tests, such as: - PulsarOrderedPartitionSplitReaderTest ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-28083) PulsarSource cannot work with object-reusing DeserializationSchema.
Yufan Sheng created FLINK-28083: --- Summary: PulsarSource cannot work with object-reusing DeserializationSchema. Key: FLINK-28083 URL: https://issues.apache.org/jira/browse/FLINK-28083 Project: Flink Issue Type: Bug Components: Connectors / Pulsar Affects Versions: 1.14.4, 1.15.0 Reporter: Yufan Sheng Fix For: 1.16.0 This issue is the same as Kafka's https://issues.apache.org/jira/browse/FLINK-25132 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [flink] flinkbot commented on pull request #19973: [FLINK-28082][Connector/Pulsar] Add end-to-end encryption support for Pulsar connector.
flinkbot commented on PR #19973: URL: https://github.com/apache/flink/pull/19973#issuecomment-1156816241 ## CI report: * f2c2ed2a6537a36091a3c0da129537a523b08d16 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28082) Support end to end encryption on Pulsar connector.
[ https://issues.apache.org/jira/browse/FLINK-28082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-28082: --- Labels: pull-request-available (was: ) > Support end to end encryption on Pulsar connector. > -- > > Key: FLINK-28082 > URL: https://issues.apache.org/jira/browse/FLINK-28082 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.16.0 >Reporter: Yufan Sheng >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0 > > > Add this Pulsar encryption support: > https://pulsar.apache.org/docs/security-encryption/ -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [flink] syhily opened a new pull request, #19973: [FLINK-28082][Connector/Pulsar] Add end-to-end encryption support for Pulsar connector.
syhily opened a new pull request, #19973: URL: https://github.com/apache/flink/pull/19973 ## What is the purpose of the change Add the end-to-end encryption support for Pulsar. ## Brief change log Add the CryptoKeyReader for both Pulsar source and sink. Add encryption config options to Pulsar sink. ## Verifying this change This change added tests and can be verified as follows: - TODO ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduces a new feature? (yes) - If yes, how is the feature documented? (JavaDocs) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-28082) Support end to end encryption on Pulsar connector.
Yufan Sheng created FLINK-28082: --- Summary: Support end to end encryption on Pulsar connector. Key: FLINK-28082 URL: https://issues.apache.org/jira/browse/FLINK-28082 Project: Flink Issue Type: Bug Components: Connectors / Pulsar Affects Versions: 1.16.0 Reporter: Yufan Sheng Fix For: 1.16.0 Add this Pulsar encryption support: https://pulsar.apache.org/docs/security-encryption/ -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (FLINK-28043) "Invalid lambda deserialization" in AvroParquetReaders
[ https://issues.apache.org/jira/browse/FLINK-28043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17554703#comment-17554703 ] Jing Ge commented on FLINK-28043: - Just want to let you know that I will find time to do it this week. > "Invalid lambda deserialization" in AvroParquetReaders > -- > > Key: FLINK-28043 > URL: https://issues.apache.org/jira/browse/FLINK-28043 > Project: Flink > Issue Type: Bug > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.15.0 >Reporter: Juntao Hu >Assignee: Juntao Hu >Priority: Minor > Labels: pull-request-available > Fix For: 1.16.0 > > > I packed a bundle jar including flink-parquet and flink-avro with > "org.apache.avro" relocated, to support PyFlink reading avro records from > parquet file, and "Invalid lambda deserialization" error occurs at runtime. I > guess this is similar to FLINK-18006 and points to MSHADE-260 > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [flink] syhily commented on pull request #19473: [FLINK-27199][Connector/Pulsar] Bump pulsar to 2.10.0
syhily commented on PR #19473: URL: https://github.com/apache/flink/pull/19473#issuecomment-1156706620 @fapaul This is PR ready, I think. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-27790) Port ADD JAR /SHOW JARS syntax implementation from SqlClient to TableEnvironment side
[ https://issues.apache.org/jira/browse/FLINK-27790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dalongliu updated FLINK-27790: -- Summary: Port ADD JAR /SHOW JARS syntax implementation from SqlClient to TableEnvironment side (was: Port ADD/ShowJar syntax implementation from SqlClient to TableEnvironment side) > Port ADD JAR /SHOW JARS syntax implementation from SqlClient to > TableEnvironment side > - > > Key: FLINK-27790 > URL: https://issues.apache.org/jira/browse/FLINK-27790 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Affects Versions: 1.16.0 >Reporter: dalongliu >Priority: Major > Fix For: 1.16.0 > > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [flink] flinkbot commented on pull request #19972: [FLINK-27399][Connector/Pulsar] Modify start cursor and stop cursor, change initial position setting logic.
flinkbot commented on PR #19972: URL: https://github.com/apache/flink/pull/19972#issuecomment-1156690944 ## CI report: * da9be03ef90e2e12873a3ef5c72996f2fa8fa964 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] syhily opened a new pull request, #19972: [FLINK-27399][Connector/Pulsar] Modify start cursor and stop cursor, change initial position setting logic.
syhily opened a new pull request, #19972: URL: https://github.com/apache/flink/pull/19972 ## What is the purpose of the change ## Brief change log This task modifies the `flink-connector-pulsar` module, adding some new mechanisms to set the initial consuming position. - Change `StartCursor`, add new useful methods, rename the confused `fromMessageTime()` method. - Introduce a new `SplitAssigner` for assigning the splits among the Pulsar readers. - Change the seeking mechanism from Pulsar consumer to Pulsar admin API. Which fixes a lot of issues. ## Verifying this change This change is already covered by existing tests, such as: - `PulsarSourceEnumeratorTest` - `PulsarOrderedPartitionSplitReaderTest` ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-27399) Pulsar connector didn't set start consuming position correctly
[
https://issues.apache.org/jira/browse/FLINK-27399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-27399:
---
Labels: pull-request-available (was: )
> Pulsar connector didn't set start consuming position correctly
> --
>
> Key: FLINK-27399
> URL: https://issues.apache.org/jira/browse/FLINK-27399
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
>Affects Versions: 1.15.0, 1.14.4, 1.16.0
>Reporter: Yufan Sheng
>Assignee: Yufan Sheng
>Priority: Major
> Labels: pull-request-available
>
> The Pulsar connector didn't use the consuming position from the checkpoint.
> They just commit the position to Pulsar after the checkpoint is complete. And
> the connector start to consume message from Pulsar directly by the offset
> stored on the Pulsar subscription.
> This causes the test could be failed in some situations. The start cursor
> (position on Pulsar) would be reset to the wrong position which causes the
> results didn't match the desired records.
> This issue fixes
> # FLINK-23944
> # FLINK-25884
> # FLINK-26177
> # FLINK-26237
> # FLINK-26721
> Although the test failure message could be various. They are truly sharing
> the same cause.
> h2. How to fix this issue:
> SourceEvent protocol for limiting the {{Consumer.seek}} operation.
> The Pulsar source needs to seek the desired consuming position when
> bootstrapping. The seeking action couldn’t be executed concurrently. We have
> designed a [new
> mechanism|https://github.com/apache/flink/pull/17119#pullrequestreview-746035072].
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[GitHub] [flink-ml] coderappee commented on pull request #109: [hotfix][flink-ml-lib][syntax]fix the forgotten static import method compareResultCollections from TestBaseUtils
coderappee commented on PR #109: URL: https://github.com/apache/flink-ml/pull/109#issuecomment-1156647134 > Hi @coderappee , thanks for creating this PR. Could you please illustrate why do we need to add these static imports? For now the test cases can still pass without these imports. Ok, I understand that why I can not build the project and pass the test cases now, because I use the latest code of flink master and do not use flink version 1.15.0. In or before flink version 1.15.0, AbstractTestBase is extended from TestBaseUtils, so any class extends from AbstractTestBase can use the static method of TestBaseUtils(compareResultCollections is one of the static method of it) But after [FLINK-27454], the AbstractTestBase does not extend from TestBaseUtils any more. It exteneds from TestLogger and can not use those static methods without static import. Ok it depends whether we close this pr or not, since if we migrate or upgrade flink version newer than 1.15.0, we still need to static import that method. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-25842) [v2] FLIP-158: Generalized incremental checkpoints
[ https://issues.apache.org/jira/browse/FLINK-25842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei reassigned FLINK-25842: Assignee: Yuan Mei > [v2] FLIP-158: Generalized incremental checkpoints > -- > > Key: FLINK-25842 > URL: https://issues.apache.org/jira/browse/FLINK-25842 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Roman Khachatryan >Assignee: Yuan Mei >Priority: Major > Fix For: 1.16.0 > > > Umbrella ticket for the 2nd iteration of [FLIP-158: Generalized incremental > checkpoints|https://cwiki.apache.org/confluence/display/FLINK/FLIP-158%3A+Generalized+incremental+checkpoints] > > 1st iteration: FLINK-21352 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (FLINK-23143) Support state migration
[ https://issues.apache.org/jira/browse/FLINK-23143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei reassigned FLINK-23143: Assignee: Hangxiang Yu > Support state migration > --- > > Key: FLINK-23143 > URL: https://issues.apache.org/jira/browse/FLINK-23143 > Project: Flink > Issue Type: Sub-task > Components: Runtime / State Backends >Reporter: Roman Khachatryan >Assignee: Hangxiang Yu >Priority: Minor > Labels: pull-request-available > Fix For: 1.16.0 > > > ChangelogKeyedStateBackend.getOrCreateKeyedState is currently used during > recovery; on 1st user access, it doesn't update metadata nor migrate state > (as opposed to other backends). > > The proposed solution is to > # wrap serializers (and maybe other objects) in getOrCreateKeyedState > # store wrapping objects in a new map keyed by state name > # pass wrapped objects to delegatedBackend.createInternalState > # on 1st user access, lookup wrapper and upgrade its wrapped serializer > This should be done for both KV/PQ states. > > See also [https://github.com/apache/flink/pull/15420#discussion_r656934791] > > cc: [~yunta] -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (FLINK-28081) Remove deprecated Hadoop specific Flink configuration options
[
https://issues.apache.org/jira/browse/FLINK-28081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Somogyi reassigned FLINK-28081:
-
Assignee: Gabor Somogyi
> Remove deprecated Hadoop specific Flink configuration options
> -
>
> Key: FLINK-28081
> URL: https://issues.apache.org/jira/browse/FLINK-28081
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Hadoop Compatibility
>Affects Versions: 1.16.0
>Reporter: Gabor Somogyi
>Assignee: Gabor Somogyi
>Priority: Major
>
> FLINK-7967 deprecated the following Flink configuration options in 1.5:
> {code:java}
> fs.hdfs.hdfsdefault
> fs.hdfs.hdfssite
> fs.hdfs.hadoopconf
> {code}
> These are deprecated for 10 minor releases so I think it's time to remove
> them.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (FLINK-28081) Remove deprecated Hadoop specific Flink configuration options
Gabor Somogyi created FLINK-28081:
-
Summary: Remove deprecated Hadoop specific Flink configuration
options
Key: FLINK-28081
URL: https://issues.apache.org/jira/browse/FLINK-28081
Project: Flink
Issue Type: Sub-task
Components: Connectors / Hadoop Compatibility
Affects Versions: 1.16.0
Reporter: Gabor Somogyi
FLINK-7967 deprecated the following Flink configuration options in 1.5:
{code:java}
fs.hdfs.hdfsdefault
fs.hdfs.hdfssite
fs.hdfs.hadoopconf
{code}
These are deprecated for 10 minor releases so I think it's time to remove them.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (FLINK-25458) Support local recovery
[ https://issues.apache.org/jira/browse/FLINK-25458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17554667#comment-17554667 ] Yuan Mei commented on FLINK-25458: -- As discussed offline and summarized here: * The first step is to support local recovery for materialized part * For the non-materialized part, double writing of changelog to both DFS and locally is very likely to affect the processing performance, hence it is optional. We will first prototype this part. > Support local recovery > -- > > Key: FLINK-25458 > URL: https://issues.apache.org/jira/browse/FLINK-25458 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Reporter: Yun Tang >Assignee: Yanfei Lei >Priority: Major > Fix For: 1.16.0 > > > Currently, changelog state-backend doesn't support local recovery. Thus, > recovery times might be sub-optimal. > > Materialized state issues: > Current periodic materialization would call state backend snapshot method > with a materialization id. However, current local state managment would rely > on checkpoint id as storing, confirming and discarding. The gap between them > would break how local recovery works. > > Non-materialized state issues: > * non-materialized state (i.e. changelog) is shared across checkpoints, and > therefore needs some tracking (in TM or hard-linking in FS) > * the writer does not enforce boundary between checkpoints (when writing to > DFS); if local stream simply duplicates DFS stream then it would break on > cleanup > * files can be shared across tasks, which will also break on cleanup -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [flink] eskabetxe commented on pull request #19925: [FLINK-27976][WebUi] Allow order jobs by name
eskabetxe commented on PR #19925: URL: https://github.com/apache/flink/pull/19925#issuecomment-1156625342 @MartijnVisser could you forward to someone that can check this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-25458) Support local recovery
[ https://issues.apache.org/jira/browse/FLINK-25458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei reassigned FLINK-25458: Assignee: Yanfei Lei > Support local recovery > -- > > Key: FLINK-25458 > URL: https://issues.apache.org/jira/browse/FLINK-25458 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Reporter: Yun Tang >Assignee: Yanfei Lei >Priority: Major > Fix For: 1.16.0 > > > Currently, changelog state-backend doesn't support local recovery. Thus, > recovery times might be sub-optimal. > > Materialized state issues: > Current periodic materialization would call state backend snapshot method > with a materialization id. However, current local state managment would rely > on checkpoint id as storing, confirming and discarding. The gap between them > would break how local recovery works. > > Non-materialized state issues: > * non-materialized state (i.e. changelog) is shared across checkpoints, and > therefore needs some tracking (in TM or hard-linking in FS) > * the writer does not enforce boundary between checkpoints (when writing to > DFS); if local stream simply duplicates DFS stream then it would break on > cleanup > * files can be shared across tasks, which will also break on cleanup -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (FLINK-27790) Port ADD/ShowJar syntax implementation from SqlClient to TableEnvironment side
[ https://issues.apache.org/jira/browse/FLINK-27790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dalongliu updated FLINK-27790: -- Summary: Port ADD/ShowJar syntax implementation from SqlClient to TableEnvironment side (was: Move ADD/Remove Jar syntax code implementation from SqlClient to TableEnvironment side) > Port ADD/ShowJar syntax implementation from SqlClient to TableEnvironment side > -- > > Key: FLINK-27790 > URL: https://issues.apache.org/jira/browse/FLINK-27790 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Affects Versions: 1.16.0 >Reporter: dalongliu >Priority: Major > Fix For: 1.16.0 > > -- This message was sent by Atlassian Jira (v8.20.7#820007)
