date:20210913

curcur edited a comment on pull request #16606:
URL: https://github.com/apache/flink/pull/16606#issuecomment-917898103

Roman and I had several long discussions on interfaces between
Materialization and
[`ChangelogKeyedStatebackend`](https://github.com/apache/flink/commit/3421b81c2502f61112bd131a7336c16e3dd30925#diff-e071e8a89527c24be4ee5ee342ad7d47c870170ef915d1407d18e998f7847f16L108).
Document here for future reference.

The main difference is between who is responsible to **keep and update** the
ChangelogKeyedStatebackend related states, denoted as `ChangelogSnapshotState`,
including three parts:

* - materialized snapshot from the underlying delegated state backend
* - non-materialized part in the current changelog
* - non-materialized changelog, from previous logs (before failover or
rescaling)

We've discussed and tried out three versions:

1. `Materialization` coupled with `ChangelogKeyedStatebackend`,
implemented in commit **fbd1e2d38ae6353506ceac8eb074bd24bdb29b62**
Where `PeriodicMaterializer` is an inner class of
`ChangelogKeyedStatebackend`
- Pros: states are shared, easy to reason about
- Cons: Coupled too closely, not flexible or extendible for
keyedstatebackend or materializer

Not to mention further, this approach is discarded during early
discussion.

2. `ChangelogSnapshotState` are kept in materializer. Materializer is
conceptually taken as a way to connect delegated state backend to changelog.
How to connect: through `ChangelogSnapshotState`, as denoted above.
implemented in commit **3421b81c2502f61112bd131a7336c16e3dd30925**

- Pros:
1. Good isolation and extensibility. Clear view the changelog
keyedstatebackend as four parts:
- log writer, delegated statebackend, materializer, and wrapper
changelogkeyedstatebackend for double writing
2. More natural to understand and implement.
- State is updated by the materializer, and accessible by
changelogKeyedStateBackend
- Materializer is part of ChangelogKeyedStateBackend

- Cons:
1. according to Roman, ChangelogKeyedStateBackend has implicit states
(like state double writes) besides the three mentioned above;
2. optimization (like batched writes) need to update materilizer as
well

3. `ChangelogSnapshotState` and its updates are kept in
ChangelogKeyedStatBackend. Materialization works as a stateless Materialization
Manager providing function utilities.
Implemented as commit **75dec43024d91b896d488a4c9e979d486228398a**
- Pros:
1. All states are wrapped in ChangelogKeyedStatBackend
2. Conceptually also works naturally

- Cons:
Circular constructor. `Materialization Manager` needs access to
`ChangelogKeyedStatBackend` to update `ChangelogSnapshotState`
`ChangelogKeyedStatBackend` is created from
StateBackend#createKeyedStateBackend.

To avoid circular construction, `Materialization Manager` has to be
exposed at the time creating ChangelogKeyedStatBackend.

@rkhachatryan what do you think Roman?

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] curcur edited a comment on pull request #16606: [FLINK-21357][runtime/statebackend]Periodic materialization for generalized incremental checkpoints



curcur edited a comment on pull request #16606:
URL: https://github.com/apache/flink/pull/16606#issuecomment-917898103


   Roman and I had several long discussions on interfaces between 
Materialization and `ChangelogKeyedStatebackend`. Document here for future 
reference.
   
   The main difference is between who is responsible to **keep and update** the 
ChangelogKeyedStatebackend related states, denoted as `ChangelogSnapshotState`, 
including three parts:
   
* - materialized snapshot from the underlying delegated state backend
* - non-materialized part in the current changelog
* - non-materialized changelog, from previous logs (before failover or 
rescaling) 
   
   We've discussed and tried out three versions:
   
   1. `Materialization` coupled with `ChangelogKeyedStatebackend`, 
   implemented in commit **fbd1e2d38ae6353506ceac8eb074bd24bdb29b62**
Where `PeriodicMaterializer` is an inner class of 
`ChangelogKeyedStatebackend`
- Pros: states are shared, easy to reason about
- Cons: Coupled too closely, not flexible or extendible for 
keyedstatebackend or materializer
   
Not to mention further, this approach is discarded during early 
discussion.

   2. `ChangelogSnapshotState` are kept in materializer. Materializer is 
conceptually taken as a way to connect delegated state backend to changelog. 
How to connect: through `ChangelogSnapshotState`, as denoted above.
   implemented in commit **3421b81c2502f61112bd131a7336c16e3dd30925**
   
   - Pros: 
 1. Good isolation and extensibility. Clear view the changelog 
keyedstatebackend as four parts: 
- log writer, delegated statebackend, materializer, and wrapper 
changelogkeyedstatebackend for double writing
 2. More natural to understand and implement.
- State is updated by the materializer, and accessible by 
changelogKeyedStateBackend
- Materializer is part of ChangelogKeyedStateBackend
   
   - Cons: 
  1. according to Roman, ChangelogKeyedStateBackend has implicit states 
(like state double writes) besides the three mentioned above; 
  2. optimization (like batched writes) need to update materilizer as 
well
   
   3. `ChangelogSnapshotState` and its updates are kept in 
ChangelogKeyedStatBackend. Materialization works as a stateless Materialization 
Manager providing function utilities.
   Implemented as commit **75dec43024d91b896d488a4c9e979d486228398a**
   - Pros:
  1. All states are wrapped in ChangelogKeyedStatBackend
  2. Conceptually also works naturally
  
   - Cons:
 Circular constructor. `Materialization Manager` needs access to 
`ChangelogKeyedStatBackend` to update `ChangelogSnapshotState`
 `ChangelogKeyedStatBackend` is created from 
StateBackend#createKeyedStateBackend. 
  
  To avoid circular construction, `Materialization Manager` has to be 
exposed at the time creating ChangelogKeyedStatBackend. 
   
   @rkhachatryan what do you think Roman?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Resolved] (FLINK-23773) KafkaPartitionSplitReader should remove empty splits from fetcher

2021-09-13 Thread Jiangjie Qin (Jira)



 [ 
https://issues.apache.org/jira/browse/FLINK-23773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin resolved FLINK-23773.
--
Resolution: Fixed

PR merged to master: fe17ca6042c570ce603bf4308775f61db1d515c9
cherry-picked to release-1.14: b26f7e7f5a0f1accda991a9304afa49369f5c553
cherry-picked to release-1.13: 763ac52092ba70dfef989d18b711400b437e6e09

 

> KafkaPartitionSplitReader should remove empty splits from fetcher
> -
>
> Key: FLINK-23773
> URL: https://issues.apache.org/jira/browse/FLINK-23773
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / Kafka
>Affects Versions: 1.14.0, 1.13.2
>Reporter: Qingsheng Ren
>Assignee: Qingsheng Ren
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.14.0, 1.13.3
>
>
> Currently if a {{KafkaPartitionSplit}} is empty (startingOffset >= 
> stoppingOffset), split reader only unsubscribes it from consumer, but doesn't 
> remove it from SplitFetcher. This will lead to consumer complaining some 
> partitions are not subscribed while fetching.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-20333) Flink standalone cluster throws metaspace OOM after submitting multiple PyFlink UDF jobs.



[ 
https://issues.apache.org/jira/browse/FLINK-20333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414734#comment-17414734
 ] 

Dian Fu commented on FLINK-20333:
-

I think it also depends on how you handle the jars (not only the UDF jars, but 
also connector jars, etc), e.g. whether placing them in the lib directory which 
are loaded by the context class loader or submitted using 
pipeline.jars/pipeline.classpaths which are loaded by the user class loader. 
Could you try to place the jars in the lib directory and see if the issue still 
exists?

> Flink standalone cluster throws metaspace OOM after submitting multiple 
> PyFlink UDF jobs.
> -
>
> Key: FLINK-20333
> URL: https://issues.apache.org/jira/browse/FLINK-20333
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python
>Reporter: Wei Zhong
>Assignee: Wei Zhong
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 1.11.3, 1.12.0
>
>
> Currently the Flink standalone cluster will throw metaspace OOM after 
> submitting multiple PyFlink UDF jobs. The root cause is that currently the 
> PyFlink classes are running in user classloader and so each job creates a 
> separate user class loader to load PyFlink related classes. There are many 
> soft references and Finalizers in memory (introduced by the underlying 
> Netty), which prevents the garbage collection of the user classloader of 
> already finished PyFlink jobs. 
> Due to their existence, it needs multiple full gc to reclaim the classloader 
> of the completed job. If only one full gc is performed before the metaspace 
> space is insufficient, then OOM will occur.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot edited a comment on pull request #17271: [FLINK-23969][connector/pulsar] Create e2e tests for pulsar connector.



flinkbot edited a comment on pull request #17271:
URL: https://github.com/apache/flink/pull/17271#issuecomment-918597722


   
   ## CI report:
   
   * 2e8733c07113a9fdede210b630f03df5349004e1 UNKNOWN
   * 09646cc9938c0127e18c4f71a973369fa4d89862 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24054)
 
   * 3d9a441ed810516ec8a417506be04f1d3ee5f239 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] becketqin merged pull request #17208: [BP-1.14][FLINK-23773][connector/kafka] Mark empty splits as finished to cleanup states in SplitFetcher



becketqin merged pull request #17208:
URL: https://github.com/apache/flink/pull/17208


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] Airblader commented on a change in pull request #17256: [FLINK-23385][table] Implement COALESCE function



Airblader commented on a change in pull request #17256:
URL: https://github.com/apache/flink/pull/17256#discussion_r707923511



##
File path: 
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CoalesceFunctionITCase.java
##
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.functions;
+
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+
+import org.junit.runners.Parameterized;
+
+import java.util.Arrays;
+import java.util.List;
+
+import static org.apache.flink.table.api.DataTypes.BIGINT;
+import static org.apache.flink.table.api.DataTypes.INT;
+import static org.apache.flink.table.api.DataTypes.STRING;
+import static org.apache.flink.table.api.Expressions.$;
+import static org.apache.flink.table.api.Expressions.call;
+import static org.apache.flink.table.api.Expressions.coalesce;
+
+/** Test COALESCE and its return type. * */
+public class CoalesceFunctionITCase extends BuiltInFunctionTestBase {
+
+@Parameterized.Parameters(name = "{index}: {0}")
+public static List testData() {
+return Arrays.asList(
+TestSpec.forFunction(BuiltInFunctionDefinitions.COALESCE)
+.onFieldsWithData(null, null, 1)
+.andDataTypes(BIGINT().nullable(), INT().nullable(), 
INT().notNull())
+.testResult(
+coalesce($("f0"), $("f1")),
+"COALESCE(f0, f1)",
+null,
+DataTypes.BIGINT().nullable())
+.testResult(
+coalesce($("f0"), $("f2")),
+"COALESCE(f0, f2)",
+1L,
+DataTypes.BIGINT().notNull())
+.testResult(
+coalesce($("f1"), $("f2")),
+"COALESCE(f1, f2)",
+1,
+DataTypes.INT().notNull())
+.testResult(
+coalesce($("f0"), 1),
+"COALESCE(f0, 1)",
+1L,
+// In this case, the return type is not null 
because we have a
+// constant in the function invocation
+DataTypes.BIGINT().notNull())
+.testResult(

Review comment:
   We could also investigate what the SQL standard says and what other 
implementations do; however, this isn't a new function, so we need to consider 
existing pipelines as well. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-24271) Add document for special char in JSON_VALUE

2021-09-13 Thread Jira



[ 
https://issues.apache.org/jira/browse/FLINK-24271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414726#comment-17414726
 ] 

Ingo Bürk commented on FLINK-24271:
---

JSON_VALUE can only return scalars. For your example you'd have to use 
JSON_QUERY.

> Add document for special char in JSON_VALUE
> ---
>
> Key: FLINK-24271
> URL: https://issues.apache.org/jira/browse/FLINK-24271
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Jingsong Lee
>Assignee: liwei li
>Priority: Minor
>  Labels: starter
>
> If user has a json string:
> {"fields": {"system.process": [0.998]}}
> It is hard to write a valid json path to get 0.998.
> The correct json path should be '$.fields.[''system.process''][0]'



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-24271) Add document for special char in JSON_VALUE

2021-09-13 Thread liwei li (Jira)



[ 
https://issues.apache.org/jira/browse/FLINK-24271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414725#comment-17414725
 ] 

liwei li commented on FLINK-24271:
--

Hi [~lzljs3620320]
The documentation shows that JSON_VALUE support * ,so i originally wanted to 
add an additional example with * in the document, but I found that the return 
value of JSON_VALUE('[10, 20, [30, 40]]','$[2][*]') was NULL, is that correct?

 

> Add document for special char in JSON_VALUE
> ---
>
> Key: FLINK-24271
> URL: https://issues.apache.org/jira/browse/FLINK-24271
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Jingsong Lee
>Assignee: liwei li
>Priority: Minor
>  Labels: starter
>
> If user has a json string:
> {"fields": {"system.process": [0.998]}}
> It is hard to write a valid json path to get 0.998.
> The correct json path should be '$.fields.[''system.process''][0]'



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot edited a comment on pull request #17271: [FLINK-23969][connector/pulsar] Create e2e tests for pulsar connector.



flinkbot edited a comment on pull request #17271:
URL: https://github.com/apache/flink/pull/17271#issuecomment-918597722


   
   ## CI report:
   
   * 2e8733c07113a9fdede210b630f03df5349004e1 UNKNOWN
   * 09646cc9938c0127e18c4f71a973369fa4d89862 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24054)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-20333) Flink standalone cluster throws metaspace OOM after submitting multiple PyFlink UDF jobs.

2021-09-13 Thread Puneet Duggal (Jira)



[ 
https://issues.apache.org/jira/browse/FLINK-20333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414723#comment-17414723
 ] 

Puneet Duggal commented on FLINK-20333:
---

[~dianfu] So what i meant was that in my case i am writing flink udf in java 
(Java Jobs) and using flink-core 1.12.0 maven dependency. And i am facing 
metaspace OOM on redeployment of jobs on a single task manager. Just wanted to 
know if this is a common and open issue or has it been resolved

> Flink standalone cluster throws metaspace OOM after submitting multiple 
> PyFlink UDF jobs.
> -
>
> Key: FLINK-20333
> URL: https://issues.apache.org/jira/browse/FLINK-20333
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python
>Reporter: Wei Zhong
>Assignee: Wei Zhong
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 1.11.3, 1.12.0
>
>
> Currently the Flink standalone cluster will throw metaspace OOM after 
> submitting multiple PyFlink UDF jobs. The root cause is that currently the 
> PyFlink classes are running in user classloader and so each job creates a 
> separate user class loader to load PyFlink related classes. There are many 
> soft references and Finalizers in memory (introduced by the underlying 
> Netty), which prevents the garbage collection of the user classloader of 
> already finished PyFlink jobs. 
> Due to their existence, it needs multiple full gc to reclaim the classloader 
> of the completed job. If only one full gc is performed before the metaspace 
> space is insufficient, then OOM will occur.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-23969) Test Pulsar source end 2 end



[ 
https://issues.apache.org/jira/browse/FLINK-23969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414715#comment-17414715
 ] 

Xintong Song commented on FLINK-23969:
--

[~syhily],

I don't think I'm the right person to review this. My knowledge in Pulsar and 
the Flink Pulsar connector is far from enough to be responsible for the quality 
of this PR.

I'll try to find someone.

> Test Pulsar source end 2 end
> 
>
> Key: FLINK-23969
> URL: https://issues.apache.org/jira/browse/FLINK-23969
> Project: Flink
>  Issue Type: Sub-task
>  Components: Connectors / Pulsar
>Reporter: Arvid Heise
>Assignee: Liu
>Priority: Blocker
>  Labels: pull-request-available, release-testing
> Fix For: 1.14.0
>
>
> Write a test application using Pulsar Source and execute it in distributed 
> fashion. Check fault-tolerance by crashing and restarting a TM.
> Ideally, we test different subscription modes and sticky keys in particular.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot edited a comment on pull request #17271: [FLINK-23969][connector/pulsar] Create e2e tests for pulsar connector.



flinkbot edited a comment on pull request #17271:
URL: https://github.com/apache/flink/pull/17271#issuecomment-918597722


   
   ## CI report:
   
   * 2e8733c07113a9fdede210b630f03df5349004e1 UNKNOWN
   * d818f8650d4c8a1d923276eac3f5d2dd373cc450 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24040)
 
   * 09646cc9938c0127e18c4f71a973369fa4d89862 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17274: [FLINK-24276][python] Remove confusing infos in Loopback mode



flinkbot edited a comment on pull request #17274:
URL: https://github.com/apache/flink/pull/17274#issuecomment-918785613


   
   ## CI report:
   
   * 71389b69c00b9279469fe1e08ec927f7ce796222 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24056)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17273: [FLINK-24267][python][docs] Update the example tutorial to use latest API



flinkbot edited a comment on pull request #17273:
URL: https://github.com/apache/flink/pull/17273#issuecomment-918785533


   
   ## CI report:
   
   * 57a8a4cbcf6e72d61e69bc95142adfd1749f9dd2 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24055)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-23493) python tests hang on Azure



[ 
https://issues.apache.org/jira/browse/FLINK-23493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414713#comment-17414713
 ] 

Xintong Song commented on FLINK-23493:
--

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24039=logs=821b528f-1eed-5598-a3b4-7f748b13f261=6bb545dd-772d-5d8c-f258-f5085fba3295=22038

> python tests hang on Azure
> --
>
> Key: FLINK-23493
> URL: https://issues.apache.org/jira/browse/FLINK-23493
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python
>Affects Versions: 1.14.0, 1.13.1, 1.12.4
>Reporter: Dawid Wysakowicz
>Assignee: Huang Xingbo
>Priority: Critical
>  Labels: test-stability
> Fix For: 1.14.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=20898=logs=821b528f-1eed-5598-a3b4-7f748b13f261=4fad9527-b9a5-5015-1b70-8356e5c91490=22829



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot edited a comment on pull request #15703: [FLINK-22275][table] Support random past for timestamp types in datagen connector



flinkbot edited a comment on pull request #15703:
URL: https://github.com/apache/flink/pull/15703#issuecomment-823980125


   
   ## CI report:
   
   * 68074dc1aea5ece8d4e11e19c15f04f85d6fc652 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23921)
 
   * fd278a5ded52e6525321de5503581a57f68b35b4 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24053)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] xintongsong closed pull request #17270: FLINK-23969 Test Pulsar source end 2 end



xintongsong closed pull request #17270:
URL: https://github.com/apache/flink/pull/17270


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] wuchong commented on pull request #15703: [FLINK-22275][table] Support random past for timestamp types in datagen connector



wuchong commented on pull request #15703:
URL: https://github.com/apache/flink/pull/15703#issuecomment-918789582


   I will merge it once build is passed. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot commented on pull request #17274: [FLINK-24276][python] Remove confusing infos in Loopback mode



flinkbot commented on pull request #17274:
URL: https://github.com/apache/flink/pull/17274#issuecomment-918785613


   
   ## CI report:
   
   * 71389b69c00b9279469fe1e08ec927f7ce796222 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot commented on pull request #17273: [FLINK-24267][python][docs] Update the example tutorial to use latest API



flinkbot commented on pull request #17273:
URL: https://github.com/apache/flink/pull/17273#issuecomment-918785533


   
   ## CI report:
   
   * 57a8a4cbcf6e72d61e69bc95142adfd1749f9dd2 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-23969) Test Pulsar source end 2 end

2021-09-13 Thread Yufan Sheng (Jira)



[ 
https://issues.apache.org/jira/browse/FLINK-23969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414706#comment-17414706
 ] 

Yufan Sheng commented on FLINK-23969:
-

[~xtsong] Arvid is on holiday. Can you review it?

> Test Pulsar source end 2 end
> 
>
> Key: FLINK-23969
> URL: https://issues.apache.org/jira/browse/FLINK-23969
> Project: Flink
>  Issue Type: Sub-task
>  Components: Connectors / Pulsar
>Reporter: Arvid Heise
>Assignee: Liu
>Priority: Blocker
>  Labels: pull-request-available, release-testing
> Fix For: 1.14.0
>
>
> Write a test application using Pulsar Source and execute it in distributed 
> fashion. Check fault-tolerance by crashing and restarting a TM.
> Ideally, we test different subscription modes and sticky keys in particular.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-20333) Flink standalone cluster throws metaspace OOM after submitting multiple PyFlink UDF jobs.



[ 
https://issues.apache.org/jira/browse/FLINK-20333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414698#comment-17414698
 ] 

Dian Fu commented on FLINK-20333:
-

[~pd17] What do you mean by "java client"?

> Flink standalone cluster throws metaspace OOM after submitting multiple 
> PyFlink UDF jobs.
> -
>
> Key: FLINK-20333
> URL: https://issues.apache.org/jira/browse/FLINK-20333
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python
>Reporter: Wei Zhong
>Assignee: Wei Zhong
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 1.11.3, 1.12.0
>
>
> Currently the Flink standalone cluster will throw metaspace OOM after 
> submitting multiple PyFlink UDF jobs. The root cause is that currently the 
> PyFlink classes are running in user classloader and so each job creates a 
> separate user class loader to load PyFlink related classes. There are many 
> soft references and Finalizers in memory (introduced by the underlying 
> Netty), which prevents the garbage collection of the user classloader of 
> already finished PyFlink jobs. 
> Due to their existence, it needs multiple full gc to reclaim the classloader 
> of the completed job. If only one full gc is performed before the metaspace 
> space is insufficient, then OOM will occur.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-23493) python tests hang on Azure



[ 
https://issues.apache.org/jira/browse/FLINK-23493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414696#comment-17414696
 ] 

Xintong Song commented on FLINK-23493:
--

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24036=logs=821b528f-1eed-5598-a3b4-7f748b13f261=6bb545dd-772d-5d8c-f258-f5085fba3295=23728

> python tests hang on Azure
> --
>
> Key: FLINK-23493
> URL: https://issues.apache.org/jira/browse/FLINK-23493
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python
>Affects Versions: 1.14.0, 1.13.1, 1.12.4
>Reporter: Dawid Wysakowicz
>Assignee: Huang Xingbo
>Priority: Critical
>  Labels: test-stability
> Fix For: 1.14.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=20898=logs=821b528f-1eed-5598-a3b4-7f748b13f261=4fad9527-b9a5-5015-1b70-8356e5c91490=22829



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot edited a comment on pull request #15703: [FLINK-22275][table] Support random past for timestamp types in datagen connector



flinkbot edited a comment on pull request #15703:
URL: https://github.com/apache/flink/pull/15703#issuecomment-823980125


   
   ## CI report:
   
   * 68074dc1aea5ece8d4e11e19c15f04f85d6fc652 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23921)
 
   * fd278a5ded52e6525321de5503581a57f68b35b4 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot commented on pull request #17274: [FLINK-24276][python] Remove confusing infos in Loopback mode



flinkbot commented on pull request #17274:
URL: https://github.com/apache/flink/pull/17274#issuecomment-918769720


   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Automated Checks
   Last check on commit 71389b69c00b9279469fe1e08ec927f7ce796222 (Tue Sep 14 
03:35:26 UTC 2021)
   
   **Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up 
to date!
   
   
   Mention the bot in a comment to re-run the automated checks.
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full 
explanation of the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot commented on pull request #17273: [FLINK-24267][python][docs] Update the example tutorial to use latest API



flinkbot commented on pull request #17273:
URL: https://github.com/apache/flink/pull/17273#issuecomment-918768969


   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Automated Checks
   Last check on commit 57a8a4cbcf6e72d61e69bc95142adfd1749f9dd2 (Tue Sep 14 
03:33:09 UTC 2021)
   
✅no warnings
   
   Mention the bot in a comment to re-run the automated checks.
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full 
explanation of the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-24276) Remove unnecessary info in Loopback mode

2021-09-13 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/FLINK-24276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated FLINK-24276:
---
Labels: pull-request-available  (was: )

> Remove unnecessary info in Loopback mode
> 
>
> Key: FLINK-24276
> URL: https://issues.apache.org/jira/browse/FLINK-24276
> Project: Flink
>  Issue Type: Improvement
>  Components: API / Python
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Huang Xingbo
>Assignee: Huang Xingbo
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.14.0, 1.15.0
>
>
> If the job runs in loopback mode, it will print unnecessary info 
> `apache_beam.typehints.native_type_compatibility - INFO - Using Any for 
> unsupported type: typing.Sequence[~T]` in the console. We need to remove this 
> confusing info.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-24267) PyFlink Table API example from docs does not work.

2021-09-13 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/FLINK-24267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated FLINK-24267:
---
Labels: pull-request-available  (was: )

> PyFlink Table API example from docs does not work.
> --
>
> Key: FLINK-24267
> URL: https://issues.apache.org/jira/browse/FLINK-24267
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python, Documentation
>Affects Versions: 1.14.0
>Reporter: Xintong Song
>Assignee: Dian Fu
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 1.14.0
>
>
> I followed the following documentation to build and install pyflink from 
> sources.
> https://ci.apache.org/projects/flink/flink-docs-master/docs/flinkdev/building/#build-pyflink
> Then I tried the WordCount.py example from the following documentation.
> https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/python/table_api_tutorial/
> This results in the following error.
> {code}
> $ python WordCount.py
> Traceback (most recent call last):
>   File "WordCount.py", line 2, in 
> from pyflink.table.descriptors import Schema, FileSystem
> ImportError: cannot import name 'FileSystem' from 'pyflink.table.descriptors' 
> (/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pyflink/table/descriptors.py)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] dianfu opened a new pull request #17273: [FLINK-24267][python][docs] Update the example tutorial to use latest API



dianfu opened a new pull request #17273:
URL: https://github.com/apache/flink/pull/17273


   
   ## What is the purpose of the change
   
   *This pull request updates the example tutorial of PyFlink to use latest API*
   
   
   ## Verifying this change
   
   This change is a trivial rework without any test coverage.
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (no)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
 - The serializers: (no)
 - The runtime per-record code paths (performance sensitive): (no)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
 - The S3 file system connector: (no)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (no)
 - If yes, how is the feature documented? (not applicable)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] HuangXingBo opened a new pull request #17274: [FLINK-24276][python] Remove confusing infos in Loopback mode



HuangXingBo opened a new pull request #17274:
URL: https://github.com/apache/flink/pull/17274


   ## What is the purpose of the change
   
   *This pull request will remove confusing infos in Loopback mode*
   
   ## Brief change log
   
 - *Remove confusing infos in Loopback mode*
   
   ## Verifying this change
   
 - *Original Tests*
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (no)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
 - The serializers: (no)
 - The runtime per-record code paths (performance sensitive): (no)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
 - The S3 file system connector: (no)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (no)
 - If yes, how is the feature documented? (not applicable)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] xuyangzhong commented on pull request #17272: [FLINK-24139][table-planner] Push down more predicates through Join in stream mode



xuyangzhong commented on pull request #17272:
URL: https://github.com/apache/flink/pull/17272#issuecomment-918768165


   Hi, @zjuwangg  
   The main reason is that in FlinkBatchProgram, this iterator time is set with 
5. In order to be consistent, i rewrite the number 5 here.
   I tried to find why we need to set iterations here. Actually, I think this 
aims to pushdown the filter condition into the join condition and can continue 
optimizing the join condition several times. 
   For example, in table api, the api named 'join' and 'where' will create a 
join relNode with a true condition and a fliter above the join relNode. The 
real condition we need will be set in the filter relNode.  The optimized 
program will do JOIN_PREDICATE_REWRITE_RULES first, which will extract the join 
condition, and then do FILTER_PREPARE_RULES, which will push the filter 
condition into the join condition. However, if we don't set the iterator, 
JOIN_PREDICATE_REWRITE_RULES finds that the join condition is always True, and 
will do nothing, which is not in our expectation. I reversed the order of 
executing these two rules and get the result i expected. I think the iterator 
time which means the repeat execution times will help avoid this kind of 
problem by repeating executing these two rules many times, which even can 
improve the problem of recursive optimization.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17272: [FLINK-24139][table-planner] Push down more predicates through Join in stream mode



flinkbot edited a comment on pull request #17272:
URL: https://github.com/apache/flink/pull/17272#issuecomment-918749449


   
   ## CI report:
   
   * eb60175d19a4988275846884b2fcbd3fc1b6f889 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24050)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17270: FLINK-23969 Test Pulsar source end 2 end



flinkbot edited a comment on pull request #17270:
URL: https://github.com/apache/flink/pull/17270#issuecomment-918480494


   
   ## CI report:
   
   * 05edbd6128a55c4e10812eb405216d7efd79bbcf Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24045)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17259: [FLINK-24251][table-common] Make default constructor of BinaryStringData construct an empty binary string again



flinkbot edited a comment on pull request #17259:
URL: https://github.com/apache/flink/pull/17259#issuecomment-918153592


   
   ## CI report:
   
   * 4c0e1cb84cde4efc161eeac74e4f07a2b80954db Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24049)
 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24007)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-24276) Remove unnecessary info in Loopback mode

2021-09-13 Thread Huang Xingbo (Jira)



 [ 
https://issues.apache.org/jira/browse/FLINK-24276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Huang Xingbo updated FLINK-24276:
-
Summary: Remove unnecessary info in Loopback mode  (was: Remove unnecessary 
info in Loopbck mode)

> Remove unnecessary info in Loopback mode
> 
>
> Key: FLINK-24276
> URL: https://issues.apache.org/jira/browse/FLINK-24276
> Project: Flink
>  Issue Type: Improvement
>  Components: API / Python
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Huang Xingbo
>Assignee: Huang Xingbo
>Priority: Major
> Fix For: 1.14.0, 1.15.0
>
>
> If the job runs in loopback mode, it will print unnecessary info 
> `apache_beam.typehints.native_type_compatibility - INFO - Using Any for 
> unsupported type: typing.Sequence[~T]` in the console. We need to remove this 
> confusing info.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-23969) Test Pulsar source end 2 end



[ 
https://issues.apache.org/jira/browse/FLINK-23969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414686#comment-17414686
 ] 

Xintong Song commented on FLINK-23969:
--

[~arvid], would you like to take a look at the PR?

> Test Pulsar source end 2 end
> 
>
> Key: FLINK-23969
> URL: https://issues.apache.org/jira/browse/FLINK-23969
> Project: Flink
>  Issue Type: Sub-task
>  Components: Connectors / Pulsar
>Reporter: Arvid Heise
>Assignee: Liu
>Priority: Blocker
>  Labels: pull-request-available, release-testing
> Fix For: 1.14.0
>
>
> Write a test application using Pulsar Source and execute it in distributed 
> fashion. Check fault-tolerance by crashing and restarting a TM.
> Ideally, we test different subscription modes and sticky keys in particular.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] Myracle commented on a change in pull request #17271: [FLINK-23969][connector/pulsar] Create e2e tests for pulsar connector.



Myracle commented on a change in pull request #17271:
URL: https://github.com/apache/flink/pull/17271#discussion_r707859625



##
File path: 
flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/testutils/runtime/PulsarRuntime.java
##
@@ -18,33 +18,36 @@
 
 package org.apache.flink.connector.pulsar.testutils.runtime;
 
-import 
org.apache.flink.connector.pulsar.testutils.runtime.container.PulsarContainerProvider;
-import 
org.apache.flink.connector.pulsar.testutils.runtime.mock.PulsarMockProvider;
+import org.apache.flink.connector.pulsar.testutils.PulsarTestEnvironment;
+import 
org.apache.flink.connector.pulsar.testutils.runtime.container.PulsarContainerRuntime;
+import 
org.apache.flink.connector.pulsar.testutils.runtime.mock.PulsarMockRuntime;
 
-import java.util.function.Supplier;
+import org.testcontainers.containers.GenericContainer;
 
 /**
- * A enum class for providing a operable pulsar runtime. We support two types 
of runtime, the
- * container and mock.
+ * A abstraction for different pulsar runtimes. Providing the common methods 
for {@link

Review comment:
   An abstraction




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] zjuwangg removed a comment on pull request #16853: [FLINK-23827][table-planner] Fix ModifiedMonotonicity inference for s…



zjuwangg removed a comment on pull request #16853:
URL: https://github.com/apache/flink/pull/16853#issuecomment-918763170


   Communicate with @cshuo offline and the new added RankTest cover 
monotonicity modification. For the UpdateFastRank optimization worked as 
expected.  
   cc @godfreyhe I think this pr can be merged, plz have a another look.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] Myasuka commented on a change in pull request #17203: [FLINK-22944][state] Optimize writing state changelog



Myasuka commented on a change in pull request #17203:
URL: https://github.com/apache/flink/pull/17203#discussion_r707877114



##
File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/state/changelog/ChangelogStateBackendHandle.java
##
@@ -48,20 +49,25 @@
 
 List getNonMaterializedStateHandles();
 
+UUID getBackendID();

Review comment:
   What the purpose of UUID acts here?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] Myasuka commented on a change in pull request #17203: [FLINK-22944][state] Optimize writing state changelog



Myasuka commented on a change in pull request #17203:
URL: https://github.com/apache/flink/pull/17203#discussion_r707874711



##
File path: 
flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java
##
@@ -506,7 +517,9 @@ public void notifyCheckpointAborted(long checkpointId) 
throws Exception {
 stateChangelogWriter,
 meta,
 stateDesc.getTtlConfig(),
-stateDesc.getDefaultValue());
+stateDesc.getDefaultValue(),
+++lastCreatedStateId);

Review comment:
   I wonder how the `lastCreatedStateId` work with restored state? Will it 
increase to the restored state id?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] zjuwangg commented on pull request #16853: [FLINK-23827][table-planner] Fix ModifiedMonotonicity inference for s…



zjuwangg commented on pull request #16853:
URL: https://github.com/apache/flink/pull/16853#issuecomment-918763170


   Communicate with @cshuo offline and the new added RankTest cover 
monotonicity modification. For the UpdateFastRank optimization worked as 
expected.  
   cc @godfreyhe I think this pr can be merged, plz have a another look.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-23826) Verify optimized scheduler performance for large-scale jobs

2021-09-13 Thread Zhu Zhu (Jira)



 [ 
https://issues.apache.org/jira/browse/FLINK-23826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhu Zhu updated FLINK-23826:

Priority: Critical  (was: Blocker)

> Verify optimized scheduler performance for large-scale jobs
> ---
>
> Key: FLINK-23826
> URL: https://issues.apache.org/jira/browse/FLINK-23826
> Project: Flink
>  Issue Type: Sub-task
>  Components: Runtime / Coordination
>Affects Versions: 1.14.0
>Reporter: Zhu Zhu
>Assignee: Zhilong Hong
>Priority: Critical
> Fix For: 1.14.0
>
>
> This ticket is used to verify the result of FLINK-21110.
> It should check if large scale jobs' scheduling are working well and the 
> scheduling performance, with a real job running on cluster. 
> The conclusion should include, for a *1 --- all-to-all-connected 
> -->1* job:
> 1. time of job initialization on master (job received -> scheduling started)
> 2. time of job scheduling and deployment (scheduling started -> all tasks in 
> INITIALIZATION)
> 3. time of job restarting on task failover (JM notified about task failure -> 
> all tasks in INITIALIZATION again)
> 4. master heap memory required



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] yittg commented on pull request #15703: [FLINK-22275][table] Support random past for timestamp types in datagen connector



yittg commented on pull request #15703:
URL: https://github.com/apache/flink/pull/15703#issuecomment-918762396


   Thanks @wuchong , i've pushed the fix. PTAL


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] cshuo commented on pull request #16853: [FLINK-23827][table-planner] Fix ModifiedMonotonicity inference for s…



cshuo commented on pull request #16853:
URL: https://github.com/apache/flink/pull/16853#issuecomment-918762153


   Thks @zjuwangg for reviewing. Currently, `ModifiedMonotonicity` is mainly 
used for inferring the rank strategy, and the added test cannot get 
`UpdateFastStrategy` for rank before the fix.
   
   Besides, I noticed an unexpected plan for the added test, thks for your 
reminder.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Issue Comment Deleted] (FLINK-24155) Translate documentation for how to configure the CheckpointFailureManager

2021-09-13 Thread Yuan Mei (Jira)



 [ 
https://issues.apache.org/jira/browse/FLINK-24155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuan Mei updated FLINK-24155:
-
Comment: was deleted

(was: merged commit 
[{{a1a645a}}|https://github.com/apache/flink/commit/a1a645ab7a41cca8409c5a91417a21ac119ff86d]
 into apache:master

merged commit 
[{{da82cb1}}|https://github.com/apache/flink/commit/da82cb13feb6aa28fde3bf1727ab6b398c0cdd68]
 into apache:release-1.14

merged commit 
[{{36a33e6}}|https://github.com/apache/flink/commit/36a33e6fefc2c2a361bff755d7f31339a5a784d0]
 into apache:release-1.13

 )

> Translate documentation for how to configure the CheckpointFailureManager
> -
>
> Key: FLINK-24155
> URL: https://issues.apache.org/jira/browse/FLINK-24155
> Project: Flink
>  Issue Type: Improvement
>  Components: chinese-translation, Documentation
>Affects Versions: 1.14.0, 1.13.2, 1.15.0
>Reporter: Piotr Nowojski
>Assignee: Yuan Mei
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.14.0, 1.13.3, 1.15.0
>
>
> Documentation added in FLINK-23916 should be translated to it's Chinese 
> counterpart. Note that this applies to three separate commits:
> merged to master as cd01d4c0279
> merged to release-1.14 as 2e769746bf2
> merged to release-1.13 as e1a71219454



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] yittg commented on a change in pull request #15703: [FLINK-22275][table] Support random past for timestamp types in datagen connector



yittg commented on a change in pull request #15703:
URL: https://github.com/apache/flink/pull/15703#discussion_r707874849



##
File path: 
flink-table/flink-table-api-java-bridge/src/main/java/org/apache/flink/connector/datagen/table/RandomGeneratorVisitor.java
##
@@ -203,6 +217,33 @@ public DataGeneratorContainer visit(DayTimeIntervalType 
dayTimeIntervalType) {
 RandomGenerator.longGenerator(config.get(min), 
config.get(max)), min, max);
 }
 
+@Override
+public DataGeneratorContainer visit(TimestampType timestampType) {

Review comment:
   I've remove the default implementation in `DataGenVisitorBase`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Closed] (FLINK-24155) Translate documentation for how to configure the CheckpointFailureManager

2021-09-13 Thread Yuan Mei (Jira)



 [ 
https://issues.apache.org/jira/browse/FLINK-24155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuan Mei closed FLINK-24155.

Resolution: Fixed

merged commit 
[{{a1a645a}}|https://github.com/apache/flink/commit/a1a645ab7a41cca8409c5a91417a21ac119ff86d]
 into apache:master

merged commit 
[{{da82cb1}}|https://github.com/apache/flink/commit/da82cb13feb6aa28fde3bf1727ab6b398c0cdd68]
 into apache:release-1.14

merged commit 
[{{36a33e6}}|https://github.com/apache/flink/commit/36a33e6fefc2c2a361bff755d7f31339a5a784d0]
 into apache:release-1.13

> Translate documentation for how to configure the CheckpointFailureManager
> -
>
> Key: FLINK-24155
> URL: https://issues.apache.org/jira/browse/FLINK-24155
> Project: Flink
>  Issue Type: Improvement
>  Components: chinese-translation, Documentation
>Affects Versions: 1.14.0, 1.13.2, 1.15.0
>Reporter: Piotr Nowojski
>Assignee: Yuan Mei
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.14.0, 1.13.3, 1.15.0
>
>
> Documentation added in FLINK-23916 should be translated to it's Chinese 
> counterpart. Note that this applies to three separate commits:
> merged to master as cd01d4c0279
> merged to release-1.14 as 2e769746bf2
> merged to release-1.13 as e1a71219454



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-24155) Translate documentation for how to configure the CheckpointFailureManager

2021-09-13 Thread Yuan Mei (Jira)



[ 
https://issues.apache.org/jira/browse/FLINK-24155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414683#comment-17414683
 ] 

Yuan Mei commented on FLINK-24155:
--

merged commit 
[{{a1a645a}}|https://github.com/apache/flink/commit/a1a645ab7a41cca8409c5a91417a21ac119ff86d]
 into apache:master

merged commit 
[{{da82cb1}}|https://github.com/apache/flink/commit/da82cb13feb6aa28fde3bf1727ab6b398c0cdd68]
 into apache:release-1.14

merged commit 
[{{36a33e6}}|https://github.com/apache/flink/commit/36a33e6fefc2c2a361bff755d7f31339a5a784d0]
 into apache:release-1.13

 

> Translate documentation for how to configure the CheckpointFailureManager
> -
>
> Key: FLINK-24155
> URL: https://issues.apache.org/jira/browse/FLINK-24155
> Project: Flink
>  Issue Type: Improvement
>  Components: chinese-translation, Documentation
>Affects Versions: 1.14.0, 1.13.2, 1.15.0
>Reporter: Piotr Nowojski
>Assignee: Yuan Mei
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.14.0, 1.13.3, 1.15.0
>
>
> Documentation added in FLINK-23916 should be translated to it's Chinese 
> counterpart. Note that this applies to three separate commits:
> merged to master as cd01d4c0279
> merged to release-1.14 as 2e769746bf2
> merged to release-1.13 as e1a71219454



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] curcur merged pull request #17257: [FLINK-24155][documentation] Sync Chinese version of documentation to configure CheckpointFailureManager



curcur merged pull request #17257:
URL: https://github.com/apache/flink/pull/17257


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (FLINK-24276) Remove unnecessary info in Loopbck mode

2021-09-13 Thread Huang Xingbo (Jira)

Huang Xingbo created FLINK-24276:


 Summary: Remove unnecessary info in Loopbck mode
 Key: FLINK-24276
 URL: https://issues.apache.org/jira/browse/FLINK-24276
 Project: Flink
  Issue Type: Improvement
  Components: API / Python
Affects Versions: 1.14.0, 1.15.0
Reporter: Huang Xingbo
Assignee: Huang Xingbo
 Fix For: 1.14.0, 1.15.0


If the job runs in loopback mode, it will print unnecessary info 
`apache_beam.typehints.native_type_compatibility - INFO - Using Any for 
unsupported type: typing.Sequence[~T]` in the console. We need to remove this 
confusing info.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-23969) Test Pulsar source end 2 end

2021-09-13 Thread Liu (Jira)



[ 
https://issues.apache.org/jira/browse/FLINK-23969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414682#comment-17414682
 ] 

Liu commented on FLINK-23969:
-

[~syhily] I have reviewed your code. The additional  Shared and Key_Shared  are 
both implemented in your code. Since your tests covers mine, maybe we can use 
your code for this ticket.

> Test Pulsar source end 2 end
> 
>
> Key: FLINK-23969
> URL: https://issues.apache.org/jira/browse/FLINK-23969
> Project: Flink
>  Issue Type: Sub-task
>  Components: Connectors / Pulsar
>Reporter: Arvid Heise
>Assignee: Liu
>Priority: Blocker
>  Labels: pull-request-available, release-testing
> Fix For: 1.14.0
>
>
> Write a test application using Pulsar Source and execute it in distributed 
> fashion. Check fault-tolerance by crashing and restarting a TM.
> Ideally, we test different subscription modes and sticky keys in particular.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] curcur merged pull request #17252: [FLINK-24155] Sync Chinese version of documentation to configure CheckpointFailureManager



curcur merged pull request #17252:
URL: https://github.com/apache/flink/pull/17252


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] zjuwangg commented on a change in pull request #17272: [FLINK-24139][table-planner] Push down more predicates through Join in stream mode



zjuwangg commented on a change in pull request #17272:
URL: https://github.com/apache/flink/pull/17272#discussion_r707869440



##
File path: 
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/plan/stream/sql/join/JoinTest.scala
##
@@ -32,6 +32,21 @@ class JoinTest extends TableTestBase {
   util.addTableSource[(Int, Long, String)]("t", 'a, 'b, 'c)
   util.addTableSource[(Long, String, Int)]("s", 'x, 'y, 'z)
 
+  @Test
+  def testDependentConditionderivationInnerJoin: Unit = {

Review comment:
   ```suggestion
 def testDependentConditionDerivationInnerJoin: Unit = {
   ```

##
File path: 
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/plan/stream/sql/join/JoinTest.scala
##
@@ -32,6 +32,21 @@ class JoinTest extends TableTestBase {
   util.addTableSource[(Int, Long, String)]("t", 'a, 'b, 'c)
   util.addTableSource[(Long, String, Int)]("s", 'x, 'y, 'z)
 
+  @Test
+  def testDependentConditionderivationInnerJoin: Unit = {
+util.verifyExecPlan("SELECT a1, b1 FROM A JOIN B ON (a1 = 1 AND b1 = 1) OR 
(a2 = 2 AND b2 = 2)")
+  }
+
+  @Test
+  def testDependentConditionderivationInnerJoinWithTrue: Unit = {

Review comment:
   ditto

##
File path: 
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/plan/stream/table/JoinTest.scala
##
@@ -31,6 +31,48 @@ import java.sql.Timestamp
   */
 class JoinTest extends TableTestBase {
 
+  @Test
+  def testDependentConditionderivationInnerJoin: Unit = {

Review comment:
   Ditto and all new added following method name should change.

##
File path: 
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/plan/stream/sql/join/JoinTest.scala
##
@@ -32,6 +32,21 @@ class JoinTest extends TableTestBase {
   util.addTableSource[(Int, Long, String)]("t", 'a, 'b, 'c)
   util.addTableSource[(Long, String, Int)]("s", 'x, 'y, 'z)
 
+  @Test
+  def testDependentConditionderivationInnerJoin: Unit = {
+util.verifyExecPlan("SELECT a1, b1 FROM A JOIN B ON (a1 = 1 AND b1 = 1) OR 
(a2 = 2 AND b2 = 2)")
+  }
+
+  @Test
+  def testDependentConditionderivationInnerJoinWithTrue: Unit = {
+util.verifyExecPlan("SELECT a1, b1 FROM A JOIN B ON (a1 = 1 AND b1 = 1) OR 
(a2 = 2 AND true)")
+  }
+
+  @Test
+  def testDependentConditionderivationInnerJoinWithNull: Unit = {

Review comment:
   ditto




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-24149) Make checkpoint self-contained and relocatable

2021-09-13 Thread Yun Tang (Jira)



[ 
https://issues.apache.org/jira/browse/FLINK-24149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414676#comment-17414676
 ] 

Yun Tang commented on FLINK-24149:
--

[~Feifan Wang] My core idea is that we should avoid any misuse from customers 
and could make this whole phase easy to use. From my view, using 
state-processor-api to rewrite is not so convenient.

We can have several ways:
*  Use a tool to modify the checkpoint meta data and call distributed file 
system to copy remote files to another namespace.
* The checkpoint directories could be copied directly to another namespace 
without any additional changes (just like what you provided in this ticket, but 
current solution cannot satify case that containing multi previous incremental 
checkpoint directories)


> Make checkpoint self-contained and relocatable
> --
>
> Key: FLINK-24149
> URL: https://issues.apache.org/jira/browse/FLINK-24149
> Project: Flink
>  Issue Type: Improvement
>  Components: Runtime / Checkpointing
>Reporter: Feifan Wang
>Priority: Major
>  Labels: pull-request-available
> Attachments: image-2021-09-08-17-06-31-560.png, 
> image-2021-09-08-17-10-28-240.png, image-2021-09-08-17-55-46-898.png, 
> image-2021-09-08-18-01-03-176.png
>
>
> h1. Backgroud
> We have many jobs with large state size in production environment. According 
> to the operation practice of these jobs and the analysis of some specific 
> problems, we believe that RocksDBStateBackend's incremental checkpoint has 
> many advantages over savepoint：
>  # Savepoint takes much longer time then incremental checkpoint in jobs with 
> large state. The figure below is a job in our production environment, it 
> takes nearly 7 minutes to complete a savepoint, while checkpoint only takes a 
> few seconds.( checkpoint after savepoint takes longer time is a problem 
> described in -FLINK-23949-)
>  !image-2021-09-08-17-55-46-898.png|width=723,height=161!
>  # Savepoint causes excessive cpu usage. The figure below shows the CPU usage 
> of the same job in the above figure :
>  !image-2021-09-08-18-01-03-176.png|width=516,height=148!
>  # Savepoint may cause excessive native memory usage and eventually cause the 
> TaskManager process memory usage to exceed the limit. (We did not further 
> investigate the cause and did not try to reproduce the problem on other large 
> state jobs, but only increased the overhead memory. So this reason may not be 
> so conclusive. )
> For the above reasons, we tend to use retained incremental checkpoint to 
> completely replace savepoint for jobs with large state size.
> h1. Problems
>  * *Problem 1 : retained incremental checkpoint difficult to clean up once 
> they used for recovery*
> This problem caused by jobs recoveryed from a retained incremental checkpoint 
> may reference files on this retained incremental checkpoint's shared 
> directory in subsequent checkpoints, even they are not in a same job 
> instance. The worst case is that the retained checkpoint will be referenced 
> one by one, forming a very long reference chain.This makes it difficult for 
> users to manage retained checkpoints. In fact, we have also suffered failures 
> caused by incorrect deletion of retained checkpoints.
> Although we can use the file handle in checkpoint metadata to figure out 
> which files can be deleted, but I think it is inappropriate to let users do 
> this.
>  * *Problem 2 : checkpoint not relocatable*
> Even if we can figure out all files referenced by a checkpoint, moving these 
> files will invalidate the checkpoint as well, because the metadata file 
> references absolute file paths.
> Since savepoint already be self-contained and relocatable (FLINK-5763), why 
> don't we use savepoint just for migrate jobs to another place ? In addition 
> to the savepoint performance problem in the background description, a very 
> important reason is that the migration requirement may come from the failure 
> of the original cluster. In this case, there is no opportunity to trigger 
> savepoint.
> h1. Proposal
>  * *job's checkpoint directory (user-defined-checkpoint-dir/) contains 
> all their state files (self-contained)*
>  As far as I know, in the current status, only the subsequent checkpoints of 
> the jobs restored from the retained checkpoint violate this constraint. One 
> possible solution is to re-upload all shared files at the first incremental 
> checkpoint after the job started, but we need to discuss how to distinguish 
> between a new job instance and a restart.
>  * *use relative file path in checkpoint metadata (relocatable)*
> Change all file references in checkpoint metadata to the relative path 
> relative to the _metadata file, so we can copy 
> user-defined-checkpoint-dir/ to any other place.
>  
> BTW, this issue is so

[GitHub] [flink] zjuwangg commented on a change in pull request #17272: [FLINK-24139][table-planner] Push down more predicates through Join in stream mode



zjuwangg commented on a change in pull request #17272:
URL: https://github.com/apache/flink/pull/17272#discussion_r707866485



##
File path: 
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/optimize/program/FlinkStreamProgram.scala
##
@@ -121,11 +121,20 @@ object FlinkStreamProgram {
   PREDICATE_PUSHDOWN,
   FlinkGroupProgramBuilder.newBuilder[StreamOptimizeContext]
 .addProgram(
-  FlinkHepRuleSetProgramBuilder.newBuilder
-.setHepRulesExecutionType(HEP_RULES_EXECUTION_TYPE.RULE_COLLECTION)
-.setHepMatchOrder(HepMatchOrder.BOTTOM_UP)
-.add(FlinkStreamRuleSets.FILTER_PREPARE_RULES)
-.build(), "filter rules")
+  FlinkGroupProgramBuilder.newBuilder[StreamOptimizeContext]
+.addProgram(
+  FlinkHepRuleSetProgramBuilder.newBuilder[StreamOptimizeContext]
+
.setHepRulesExecutionType(HEP_RULES_EXECUTION_TYPE.RULE_SEQUENCE)
+.setHepMatchOrder(HepMatchOrder.BOTTOM_UP)
+.add(FlinkStreamRuleSets.JOIN_PREDICATE_REWRITE_RULES)
+.build(), "join predicate rewrite")
+.addProgram(
+  FlinkHepRuleSetProgramBuilder.newBuilder
+
.setHepRulesExecutionType(HEP_RULES_EXECUTION_TYPE.RULE_COLLECTION)
+.setHepMatchOrder(HepMatchOrder.BOTTOM_UP)
+.add(FlinkStreamRuleSets.FILTER_PREPARE_RULES)
+.build(), "filter rules")
+.setIterations(5).build(), "predicate rewrite")

Review comment:
   Why we need set iterations and why chose 5 as the iteration number?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot commented on pull request #17272: [FLINK-24139][table-planner] Push down more predicates through Join in stream mode



flinkbot commented on pull request #17272:
URL: https://github.com/apache/flink/pull/17272#issuecomment-918749449


   
   ## CI report:
   
   * eb60175d19a4988275846884b2fcbd3fc1b6f889 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17271: [FLINK-23969][connector/pulsar] Create e2e tests for pulsar connector.



flinkbot edited a comment on pull request #17271:
URL: https://github.com/apache/flink/pull/17271#issuecomment-918597722


   
   ## CI report:
   
   * 2e8733c07113a9fdede210b630f03df5349004e1 UNKNOWN
   * d818f8650d4c8a1d923276eac3f5d2dd373cc450 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24040)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17259: [FLINK-24251][table-common] Make default constructor of BinaryStringData construct an empty binary string again



flinkbot edited a comment on pull request #17259:
URL: https://github.com/apache/flink/pull/17259#issuecomment-918153592


   
   ## CI report:
   
   * 4c0e1cb84cde4efc161eeac74e4f07a2b80954db Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24007)
 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24049)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] zjuwangg commented on a change in pull request #17272: [FLINK-24139][table-planner] Push down more predicates through Join in stream mode



zjuwangg commented on a change in pull request #17272:
URL: https://github.com/apache/flink/pull/17272#discussion_r707865281



##
File path: 
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/rules/FlinkStreamRuleSets.scala
##
@@ -159,6 +159,12 @@ object FlinkStreamRuleSets {
 CoreRules.FILTER_MERGE
   )
 
+  /**
+   * RuleSet to extracts sub-condition which can be pushed into join inputs

Review comment:
   ```suggestion
  * RuleSet to extract sub-condition which can be pushed into join inputs
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Closed] (FLINK-24219) Translate "SET Statements" page of "SQL" into Chinese

2021-09-13 Thread Jark Wu (Jira)



 [ 
https://issues.apache.org/jira/browse/FLINK-24219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jark Wu closed FLINK-24219.
---
Fix Version/s: 1.15.0
   Resolution: Fixed

Fixed in master: 94c507754f9328e8706e271aa3c8369472a3bfa5

> Translate "SET Statements" page of "SQL" into Chinese
> -
>
> Key: FLINK-24219
> URL: https://issues.apache.org/jira/browse/FLINK-24219
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Reporter: wuguihu
>Assignee: wuguihu
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 1.15.0
>
>
> [https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/table/sql/set/]
> docs/content.zh/docs/dev/table/sql/set.md



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] wuchong merged pull request #17223: [FLINK-24219][doc]Translate "SET Statements" page of "SQL" into Chinese



wuchong merged pull request #17223:
URL: https://github.com/apache/flink/pull/17223


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] tsreaper commented on pull request #17259: [FLINK-24251][table-common] Make default constructor of BinaryStringData construct an empty binary string again



tsreaper commented on pull request #17259:
URL: https://github.com/apache/flink/pull/17259#issuecomment-918743161


@flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Assigned] (FLINK-24267) PyFlink Table API example from docs does not work.



 [ 
https://issues.apache.org/jira/browse/FLINK-24267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dian Fu reassigned FLINK-24267:
---

Assignee: Dian Fu  (was: Huang Xingbo)

> PyFlink Table API example from docs does not work.
> --
>
> Key: FLINK-24267
> URL: https://issues.apache.org/jira/browse/FLINK-24267
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python, Documentation
>Affects Versions: 1.14.0
>Reporter: Xintong Song
>Assignee: Dian Fu
>Priority: Critical
> Fix For: 1.14.0
>
>
> I followed the following documentation to build and install pyflink from 
> sources.
> https://ci.apache.org/projects/flink/flink-docs-master/docs/flinkdev/building/#build-pyflink
> Then I tried the WordCount.py example from the following documentation.
> https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/python/table_api_tutorial/
> This results in the following error.
> {code}
> $ python WordCount.py
> Traceback (most recent call last):
>   File "WordCount.py", line 2, in 
> from pyflink.table.descriptors import Schema, FileSystem
> ImportError: cannot import name 'FileSystem' from 'pyflink.table.descriptors' 
> (/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pyflink/table/descriptors.py)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot edited a comment on pull request #17270: FLINK-23969 Test Pulsar source end 2 end



flinkbot edited a comment on pull request #17270:
URL: https://github.com/apache/flink/pull/17270#issuecomment-918480494


   
   ## CI report:
   
   * 4dea0cfa8f0f164e0adef4f606f4b20f7327032c Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24035)
 
   * 05edbd6128a55c4e10812eb405216d7efd79bbcf Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24045)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot commented on pull request #17272: [FLINK-24139][table-planner] Push down more predicates through Join in stream mode



flinkbot commented on pull request #17272:
URL: https://github.com/apache/flink/pull/17272#issuecomment-918737626


   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Automated Checks
   Last check on commit eb60175d19a4988275846884b2fcbd3fc1b6f889 (Tue Sep 14 
02:16:22 UTC 2021)
   
   **Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up 
to date!
* **This pull request references an unassigned [Jira 
ticket](https://issues.apache.org/jira/browse/FLINK-24139).** According to the 
[code contribution 
guide](https://flink.apache.org/contributing/contribute-code.html), tickets 
need to be assigned before starting with the implementation work.
   
   
   Mention the bot in a comment to re-run the automated checks.
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full 
explanation of the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Assigned] (FLINK-24267) PyFlink Table API example from docs does not work.

2021-09-13 Thread Huang Xingbo (Jira)



 [ 
https://issues.apache.org/jira/browse/FLINK-24267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Huang Xingbo reassigned FLINK-24267:


Assignee: Huang Xingbo

> PyFlink Table API example from docs does not work.
> --
>
> Key: FLINK-24267
> URL: https://issues.apache.org/jira/browse/FLINK-24267
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python, Documentation
>Affects Versions: 1.14.0
>Reporter: Xintong Song
>Assignee: Huang Xingbo
>Priority: Critical
> Fix For: 1.14.0
>
>
> I followed the following documentation to build and install pyflink from 
> sources.
> https://ci.apache.org/projects/flink/flink-docs-master/docs/flinkdev/building/#build-pyflink
> Then I tried the WordCount.py example from the following documentation.
> https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/python/table_api_tutorial/
> This results in the following error.
> {code}
> $ python WordCount.py
> Traceback (most recent call last):
>   File "WordCount.py", line 2, in 
> from pyflink.table.descriptors import Schema, FileSystem
> ImportError: cannot import name 'FileSystem' from 'pyflink.table.descriptors' 
> (/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pyflink/table/descriptors.py)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (FLINK-24267) PyFlink Table API example from docs does not work.



 [ 
https://issues.apache.org/jira/browse/FLINK-24267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xintong Song reassigned FLINK-24267:


Assignee: Xingbo Huang

> PyFlink Table API example from docs does not work.
> --
>
> Key: FLINK-24267
> URL: https://issues.apache.org/jira/browse/FLINK-24267
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python, Documentation
>Affects Versions: 1.14.0
>Reporter: Xintong Song
>Assignee: Xingbo Huang
>Priority: Critical
> Fix For: 1.14.0
>
>
> I followed the following documentation to build and install pyflink from 
> sources.
> https://ci.apache.org/projects/flink/flink-docs-master/docs/flinkdev/building/#build-pyflink
> Then I tried the WordCount.py example from the following documentation.
> https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/python/table_api_tutorial/
> This results in the following error.
> {code}
> $ python WordCount.py
> Traceback (most recent call last):
>   File "WordCount.py", line 2, in 
> from pyflink.table.descriptors import Schema, FileSystem
> ImportError: cannot import name 'FileSystem' from 'pyflink.table.descriptors' 
> (/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pyflink/table/descriptors.py)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-23228) Translate "Stateful Stream Processing" page into Chinese

2021-09-13 Thread kevin (Jira)



 [ 
https://issues.apache.org/jira/browse/FLINK-23228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kevin updated FLINK-23228:
--
Labels: documentation pendingPR pull-request-available  (was: documentation 
pull-request-available)

> Translate "Stateful Stream Processing" page into Chinese
> 
>
> Key: FLINK-23228
> URL: https://issues.apache.org/jira/browse/FLINK-23228
> Project: Flink
>  Issue Type: Improvement
>  Components: chinese-translation, Documentation
>Affects Versions: 1.13.0
>Reporter: kevin
>Assignee: kevin
>Priority: Major
>  Labels: documentation, pendingPR, pull-request-available
> Fix For: 1.13.0
>
>   Original Estimate: 120h
>  Time Spent: 20h
>  Remaining Estimate: 0h
>
> The page url is 
> [https://ci.apache.org/projects/flink/flink-docs-release-1.13/zh/docs/concepts/stateful-stream-processing/
>  
> |https://ci.apache.org/projects/flink/flink-docs-release-1.13/zh/docs/concepts/stateful-stream-processing/]
> The markdown file is located in 
> *docs/content.zh/docs/concepts/stateful-stream-processing.md*
> This page needs to be translated to Chinese, currently, only title was 
> translated to Chinese.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-24139) Push down more predicates through Join in stream mode

2021-09-13 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/FLINK-24139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated FLINK-24139:
---
Labels: pull-request-available  (was: )

> Push down more predicates through Join in stream mode
> -
>
> Key: FLINK-24139
> URL: https://issues.apache.org/jira/browse/FLINK-24139
> Project: Flink
>  Issue Type: Improvement
>  Components: Table SQL / Planner
>Reporter: Alexander Trushev
>Priority: Minor
>  Labels: pull-request-available
> Attachments: q13_after.json, q13_after.png, q13_after.txt, 
> q13_before.json, q13_before.png, q13_before.txt
>
>
> h3. Context
> Rule {{JoinDependentConditionDerivationRule}} is introduced in FLINK-12509. 
> This rule rewrites join condition in such way that more predicates can be 
> pushed down through join. For example,
>  # Source A = [a0, a1, a2], source B = [b0, b1]
>  # {code:sql}select * from A join B on a0 = b0 where (a1 = 0 and b1 = 0) or 
> a2 = 0{code}
>  # {{JoinDependentConditionDerivationRule}} transforms condition ((a1 and b1) 
> or a2) to (((a1 and b1) or a2) and (a1 or a2))
>  # {{JoinConditionPushRule}} pushes (a1 or a2) to A source
> It is a good optimization that can lead to performance improvement of query 
> execution.
>  Currently, {{JoinDependentConditionDerivationRule}} is used only in batch 
> mode.
> h3. Proposal
> Enable {{JoinDependentConditionDerivationRule}} in stream mode.
> h3. Benefit
> Experiment based on [https://github.com/ververica/flink-sql-benchmark]
>  Cluster – 4 nodes each 2 slots
>  Dataset – tpcds_bin_orc_20
>  Before – 1.14.0-rc0
>  After – 1.14.0-rc0 + patched {{FlinkStreamProgram}} including 
> {{JoinDependentConditionDerivationRule}}
> ||TPC-DS 20 GB||Before||After||
> |q13 stream mode|83 s|8 s|
> Query plan, stream graph, dashboard visualization before and after the patch 
> are in the attachment
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] xuyangzhong opened a new pull request #17272: [FLINK-24139][table-planner] Push down more predicates through Join in stream mode



xuyangzhong opened a new pull request #17272:
URL: https://github.com/apache/flink/pull/17272


   ## What is the purpose of the change
   
   Enable JoinDependentConditionDerivationRule in stream mode.
   
   
   ## Brief change log
   
 - *add a new rule set in FlinkStreamRuleSets*
 - *add a new program in FlinkStreamProgram*
 - *add some test case in stream/table/JoinTest, stream/sqlJoinTest, 
stream/table/JoinItCase, stream/sql/JoinItCase*
   
   
   ## Verifying this change
   
   Some test cases added in stream/table/JoinTest, stream/sqlJoinTest, 
stream/table/JoinItCase and stream/sql/JoinItCase can verify this change.
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): no
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: no
 - The serializers: no
 - The runtime per-record code paths (performance sensitive): no
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
 - The S3 file system connector: no
   
   ## Documentation
   
 - Does this pull request introduce a new feature? no
 - If yes, how is the feature documented? 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Closed] (FLINK-24137) Python tests fail with "Exception in thread read_grpc_client_inputs"



 [ 
https://issues.apache.org/jira/browse/FLINK-24137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xintong Song closed FLINK-24137.

Resolution: Fixed

Thanks [~dianfu]. I'm closing this ticket for now. We can reopen it if the 
problem happens again.

> Python tests fail with "Exception in thread read_grpc_client_inputs"
> 
>
> Key: FLINK-24137
> URL: https://issues.apache.org/jira/browse/FLINK-24137
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python
>Affects Versions: 1.14.0
>Reporter: Dawid Wysakowicz
>Assignee: Dian Fu
>Priority: Blocker
>  Labels: pull-request-available, test-stability
> Fix For: 1.14.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=23443=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=c67e71ed-6451-5d26-8920-5a8cf9651901=24681
> {code}
> Sep 01 02:26:21 E   Caused by: java.lang.RuntimeException: 
> Failed to create stage bundle factory! INFO:root:Initializing python harness: 
> /__w/1/s/flink-python/pyflink/fn_execution/beam/beam_boot.py --id=1-1 
> --provision_endpoint=localhost:44544
> Sep 01 02:26:21 E   
> Sep 01 02:26:21 E at 
> org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.createStageBundleFactory(BeamPythonFunctionRunner.java:566)
> Sep 01 02:26:21 E at 
> org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.open(BeamPythonFunctionRunner.java:255)
> Sep 01 02:26:21 E at 
> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.open(AbstractPythonFunctionOperator.java:131)
> Sep 01 02:26:21 E at 
> org.apache.flink.streaming.api.operators.python.AbstractOneInputPythonFunctionOperator.open(AbstractOneInputPythonFunctionOperator.java:116)
> Sep 01 02:26:21 E at 
> org.apache.flink.streaming.api.operators.python.PythonProcessOperator.open(PythonProcessOperator.java:59)
> Sep 01 02:26:21 E at 
> org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:110)
> Sep 01 02:26:21 E at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:691)
> Sep 01 02:26:21 E at 
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
> Sep 01 02:26:21 E at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:667)
> Sep 01 02:26:21 E at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:639)
> Sep 01 02:26:21 E at 
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
> Sep 01 02:26:21 E at 
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927)
> Sep 01 02:26:21 E at 
> org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
> Sep 01 02:26:21 E at 
> org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
> Sep 01 02:26:21 E at java.lang.Thread.run(Thread.java:748)
> Sep 01 02:26:21 E   Caused by: 
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>  java.lang.IllegalStateException: Process died with exit code 0
> Sep 01 02:26:21 E at 
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2050)
> Sep 01 02:26:21 E at 
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.get(LocalCache.java:3952)
> Sep 01 02:26:21 E at 
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
> Sep 01 02:26:21 E at 
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)
> Sep 01 02:26:21 E at 
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4964)
> Sep 01 02:26:21 E at 
> org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$SimpleStageBundleFactory.(DefaultJobBundleFactory.java:451)
> Sep 01 02:26:21 E at 
> org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$SimpleStageBundleFactory.(DefaultJobBundleFactory.java:436)
> Sep 01 02:26:21 E at 
>

[jira] [Closed] (FLINK-24212) kerberos krb5.conf file is mounted as empty directory, not the expected file



 [ 
https://issues.apache.org/jira/browse/FLINK-24212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xintong Song closed FLINK-24212.

Resolution: Fixed

Fixed via
- master (1.15): d532f5fc81f9acf611541cab440eb3d5a9de08cc
- release-1.14: 742ba0a171fc909a6e65b9581ce76ea35862c341

> kerberos krb5.conf file is mounted as empty directory, not the expected file
> 
>
> Key: FLINK-24212
> URL: https://issues.apache.org/jira/browse/FLINK-24212
> Project: Flink
>  Issue Type: Bug
>  Components: Deployment / Kubernetes
>Affects Versions: 1.14.0, 1.15.0
>Reporter: LiuZeshan
>Assignee: LiuZeshan
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.14.0, 1.15.0
>
> Attachments: mykrb5conf.yaml
>
>
> From FLINK-18971，we can mount kerberos krb5 conf file to pod with path 
> /etc/krb5.conf，however if the krb5 conf file is not named krb5.conf (e.g 
> named mykrb5.conf)，the mount path /etc/krb5.conf in pod will be an empty 
> directory, not a file that we expect.
> {code:java}
> root@mykrb5-conf-test-6dd5c76f87-vfwh5:/# ls /etc/krb5.conf/ -la
> total 8
> drwxrwxrwx 2 root root 4096 Sep  8 10:42 .
> drwxr-xr-x 1 root root 4096 Sep  8 10:42 ..{code}
>  
>  The reason is that, the code  in 
> [KerberosMountDecrator#decroateFlinkPod|https://github.com/apache/flink/blob/7e91e82fca999ddefa7ebdf198b8cfd3b6998b8b/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/decorators/KerberosMountDecorator.java#L110],
>  we create the deployment like this: 
> {code:java}
> ...
> volumeMounts:
> - mountPath: /etc/krb5.conf
>   name: my-krb5conf-volume
>   subPath: krb5.conf
> ...
>   volumes:
>   - configMap:
>   defaultMode: 420
>   items:
>   - key: mykrb5.conf
> path: mykrb5.conf
>   name: my-krb5conf
> name: my-krb5conf-volume
> {code}
> path value should be set to const value "krb5.conf", not the file name that 
> user provide (path: mykrb5.conf).
>  we can use the yaml description file attachment to reproduce the problem.  
> [^mykrb5conf.yaml]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-24268) Documentation for building PyFlink does not show correctly.



 [ 
https://issues.apache.org/jira/browse/FLINK-24268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dian Fu updated FLINK-24268:

Fix Version/s: (was: 1.14.0)

> Documentation for building PyFlink does not show correctly.
> ---
>
> Key: FLINK-24268
> URL: https://issues.apache.org/jira/browse/FLINK-24268
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python, Documentation
>Affects Versions: 1.14.0
>Reporter: Xintong Song
>Priority: Major
> Attachments: 截屏2021-09-13 下午2.53.41.png
>
>
> https://ci.apache.org/projects/flink/flink-docs-master/docs/flinkdev/building/#build-pyflink
> "The sdist package of xxx will be found under..." where xxx is not showing.
>  !截屏2021-09-13 下午2.53.41.png! 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] xintongsong closed pull request #17198: [FLINK-24212][kerbernets] fix the problem that kerberos krb5.conf file is mounted as empty directory, not a expected file



xintongsong closed pull request #17198:
URL: https://github.com/apache/flink/pull/17198


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-24212) kerberos krb5.conf file is mounted as empty directory, not the expected file



 [ 
https://issues.apache.org/jira/browse/FLINK-24212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xintong Song updated FLINK-24212:
-
Fix Version/s: (was: 1.13.3)

> kerberos krb5.conf file is mounted as empty directory, not the expected file
> 
>
> Key: FLINK-24212
> URL: https://issues.apache.org/jira/browse/FLINK-24212
> Project: Flink
>  Issue Type: Bug
>  Components: Deployment / Kubernetes
>Affects Versions: 1.14.0, 1.15.0
>Reporter: LiuZeshan
>Assignee: LiuZeshan
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.14.0, 1.15.0
>
> Attachments: mykrb5conf.yaml
>
>
> From FLINK-18971，we can mount kerberos krb5 conf file to pod with path 
> /etc/krb5.conf，however if the krb5 conf file is not named krb5.conf (e.g 
> named mykrb5.conf)，the mount path /etc/krb5.conf in pod will be an empty 
> directory, not a file that we expect.
> {code:java}
> root@mykrb5-conf-test-6dd5c76f87-vfwh5:/# ls /etc/krb5.conf/ -la
> total 8
> drwxrwxrwx 2 root root 4096 Sep  8 10:42 .
> drwxr-xr-x 1 root root 4096 Sep  8 10:42 ..{code}
>  
>  The reason is that, the code  in 
> [KerberosMountDecrator#decroateFlinkPod|https://github.com/apache/flink/blob/7e91e82fca999ddefa7ebdf198b8cfd3b6998b8b/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/decorators/KerberosMountDecorator.java#L110],
>  we create the deployment like this: 
> {code:java}
> ...
> volumeMounts:
> - mountPath: /etc/krb5.conf
>   name: my-krb5conf-volume
>   subPath: krb5.conf
> ...
>   volumes:
>   - configMap:
>   defaultMode: 420
>   items:
>   - key: mykrb5.conf
> path: mykrb5.conf
>   name: my-krb5conf
> name: my-krb5conf-volume
> {code}
> path value should be set to const value "krb5.conf", not the file name that 
> user provide (path: mykrb5.conf).
>  we can use the yaml description file attachment to reproduce the problem.  
> [^mykrb5conf.yaml]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-24212) kerberos krb5.conf file is mounted as empty directory, not the expected file



 [ 
https://issues.apache.org/jira/browse/FLINK-24212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xintong Song updated FLINK-24212:
-
Fix Version/s: (was: 1.12.6)

> kerberos krb5.conf file is mounted as empty directory, not the expected file
> 
>
> Key: FLINK-24212
> URL: https://issues.apache.org/jira/browse/FLINK-24212
> Project: Flink
>  Issue Type: Bug
>  Components: Deployment / Kubernetes
>Affects Versions: 1.14.0, 1.12.5, 1.13.2, 1.15.0
>Reporter: LiuZeshan
>Assignee: LiuZeshan
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.14.0, 1.13.3, 1.15.0
>
> Attachments: mykrb5conf.yaml
>
>
> From FLINK-18971，we can mount kerberos krb5 conf file to pod with path 
> /etc/krb5.conf，however if the krb5 conf file is not named krb5.conf (e.g 
> named mykrb5.conf)，the mount path /etc/krb5.conf in pod will be an empty 
> directory, not a file that we expect.
> {code:java}
> root@mykrb5-conf-test-6dd5c76f87-vfwh5:/# ls /etc/krb5.conf/ -la
> total 8
> drwxrwxrwx 2 root root 4096 Sep  8 10:42 .
> drwxr-xr-x 1 root root 4096 Sep  8 10:42 ..{code}
>  
>  The reason is that, the code  in 
> [KerberosMountDecrator#decroateFlinkPod|https://github.com/apache/flink/blob/7e91e82fca999ddefa7ebdf198b8cfd3b6998b8b/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/decorators/KerberosMountDecorator.java#L110],
>  we create the deployment like this: 
> {code:java}
> ...
> volumeMounts:
> - mountPath: /etc/krb5.conf
>   name: my-krb5conf-volume
>   subPath: krb5.conf
> ...
>   volumes:
>   - configMap:
>   defaultMode: 420
>   items:
>   - key: mykrb5.conf
> path: mykrb5.conf
>   name: my-krb5conf
> name: my-krb5conf-volume
> {code}
> path value should be set to const value "krb5.conf", not the file name that 
> user provide (path: mykrb5.conf).
>  we can use the yaml description file attachment to reproduce the problem.  
> [^mykrb5conf.yaml]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-24212) kerberos krb5.conf file is mounted as empty directory, not the expected file



 [ 
https://issues.apache.org/jira/browse/FLINK-24212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xintong Song updated FLINK-24212:
-
Fix Version/s: 1.12.6

> kerberos krb5.conf file is mounted as empty directory, not the expected file
> 
>
> Key: FLINK-24212
> URL: https://issues.apache.org/jira/browse/FLINK-24212
> Project: Flink
>  Issue Type: Bug
>  Components: Deployment / Kubernetes
>Affects Versions: 1.14.0, 1.12.5, 1.13.2, 1.15.0
>Reporter: LiuZeshan
>Assignee: LiuZeshan
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.14.0, 1.12.6, 1.13.3, 1.15.0
>
> Attachments: mykrb5conf.yaml
>
>
> From FLINK-18971，we can mount kerberos krb5 conf file to pod with path 
> /etc/krb5.conf，however if the krb5 conf file is not named krb5.conf (e.g 
> named mykrb5.conf)，the mount path /etc/krb5.conf in pod will be an empty 
> directory, not a file that we expect.
> {code:java}
> root@mykrb5-conf-test-6dd5c76f87-vfwh5:/# ls /etc/krb5.conf/ -la
> total 8
> drwxrwxrwx 2 root root 4096 Sep  8 10:42 .
> drwxr-xr-x 1 root root 4096 Sep  8 10:42 ..{code}
>  
>  The reason is that, the code  in 
> [KerberosMountDecrator#decroateFlinkPod|https://github.com/apache/flink/blob/7e91e82fca999ddefa7ebdf198b8cfd3b6998b8b/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/decorators/KerberosMountDecorator.java#L110],
>  we create the deployment like this: 
> {code:java}
> ...
> volumeMounts:
> - mountPath: /etc/krb5.conf
>   name: my-krb5conf-volume
>   subPath: krb5.conf
> ...
>   volumes:
>   - configMap:
>   defaultMode: 420
>   items:
>   - key: mykrb5.conf
> path: mykrb5.conf
>   name: my-krb5conf
> name: my-krb5conf-volume
> {code}
> path value should be set to const value "krb5.conf", not the file name that 
> user provide (path: mykrb5.conf).
>  we can use the yaml description file attachment to reproduce the problem.  
> [^mykrb5conf.yaml]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-24212) kerberos krb5.conf file is mounted as empty directory, not the expected file



 [ 
https://issues.apache.org/jira/browse/FLINK-24212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xintong Song updated FLINK-24212:
-
Affects Version/s: (was: 1.13.2)

> kerberos krb5.conf file is mounted as empty directory, not the expected file
> 
>
> Key: FLINK-24212
> URL: https://issues.apache.org/jira/browse/FLINK-24212
> Project: Flink
>  Issue Type: Bug
>  Components: Deployment / Kubernetes
>Affects Versions: 1.14.0, 1.15.0
>Reporter: LiuZeshan
>Assignee: LiuZeshan
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.14.0, 1.13.3, 1.15.0
>
> Attachments: mykrb5conf.yaml
>
>
> From FLINK-18971，we can mount kerberos krb5 conf file to pod with path 
> /etc/krb5.conf，however if the krb5 conf file is not named krb5.conf (e.g 
> named mykrb5.conf)，the mount path /etc/krb5.conf in pod will be an empty 
> directory, not a file that we expect.
> {code:java}
> root@mykrb5-conf-test-6dd5c76f87-vfwh5:/# ls /etc/krb5.conf/ -la
> total 8
> drwxrwxrwx 2 root root 4096 Sep  8 10:42 .
> drwxr-xr-x 1 root root 4096 Sep  8 10:42 ..{code}
>  
>  The reason is that, the code  in 
> [KerberosMountDecrator#decroateFlinkPod|https://github.com/apache/flink/blob/7e91e82fca999ddefa7ebdf198b8cfd3b6998b8b/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/decorators/KerberosMountDecorator.java#L110],
>  we create the deployment like this: 
> {code:java}
> ...
> volumeMounts:
> - mountPath: /etc/krb5.conf
>   name: my-krb5conf-volume
>   subPath: krb5.conf
> ...
>   volumes:
>   - configMap:
>   defaultMode: 420
>   items:
>   - key: mykrb5.conf
> path: mykrb5.conf
>   name: my-krb5conf
> name: my-krb5conf-volume
> {code}
> path value should be set to const value "krb5.conf", not the file name that 
> user provide (path: mykrb5.conf).
>  we can use the yaml description file attachment to reproduce the problem.  
> [^mykrb5conf.yaml]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-24212) kerberos krb5.conf file is mounted as empty directory, not the expected file



 [ 
https://issues.apache.org/jira/browse/FLINK-24212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xintong Song updated FLINK-24212:
-
Affects Version/s: (was: 1.12.5)

> kerberos krb5.conf file is mounted as empty directory, not the expected file
> 
>
> Key: FLINK-24212
> URL: https://issues.apache.org/jira/browse/FLINK-24212
> Project: Flink
>  Issue Type: Bug
>  Components: Deployment / Kubernetes
>Affects Versions: 1.14.0, 1.13.2, 1.15.0
>Reporter: LiuZeshan
>Assignee: LiuZeshan
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.14.0, 1.13.3, 1.15.0
>
> Attachments: mykrb5conf.yaml
>
>
> From FLINK-18971，we can mount kerberos krb5 conf file to pod with path 
> /etc/krb5.conf，however if the krb5 conf file is not named krb5.conf (e.g 
> named mykrb5.conf)，the mount path /etc/krb5.conf in pod will be an empty 
> directory, not a file that we expect.
> {code:java}
> root@mykrb5-conf-test-6dd5c76f87-vfwh5:/# ls /etc/krb5.conf/ -la
> total 8
> drwxrwxrwx 2 root root 4096 Sep  8 10:42 .
> drwxr-xr-x 1 root root 4096 Sep  8 10:42 ..{code}
>  
>  The reason is that, the code  in 
> [KerberosMountDecrator#decroateFlinkPod|https://github.com/apache/flink/blob/7e91e82fca999ddefa7ebdf198b8cfd3b6998b8b/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/decorators/KerberosMountDecorator.java#L110],
>  we create the deployment like this: 
> {code:java}
> ...
> volumeMounts:
> - mountPath: /etc/krb5.conf
>   name: my-krb5conf-volume
>   subPath: krb5.conf
> ...
>   volumes:
>   - configMap:
>   defaultMode: 420
>   items:
>   - key: mykrb5.conf
> path: mykrb5.conf
>   name: my-krb5conf
> name: my-krb5conf-volume
> {code}
> path value should be set to const value "krb5.conf", not the file name that 
> user provide (path: mykrb5.conf).
>  we can use the yaml description file attachment to reproduce the problem.  
> [^mykrb5conf.yaml]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-24212) kerberos krb5.conf file is mounted as empty directory, not the expected file



 [ 
https://issues.apache.org/jira/browse/FLINK-24212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xintong Song updated FLINK-24212:
-
Affects Version/s: (was: 1.13.1)
   (was: 1.12.1)
   1.15.0
   1.14.0
   1.12.5

> kerberos krb5.conf file is mounted as empty directory, not the expected file
> 
>
> Key: FLINK-24212
> URL: https://issues.apache.org/jira/browse/FLINK-24212
> Project: Flink
>  Issue Type: Bug
>  Components: Deployment / Kubernetes
>Affects Versions: 1.14.0, 1.12.5, 1.13.2, 1.15.0
>Reporter: LiuZeshan
>Assignee: LiuZeshan
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.14.1
>
> Attachments: mykrb5conf.yaml
>
>
> From FLINK-18971，we can mount kerberos krb5 conf file to pod with path 
> /etc/krb5.conf，however if the krb5 conf file is not named krb5.conf (e.g 
> named mykrb5.conf)，the mount path /etc/krb5.conf in pod will be an empty 
> directory, not a file that we expect.
> {code:java}
> root@mykrb5-conf-test-6dd5c76f87-vfwh5:/# ls /etc/krb5.conf/ -la
> total 8
> drwxrwxrwx 2 root root 4096 Sep  8 10:42 .
> drwxr-xr-x 1 root root 4096 Sep  8 10:42 ..{code}
>  
>  The reason is that, the code  in 
> [KerberosMountDecrator#decroateFlinkPod|https://github.com/apache/flink/blob/7e91e82fca999ddefa7ebdf198b8cfd3b6998b8b/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/decorators/KerberosMountDecorator.java#L110],
>  we create the deployment like this: 
> {code:java}
> ...
> volumeMounts:
> - mountPath: /etc/krb5.conf
>   name: my-krb5conf-volume
>   subPath: krb5.conf
> ...
>   volumes:
>   - configMap:
>   defaultMode: 420
>   items:
>   - key: mykrb5.conf
> path: mykrb5.conf
>   name: my-krb5conf
> name: my-krb5conf-volume
> {code}
> path value should be set to const value "krb5.conf", not the file name that 
> user provide (path: mykrb5.conf).
>  we can use the yaml description file attachment to reproduce the problem.  
> [^mykrb5conf.yaml]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-24212) kerberos krb5.conf file is mounted as empty directory, not the expected file



 [ 
https://issues.apache.org/jira/browse/FLINK-24212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xintong Song updated FLINK-24212:
-
Fix Version/s: (was: 1.14.1)
   1.15.0
   1.14.0

> kerberos krb5.conf file is mounted as empty directory, not the expected file
> 
>
> Key: FLINK-24212
> URL: https://issues.apache.org/jira/browse/FLINK-24212
> Project: Flink
>  Issue Type: Bug
>  Components: Deployment / Kubernetes
>Affects Versions: 1.14.0, 1.12.5, 1.13.2, 1.15.0
>Reporter: LiuZeshan
>Assignee: LiuZeshan
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.14.0, 1.15.0
>
> Attachments: mykrb5conf.yaml
>
>
> From FLINK-18971，we can mount kerberos krb5 conf file to pod with path 
> /etc/krb5.conf，however if the krb5 conf file is not named krb5.conf (e.g 
> named mykrb5.conf)，the mount path /etc/krb5.conf in pod will be an empty 
> directory, not a file that we expect.
> {code:java}
> root@mykrb5-conf-test-6dd5c76f87-vfwh5:/# ls /etc/krb5.conf/ -la
> total 8
> drwxrwxrwx 2 root root 4096 Sep  8 10:42 .
> drwxr-xr-x 1 root root 4096 Sep  8 10:42 ..{code}
>  
>  The reason is that, the code  in 
> [KerberosMountDecrator#decroateFlinkPod|https://github.com/apache/flink/blob/7e91e82fca999ddefa7ebdf198b8cfd3b6998b8b/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/decorators/KerberosMountDecorator.java#L110],
>  we create the deployment like this: 
> {code:java}
> ...
> volumeMounts:
> - mountPath: /etc/krb5.conf
>   name: my-krb5conf-volume
>   subPath: krb5.conf
> ...
>   volumes:
>   - configMap:
>   defaultMode: 420
>   items:
>   - key: mykrb5.conf
> path: mykrb5.conf
>   name: my-krb5conf
> name: my-krb5conf-volume
> {code}
> path value should be set to const value "krb5.conf", not the file name that 
> user provide (path: mykrb5.conf).
>  we can use the yaml description file attachment to reproduce the problem.  
> [^mykrb5conf.yaml]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-24212) kerberos krb5.conf file is mounted as empty directory, not the expected file



 [ 
https://issues.apache.org/jira/browse/FLINK-24212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xintong Song updated FLINK-24212:
-
Fix Version/s: 1.13.3

> kerberos krb5.conf file is mounted as empty directory, not the expected file
> 
>
> Key: FLINK-24212
> URL: https://issues.apache.org/jira/browse/FLINK-24212
> Project: Flink
>  Issue Type: Bug
>  Components: Deployment / Kubernetes
>Affects Versions: 1.14.0, 1.12.5, 1.13.2, 1.15.0
>Reporter: LiuZeshan
>Assignee: LiuZeshan
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.14.0, 1.13.3, 1.15.0
>
> Attachments: mykrb5conf.yaml
>
>
> From FLINK-18971，we can mount kerberos krb5 conf file to pod with path 
> /etc/krb5.conf，however if the krb5 conf file is not named krb5.conf (e.g 
> named mykrb5.conf)，the mount path /etc/krb5.conf in pod will be an empty 
> directory, not a file that we expect.
> {code:java}
> root@mykrb5-conf-test-6dd5c76f87-vfwh5:/# ls /etc/krb5.conf/ -la
> total 8
> drwxrwxrwx 2 root root 4096 Sep  8 10:42 .
> drwxr-xr-x 1 root root 4096 Sep  8 10:42 ..{code}
>  
>  The reason is that, the code  in 
> [KerberosMountDecrator#decroateFlinkPod|https://github.com/apache/flink/blob/7e91e82fca999ddefa7ebdf198b8cfd3b6998b8b/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/decorators/KerberosMountDecorator.java#L110],
>  we create the deployment like this: 
> {code:java}
> ...
> volumeMounts:
> - mountPath: /etc/krb5.conf
>   name: my-krb5conf-volume
>   subPath: krb5.conf
> ...
>   volumes:
>   - configMap:
>   defaultMode: 420
>   items:
>   - key: mykrb5.conf
> path: mykrb5.conf
>   name: my-krb5conf
> name: my-krb5conf-volume
> {code}
> path value should be set to const value "krb5.conf", not the file name that 
> user provide (path: mykrb5.conf).
>  we can use the yaml description file attachment to reproduce the problem.  
> [^mykrb5conf.yaml]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (FLINK-24212) kerberos krb5.conf file is mounted as empty directory, not the expected file



 [ 
https://issues.apache.org/jira/browse/FLINK-24212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xintong Song updated FLINK-24212:
-
Priority: Major  (was: Minor)

> kerberos krb5.conf file is mounted as empty directory, not the expected file
> 
>
> Key: FLINK-24212
> URL: https://issues.apache.org/jira/browse/FLINK-24212
> Project: Flink
>  Issue Type: Bug
>  Components: Deployment / Kubernetes
>Affects Versions: 1.12.1, 1.13.1, 1.13.2
>Reporter: LiuZeshan
>Assignee: LiuZeshan
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.14.1
>
> Attachments: mykrb5conf.yaml
>
>
> From FLINK-18971，we can mount kerberos krb5 conf file to pod with path 
> /etc/krb5.conf，however if the krb5 conf file is not named krb5.conf (e.g 
> named mykrb5.conf)，the mount path /etc/krb5.conf in pod will be an empty 
> directory, not a file that we expect.
> {code:java}
> root@mykrb5-conf-test-6dd5c76f87-vfwh5:/# ls /etc/krb5.conf/ -la
> total 8
> drwxrwxrwx 2 root root 4096 Sep  8 10:42 .
> drwxr-xr-x 1 root root 4096 Sep  8 10:42 ..{code}
>  
>  The reason is that, the code  in 
> [KerberosMountDecrator#decroateFlinkPod|https://github.com/apache/flink/blob/7e91e82fca999ddefa7ebdf198b8cfd3b6998b8b/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/decorators/KerberosMountDecorator.java#L110],
>  we create the deployment like this: 
> {code:java}
> ...
> volumeMounts:
> - mountPath: /etc/krb5.conf
>   name: my-krb5conf-volume
>   subPath: krb5.conf
> ...
>   volumes:
>   - configMap:
>   defaultMode: 420
>   items:
>   - key: mykrb5.conf
> path: mykrb5.conf
>   name: my-krb5conf
> name: my-krb5conf-volume
> {code}
> path value should be set to const value "krb5.conf", not the file name that 
> user provide (path: mykrb5.conf).
>  we can use the yaml description file attachment to reproduce the problem.  
> [^mykrb5conf.yaml]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-24268) Documentation for building PyFlink does not show correctly.



[ 
https://issues.apache.org/jira/browse/FLINK-24268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414666#comment-17414666
 ] 

Xintong Song commented on FLINK-24268:
--

True, I need to update my browser bookmarks XD.
Closing this.

> Documentation for building PyFlink does not show correctly.
> ---
>
> Key: FLINK-24268
> URL: https://issues.apache.org/jira/browse/FLINK-24268
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python, Documentation
>Affects Versions: 1.14.0
>Reporter: Xintong Song
>Priority: Major
> Fix For: 1.14.0
>
> Attachments: 截屏2021-09-13 下午2.53.41.png
>
>
> https://ci.apache.org/projects/flink/flink-docs-master/docs/flinkdev/building/#build-pyflink
> "The sdist package of xxx will be found under..." where xxx is not showing.
>  !截屏2021-09-13 下午2.53.41.png! 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Closed] (FLINK-24268) Documentation for building PyFlink does not show correctly.



 [ 
https://issues.apache.org/jira/browse/FLINK-24268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xintong Song closed FLINK-24268.

Resolution: Invalid

> Documentation for building PyFlink does not show correctly.
> ---
>
> Key: FLINK-24268
> URL: https://issues.apache.org/jira/browse/FLINK-24268
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python, Documentation
>Affects Versions: 1.14.0
>Reporter: Xintong Song
>Priority: Major
> Fix For: 1.14.0
>
> Attachments: 截屏2021-09-13 下午2.53.41.png
>
>
> https://ci.apache.org/projects/flink/flink-docs-master/docs/flinkdev/building/#build-pyflink
> "The sdist package of xxx will be found under..." where xxx is not showing.
>  !截屏2021-09-13 下午2.53.41.png! 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot edited a comment on pull request #17270: FLINK-23969 Test Pulsar source end 2 end



flinkbot edited a comment on pull request #17270:
URL: https://github.com/apache/flink/pull/17270#issuecomment-918480494


   
   ## CI report:
   
   * 4dea0cfa8f0f164e0adef4f606f4b20f7327032c Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24035)
 
   * 05edbd6128a55c4e10812eb405216d7efd79bbcf UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Closed] (FLINK-24206) PulsarSourceITCase fails with "Consumer not found"



 [ 
https://issues.apache.org/jira/browse/FLINK-24206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xintong Song closed FLINK-24206.

Resolution: Fixed

Fixed via
- master (1.15): 5474f89c8e39d97b6f1b94cbbae3d2c9dd116bf6
- release-1.14: 28677f34bad1ffb4f405a4550536969eb9621d05

> PulsarSourceITCase fails with "Consumer not found"
> --
>
> Key: FLINK-24206
> URL: https://issues.apache.org/jira/browse/FLINK-24206
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / Pulsar
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Dawid Wysakowicz
>Assignee: Yufan Sheng
>Priority: Major
>  Labels: pull-request-available, test-stability
> Fix For: 1.14.0, 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=23732=logs=fc5181b0-e452-5c8f-68de-1097947f6483=995c650b-6573-581c-9ce6-7ad4cc038461=24650
> {code}
> Sep 08 05:08:35 Caused by: org.apache.flink.runtime.JobException: Recovery is 
> suppressed by NoRestartBackoffTimeStrategy
> Sep 08 05:08:35   at 
> org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:138)
> Sep 08 05:08:35   at 
> org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:82)
> Sep 08 05:08:35   at 
> org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:228)
> Sep 08 05:08:35   at 
> org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:218)
> Sep 08 05:08:35   at 
> org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:209)
> Sep 08 05:08:35   at 
> org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:679)
> Sep 08 05:08:35   at 
> org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:79)
> Sep 08 05:08:35   at 
> org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:444)
> Sep 08 05:08:35   at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown 
> Source)
> Sep 08 05:08:35   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Sep 08 05:08:35   at java.lang.reflect.Method.invoke(Method.java:498)
> Sep 08 05:08:35   at 
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRpcInvocation$1(AkkaRpcActor.java:316)
> Sep 08 05:08:35   at 
> org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
> Sep 08 05:08:35   at 
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:314)
> Sep 08 05:08:35   at 
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:217)
> Sep 08 05:08:35   at 
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:78)
> Sep 08 05:08:35   at 
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:163)
> Sep 08 05:08:35   at 
> akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24)
> Sep 08 05:08:35   at 
> akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20)
> Sep 08 05:08:35   at 
> scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
> Sep 08 05:08:35   at 
> scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
> Sep 08 05:08:35   at 
> akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20)
> Sep 08 05:08:35   at 
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> Sep 08 05:08:35   at 
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
> Sep 08 05:08:35   at 
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
> Sep 08 05:08:35   at akka.actor.Actor.aroundReceive(Actor.scala:537)
> Sep 08 05:08:35   at akka.actor.Actor.aroundReceive$(Actor.scala:535)
> Sep 08 05:08:35   at 
> akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220)
> Sep 08 05:08:35   at 
> akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
> Sep 08 05:08:35   at akka.actor.ActorCell.invoke(ActorCell.scala:548)
> Sep 08 05:08:35   at 
> akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
> Sep 08 05:08:35   at akka.dispatch.Mailbox.run(Mailbox.scala:231)
> Sep 08 05:08:35   at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
> Sep 08 05:08:35   at 
> java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> Sep 08 05:08:35   at 
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> Sep 08 05:08:35   at 
>

[GitHub] [flink] xintongsong closed pull request #17255: [FLINK-24206][connector/pulsar] Close the pulsar client properly



xintongsong closed pull request #17255:
URL: https://github.com/apache/flink/pull/17255


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-24206) PulsarSourceITCase fails with "Consumer not found"



 [ 
https://issues.apache.org/jira/browse/FLINK-24206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xintong Song updated FLINK-24206:
-
Fix Version/s: (was: 1.15)
   1.15.0

> PulsarSourceITCase fails with "Consumer not found"
> --
>
> Key: FLINK-24206
> URL: https://issues.apache.org/jira/browse/FLINK-24206
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / Pulsar
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Dawid Wysakowicz
>Assignee: Yufan Sheng
>Priority: Major
>  Labels: pull-request-available, test-stability
> Fix For: 1.14.0, 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=23732=logs=fc5181b0-e452-5c8f-68de-1097947f6483=995c650b-6573-581c-9ce6-7ad4cc038461=24650
> {code}
> Sep 08 05:08:35 Caused by: org.apache.flink.runtime.JobException: Recovery is 
> suppressed by NoRestartBackoffTimeStrategy
> Sep 08 05:08:35   at 
> org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:138)
> Sep 08 05:08:35   at 
> org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:82)
> Sep 08 05:08:35   at 
> org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:228)
> Sep 08 05:08:35   at 
> org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:218)
> Sep 08 05:08:35   at 
> org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:209)
> Sep 08 05:08:35   at 
> org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:679)
> Sep 08 05:08:35   at 
> org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:79)
> Sep 08 05:08:35   at 
> org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:444)
> Sep 08 05:08:35   at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown 
> Source)
> Sep 08 05:08:35   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Sep 08 05:08:35   at java.lang.reflect.Method.invoke(Method.java:498)
> Sep 08 05:08:35   at 
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRpcInvocation$1(AkkaRpcActor.java:316)
> Sep 08 05:08:35   at 
> org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
> Sep 08 05:08:35   at 
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:314)
> Sep 08 05:08:35   at 
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:217)
> Sep 08 05:08:35   at 
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:78)
> Sep 08 05:08:35   at 
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:163)
> Sep 08 05:08:35   at 
> akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24)
> Sep 08 05:08:35   at 
> akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20)
> Sep 08 05:08:35   at 
> scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
> Sep 08 05:08:35   at 
> scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
> Sep 08 05:08:35   at 
> akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20)
> Sep 08 05:08:35   at 
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> Sep 08 05:08:35   at 
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
> Sep 08 05:08:35   at 
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
> Sep 08 05:08:35   at akka.actor.Actor.aroundReceive(Actor.scala:537)
> Sep 08 05:08:35   at akka.actor.Actor.aroundReceive$(Actor.scala:535)
> Sep 08 05:08:35   at 
> akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220)
> Sep 08 05:08:35   at 
> akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
> Sep 08 05:08:35   at akka.actor.ActorCell.invoke(ActorCell.scala:548)
> Sep 08 05:08:35   at 
> akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
> Sep 08 05:08:35   at akka.dispatch.Mailbox.run(Mailbox.scala:231)
> Sep 08 05:08:35   at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
> Sep 08 05:08:35   at 
> java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> Sep 08 05:08:35   at 
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> Sep 08 05:08:35   at 
> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> Sep 08 05:08:35   at 
>

[GitHub] [flink] flinkbot edited a comment on pull request #17250: [FLINK-24266][checkpoint] Log improvement for aborting checkpoint due to tasks are finishing



flinkbot edited a comment on pull request #17250:
URL: https://github.com/apache/flink/pull/17250#issuecomment-917910682


   
   ## CI report:
   
   * 56d6e1ca4607d829725d200595e05fd044ba34d7 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23980)
 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24041)
 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24022)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Comment Edited] (FLINK-24269) Rename methods around final checkpoints

2021-09-13 Thread liwei li (Jira)



[ 
https://issues.apache.org/jira/browse/FLINK-24269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414224#comment-17414224
 ] 

liwei li edited comment on FLINK-24269 at 9/14/21, 1:12 AM:


Is it just renaming? Can I try it? Meanwhile, by modifying it, I can learn more 
about the mechanism of checkpoint. thx.


was (Author: liliwei):
Is it just renaming? Can I try it? Meanwhile, I just want to know more about 
the mechanism of checkpoint. thx.

> Rename methods around final checkpoints
> ---
>
> Key: FLINK-24269
> URL: https://issues.apache.org/jira/browse/FLINK-24269
> Project: Flink
>  Issue Type: Sub-task
>  Components: Runtime / Checkpointing
>Reporter: Dawid Wysakowicz
>Priority: Minor
> Fix For: 1.15.0
>
>
> We should rename:
> * {{TaskStateSnapshot.isFinishedOnRestore()}} to {{isTaskDeployedAsFinished}}
> * {{TaskStateSnapshot.isOperatorsFinished()}} to {{isTaskFinished}}
> * {{PendingCheckpoint#updateNonFinishedOnRestoreOperatorState}} to 
> {{updateOperatorState}}
> For context see: 
> https://github.com/apache/flink/pull/16655#issuecomment-899603149



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot edited a comment on pull request #17269: (1.14) [FLINK-24266][checkpoint] Log improvement for aborting checkpoint due to tasks are finishing



flinkbot edited a comment on pull request #17269:
URL: https://github.com/apache/flink/pull/17269#issuecomment-918355258


   
   ## CI report:
   
   * 8d3946fa4502bdeab94448c26f9c7d29842064dc Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24032)
 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24042)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17250: [FLINK-24266][checkpoint] Log improvement for aborting checkpoint due to tasks are finishing



flinkbot edited a comment on pull request #17250:
URL: https://github.com/apache/flink/pull/17250#issuecomment-917910682


   
   ## CI report:
   
   * 56d6e1ca4607d829725d200595e05fd044ba34d7 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24022)
 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23980)
 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24041)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] gaoyunhaii commented on pull request #17250: [FLINK-24266][checkpoint] Log improvement for aborting checkpoint due to tasks are finishing



gaoyunhaii commented on pull request #17250:
URL: https://github.com/apache/flink/pull/17250#issuecomment-918696753


   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] gaoyunhaii commented on pull request #17269: (1.14) [FLINK-24266][checkpoint] Log improvement for aborting checkpoint due to tasks are finishing



gaoyunhaii commented on pull request #17269:
URL: https://github.com/apache/flink/pull/17269#issuecomment-918696608


   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17253: [FLINK-24182][task] Allow tasks to cleanly cancel without any interruptions



flinkbot edited a comment on pull request #17253:
URL: https://github.com/apache/flink/pull/17253#issuecomment-917975072


   
   ## CI report:
   
   * d843dafa8429067004b68759a7187787635d Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24034)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #17271: [FLINK-23969][connector/pulsar] Create e2e tests for pulsar connector.



flinkbot edited a comment on pull request #17271:
URL: https://github.com/apache/flink/pull/17271#issuecomment-918597722


   
   ## CI report:
   
   * 2e8733c07113a9fdede210b630f03df5349004e1 UNKNOWN
   * d818f8650d4c8a1d923276eac3f5d2dd373cc450 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24040)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-23397) [DOCS] task_failure_recovery page return 404

2021-09-13 Thread Flink Jira Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/FLINK-23397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Flink Jira Bot updated FLINK-23397:
---
Labels: pull-request-available stale-major  (was: pull-request-available)

I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help 
the community manage its development. I see this issues has been marked as 
Major but is unassigned and neither itself nor its Sub-Tasks have been updated 
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this 
ticket is a Major, please either assign yourself or give an update. Afterwards, 
please remove the label or in 7 days the issue will be deprioritized.


> [DOCS] task_failure_recovery page return 404
> 
>
> Key: FLINK-23397
> URL: https://issues.apache.org/jira/browse/FLINK-23397
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Reporter: Dino Zhang
>Priority: Major
>  Labels: pull-request-available, stale-major
> Attachments: image-2021-07-15-17-32-39-423.png
>
>
>  
> [https://ci.apache.org/projects/flink/flink-docs-master/docs/ops/state/task_failure_recovery/]
>  
> [https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/config/#advanced-fault-tolerance-options]
> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/config/#full-jobmanager-options
>  
>  
> In the above page, clicking here will return a 404
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot edited a comment on pull request #17271: [FLINK-23969][connector/pulsar] Create e2e tests for pulsar connector.