[jira] [Created] (FLINK-33602) Pulsar connector should be compatible with Flink 1.18

[Zili Chen (Jira)]

Zili Chen created FLINK-33602:
-

 Summary: Pulsar connector should be compatible with Flink 1.18
 Key: FLINK-33602
 URL: https://issues.apache.org/jira/browse/FLINK-33602
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Pulsar
Affects Versions: 1.18.0, pulsar-4.1.0
Reporter: Zili Chen
Assignee: Zili Chen


Currently, the build and test job always fails - 
https://github.com/apache/flink-connector-pulsar/actions/runs/6937440214



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-32625) MiniClusterTestEnvironment supports customized MiniClusterResourceConfiguration

[Zili Chen (Jira)]

Zili Chen created FLINK-32625:
-

 Summary: MiniClusterTestEnvironment supports customized 
MiniClusterResourceConfiguration
 Key: FLINK-32625
 URL: https://issues.apache.org/jira/browse/FLINK-32625
 Project: Flink
  Issue Type: Improvement
  Components: Test Infrastructure
Reporter: Zili Chen
Assignee: Zili Chen






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-32612) Upgrade Pulsar version to 3.0.0 for the connector

[Zili Chen (Jira)]

Zili Chen created FLINK-32612:
-

 Summary: Upgrade Pulsar version to 3.0.0 for the connector
 Key: FLINK-32612
 URL: https://issues.apache.org/jira/browse/FLINK-32612
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Pulsar
Reporter: Zili Chen
Assignee: Zili Chen






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-32585) Filter javax.xml.bind:jaxb-api false positive for Pulsar connector

[Zili Chen (Jira)]

Zili Chen created FLINK-32585:
-

 Summary: Filter javax.xml.bind:jaxb-api false positive for Pulsar 
connector
 Key: FLINK-32585
 URL: https://issues.apache.org/jira/browse/FLINK-32585
 Project: Flink
  Issue Type: Improvement
  Components: Build System / CI
Reporter: Zili Chen
Assignee: Zili Chen






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release flink-connector-pulsar 3.0.1, release candidate #1

[Zili Chen]

+1

I verified

+ LICENSE and NOTICE present
+ Checksum and GPG sign matches
+ No unexpected binaries in the source release
+ Build from source and run unit tests with JDK 17 on macOS M1

On 2023/05/25 16:18:51 Leonard Xu wrote:
> Hey all,
> 
> Please review and vote on the release candidate #1 for the version 3.0.1 of 
> the
> Apache Flink Pulsar Connector as follows:
> 
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
> 
> The complete staging area is available for your review, which includes:
> JIRA release notes [1],
> The official Apache source release to be deployed to dist.apache.org [2], 
> which are signed with the key with 
> fingerprint5B2F6608732389AEB67331F5B197E1F1108998AD [3],
> All artifacts to be deployed to the Maven Central Repository [4],
> Source code tag v3.0.1-rc1 [5],
> Website pull request listing the new release [6].
> The vote will be open for at least 72 hours. It is adopted by majority 
> approval, with at least 3 PMC affirmative votes.
> 
> 
> Best,
> Leonard
> 
> [1] 
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352640
> [2] 
> https://dist.apache.org/repos/dist/dev/flink/flink-connector-pulsar-3.0.1-rc1/
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4] https://repository.apache.org/content/repositories/orgapacheflink-1641/
> [5] https://github.com/apache/flink-connector-pulsar/tree/v3.0.1-rc1
> [6] https://github.com/apache/flink-web/pull/655
> 
> 

[jira] [Created] (FLINK-31748) Adapt SplitFetcherManager#removeSplit for flink-connector-pulsar

[Zili Chen (Jira)]

Zili Chen created FLINK-31748:
-

 Summary: Adapt SplitFetcherManager#removeSplit for 
flink-connector-pulsar
 Key: FLINK-31748
 URL: https://issues.apache.org/jira/browse/FLINK-31748
 Project: Flink
  Issue Type: Sub-task
Reporter: Zili Chen
Assignee: Yufan Sheng






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31206) Broken links on flink.apache.org

[Zili Chen (Jira)]

Zili Chen created FLINK-31206:
-

 Summary: Broken links on flink.apache.org
 Key: FLINK-31206
 URL: https://issues.apache.org/jira/browse/FLINK-31206
 Project: Flink
  Issue Type: Bug
Reporter: Zili Chen


Previously page link 
https://flink.apache.org/contribute/code-style-and-quality/preamble/ is broken, 
new link is 
https://flink.apache.org/how-to-contribute/code-style-and-quality-preamble/.

Shall we set up a redirection or just let those broken links wait for 
maintainers fixing?

cc [~martijnvisser]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29436) Upgrade Spotless Maven Plugin to 2.27.0 for running on Java 17

[Zili Chen (Jira)]

Zili Chen created FLINK-29436:
-

 Summary: Upgrade Spotless Maven Plugin to 2.27.0 for running on 
Java 17
 Key: FLINK-29436
 URL: https://issues.apache.org/jira/browse/FLINK-29436
 Project: Flink
  Issue Type: Bug
Reporter: Zili Chen
Assignee: Zili Chen


This blocker is fixed by: https://github.com/diffplug/spotless/pull/1224 and 
https://github.com/diffplug/spotless/pull/1228.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-26214) Publish Kubernetes operator to container registry

[Zili Chen (Jira)]

Zili Chen created FLINK-26214:
-

 Summary: Publish Kubernetes operator to container registry
 Key: FLINK-26214
 URL: https://issues.apache.org/jira/browse/FLINK-26214
 Project: Flink
  Issue Type: Sub-task
Reporter: Zili Chen


Created from 
https://github.com/apache/flink-kubernetes-operator/pull/4#issuecomment-1042717476.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-20139) Enrich logs when MiniDispatcher shutting down

[Zili Chen (Jira)]

Zili Chen created FLINK-20139:
-

 Summary: Enrich logs when MiniDispatcher shutting down
 Key: FLINK-20139
 URL: https://issues.apache.org/jira/browse/FLINK-20139
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Reporter: Zili Chen
Assignee: Zili Chen






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16637) Flink YARN app terminated before the client receives the result

[Zili Chen (Jira)]

Zili Chen created FLINK-16637:
-

 Summary: Flink YARN app terminated before the client receives the 
result
 Key: FLINK-16637
 URL: https://issues.apache.org/jira/browse/FLINK-16637
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.10.0
Reporter: Zili Chen
 Attachments: yarn.log

See also 
[https://lists.apache.org/x/thread.html/rcadbd6ceede422bac8d4483fd0cdae58659fbff78533a399eb136743@%3Cdev.flink.apache.org%3E]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16562) Handle JobManager termination future in place

[Zili Chen (Jira)]

Zili Chen created FLINK-16562:
-

 Summary: Handle JobManager termination future in place
 Key: FLINK-16562
 URL: https://issues.apache.org/jira/browse/FLINK-16562
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.11.0


After FLINK-11843 {{Dispatcher}} becomes a {{PermanentlyFencedRpcEndpoint}} and 
will be created as different instance in difference leader epoch. Thus, we 
don't have {{jobManagerTerminationFutures}} crosses multiple leader epoch that 
should be handled. Given the truth, we can remove 
{{jobManagerTerminationFutures}} field in {{Dispatcher}} and handle those 
futures in place, which will simplify the code and helps on further refactoring.

CC [~trohrmann]

I will create a branch later this week.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16458) Bump japicmp.referenceVersion to 1.10.0

[Zili Chen (Jira)]

Zili Chen created FLINK-16458:
-

 Summary: Bump japicmp.referenceVersion to 1.10.0
 Key: FLINK-16458
 URL: https://issues.apache.org/jira/browse/FLINK-16458
 Project: Flink
  Issue Type: Bug
  Components: Build System
Reporter: Zili Chen
Assignee: Zili Chen






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16427) Remove directly throw ProgramInvocationExceptions in RemoteStreamEnvironment

[Zili Chen (Jira)]

Zili Chen created FLINK-16427:
-

 Summary: Remove directly throw ProgramInvocationExceptions in 
RemoteStreamEnvironment
 Key: FLINK-16427
 URL: https://issues.apache.org/jira/browse/FLINK-16427
 Project: Flink
  Issue Type: Bug
  Components: API / DataStream
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16206) Support JSON_ARRAYAGG for blink planner

[Zili Chen (Jira)]

Zili Chen created FLINK-16206:
-

 Summary: Support JSON_ARRAYAGG for blink planner
 Key: FLINK-16206
 URL: https://issues.apache.org/jira/browse/FLINK-16206
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Zili Chen
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16205) Support JSON_OBJECTAGG for blink planner

[Zili Chen (Jira)]

Zili Chen created FLINK-16205:
-

 Summary: Support JSON_OBJECTAGG for blink planner
 Key: FLINK-16205
 URL: https://issues.apache.org/jira/browse/FLINK-16205
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Zili Chen
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16203) Support JSON_OBJECT for blink planner

[Zili Chen (Jira)]

Zili Chen created FLINK-16203:
-

 Summary: Support JSON_OBJECT for blink planner
 Key: FLINK-16203
 URL: https://issues.apache.org/jira/browse/FLINK-16203
 Project: Flink
  Issue Type: Sub-task
Reporter: Zili Chen
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16202) Support JSON_QUERY for blink planner

[Zili Chen (Jira)]

Zili Chen created FLINK-16202:
-

 Summary: Support JSON_QUERY for blink planner
 Key: FLINK-16202
 URL: https://issues.apache.org/jira/browse/FLINK-16202
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Zili Chen
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16204) Support JSON_ARRAY for blink planner

[Zili Chen (Jira)]

Zili Chen created FLINK-16204:
-

 Summary: Support JSON_ARRAY for blink planner
 Key: FLINK-16204
 URL: https://issues.apache.org/jira/browse/FLINK-16204
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Zili Chen
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16201) Support JSON_VALUE for blink planner

[Zili Chen (Jira)]

Zili Chen created FLINK-16201:
-

 Summary: Support JSON_VALUE for blink planner
 Key: FLINK-16201
 URL: https://issues.apache.org/jira/browse/FLINK-16201
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Zili Chen
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16200) Support JSON_EXISTS for blink planner

[Zili Chen (Jira)]

Zili Chen created FLINK-16200:
-

 Summary: Support JSON_EXISTS for blink planner
 Key: FLINK-16200
 URL: https://issues.apache.org/jira/browse/FLINK-16200
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Zili Chen
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16199) Support IS JSON predicate for blink planner

[Zili Chen (Jira)]

Zili Chen created FLINK-16199:
-

 Summary: Support IS JSON predicate for blink planner
 Key: FLINK-16199
 URL: https://issues.apache.org/jira/browse/FLINK-16199
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16185) Support JSON functions provides by calcite for blink planner

[Zili Chen (Jira)]

Zili Chen created FLINK-16185:
-

 Summary: Support JSON functions provides by calcite for blink 
planner
 Key: FLINK-16185
 URL: https://issues.apache.org/jira/browse/FLINK-16185
 Project: Flink
  Issue Type: New Feature
  Components: Table SQL / API
Reporter: Zili Chen
 Fix For: 1.11.0


Calcite 1.20 provides several built-in JSON functions(listed below). In order 
to enrich our builtin UDFs and according to our internal user experiences I'd 
propose to support those functions for blink planner.

cc [~jark] [~docete] what do you think?

JSON_EXISTS
JSON_QUERY
JSON_OBJECT
JSON_ARRAY
IS_JSON_ARRAY
IS_JSON_OBJECT
IS_JSON_SCALAR
IS_JSON_VALUE
IS_NOT_JSON_ARRAY
IS_NOT_JSON_OBJECT
IS_NOT_JSON_SCALAR
IS_NOT_JSON_VALUE
JSON_VALUE
JSON_VALUE_ANY
JSON_VALUE_EXPRESSION

JSON_OBJECTAGG
JSON_ARRAYAGG



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16143) Turn on more date time functions of blink planner

[Zili Chen (Jira)]

Zili Chen created FLINK-16143:
-

 Summary: Turn on more date time functions of blink planner
 Key: FLINK-16143
 URL: https://issues.apache.org/jira/browse/FLINK-16143
 Project: Flink
  Issue Type: New Feature
  Components: Table SQL / Planner
Reporter: Zili Chen
 Fix For: 1.11.0


Currently blink planner has a series of built-in function such as

DATEDIFF
DATE_ADD
DATE_SUB

which doesn't get into used. We could add the necessary register, generate and 
convert code to make it available in production scope.

 

what do you think [~jark]?

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15977) Update pull request template to include Kubernetes as deployment candidates

[Zili Chen (Jira)]

Zili Chen created FLINK-15977:
-

 Summary: Update pull request template to include Kubernetes as 
deployment candidates
 Key: FLINK-15977
 URL: https://issues.apache.org/jira/browse/FLINK-15977
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Reporter: Zili Chen
Assignee: Zili Chen






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15967) Examples use nightly kafka connector

[Zili Chen (Jira)]

Zili Chen created FLINK-15967:
-

 Summary: Examples use nightly kafka connector
 Key: FLINK-15967
 URL: https://issues.apache.org/jira/browse/FLINK-15967
 Project: Flink
  Issue Type: Improvement
  Components: Examples
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.11.0


{{StateMachineExample}} & {{KafkaEventsGeneratorJob}} still use kafka010 
connector, it would be an improvement we show the ability of nightly connector 
version.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15536) Revert removal of ConfigConstants.YARN_MAX_FAILED_CONTAINERS

[Zili Chen (Jira)]

Zili Chen created FLINK-15536:
-

 Summary: Revert removal of 
ConfigConstants.YARN_MAX_FAILED_CONTAINERS
 Key: FLINK-15536
 URL: https://issues.apache.org/jira/browse/FLINK-15536
 Project: Flink
  Issue Type: Test
  Components: Runtime / Configuration
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.11.0


FLINK-15359 cleans up some constants with no power. However, 
{{ConfigConstants.YARN_MAX_FAILED_CONTAINERS}} inside {{ConfigConstants}} is 
{{Public}} so that we should revert the removal of it for b/w compatibility.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15510) Pretty Print StreamGraph JSON Plan

[Zili Chen (Jira)]

Zili Chen created FLINK-15510:
-

 Summary: Pretty Print StreamGraph JSON Plan
 Key: FLINK-15510
 URL: https://issues.apache.org/jira/browse/FLINK-15510
 Project: Flink
  Issue Type: Improvement
  Components: API / DataStream
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.11.0


Currently we print StreamGraph JSON as string without pretty printer. The 
output is less than awesome. We can simply switch to {{toPrettyString}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15496) Remove RegisterApplicationMasterResponseReflector

[Zili Chen (Jira)]

Zili Chen created FLINK-15496:
-

 Summary: Remove RegisterApplicationMasterResponseReflector
 Key: FLINK-15496
 URL: https://issues.apache.org/jira/browse/FLINK-15496
 Project: Flink
  Issue Type: Improvement
  Components: Deployment / YARN
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.11.0


{{RegisterApplicationMasterResponseReflector}} is no longer needed. Directly 
call {{registerApplicationMasterResponse.getContainersFromPreviousAttempts()}} 
is viable.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15454) ClusterEntrypoint#installSecurityContext is now unique

[Zili Chen (Jira)]

Zili Chen created FLINK-15454:
-

 Summary: ClusterEntrypoint#installSecurityContext is now unique
 Key: FLINK-15454
 URL: https://issues.apache.org/jira/browse/FLINK-15454
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Affects Versions: 1.11.0
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.11.0


So that we can remove some dead/duplicated codes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15359) Remove unused YarnConfigOptions

[Zili Chen (Jira)]

Zili Chen created FLINK-15359:
-

 Summary: Remove unused YarnConfigOptions
 Key: FLINK-15359
 URL: https://issues.apache.org/jira/browse/FLINK-15359
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Configuration
Reporter: Zili Chen
 Fix For: 1.11.0


There are several unused {{YarnConfigOptions}}. Remove them for preventing 
misunderstanding.


- {{yarn.appmaster.rpc.address}}
- {{yarn.appmaster.rpc.port}}
- {{yarn.maximum-failed-containers}}





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15341) Reset context classload in PackagedProgramUtils#createJobGraph

[Zili Chen (Jira)]

Zili Chen created FLINK-15341:
-

 Summary: Reset context classload in 
PackagedProgramUtils#createJobGraph
 Key: FLINK-15341
 URL: https://issues.apache.org/jira/browse/FLINK-15341
 Project: Flink
  Issue Type: Bug
  Components: Client / Job Submission
Affects Versions: 1.10.0
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.10.0, 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15312) Remove PlanExposingEnvironment

[Zili Chen (Jira)]

Zili Chen created FLINK-15312:
-

 Summary: Remove PlanExposingEnvironment
 Key: FLINK-15312
 URL: https://issues.apache.org/jira/browse/FLINK-15312
 Project: Flink
  Issue Type: Sub-task
  Components: API / DataSet
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.11.0


We don't need it. {{env.createProgramPlan}} is sufficient, which is internal 
and can be used in testing.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15299) Move ClientUtils#submitJob & ClientUtils#submitJobAndWaitForResult to flink-test-utils

[Zili Chen (Jira)]

Zili Chen created FLINK-15299:
-

 Summary: Move ClientUtils#submitJob & 
ClientUtils#submitJobAndWaitForResult to flink-test-utils
 Key: FLINK-15299
 URL: https://issues.apache.org/jira/browse/FLINK-15299
 Project: Flink
  Issue Type: Improvement
  Components: Tests
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.11.0


Now we don't use these methods in production any more and it doesn't attend to 
server as productive methods. Move to flink-test-utils.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15297) Do not throw exception if YARN Application switched to FINISHED immediately after deployed in YarnClusterDescriptor#startAppMaster

[Zili Chen (Jira)]

Zili Chen created FLINK-15297:
-

 Summary: Do not throw exception if YARN Application switched to 
FINISHED immediately after deployed in YarnClusterDescriptor#startAppMaster
 Key: FLINK-15297
 URL: https://issues.apache.org/jira/browse/FLINK-15297
 Project: Flink
  Issue Type: Bug
  Components: Deployment / YARN
Reporter: Zili Chen
Assignee: Zili Chen


Currently we throw an exception in {{YarnClusterDescriptor#startAppMaster}} if 
we first detect {{FINISHED}} before {{RUNNING}}. However, it is possible a 
legal state that the application finished normally immediately.

Right now we always try to connect the Dispatcher so it may be fine to throw 
the exception a bit earlier(otherwise when connect to a closed cluster an 
exception thrown also), but it is semantically wrong. Internally we have a code 
path that only required to report the ApplicationReport and it causes trouble.

cc [~trohrmann] what do you think?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15296) Support print executor-specific help information from CLI

[Zili Chen (Jira)]

Zili Chen created FLINK-15296:
-

 Summary: Support print executor-specific help information from CLI
 Key: FLINK-15296
 URL: https://issues.apache.org/jira/browse/FLINK-15296
 Project: Flink
  Issue Type: Sub-task
Reporter: Zili Chen






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15295) Mark CustomCommandLine and its subclass as deprecated

[Zili Chen (Jira)]

Zili Chen created FLINK-15295:
-

 Summary: Mark CustomCommandLine and its subclass as deprecated
 Key: FLINK-15295
 URL: https://issues.apache.org/jira/browse/FLINK-15295
 Project: Flink
  Issue Type: Sub-task
Reporter: Zili Chen






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15294) Deprecate CustomCommandLine

[Zili Chen (Jira)]

Zili Chen created FLINK-15294:
-

 Summary: Deprecate CustomCommandLine
 Key: FLINK-15294
 URL: https://issues.apache.org/jira/browse/FLINK-15294
 Project: Flink
  Issue Type: Task
  Components: Command Line Client
Reporter: Zili Chen


See also discussion in FLINK-15179. We'd like to migrate {{CustomCommandLine}} 
abstraction to Executor(FLIP-73) abstraction.

cc [~fly_in_gis] [~kkl0u]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15207) japicmp reference version is stale

[Zili Chen (Jira)]

Zili Chen created FLINK-15207:
-

 Summary: japicmp reference version is stale
 Key: FLINK-15207
 URL: https://issues.apache.org/jira/browse/FLINK-15207
 Project: Flink
  Issue Type: Bug
  Components: Build System
Reporter: Zili Chen


{{jamicmp.referenceVersion}} property is still {{1.8.0}}. Given now we're in a 
releasing process I'm not sure what to do with this. Besides, there is a typo 
in the property.

cc [~chesnay]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15201) Remove verifications in detach execution

[Zili Chen (Jira)]

Zili Chen created FLINK-15201:
-

 Summary: Remove verifications in detach execution
 Key: FLINK-15201
 URL: https://issues.apache.org/jira/browse/FLINK-15201
 Project: Flink
  Issue Type: Improvement
  Components: Client / Job Submission
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.10.0


>From [~aljoscha]: I think we actually don't need these "verifications" 
>anymore, with the new architecture where the Executor is called inside the 
>execute() method and where we don't actually "hijack" the method anymore, we 
>can actually have multiple execute() calls. We can address that in a 
>follow-up, though.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15134) Delete temporary files created in YarnClusterDescriptor

[Zili Chen (Jira)]

Zili Chen created FLINK-15134:
-

 Summary: Delete temporary files created in YarnClusterDescriptor
 Key: FLINK-15134
 URL: https://issues.apache.org/jira/browse/FLINK-15134
 Project: Flink
  Issue Type: Improvement
  Components: Client / Job Submission
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.10.0


We create temporary files for storing {{flink-conf}} & {{JobGraph}}. Although 
we ask for deleting them on exit, for a long running {{YarnClusterDescriptor}} 
it is possibly a resource leak. We can delete the file when they are no longer 
used. The delete hook will then silently fail on deletion but it is expected.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15090) Reverse the dependency from flink-streaming-java to flink-client

[Zili Chen (Jira)]

Zili Chen created FLINK-15090:
-

 Summary: Reverse the dependency from flink-streaming-java to 
flink-client
 Key: FLINK-15090
 URL: https://issues.apache.org/jira/browse/FLINK-15090
 Project: Flink
  Issue Type: Improvement
Reporter: Zili Chen
 Fix For: 1.11.0


After FLIP-73 the dependencies are minor. Tasks I can find are

1. Move {{StreamGraphTranslator}} to {{flink-client}}.
2. Implement similar context environment of streaming as {{flink-java}}. 
Set/Unset as context along with {{ExecutionEnvironment}}.

After this task we still have a dependency from {{flink-streaming-java}} to 
{{flink-java}} because some input formats dependencies. We can break the 
dependencies as follow-ups.

cc [~aljoscha]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15086) JobCluster may not be shutdown if executeAsync called

[Zili Chen (Jira)]

Zili Chen created FLINK-15086:
-

 Summary: JobCluster may not be shutdown if executeAsync called
 Key: FLINK-15086
 URL: https://issues.apache.org/jira/browse/FLINK-15086
 Project: Flink
  Issue Type: Bug
  Components: Client / Job Submission, Runtime / Coordination
Reporter: Zili Chen


If a JobCluster started in attached execution mode, it will wait for a client 
requesting its result and shutdown itself. However, there is no permission that 
a user get {{JobClient}} from {{executeAsync}} call {{getJobExecutionResult}} 
so the cluster may remain.

cc [~aljoscha] [~kkl0u]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15072) Hijack executeAsync instead of execute in context environment

[Zili Chen (Jira)]

Zili Chen created FLINK-15072:
-

 Summary: Hijack executeAsync instead of execute in context 
environment
 Key: FLINK-15072
 URL: https://issues.apache.org/jira/browse/FLINK-15072
 Project: Flink
  Issue Type: Sub-task
  Components: Client / Job Submission
Affects Versions: 1.10.0
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.10.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15048) Properly set internal.jobgraph-path when deploy job cluster

[Zili Chen (Jira)]

Zili Chen created FLINK-15048:
-

 Summary: Properly set internal.jobgraph-path when deploy job 
cluster
 Key: FLINK-15048
 URL: https://issues.apache.org/jira/browse/FLINK-15048
 Project: Flink
  Issue Type: Bug
  Components: Deployment / YARN
Affects Versions: 1.10.0
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.10.0


Currently we don't ship this config. It works occasionally because we use the 
default value as the actual value.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15047) YarnDistributedCacheITCase is unstable

[Zili Chen (Jira)]

Zili Chen created FLINK-15047:
-

 Summary: YarnDistributedCacheITCase is unstable
 Key: FLINK-15047
 URL: https://issues.apache.org/jira/browse/FLINK-15047
 Project: Flink
  Issue Type: Bug
  Components: Tests
Affects Versions: 1.10.0
Reporter: Zili Chen
 Fix For: 1.10.0


See also https://api.travis-ci.com/v3/job/262854881/log.txt

cc [~ZhenqiuHuang]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14969) Refactor CliFrontendRunWithYarnTest reflect to new execution codepath

[Zili Chen (Jira)]

Zili Chen created FLINK-14969:
-

 Summary: Refactor CliFrontendRunWithYarnTest reflect to new 
execution codepath
 Key: FLINK-14969
 URL: https://issues.apache.org/jira/browse/FLINK-14969
 Project: Flink
  Issue Type: Improvement
  Components: Tests
Reporter: Zili Chen
Assignee: Zili Chen


After commit 8379bc5473e7f054b53cf9b3e532f4f7f974fe29 merged as part of 
FLINK-14851, {{CliFrontendRunWithYarnTest}} currently runs in a different way 
from that was previously. Thus we need to refactor 
{{CliFrontendRunWithYarnTest}} reflect to the new execution codepath.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14964) Support configure remote flink jar

[Zili Chen (Jira)]

Zili Chen created FLINK-14964:
-

 Summary: Support configure remote flink jar
 Key: FLINK-14964
 URL: https://issues.apache.org/jira/browse/FLINK-14964
 Project: Flink
  Issue Type: Improvement
  Components: Deployment / YARN
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.10.0


Corresponding discussion happens on 
[https://lists.apache.org/x/thread.html/a2fef19f6925d43dd2cf3f9a46a2263b20562bd61d4da970f609568d@%3Cdev.flink.apache.org%3E]

This ticket focuses on support configure remote flink jar. That is, when user 
configure flink jar as "hdfs://..." we also respect the pattern and instead of 
uploading from local, directly register as YARN's {{LocalResource}}.

This improvement will speed up the deployment on YARN.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14958) ProgramTargetDescriptor#jobID possibly can be of type JobID instead of String

[Zili Chen (Jira)]

Zili Chen created FLINK-14958:
-

 Summary: ProgramTargetDescriptor#jobID possibly can be of type 
JobID instead of String
 Key: FLINK-14958
 URL: https://issues.apache.org/jira/browse/FLINK-14958
 Project: Flink
  Issue Type: Improvement
Reporter: Zili Chen






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14957) Remove no power option -yst

[Zili Chen (Jira)]

Zili Chen created FLINK-14957:
-

 Summary: Remove no power option -yst
 Key: FLINK-14957
 URL: https://issues.apache.org/jira/browse/FLINK-14957
 Project: Flink
  Issue Type: Improvement
  Components: Command Line Client, Deployment / YARN
Reporter: Zili Chen
 Fix For: 1.10.0


As codebase moved, {{FlinkYarnSessionCli#streaming}} a.k.a. "-yst" is no power 
any more. We could remove the option now.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14948) Implement shutDownCluster for MiniClusterClient

[Zili Chen (Jira)]

Zili Chen created FLINK-14948:
-

 Summary: Implement shutDownCluster for MiniClusterClient
 Key: FLINK-14948
 URL: https://issues.apache.org/jira/browse/FLINK-14948
 Project: Flink
  Issue Type: Improvement
  Components: Client / Job Submission
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.10.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14947) Implement LocalExecutor as new Executor interface

[Zili Chen (Jira)]

Zili Chen created FLINK-14947:
-

 Summary: Implement LocalExecutor as new Executor interface
 Key: FLINK-14947
 URL: https://issues.apache.org/jira/browse/FLINK-14947
 Project: Flink
  Issue Type: Sub-task
  Components: Client / Job Submission
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.10.0


We can replace {{PlanExecutor}} things with new Executor interface. One of this 
series is implement a {{LocalExecutor}} that execute pipeline within a 
{{MiniCluster}}. For correctly lifecycle management I would wait for 
FLINK-14762 being merged.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14916) Asynchronously deploy cluster in AbstractJobClusterExecutor#execute

[Zili Chen (Jira)]

Zili Chen created FLINK-14916:
-

 Summary: Asynchronously deploy cluster in 
AbstractJobClusterExecutor#execute
 Key: FLINK-14916
 URL: https://issues.apache.org/jira/browse/FLINK-14916
 Project: Flink
  Issue Type: Sub-task
Reporter: Zili Chen


https://github.com/apache/flink/pull/10251#discussion_r348933298

since {{execute}} returns {{CompletableFuture}} we'd better run 
deployment action asynchronously.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14915) SchedulingStrategyFactory#createInstance might not need to know JobGraph

[Zili Chen (Jira)]

Zili Chen created FLINK-14915:
-

 Summary: SchedulingStrategyFactory#createInstance might not need 
to know JobGraph
 Key: FLINK-14915
 URL: https://issues.apache.org/jira/browse/FLINK-14915
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Reporter: Zili Chen


[~zhuzh] I just notice that {{SchedulingStrategyFactory#createInstance}} take a 
parameter {{JobGraph}} but neither the parameter is in use nor this 
class/method should know {{JobGraph}}. Could you explain why we need it or we 
can safely remove the parameter so that we get rid of confusing parameter?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14885) YarnClusterDescriptor should not know about detached option

[Zili Chen (Jira)]

Zili Chen created FLINK-14885:
-

 Summary: YarnClusterDescriptor should not know about detached 
option
 Key: FLINK-14885
 URL: https://issues.apache.org/jira/browse/FLINK-14885
 Project: Flink
  Issue Type: Sub-task
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.10.0


YarnClusterDescriptor should not know about detached option. We pass the 
parameter on deploying, instead of one of fields. This refactor won't change 
too much since the field is nearly dummy.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14873) Make PackagedProgram#savepointSettings final

[Zili Chen (Jira)]

Zili Chen created FLINK-14873:
-

 Summary: Make PackagedProgram#savepointSettings final
 Key: FLINK-14873
 URL: https://issues.apache.org/jira/browse/FLINK-14873
 Project: Flink
  Issue Type: Improvement
  Components: Client / Job Submission
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.10.0


This field is logically final and with we move the initialization into 
{{Builder}} we can make it final.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14867) Move TextInputFormat & TextOutputFormat to flink-core

[Zili Chen (Jira)]

Zili Chen created FLINK-14867:
-

 Summary: Move TextInputFormat & TextOutputFormat to flink-core
 Key: FLINK-14867
 URL: https://issues.apache.org/jira/browse/FLINK-14867
 Project: Flink
  Issue Type: Improvement
  Components: API / DataSet, API / DataStream
Reporter: Zili Chen
Assignee: Zili Chen


This is one step to decouple the dependency from flink-streaming-java to 
flink-java. We already have a package {{o.a.f.core.io}} for these formats.

However, it possibly suffers from b/w compatibility issue so that we 
unfortunately move it under the same package in flink-core module.

CC [~aljoscha] also do you know how to verify whether or not such compatibility 
issue would happen?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14839) JobGraph#userJars & JobGraph#classpaths are non-null

[Zili Chen (Jira)]

Zili Chen created FLINK-14839:
-

 Summary: JobGraph#userJars & JobGraph#classpaths are non-null
 Key: FLINK-14839
 URL: https://issues.apache.org/jira/browse/FLINK-14839
 Project: Flink
  Issue Type: Improvement
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.10.0


So we can improve our codes a bit.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14797) Remove power mockito from RemoteStreamExecutionEnvironmentTest

[Zili Chen (Jira)]

Zili Chen created FLINK-14797:
-

 Summary: Remove power mockito from 
RemoteStreamExecutionEnvironmentTest
 Key: FLINK-14797
 URL: https://issues.apache.org/jira/browse/FLINK-14797
 Project: Flink
  Issue Type: Test
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.10.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14762) Implement ClusterClientJobClientAdapter

[Zili Chen (Jira)]

Zili Chen created FLINK-14762:
-

 Summary: Implement ClusterClientJobClientAdapter
 Key: FLINK-14762
 URL: https://issues.apache.org/jira/browse/FLINK-14762
 Project: Flink
  Issue Type: Sub-task
  Components: Client / Job Submission
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.10.0


{{ClusterClientJobClientAdapter}} is a minimum viable prototype of 
{{JobClient}} interface. It adapts a {{ClusterClient}} as a {{JobClient}} which 
associates with the specific {{JobID}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14761) ProgramDeployer#deployJobOnNewCluster potentially leak cluster

[Zili Chen (Jira)]

Zili Chen created FLINK-14761:
-

 Summary: ProgramDeployer#deployJobOnNewCluster potentially leak 
cluster
 Key: FLINK-14761
 URL: https://issues.apache.org/jira/browse/FLINK-14761
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Client
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.10.0


Currently {{ProgramDeployer#deployJobOnNewCluster}} always deploys cluster in 
attached mode. If {{awaitJobResult == false}} the cluster won't be closed 
because Dispatcher wait for delivering the result.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14759) Remove unused class TaskManagerCliOptions

[Zili Chen (Jira)]

Zili Chen created FLINK-14759:
-

 Summary: Remove unused class TaskManagerCliOptions
 Key: FLINK-14759
 URL: https://issues.apache.org/jira/browse/FLINK-14759
 Project: Flink
  Issue Type: Sub-task
Reporter: Zili Chen






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14742) Unstable tests TaskExecutorTest#testSubmitTaskBeforeAcceptSlot

[Zili Chen (Jira)]

Zili Chen created FLINK-14742:
-

 Summary: Unstable tests 
TaskExecutorTest#testSubmitTaskBeforeAcceptSlot
 Key: FLINK-14742
 URL: https://issues.apache.org/jira/browse/FLINK-14742
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.10.0
Reporter: Zili Chen


deadlock.


{code}
"main" #1 prio=5 os_prio=0 tid=0x7f1f8800b800 nid=0x356 waiting on 
condition [0x7f1f8e65c000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x86e9e9c0> (a 
java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at 
java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1693)
at 
java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at 
java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1729)
at 
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at 
org.apache.flink.runtime.taskexecutor.TaskExecutorTest.testSubmitTaskBeforeAcceptSlot(TaskExecutorTest.java:1108)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
{code}


full log https://api.travis-ci.org/v3/job/611275566/log.txt



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14699) Move ClosureCleaner to flink-core

[Zili Chen (Jira)]

Zili Chen created FLINK-14699:
-

 Summary: Move ClosureCleaner to flink-core
 Key: FLINK-14699
 URL: https://issues.apache.org/jira/browse/FLINK-14699
 Project: Flink
  Issue Type: Improvement
Reporter: Zili Chen
 Fix For: 1.10.0


{{ClosureCleaner}} is currently under {{flink-java}}. However, it doesn't stick 
to {{flink-java}} and used in {{flink-streaming-java}}. IMHO 
{{flink-streaming-java}} should not base on {{flink-java}}( 
{{flink-batch-java}} in fact ). Thus, I propose to move {{ClosureCleaner}} to 
{{flink-core}}.

CC [~chesnay] [~aljoscha]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14685) ZooKeeperCheckpointIDCounter forever broken if once loss connection with ZK

[Zili Chen (Jira)]

Zili Chen created FLINK-14685:
-

 Summary: ZooKeeperCheckpointIDCounter forever broken if once loss 
connection with ZK
 Key: FLINK-14685
 URL: https://issues.apache.org/jira/browse/FLINK-14685
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing, Runtime / Coordination
Affects Versions: 1.10.0
Reporter: Zili Chen
 Fix For: 1.10.0


Currently, if {{ZooKeeperCheckpointIDCounter}} suffers SUSPENDED state i.e. 
connection loss, it will set the state as invalid so that all checkpoint id 
counter operations succeed will fail.

Although couple with JM leadership management we will generate a new id counter 
on re-granted leadership so that it is not a problem so far, the semantic is 
wrong because id counter should only check whether current state is 
SUSPENDED/LOST. 

It is also a blocker upgrading to Curator 4.2 and [~lamber-ken] provides a 
[fix|https://github.com/BigDataArtisans/flink/commit/bd146ddcd1d9e0501f7e792875f5887edb8b7299]
 there.

Besides, in product scenario we once noticed that JM didn't re-elected(it 
shouldn't happen after [~trohrmann] add linearized leader operation) on 
SUSPENDED-RECONNECTED very fast so that a JM runs with a broken ID counter.

I think it is reasonable we pick [~lamber-ken]'s commit as a separated issue 
and fix this wrong semantic.

CC [~GJL] [~azagrebin]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14627) Refactor ExecutionGraph creation in tests as TestingExecutionGraphBuilder

[Zili Chen (Jira)]

Zili Chen created FLINK-14627:
-

 Summary: Refactor ExecutionGraph creation in tests as 
TestingExecutionGraphBuilder
 Key: FLINK-14627
 URL: https://issues.apache.org/jira/browse/FLINK-14627
 Project: Flink
  Issue Type: Improvement
  Components: Tests
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.10.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14593) Make ClusterClient an interface

[Zili Chen (Jira)]

Zili Chen created FLINK-14593:
-

 Summary: Make ClusterClient an interface
 Key: FLINK-14593
 URL: https://issues.apache.org/jira/browse/FLINK-14593
 Project: Flink
  Issue Type: Sub-task
  Components: Client / Job Submission
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.10.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14565) Shutdown SystemResourcesCounter on (JM|TM)MetricGroup closed

[Zili Chen (Jira)]

Zili Chen created FLINK-14565:
-

 Summary: Shutdown SystemResourcesCounter on (JM|TM)MetricGroup 
closed
 Key: FLINK-14565
 URL: https://issues.apache.org/jira/browse/FLINK-14565
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Metrics
Reporter: Zili Chen
Assignee: Zili Chen


Currently, we start SystemResourcesCounter when initialize (JM|TM)MetricGroup. 
This thread doesn't exit on (JM|TM)MetricGroup closed and even there is not 
exit logic of them.

It possibly causes thread leak. For example, on our platform which supports 
previewing sample SQL execution, it starts a MiniCluster in the same process as 
the platform. When the preview job finished MiniCluster closed and also 
(JM|TM)MetricGroup. However these SystemResourcesCounter threads remain.

I propose when creating SystemResourcesCounter, track it in (JM|TM)MetricGroup, 
and on (JM|TM)MetricGroup closed, shutdown SystemResourcesCounter. This way, we 
survive from thread leaks.

CC [~chesnay] [~trohrmann]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] Becket Qin joins the Flink PMC

[Zili Chen]

Congratulations Becket!

Best,
tison.


Congxian Qiu  于2019年10月29日周二 上午9:53写道：

> Congratulations Becket!
>
> Best,
> Congxian
>
>
> Wei Zhong  于2019年10月29日周二 上午9:42写道：
>
> > Congratulations Becket!
> >
> > Best,
> > Wei
> >
> > > 在 2019年10月29日，09:36，Paul Lam  写道：
> > >
> > > Congrats Becket!
> > >
> > > Best,
> > > Paul Lam
> > >
> > >> 在 2019年10月29日，02:18，Xingcan Cui  写道：
> > >>
> > >> Congratulations, Becket!
> > >>
> > >> Best,
> > >> Xingcan
> > >>
> > >>> On Oct 28, 2019, at 1:23 PM, Xuefu Z  wrote:
> > >>>
> > >>> Congratulations, Becket!
> > >>>
> > >>> On Mon, Oct 28, 2019 at 10:08 AM Zhu Zhu  wrote:
> > >>>
> >  Congratulations Becket!
> > 
> >  Thanks,
> >  Zhu Zhu
> > 
> >  Peter Huang  于2019年10月29日周二 上午1:01写道：
> > 
> > > Congratulations Becket Qin!
> > >
> > >
> > > Best Regards
> > > Peter Huang
> > >
> > > On Mon, Oct 28, 2019 at 9:19 AM Rong Rong 
> > wrote:
> > >
> > >> Congratulations Becket!!
> > >>
> > >> --
> > >> Rong
> > >>
> > >> On Mon, Oct 28, 2019, 7:53 AM Jark Wu  wrote:
> > >>
> > >>> Congratulations Becket!
> > >>>
> > >>> Best,
> > >>> Jark
> > >>>
> > >>> On Mon, 28 Oct 2019 at 20:26, Benchao Li 
> >  wrote:
> > >>>
> >  Congratulations Becket.
> > 
> >  Dian Fu  于2019年10月28日周一 下午7:22写道：
> > 
> > > Congrats, Becket.
> > >
> > >> 在 2019年10月28日，下午6:07，Fabian Hueske  写道：
> > >>
> > >> Hi everyone,
> > >>
> > >> I'm happy to announce that Becket Qin has joined the Flink
> PMC.
> > >> Let's congratulate and welcome Becket as a new member of the
> > > Flink
> > >>> PMC!
> > >>
> > >> Cheers,
> > >> Fabian
> > >
> > >
> > 
> >  --
> > 
> >  Benchao Li
> >  School of Electronics Engineering and Computer Science, Peking
> > >> University
> >  Tel:+86-15650713730
> >  Email: libenc...@gmail.com; libenc...@pku.edu.cn
> > 
> > >>>
> > >>
> > >
> > 
> > >>>
> > >>>
> > >>> --
> > >>> Xuefu Zhang
> > >>>
> > >>> "In Honey We Trust!"
> > >>
> > >
> >
> >
>

Re: [DISCUSS] What do we gain by supporting customized High-Availability services

[Zili Chen]

Thanks for your clarification Till.

For terminology pluggable and customizable, I am mainly concerning about
interface audience issue. Pluggable means we have multiple high-availability
implementation but closed to extend in user scope; customizable means
high-availability interface are stable and user-facing.

I agree that we should try to keep backward compatibility if possible. I
start
this thread due to being confused how I progress FLINK-10333. Specifically,
how to deal with compatibility things.

Given that it is an internal interface I think it is reasonable we
evolve it for
supporting leader store based high-availability storage in a minor version
bump. I'm going to start a discuss thread next week for gathering wider
feedbacks beyond the original (maybe limited) JIRA, which also calls of
review among community members. What do you think?

Best,
tison.


Till Rohrmann  于2019年10月21日周一 下午5:21写道：

> Hi Tison,
>
> I'm not sure whether I fully understand your distinction between
> customizable and pluggable. Maybe you could clarify your ideas a bit
> because you seem to favour support for pluggable implementations.
>
> Maybe let me try to answer some other questions you raised. With the
> HighAvailabilityServices interface and the functionality to load custom
> implementations specified via `high-availability: FQDN` it is indeed
> possible to provide a custom implementation of the HA services. This is,
> however, pretty much a power user feature where users have to implement
> against an internal API. As such, it can be subject to change and we don't
> give guarantees that these interfaces won't change. Of course, if possible,
> we should extend/change it in a way that we guarantee backwards
> compatibility.
>
> You are right that at the moment this interface is not stable enough for
> being public API. Once this changes and we are happy with it, then we can
> think about making it a public API and documenting it properly. Afaik,
> there is no documentation how to implement your own HA services at the
> moment. This underlines as well that this interface is an internal API.
>
> Cheers,
> Till
>
> On Fri, Oct 18, 2019 at 5:56 AM Zili Chen  wrote:
>
> > Another perspective is that a stable, carefully-designed interface with
> > clear semantic
> > could be safer to customize.
> >
> > Following the discussion in FLINK-10333 our JobGraphStore is actually
> > required
> > performing write operation only with leadership,  which is a basic
> > requirement
> > for coordination rather than an implementation detail.
> > Thus it depends on LeaderElectionService(in the design, we narrow the
> > specific interface
> > as LeaderStore). HighAvailabilityServices#getJobGraphStore() infers a
> > implicit field for
> > that which is hard to express the relationship between them.
> >
> > If the interface is unstable(also we introduce a ClientHAService for
> > separate concern and
> > have to keep b/w comp. for customized), we'd better keep it internal for
> > freely evolution.
> > And when we try to support customized, it would be helpful to start a
> > proposal to revisit
> > the interface to be well-designed and stable. ref[2].
> >
> > In short, current high-availability services as well as
> > runtime/coordination is still under
> > development and active evolution. It is possibly not a good time for make
> > it public and
> > customizable.
> >
> > Best,
> > tison.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-10333
> > [2]
> >
> >
> https://github.com/apache/pulsar/wiki/PIP-45%3A-Pluggable-metadata-interface
> >
> >
> > Zili Chen  于2019年10月17日周四 下午8:13写道：
> >
> > > A challenge is how we ensure the support for customized implementation.
> > > When
> > > we introduce JobGraphStore#releaseJobGraph we actually change quite a
> bit
> > > codepath
> > > in Dispatcher. While we are unable to test arbitrarily customized
> > > implementation our
> > > compatibility promise is actually no more than compilation compatible.
> > >
> > > Customer should still be required to be familiar with implementation
> > > details to figure
> > > out the fitment when they bump Flink version. This effort requires also
> > > and no extra
> > > when we support pluggable strategy. In another word, a customized
> support
> > > tends
> > > to hide the challenge when customer want to use their own
> implementation.
> > >
> >
>

Re: [VOTE] FLIP-81: Executor-related new ConfigOptions

[Zili Chen]

Thanks for starting this voting process. I have looked at the FLIP and the
discussion
thread. These options added make sense to me.

+1 from my side.

Best,
tison.


Kostas Kloudas  于2019年10月23日周三 下午4:12写道：

> Hi all,
>
> This is the voting thread for FLIP-81, as the title says.
>
> The FLIP can be found in [1] and the discussion thread in [2].
>
> As per the bylaws, the vote will stay open until Friday 26-10 (3 days)
> or until it gets 3 votes.
>
> Thank you,
> Kostas
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=133631524
> [2]
> https://lists.apache.org/thread.html/a4dd8e0c7b79350c59f5afefc1bc583dac99abcf94caaa8c22017974@%3Cdev.flink.apache.org%3E
>

[jira] [Created] (FLINK-14499) MetricRegistry#getMetricQueryServiceGatewayRpcAddress is Nonnull

[Zili Chen (Jira)]

Zili Chen created FLINK-14499:
-

 Summary: MetricRegistry#getMetricQueryServiceGatewayRpcAddress is 
Nonnull
 Key: FLINK-14499
 URL: https://issues.apache.org/jira/browse/FLINK-14499
 Project: Flink
  Issue Type: Bug
Reporter: Zili Chen


As codebase moved, I suspect 
{{MetricRegistry#getMetricQueryServiceGatewayRpcAddress}} is now non-null. One 
can try to figure it out and if so, refactor related code a bit.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14496) Exclude detach flag from ClusterClient

[Zili Chen (Jira)]

Zili Chen created FLINK-14496:
-

 Summary: Exclude detach flag from ClusterClient
 Key: FLINK-14496
 URL: https://issues.apache.org/jira/browse/FLINK-14496
 Project: Flink
  Issue Type: Sub-task
  Components: Client / Job Submission
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.10.0


According to the consensus of FLIP-74, we're working towards an entirely 
asynchronous interface {{ClusterClient}}. Thus submission is naturally 
asynchronous, which means we need not {{detach}} flag any more but leave the 
decision to the caller.

This ticket tracks effort to exclude detach flag from {{ClusterClient}} and 
make submit job graph an asynchronous method. Porting other methods to their 
respective asynchronous version could be a follow-up.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Accept Stateful Functions into Apache Flink

[Zili Chen]

+1 (non-binding)

Best,
tison.


Kostas Kloudas  于2019年10月21日周一 下午11:49写道：

> +1 (binding)
>
> On Mon, Oct 21, 2019 at 5:18 PM Aljoscha Krettek 
> wrote:
> >
> > +1 (binding)
> >
> > Aljoscha
> >
> > > On 21. Oct 2019, at 16:18, Thomas Weise  wrote:
> > >
> > > +1 (binding)
> > >
> > >
> > > On Mon, Oct 21, 2019 at 7:10 AM Timo Walther 
> wrote:
> > >
> > >> +1 (binding)
> > >>
> > >> Thanks,
> > >> Timo
> > >>
> > >>
> > >> On 21.10.19 15:59, Till Rohrmann wrote:
> > >>> +1 (binding)
> > >>>
> > >>> Cheers,
> > >>> Till
> > >>>
> > >>> On Mon, Oct 21, 2019 at 12:13 PM Robert Metzger  >
> > >> wrote:
> > >>>
> >  +1 (binding)
> > 
> >  On Mon, Oct 21, 2019 at 12:06 PM Stephan Ewen 
> wrote:
> > 
> > > This is the official vote whether to accept the Stateful Functions
> code
> > > contribution to Apache Flink.
> > >
> > > The current Stateful Functions code, documentation, and website
> can be
> > > found here:
> > > https://statefun.io/
> > > https://github.com/ververica/stateful-functions
> > >
> > > This vote should capture whether the Apache Flink community is
> > >> interested
> > > in accepting, maintaining, and evolving Stateful Functions.
> > >
> > > Reiterating my original motivation, I believe that this project is
> a
> >  great
> > > match for Apache Flink, because it helps Flink to grow the
> community
> >  into a
> > > new set of use cases. We see current users interested in such use
> > >> cases,
> > > but they are not well supported by Flink as it currently is.
> > >
> > > I also personally commit to put time into making sure this
> integrates
> >  well
> > > with Flink and that we grow contributors and committers to maintain
> > >> this
> > > new component well.
> > >
> > > This is a "Adoption of a new Codebase" vote as per the Flink bylaws
> > >> [1].
> > > Only PMC votes are binding. The vote will be open at least 6 days
> > > (excluding weekends), meaning until Tuesday Oct.29th 12:00 UTC, or
> > >> until
> >  we
> > > achieve the 2/3rd majority.
> > >
> > > Happy voting!
> > >
> > > Best,
> > > Stephan
> > >
> > > [1]
> > >
> > 
> > >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120731026
> > >>
> > >>
> > >>
> >
>

[jira] [Created] (FLINK-14462) Remove JobGraph#allowQueuedScheduling flag because it is always true

[Zili Chen (Jira)]

Zili Chen created FLINK-14462:
-

 Summary: Remove JobGraph#allowQueuedScheduling flag because it is 
always true
 Key: FLINK-14462
 URL: https://issues.apache.org/jira/browse/FLINK-14462
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Configuration
Affects Versions: 1.10.0
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.10.0


CC [~trohrmann][~zhuzh]

The only point {{#setAllowQueuedScheduling(false)}} is in 
{{JobGraphGenerator}}. IIRC we always {{#setAllowQueuedScheduling(true)}} after 
the generation and before the submission. For reduce confusion I propose to 
remove {{JobGraph#allowQueuedScheduling}} and refactor the related logic to all 
respect {{true}}.

This flag is originally used for configuring different resource allocation 
strategy between legacy mode and FLIP-6 arch. And there remains branches in 
{{Scheduler}} which might cause further confusion.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14461) Remove unused sessionTimeout from JobGraph

[Zili Chen (Jira)]

Zili Chen created FLINK-14461:
-

 Summary: Remove unused sessionTimeout from JobGraph
 Key: FLINK-14461
 URL: https://issues.apache.org/jira/browse/FLINK-14461
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Configuration
Affects Versions: 1.10.0
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.10.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] Apache Flink 1.9.1 released

[Zili Chen]

Thanks a lot for being release manager Jark. Great work!

Best,
tison.


Till Rohrmann  于2019年10月19日周六 下午10:15写道：

> Thanks a lot for being our release manager Jark and thanks to everyone who
> has helped to make this release possible.
>
> Cheers,
> Till
>
> On Sat, Oct 19, 2019 at 3:26 PM Jark Wu  wrote:
>
>>  Hi,
>>
>> The Apache Flink community is very happy to announce the release of
>> Apache Flink 1.9.1, which is the first bugfix release for the Apache Flink
>> 1.9 series.
>>
>> Apache Flink® is an open-source stream processing framework for
>> distributed, high-performing, always-available, and accurate data streaming
>> applications.
>>
>> The release is available for download at:
>> https://flink.apache.org/downloads.html
>>
>> Please check out the release blog post for an overview of the
>> improvements for this bugfix release:
>> https://flink.apache.org/news/2019/10/18/release-1.9.1.html
>>
>> The full release notes are available in Jira:
>> https://issues.apache.org/jira/projects/FLINK/versions/12346003
>>
>> We would like to thank all contributors of the Apache Flink community who
>> helped to verify this release and made this release possible!
>> Great thanks to @Jincheng for helping finalize this release.
>>
>> Regards,
>> Jark Wu
>>
>

[jira] [Created] (FLINK-14457) Shift down ClusterClient#configuration

[Zili Chen (Jira)]

Zili Chen created FLINK-14457:
-

 Summary: Shift down ClusterClient#configuration
 Key: FLINK-14457
 URL: https://issues.apache.org/jira/browse/FLINK-14457
 Project: Flink
  Issue Type: Sub-task
  Components: Client / Job Submission
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.10.0


Toward a {{ClusterClient}} interface. A follow up could be figure out what 
configuration is exactly used. I suspect we don't have to hold this 
configuration object.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14456) Remove or shift down all field in ClusterClient

[Zili Chen (Jira)]

Zili Chen created FLINK-14456:
-

 Summary: Remove or shift down all field in ClusterClient
 Key: FLINK-14456
 URL: https://issues.apache.org/jira/browse/FLINK-14456
 Project: Flink
  Issue Type: Sub-task
  Components: Client / Job Submission
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.10.0


Towards a {{ClusterClient}} interface.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] What do we gain by supporting customized High-Availability services

[Zili Chen]

Another perspective is that a stable, carefully-designed interface with
clear semantic
could be safer to customize.

Following the discussion in FLINK-10333 our JobGraphStore is actually
required
performing write operation only with leadership,  which is a basic
requirement
for coordination rather than an implementation detail.
Thus it depends on LeaderElectionService(in the design, we narrow the
specific interface
as LeaderStore). HighAvailabilityServices#getJobGraphStore() infers a
implicit field for
that which is hard to express the relationship between them.

If the interface is unstable(also we introduce a ClientHAService for
separate concern and
have to keep b/w comp. for customized), we'd better keep it internal for
freely evolution.
And when we try to support customized, it would be helpful to start a
proposal to revisit
the interface to be well-designed and stable. ref[2].

In short, current high-availability services as well as
runtime/coordination is still under
development and active evolution. It is possibly not a good time for make
it public and
customizable.

Best,
tison.

[1] https://issues.apache.org/jira/browse/FLINK-10333
[2]
https://github.com/apache/pulsar/wiki/PIP-45%3A-Pluggable-metadata-interface


Zili Chen  于2019年10月17日周四 下午8:13写道：

> A challenge is how we ensure the support for customized implementation.
> When
> we introduce JobGraphStore#releaseJobGraph we actually change quite a bit
> codepath
> in Dispatcher. While we are unable to test arbitrarily customized
> implementation our
> compatibility promise is actually no more than compilation compatible.
>
> Customer should still be required to be familiar with implementation
> details to figure
> out the fitment when they bump Flink version. This effort requires also
> and no extra
> when we support pluggable strategy. In another word, a customized support
> tends
> to hide the challenge when customer want to use their own implementation.
>

Re: [NOTICE] Binary licensing is now auto-generated

[Zili Chen]

Thanks for this improvement Chesnay ;-)

Best,
tison.


Chesnay Schepler  于2019年10月17日周四 下午9:37写道：

> Hello,
>
> I just merged FLINK-14008 to 1.8, 1.9 and 1.10, which means that from
> now on the tricky part of the binary licensing (NOTICE-binary,
> licenses-binary) is automatically generated during the release process.
>
> As such these files have been removed from the root directory of the
> project (thus, you don't have to update these things anymore ;)).
>
> This also means that only builds of flink-dist that were built as part
> of the release process will have these files attached.
>
> I have updated the Licensing guide
>  accordingly.
>
>

Re: [DISCUSS] What do we gain by supporting customized High-Availability services

[Zili Chen]

A challenge is how we ensure the support for customized implementation. When
we introduce JobGraphStore#releaseJobGraph we actually change quite a bit
codepath
in Dispatcher. While we are unable to test arbitrarily customized
implementation our
compatibility promise is actually no more than compilation compatible.

Customer should still be required to be familiar with implementation
details to figure
out the fitment when they bump Flink version. This effort requires also and
no extra
when we support pluggable strategy. In another word, a customized support
tends
to hide the challenge when customer want to use their own implementation.

[DISCUSS] What do we gain by supporting customized High-Availability services

[Zili Chen]

Hi devs,

Recently the community excludes customize support on new restart strategies[1],
which reminds
me to think of which kind of customized support a framework like Flink
should provides.

The key idea is pluggable is not customizable.

We might handle a series of implementation of restart strategies as well as
high-availability
services in our codebase. But it has a fixed size, which is definitely
different from support
arbitrarily customized.

For a services like high-availability services, it underneath relies on
quite a lot of runtime
implementations. For example, JobGraphStore supports #releaseJobGraphStore
originally
due to ZK lock strategy; getJobManagerRetriever requires default address
because
StandaloneHighAvailabilityServices is non-ha and pre-configured.

This kind of interfaces, however, are possibly evolves with flink runtime
implementation such
as cluster management and coordination details. If we support customizing
it, it means
such internal a high-availability services becomes public interfaces. If we
keep it pluggable,
we can extend it reacting to runtime evolution, ensuring the
implementations stay in a fixed
set; while introducing new implementation(such as etcd[2] or MapDB[3]) if
they are good fit.

We don't have a customize support on ResourceManager although it is
pluggable that
others can implement a kubernetes resource manager[4]. Maybe this is a
better way
how we handle high-availability services. Pluggable, but not customizable.

Looking forward to your ideas. To be clear, I'm not trying to drop it now,
but I'm a bit
confusing about this topic and would like to turn to the wisdom in our
community.

Best,
tison.

[1]
https://lists.apache.org/x/thread.html/6ed95eb6a91168dba09901e158bc1b6f4b08f1e176db4641f79de765@%3Cdev.flink.apache.org%3E
[2] https://issues.apache.org/jira/browse/FLINK-11105
[3]
https://lists.apache.org/x/thread.html/eae4cbdf6dac466bc0247e3bc1a7a69fe7e1db7a512fcd607e9c081b@%3Cuser.flink.apache.org%3E
[4] https://github.com/tianchen92/flink/tree/k8s-master/flink-kubernete

[jira] [Created] (FLINK-14434) Dispatcher#createJobManagerRunner should returns on creation succeed, not after startJobManagerRunner

[Zili Chen (Jira)]

Zili Chen created FLINK-14434:
-

 Summary: Dispatcher#createJobManagerRunner should returns on 
creation succeed, not after startJobManagerRunner
 Key: FLINK-14434
 URL: https://issues.apache.org/jira/browse/FLINK-14434
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.10.0
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.10.0


In an edge case, let's said

1) job finished nearly immediately
2) Dispatcher has been suspended in {{#startJobManagerRunner}} after 
{{jobManagerRunner.start();}} but before {{return jobManagerRunner;}}

due to

1) we put {{jobManagerRunnerFutures}} with {{#startJobManagerRunner}} finished.
2) the creation of JobManagerRunner doesn't happen in MainThread.

it is a possible execution order

1) JobManagerRunner created in akka-dispatcher thread
2) then apply {{Dispatcher#startJobManagerRunner}}
3) until {{jobManagerRunner.start();}} and before {{return jobManagerRunner;}}
4) this thread suspended
5) job finished, execute callback on MainThread
6) {{jobManagerRunnerFutures.get(jobID).getNow(null)}} returns {{null}} because 
akka-dispatcher thread doesn't {{return jobManagerRunner;}}
7) it report {{There is a newer JobManagerRunner for the job}} but actually not.

**Solution**

Two perspective but we can even have them both.

1. return {{jobManagerRunnerFuture}} in {{#createJobManagerRunner}}, let 
{{#startJobManagerRunner}} an action
2. on JobManagerRunner created, execute {{#startJobManagerRunner}} in 
MainThread.

CC [~trohrmann]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-77: Introduce ConfigOptions with Data Types

[Zili Chen]

+1

Best,
tison.


Aljoscha Krettek  于2019年10月16日周三 下午6:25写道：

> +1
>
> > On 16. Oct 2019, at 10:30, Timo Walther  wrote:
> >
> > +1
> >
> > Thanks,
> > Timo
> >
> > On 15.10.19 17:07, Till Rohrmann wrote:
> >> Sorry for the confusion. I should have checked with an external mail
> >> client. Thanks a lot for the clarification.
> >>
> >> Cheers,
> >> Till
> >>
> >> On Tue, Oct 15, 2019 at 2:07 PM Jark Wu  wrote:
> >>
> >>> +1
> >>>
> >>> It is a separate [VOTE] thread in my Mail client.
> >>>
> >>> Best,
> >>> Jark
> >>>
>  在 2019年10月15日，18:58，Dawid Wysakowicz  写道：
> 
>  Hi Till,
> 
>  It should and it actually is.
> 
>  I think it's some feature of gmail that it groups conversations with a
>  similar topic into a conversation. (I also see it this way in gmail,
> but
>  I see it as separate threads in my mail client)
> 
>  You can see that it is a separate thread e.g. in the ML archives:
> 
>  This thread:
> 
> >>>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-77-Introduce-ConfigOptions-with-Data-Types-td33999.html
>  Discuss thread:
> 
> >>>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-77-Introduce-ConfigOptions-with-Data-Types-td33902.html
>  Best,
> 
>  Dawid
> 
>  On 15/10/2019 12:00, Till Rohrmann wrote:
> > Shouldn't the voting happen in a distinct [VOTE] thread?
> >
> > Cheers,
> > Till
> >
> > On Tue, Oct 15, 2019 at 10:46 AM Kurt Young 
> wrote:
> >
> >> +1
> >>
> >> Best,
> >> Kurt
> >>
> >>
> >> On Tue, Oct 15, 2019 at 9:30 AM Dawid Wysakowicz <
> >>> dwysakow...@apache.org>
> >> wrote:
> >>
> >>> Hi everyone,
> >>> I would like to start a vote on FLIP-77.
> >>>
> >>> Please vote for the following design document:
> >>>
> >>>
> >>>
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-77%3A+Introduce+ConfigOptions+with+Data+Types
> >>> The discussions can be found at:
> >>>
> >>>
> >>>
> >>>
> https://lists.apache.org/thread.html/a56c6b52e5f828d4a737602b031e36b5dd6eaa97557306696a8063a9@%3Cdev.flink.apache.org%3E
> >>>
> https://lists.apache.org/thread.html/6f98eae7879764aa3a3991311d700a5ccdb713dbe345e6ecf514b2d7@%3Cdev.flink.apache.org%3E
> >>> This voting will be open for at least 72 hours. I'll try to close
> it
> >>> on
> >>> 2019-10-18 8:00 UTC, unless there is an objection or not enough
> votes.
> >>>
> >>> Best,
> >>>
> >>> Dawid
> >>>
> >>>
> >>>
> >>>
> >
>
>

Re: [DISCUSS] FLIP policy for introducing config option keys

[Zili Chen]

Hi Jark & Hequn,

Do you stick to introduce a looser FLIP? We possibly introduce a redundant
extra type
of community consensus if we are able to just reuse the process of current
FLIP. Given
the activity of our community I don't think it costs too much for a config
option keys
change with 3 days at least voting required >3 committer votes.

Best,
tison.


Hequn Cheng  于2019年10月16日周三 下午2:29写道：

> Hi all,
>
> +1 to have a looser FLIP policy for these API changes.
>
> I think the concerns raised above are all valid. Besides the feedbacks from
> @Jark, if we want to keep track of these changes, maybe we can create a new
> kind of FLIP that is dedicated to these minor API changes? For example, we
> can add a single wiki page and list all related JIRAs in it. The design
> details can be described in the JIRA.
> Another option is to simply add a new JIRA label to track these changes.
>
> What do you think?
>
> Best, Hequn
>
> On Tue, Oct 15, 2019 at 8:43 PM Zili Chen  wrote:
>
> > The naming concern above can be a separated issue since it looks also
> > affect FLIP-54 and isn't limited for config option changes FLIP.
> >
> > Best,
> > tison.
> >
> >
> > Aljoscha Krettek  于2019年10月15日周二 下午8:37写道：
> >
> > > Another PR that introduces new config options:
> > > https://github.com/apache/flink/pull/9759
> > >
> > > > On 15. Oct 2019, at 14:31, Zili Chen  wrote:
> > > >
> > > > Hi Aljoscha & Dawid & Kostas,
> > > >
> > > > I agree that changes on config option keys deserve a FLIP and it is
> > > > reasonable
> > > > we commit the changes with a standard FLIP process so that ensure the
> > > change
> > > > given proper visibility.
> > > >
> > > > My concern is about naming. Given FLIP-73 as an example, if FLIPs
> > > > associated to
> > > > FLIP-73(actually can be regarded as sub-FLIP of it) grows FLIP
> numbers
> > > and
> > > > appears
> > > > like FLIP-80 FLIP-85 FLIP-91 and so on, then we possibly run into a
> > state
> > > > flooded by
> > > > quite a few config option only FLIP. Maybe it makes sense to number
> > these
> > > > FLIP as
> > > > FLIP-73.1 FLIP-73.2, which shows the association and doesn't pollute
> > > other
> > > > FLIPs.
> > > >
> > > > Remind the general thoughts, IMO changes on config option keys
> deserve
> > a
> > > > standard
> > > > FLIP process, e.g. FLIP-61.
> > > >
> > > > Best,
> > > > tison.
> > > >
> > > >
> > > > Kostas Kloudas  于2019年10月15日周二 下午8:20写道：
> > > >
> > > >> Hi Aljoscha,
> > > >>
> > > >> Given that config option keys are user-facing and any change there
> is
> > > >> breaking, I think there should be a discussion about them and a
> FLIP,
> > > >> where people have to actually vote for it seems to be the right
> place.
> > > >> I understand that this is tedious (and actually I will also have to
> > > >> open some FLIPs as part of FLIP-73), but this contributes to the
> > > >> uniformity of our parameters and also giving them some more
> > > >> visibility.
> > > >>
> > > >> Cheers,
> > > >> Kostas
> > > >>
> > > >> On Tue, Oct 15, 2019 at 2:05 PM Aljoscha Krettek <
> aljos...@apache.org
> > >
> > > >> wrote:
> > > >>>
> > > >>> Hi Everyone,
> > > >>>
> > > >>> The title says it all, do you think we need to cover all config
> > options
> > > >> that we introduce/change by FLIPs? I was thinking about this because
> > of
> > > the
> > > >> FLIP-73 work, which will introduce some new config options and also
> > > because
> > > >> I just spotted a PR [1] that introduces some config options.
> > > >>>
> > > >>> Best,
> > > >>> Aljoscha
> > > >>>
> > > >>> [1] https://github.com/apache/flink/pull/9836
> > > >>
> > >
> > >
> >
>

Re: [DISCUSS] FLIP policy for introducing config option keys

[Zili Chen]

The naming concern above can be a separated issue since it looks also
affect FLIP-54 and isn't limited for config option changes FLIP.

Best,
tison.


Aljoscha Krettek  于2019年10月15日周二 下午8:37写道：

> Another PR that introduces new config options:
> https://github.com/apache/flink/pull/9759
>
> > On 15. Oct 2019, at 14:31, Zili Chen  wrote:
> >
> > Hi Aljoscha & Dawid & Kostas,
> >
> > I agree that changes on config option keys deserve a FLIP and it is
> > reasonable
> > we commit the changes with a standard FLIP process so that ensure the
> change
> > given proper visibility.
> >
> > My concern is about naming. Given FLIP-73 as an example, if FLIPs
> > associated to
> > FLIP-73(actually can be regarded as sub-FLIP of it) grows FLIP numbers
> and
> > appears
> > like FLIP-80 FLIP-85 FLIP-91 and so on, then we possibly run into a state
> > flooded by
> > quite a few config option only FLIP. Maybe it makes sense to number these
> > FLIP as
> > FLIP-73.1 FLIP-73.2, which shows the association and doesn't pollute
> other
> > FLIPs.
> >
> > Remind the general thoughts, IMO changes on config option keys deserve a
> > standard
> > FLIP process, e.g. FLIP-61.
> >
> > Best,
> > tison.
> >
> >
> > Kostas Kloudas  于2019年10月15日周二 下午8:20写道：
> >
> >> Hi Aljoscha,
> >>
> >> Given that config option keys are user-facing and any change there is
> >> breaking, I think there should be a discussion about them and a FLIP,
> >> where people have to actually vote for it seems to be the right place.
> >> I understand that this is tedious (and actually I will also have to
> >> open some FLIPs as part of FLIP-73), but this contributes to the
> >> uniformity of our parameters and also giving them some more
> >> visibility.
> >>
> >> Cheers,
> >> Kostas
> >>
> >> On Tue, Oct 15, 2019 at 2:05 PM Aljoscha Krettek 
> >> wrote:
> >>>
> >>> Hi Everyone,
> >>>
> >>> The title says it all, do you think we need to cover all config options
> >> that we introduce/change by FLIPs? I was thinking about this because of
> the
> >> FLIP-73 work, which will introduce some new config options and also
> because
> >> I just spotted a PR [1] that introduces some config options.
> >>>
> >>> Best,
> >>> Aljoscha
> >>>
> >>> [1] https://github.com/apache/flink/pull/9836
> >>
>
>

Re: [DISCUSS] FLIP policy for introducing config option keys

[Zili Chen]

Hi Aljoscha & Dawid & Kostas,

I agree that changes on config option keys deserve a FLIP and it is
reasonable
we commit the changes with a standard FLIP process so that ensure the change
given proper visibility.

My concern is about naming. Given FLIP-73 as an example, if FLIPs
associated to
FLIP-73(actually can be regarded as sub-FLIP of it) grows FLIP numbers and
appears
like FLIP-80 FLIP-85 FLIP-91 and so on, then we possibly run into a state
flooded by
quite a few config option only FLIP. Maybe it makes sense to number these
FLIP as
FLIP-73.1 FLIP-73.2, which shows the association and doesn't pollute other
FLIPs.

Remind the general thoughts, IMO changes on config option keys deserve a
standard
FLIP process, e.g. FLIP-61.

Best,
tison.


Kostas Kloudas  于2019年10月15日周二 下午8:20写道：

> Hi Aljoscha,
>
> Given that config option keys are user-facing and any change there is
> breaking, I think there should be a discussion about them and a FLIP,
> where people have to actually vote for it seems to be the right place.
> I understand that this is tedious (and actually I will also have to
> open some FLIPs as part of FLIP-73), but this contributes to the
> uniformity of our parameters and also giving them some more
> visibility.
>
> Cheers,
> Kostas
>
> On Tue, Oct 15, 2019 at 2:05 PM Aljoscha Krettek 
> wrote:
> >
> > Hi Everyone,
> >
> > The title says it all, do you think we need to cover all config options
> that we introduce/change by FLIPs? I was thinking about this because of the
> FLIP-73 work, which will introduce some new config options and also because
> I just spotted a PR [1] that introduces some config options.
> >
> > Best,
> > Aljoscha
> >
> > [1] https://github.com/apache/flink/pull/9836
>

[jira] [Created] (FLINK-14392) Introduce JobClient API(FLIP-74)

[Zili Chen (Jira)]

Zili Chen created FLINK-14392:
-

 Summary: Introduce JobClient API(FLIP-74)
 Key: FLINK-14392
 URL: https://issues.apache.org/jira/browse/FLINK-14392
 Project: Flink
  Issue Type: New Feature
  Components: Client / Job Submission, Command Line Client
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.10.0


This is the umbrella issue to track all efforts toward {{JobClient}} proposed 
in 
[FLIP-74|https://cwiki.apache.org/confluence/display/FLINK/FLIP-74%3A+Flink+JobClient+API].



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-74: Flink JobClient API

[Zili Chen]

Hi all,

+1 from my side.

Given the current state of this voting thread, FLIP-74 is accepted
with 3 binding vote and 2 non-binding vote. Thanks for your
participation!

I will update the wiki to reflect that the result of the vote.

Best,
tison.


Zili Chen  于2019年10月11日周五 下午8:48写道：

> Well. Then I'd remove the requirement to change cancelWithSavepoint
> but remain why we exclude it from JobClient.
>
> We might still change signature to completable future for a consistent
> async view of ClusterClient but it is quite an implement detail and we
> don't stick to it on FLIP level.
>
> Best,
> tison.
>
>
> Kostas Kloudas  于2019年10月11日周五 下午7:36写道：
>
>> Hi Tison,
>>
>> Thanks for integrating the comments!
>>
>> +1 for accepting the FLIP from my side.
>> What I meant is that in the Proposed Changes section, the FLIP still
>> has that the cancelWithSavepoin(jobId, savepointDir) of the
>> clusterClient should change to return a CompletableFuture. I believe
>> that this change is redundant as we will not need it for the
>> JobClient. I should have been more clear on what I meant before.
>>
>> Cheers,
>> Kostas
>>
>> On Fri, Oct 11, 2019 at 11:51 AM Zili Chen  wrote:
>> >
>> > Hi Kostas,
>> >
>> > Thanks for your reply.
>> >
>> > (1) cancelWithSavepoint() has already been excluded from the FLIP. But
>> > to emphasize that we make the decision to exclude it I add it to reject
>> > alternatives.
>> >
>> > (2) Updated FLIP to reflect the consensus :-)
>> >
>> > Best,
>> > tison.
>> >
>> >
>> > Kostas Kloudas  于2019年10月11日周五 下午5:12写道：
>> >
>> > > Hi all,
>> > >
>> > > I only have two minor comments before voting and they have to do with
>> > > the following:
>> > >
>> > > 1) In the discussion, we agreed to remove the cancelWithSavepoint()
>> > > from the JobClient as this is deprecated in the rest API. This is not
>> > > in the FLIP.
>> > > 2) The section "ClusterDescriptor or Executor(FLIP-73)(integration)"
>> > > does not reflect our discussion where we said that for now only the
>> > > Executor#execute() will give you the JobClient and there will be a
>> > > separate discussion about alternative ways of exposing the JobClient.
>> > >
>> > > I think that these points should be updated in order for the FLIP to
>> > > reflect the discussion in the ML thread.
>> > >
>> > > Cheers,
>> > > Kostas
>> > >
>> > > On Fri, Oct 11, 2019 at 10:58 AM Biao Liu  wrote:
>> > > >
>> > > > +1 (non-binding), glad to have this improvement!
>> > > >
>> > > > Thanks,
>> > > > Biao /'bɪ.aʊ/
>> > > >
>> > > >
>> > > >
>> > > > On Fri, 11 Oct 2019 at 14:44, Jeff Zhang  wrote:
>> > > >
>> > > > > +1， overall design make sense to me
>> > > > >
>> > > > > SHI Xiaogang  于2019年10月11日周五 上午11:15写道：
>> > > > >
>> > > > > > +1. The interface looks fine to me.
>> > > > > >
>> > > > > > Regards,
>> > > > > > Xiaogang
>> > > > > >
>> > > > > > Zili Chen  于2019年10月9日周三 下午2:36写道：
>> > > > > >
>> > > > > > > Given the ongoing FlinkForward Berlin event, I'm going to
>> extend
>> > > > > > > this vote thread with a bit of period, said until Oct.
>> > > 11th(Friday).
>> > > > > > >
>> > > > > > > Best,
>> > > > > > > tison.
>> > > > > > >
>> > > > > > >
>> > > > > > > Zili Chen  于2019年10月7日周一 下午4:15写道：
>> > > > > > >
>> > > > > > > > Hi all,
>> > > > > > > >
>> > > > > > > > I would like to start the vote for FLIP-74[1], which is
>> > > discussed and
>> > > > > > > > reached a consensus in the discussion thread[2].
>> > > > > > > >
>> > > > > > > > The vote will be open util Oct. 9th(72h starting on
>> Oct.7th),
>> > > unless
>> > > > > > > > there is an objection or not  enough votes.
>> > > > > > > >
>> > > > > > > > Best,
>> > > > > > > > tison.
>> > > > > > > >
>> > > > > > > > [1]
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-74%3A+Flink+JobClient+API
>> > > > > > > > [2]
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > >
>> https://lists.apache.org/x/thread.html/b2e22a45aeb94a8d06b50c4de078f7b23d9ff08b8226918a1a903768@%3Cdev.flink.apache.org%3E
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Best Regards
>> > > > >
>> > > > > Jeff Zhang
>> > > > >
>> > >
>>
>

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[Zili Chen]

+1 to add Stateful Function to FLINK core repository.

Best,
tison.


Becket Qin  于2019年10月14日周一 下午4:16写道：

> +1 to adding Stateful Function to Flink. It is a very useful addition to
> the Flink ecosystem.
>
> Given this is essentially a new top-level / first-citizen API of Flink, it
> seems better to have it the Flink core repo. This will also avoid letting
> this important new API to be blocked on potential problems of maintaining
> multiple different repositories.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Sun, Oct 13, 2019 at 4:48 AM Hequn Cheng  wrote:
>
>> Hi Stephan,
>>
>> Big +1 for adding this to Apache Flink!
>>
>> As for the problem of whether this should be added to the Flink main
>> repository, from my side, I prefer to put it in the main repository. Not
>> only Stateful Functions shares very close relations with the current Flink,
>> but also other libs or modules in Flink can make use of it the other way
>> round in the future. At that time the Flink API stack would also be changed
>> a bit and this would be cool.
>>
>> Best, Hequn
>>
>> On Sat, Oct 12, 2019 at 9:16 PM Biao Liu  wrote:
>>
>>> Hi Stehpan,
>>>
>>> +1 for having Stateful Functions in Flink.
>>>
>>> Before discussing which repository it should belong, I was wondering if
>>> we have reached an agreement of "splitting flink repository" as Piotr
>>> mentioned or not. It seems that it's just no more further discussion.
>>> It's OK for me to add it to core repository. After all almost everything
>>> is in core repository now. But if we decide to split the core repository
>>> someday, I tend to create a separate repository for Stateful Functions. It
>>> might be good time to take the first step of splitting.
>>>
>>> Thanks,
>>> Biao /'bɪ.aʊ/
>>>
>>>
>>>
>>> On Sat, 12 Oct 2019 at 19:31, Yu Li  wrote:
>>>
 Hi Stephan,

 Big +1 for adding stateful functions to Flink. I believe a lot of user
 would be interested to try this out and I could imagine how this could
 contribute to reduce the TCO for business requiring both streaming
 processing and stateful functions.

 And my 2 cents is to put it into flink core repository since I could
 see a tight connection between this library and flink state.

 Best Regards,
 Yu


 On Sat, 12 Oct 2019 at 17:31, jincheng sun 
 wrote:

> Hi Stephan,
>
> bit +1 for adding this great features to Apache Flink.
>
> Regarding where we should place it, put it into Flink core repository
> or create a separate repository? I prefer put it into main repository and
> looking forward the more detail discussion for this decision.
>
> Best,
> Jincheng
>
>
> Jingsong Li  于2019年10月12日周六 上午11:32写道：
>
>> Hi Stephan,
>>
>> big +1 for this contribution. It provides another user interface that
>> is easy to use and popular at this time. these functions, It's hard for
>> users to write in SQL/TableApi, while using DataStream is too complex.
>> (We've done some stateFun kind jobs using DataStream before). With
>> statefun, it is very easy.
>>
>> I think it's also a good opportunity to exercise Flink's core
>> capabilities. I looked at stateful-functions-flink briefly, it is very
>> interesting. I think there are many other things Flink can improve. So I
>> think it's a better thing to put it into Flink, and the improvement for 
>> it
>> will be more natural in the future.
>>
>> Best,
>> Jingsong Lee
>>
>> On Fri, Oct 11, 2019 at 7:33 PM Dawid Wysakowicz <
>> dwysakow...@apache.org> wrote:
>>
>>> Hi Stephan,
>>>
>>> I think this is a nice library, but what I like more about it is
>>> that it suggests exploring different use-cases. I think it definitely 
>>> makes
>>> sense for the Flink community to explore more lightweight applications 
>>> that
>>> reuses resources. Therefore I definitely think it is a good idea for 
>>> Flink
>>> community to accept this contribution and help maintaining it.
>>>
>>> Personally I'd prefer to have it in a separate repository. There
>>> were a few discussions before where different people were suggesting to
>>> extract connectors and other libraries to separate repositories. 
>>> Moreover I
>>> think it could serve as an example for the Flink ecosystem website[1]. 
>>> This
>>> could be the first project in there and give a good impression that the
>>> community sees potential in the ecosystem website.
>>>
>>> Lastly, I'm wondering if this should go through PMC vote according
>>> to our bylaws[2]. In the end the suggestion is to adopt an existing code
>>> base as is. It also proposes a new programs concept that could result 
>>> in a
>>> shift of priorities for the community in a long run.
>>>
>>> Best,
>>>
>>> Dawid
>>>
>>> [1]
>>> 

Re: [VOTE] FLIP-74: Flink JobClient API

[Zili Chen]

Well. Then I'd remove the requirement to change cancelWithSavepoint
but remain why we exclude it from JobClient.

We might still change signature to completable future for a consistent
async view of ClusterClient but it is quite an implement detail and we
don't stick to it on FLIP level.

Best,
tison.


Kostas Kloudas  于2019年10月11日周五 下午7:36写道：

> Hi Tison,
>
> Thanks for integrating the comments!
>
> +1 for accepting the FLIP from my side.
> What I meant is that in the Proposed Changes section, the FLIP still
> has that the cancelWithSavepoin(jobId, savepointDir) of the
> clusterClient should change to return a CompletableFuture. I believe
> that this change is redundant as we will not need it for the
> JobClient. I should have been more clear on what I meant before.
>
> Cheers,
> Kostas
>
> On Fri, Oct 11, 2019 at 11:51 AM Zili Chen  wrote:
> >
> > Hi Kostas,
> >
> > Thanks for your reply.
> >
> > (1) cancelWithSavepoint() has already been excluded from the FLIP. But
> > to emphasize that we make the decision to exclude it I add it to reject
> > alternatives.
> >
> > (2) Updated FLIP to reflect the consensus :-)
> >
> > Best,
> > tison.
> >
> >
> > Kostas Kloudas  于2019年10月11日周五 下午5:12写道：
> >
> > > Hi all,
> > >
> > > I only have two minor comments before voting and they have to do with
> > > the following:
> > >
> > > 1) In the discussion, we agreed to remove the cancelWithSavepoint()
> > > from the JobClient as this is deprecated in the rest API. This is not
> > > in the FLIP.
> > > 2) The section "ClusterDescriptor or Executor(FLIP-73)(integration)"
> > > does not reflect our discussion where we said that for now only the
> > > Executor#execute() will give you the JobClient and there will be a
> > > separate discussion about alternative ways of exposing the JobClient.
> > >
> > > I think that these points should be updated in order for the FLIP to
> > > reflect the discussion in the ML thread.
> > >
> > > Cheers,
> > > Kostas
> > >
> > > On Fri, Oct 11, 2019 at 10:58 AM Biao Liu  wrote:
> > > >
> > > > +1 (non-binding), glad to have this improvement!
> > > >
> > > > Thanks,
> > > > Biao /'bɪ.aʊ/
> > > >
> > > >
> > > >
> > > > On Fri, 11 Oct 2019 at 14:44, Jeff Zhang  wrote:
> > > >
> > > > > +1， overall design make sense to me
> > > > >
> > > > > SHI Xiaogang  于2019年10月11日周五 上午11:15写道：
> > > > >
> > > > > > +1. The interface looks fine to me.
> > > > > >
> > > > > > Regards,
> > > > > > Xiaogang
> > > > > >
> > > > > > Zili Chen  于2019年10月9日周三 下午2:36写道：
> > > > > >
> > > > > > > Given the ongoing FlinkForward Berlin event, I'm going to
> extend
> > > > > > > this vote thread with a bit of period, said until Oct.
> > > 11th(Friday).
> > > > > > >
> > > > > > > Best,
> > > > > > > tison.
> > > > > > >
> > > > > > >
> > > > > > > Zili Chen  于2019年10月7日周一 下午4:15写道：
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I would like to start the vote for FLIP-74[1], which is
> > > discussed and
> > > > > > > > reached a consensus in the discussion thread[2].
> > > > > > > >
> > > > > > > > The vote will be open util Oct. 9th(72h starting on Oct.7th),
> > > unless
> > > > > > > > there is an objection or not  enough votes.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > tison.
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-74%3A+Flink+JobClient+API
> > > > > > > > [2]
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> https://lists.apache.org/x/thread.html/b2e22a45aeb94a8d06b50c4de078f7b23d9ff08b8226918a1a903768@%3Cdev.flink.apache.org%3E
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best Regards
> > > > >
> > > > > Jeff Zhang
> > > > >
> > >
>

Re: [VOTE] FLIP-74: Flink JobClient API

[Zili Chen]

Hi Kostas,

Thanks for your reply.

(1) cancelWithSavepoint() has already been excluded from the FLIP. But
to emphasize that we make the decision to exclude it I add it to reject
alternatives.

(2) Updated FLIP to reflect the consensus :-)

Best,
tison.


Kostas Kloudas  于2019年10月11日周五 下午5:12写道：

> Hi all,
>
> I only have two minor comments before voting and they have to do with
> the following:
>
> 1) In the discussion, we agreed to remove the cancelWithSavepoint()
> from the JobClient as this is deprecated in the rest API. This is not
> in the FLIP.
> 2) The section "ClusterDescriptor or Executor(FLIP-73)(integration)"
> does not reflect our discussion where we said that for now only the
> Executor#execute() will give you the JobClient and there will be a
> separate discussion about alternative ways of exposing the JobClient.
>
> I think that these points should be updated in order for the FLIP to
> reflect the discussion in the ML thread.
>
> Cheers,
> Kostas
>
> On Fri, Oct 11, 2019 at 10:58 AM Biao Liu  wrote:
> >
> > +1 (non-binding), glad to have this improvement!
> >
> > Thanks,
> > Biao /'bɪ.aʊ/
> >
> >
> >
> > On Fri, 11 Oct 2019 at 14:44, Jeff Zhang  wrote:
> >
> > > +1， overall design make sense to me
> > >
> > > SHI Xiaogang  于2019年10月11日周五 上午11:15写道：
> > >
> > > > +1. The interface looks fine to me.
> > > >
> > > > Regards,
> > > > Xiaogang
> > > >
> > > > Zili Chen  于2019年10月9日周三 下午2:36写道：
> > > >
> > > > > Given the ongoing FlinkForward Berlin event, I'm going to extend
> > > > > this vote thread with a bit of period, said until Oct.
> 11th(Friday).
> > > > >
> > > > > Best,
> > > > > tison.
> > > > >
> > > > >
> > > > > Zili Chen  于2019年10月7日周一 下午4:15写道：
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I would like to start the vote for FLIP-74[1], which is
> discussed and
> > > > > > reached a consensus in the discussion thread[2].
> > > > > >
> > > > > > The vote will be open util Oct. 9th(72h starting on Oct.7th),
> unless
> > > > > > there is an objection or not  enough votes.
> > > > > >
> > > > > > Best,
> > > > > > tison.
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-74%3A+Flink+JobClient+API
> > > > > > [2]
> > > > > >
> > > > >
> > > >
> > >
> https://lists.apache.org/x/thread.html/b2e22a45aeb94a8d06b50c4de078f7b23d9ff08b8226918a1a903768@%3Cdev.flink.apache.org%3E
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > Best Regards
> > >
> > > Jeff Zhang
> > >
>

Re: [DISCUSS] FLIP-73: Introducing Executors for job submission

[Zili Chen]

Thanks for your explanation Kostas.

I agree that Clients are independent from the Executors. From
your text I wonder one thing that whether Executor#execute
returns a cluster client or a job client? As discussed previously
I think conceptually it is a job client?

Best,
tison.


Kostas Kloudas  于2019年10月10日周四 下午5:08写道：

> Hi Tison,
>
> I would say that as a first step, and until we see that the interfaces
> we introduce cover all intended purposes, we keep the Executors
> non-public.
> From the previous discussion, I think that in general the Clients are
> independent from the Executors, as the Executors simply use the
> clients to submit jobs and return a cluster client.
>
> Cheers,
> Kostas
>
> On Wed, Oct 9, 2019 at 7:01 PM Zili Chen  wrote:
> >
> > Hi Kostas & Aljoscha,
> >
> > I'm drafting a plan exposing multi-layered clients. It is mainly about
> > how we distinguish different layers and what clients we're going to
> > expose.
> >
> > In FLIP-73 scope I'd like to ask a question that whether or not Executor
> > becomes a public interface that can be made use of by downstream
> > project developer? Or it just an internal concept for unifying job
> > submission?
> > If it is the latter, I'm feeling multi-layer client topic is totally
> > independent from
> > Executor.
> >
> > Best,
> > tison.
> >
> >
> > Thomas Weise  于2019年10月5日周六 上午12:17写道：
> >
> > > It might be useful to mention on FLIP-73 that the intention for
> > > Executor.execute is to be an asynchronous API once it becomes public
> and
> > > also refer to FLIP-74 as such.
> > >
> > >
> > > On Fri, Oct 4, 2019 at 2:52 AM Aljoscha Krettek 
> > > wrote:
> > >
> > > > Hi Tison,
> > > >
> > > > I agree, for now the async Executor.execute() is an internal detail
> but
> > > > during your work for FLIP-74 it will probably also reach the public
> API.
> > > >
> > > > Best,
> > > > Aljoscha
> > > >
> > > > > On 4. Oct 2019, at 11:39, Zili Chen  wrote:
> > > > >
> > > > > Hi Aljoscha,
> > > > >
> > > > > After clearly narrow the scope of this FLIP it looks good to me the
> > > > > interface
> > > > > Executor and its discovery so that I'm glad to see the vote thread.
> > > > >
> > > > > As you said, we should still discuss on implementation details but
> I
> > > > don't
> > > > > think
> > > > > it should be a blocker of the vote thread because a vote means we
> > > > generally
> > > > > agree on the motivation and overall design.
> > > > >
> > > > > As for Executor.execute() to be async, it is much better than we
> keep
> > > the
> > > > > difference between sync/async in this level. But I'd like to note
> that
> > > it
> > > > > only
> > > > > works internally for now because user-facing interface is still
> > > > env.execute
> > > > > which block and return a JobExecutionResult. I'm afraid that there
> are
> > > > > several
> > > > > people depends on the result for doing post execution process,
> although
> > > > it
> > > > > doesn't
> > > > > work on current per-job mode.
> > > > >
> > > > > Best,
> > > > > tison.
> > > > >
> > > > >
> > > > > Aljoscha Krettek  于2019年10月4日周五 下午4:40写道：
> > > > >
> > > > >> Do you all think we could agree on the basic executor primitives
> and
> > > > start
> > > > >> voting on this FLIP? There are still some implementation details
> but I
> > > > >> think we can discuss/tackle them when we get to them and the
> various
> > > > people
> > > > >> implementing this should be in close collaboration.
> > > > >>
> > > > >> Best,
> > > > >> Aljoscha
> > > > >>
> > > > >>> On 4. Oct 2019, at 10:15, Aljoscha Krettek 
> > > > wrote:
> > > > >>>
> > > > >>> Hi,
> > > > >>>
> > > > >>> I think the end goal is to have only one environment per API,
> but I
> > > > >> think we won’t be able to achieve that in the short-term because
> of
> > > > >> bac

Re: [DISCUSS] FLIP-73: Introducing Executors for job submission

[Zili Chen]

Hi Kostas & Aljoscha,

I'm drafting a plan exposing multi-layered clients. It is mainly about
how we distinguish different layers and what clients we're going to
expose.

In FLIP-73 scope I'd like to ask a question that whether or not Executor
becomes a public interface that can be made use of by downstream
project developer? Or it just an internal concept for unifying job
submission?
If it is the latter, I'm feeling multi-layer client topic is totally
independent from
Executor.

Best,
tison.


Thomas Weise  于2019年10月5日周六 上午12:17写道：

> It might be useful to mention on FLIP-73 that the intention for
> Executor.execute is to be an asynchronous API once it becomes public and
> also refer to FLIP-74 as such.
>
>
> On Fri, Oct 4, 2019 at 2:52 AM Aljoscha Krettek 
> wrote:
>
> > Hi Tison,
> >
> > I agree, for now the async Executor.execute() is an internal detail but
> > during your work for FLIP-74 it will probably also reach the public API.
> >
> > Best,
> > Aljoscha
> >
> > > On 4. Oct 2019, at 11:39, Zili Chen  wrote:
> > >
> > > Hi Aljoscha,
> > >
> > > After clearly narrow the scope of this FLIP it looks good to me the
> > > interface
> > > Executor and its discovery so that I'm glad to see the vote thread.
> > >
> > > As you said, we should still discuss on implementation details but I
> > don't
> > > think
> > > it should be a blocker of the vote thread because a vote means we
> > generally
> > > agree on the motivation and overall design.
> > >
> > > As for Executor.execute() to be async, it is much better than we keep
> the
> > > difference between sync/async in this level. But I'd like to note that
> it
> > > only
> > > works internally for now because user-facing interface is still
> > env.execute
> > > which block and return a JobExecutionResult. I'm afraid that there are
> > > several
> > > people depends on the result for doing post execution process, although
> > it
> > > doesn't
> > > work on current per-job mode.
> > >
> > > Best,
> > > tison.
> > >
> > >
> > > Aljoscha Krettek  于2019年10月4日周五 下午4:40写道：
> > >
> > >> Do you all think we could agree on the basic executor primitives and
> > start
> > >> voting on this FLIP? There are still some implementation details but I
> > >> think we can discuss/tackle them when we get to them and the various
> > people
> > >> implementing this should be in close collaboration.
> > >>
> > >> Best,
> > >> Aljoscha
> > >>
> > >>> On 4. Oct 2019, at 10:15, Aljoscha Krettek 
> > wrote:
> > >>>
> > >>> Hi,
> > >>>
> > >>> I think the end goal is to have only one environment per API, but I
> > >> think we won’t be able to achieve that in the short-term because of
> > >> backwards compatibility. This is most notable with the context
> > environment,
> > >> preview environments etc.
> > >>>
> > >>> To keep this FLIP very slim we can make this only about the executors
> > >> and executor discovery. Anything else like job submission semantics,
> > >> detached mode, … can be tackled after this. If we don’t focus I’m
> afraid
> > >> this will drag on for quite a while.
> > >>>
> > >>> One thing I would like to propose to make this easier is to change
> > >> Executor.execute() to return a CompletableFuture and to completely
> > remove
> > >> the “detached” logic from ClusterClient. That way, the new components
> > make
> > >> no distinction between “detached” and “attached” but we can still do
> it
> > in
> > >> the CLI (via the ContextEnvironment) to support the existing
> “detached”
> > >> behaviour of the CLI that users expect. What do you think about this?
> > >>>
> > >>> Best,
> > >>> Aljoscha
> > >>>
> > >>>> On 3. Oct 2019, at 10:03, Zili Chen  wrote:
> > >>>>
> > >>>> Thanks for your explanation Kostas to make it clear subtasks under
> > >> FLIP-73.
> > >>>>
> > >>>> As you described, changes of Environment are included in this FLIP.
> > For
> > >>>> "each
> > >>>> API to have a single Environment", it could be helpful to describe
> > which
> > >>>> 

Re: [VOTE] FLIP-74: Flink JobClient API

[Zili Chen]

Given the ongoing FlinkForward Berlin event, I'm going to extend
this vote thread with a bit of period, said until Oct. 11th(Friday).

Best,
tison.


Zili Chen  于2019年10月7日周一 下午4:15写道：

> Hi all,
>
> I would like to start the vote for FLIP-74[1], which is discussed and
> reached a consensus in the discussion thread[2].
>
> The vote will be open util Oct. 9th(72h starting on Oct.7th), unless
> there is an objection or not  enough votes.
>
> Best,
> tison.
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-74%3A+Flink+JobClient+API
> [2]
> https://lists.apache.org/x/thread.html/b2e22a45aeb94a8d06b50c4de078f7b23d9ff08b8226918a1a903768@%3Cdev.flink.apache.org%3E
>

[jira] [Created] (FLINK-14343) Remove uncompleted YARNHighAvailabilityService

[Zili Chen (Jira)]

Zili Chen created FLINK-14343:
-

 Summary: Remove uncompleted YARNHighAvailabilityService
 Key: FLINK-14343
 URL: https://issues.apache.org/jira/browse/FLINK-14343
 Project: Flink
  Issue Type: Task
  Components: Runtime / Coordination
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.10.0


Corresponding mailing list 
[thread|https://lists.apache.org/x/thread.html/6022f2124be91e3f4667d61a977ea0639e2c19286560d6d1cb874792@%3Cdev.flink.apache.org%3E].

Noticed that there are several stale & uncompleted high-availability services 
implementations, I start this thread in order to see whether or not we can 
remove them for a
clean codebase.

Below are all of classes I noticed.

- YarnHighAvailabilityServices
- AbstractYarnNonHaServices
- YarnIntraNonHaMasterServices
- YarnPreConfiguredMasterNonHaServices
- SingleLeaderElectionService
- FsNegativeRunningJobsRegistry
(as well as their dedicated tests)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Release 1.9.1, release candidate #1

[Zili Chen]

Hi Jark,

I notice a critical bug[1] is marked resolved in 1.9.1 but given 1.9.1
has been cut I'd like to throw the issue here so that we're sure
whether or not it is included in 1.9.1.

Best,
tison.

[1] https://issues.apache.org/jira/browse/FLINK-14315


Jark Wu  于2019年9月30日周一 下午3:25写道：

>  Hi everyone,
>
> Please review and vote on the release candidate #1 for the version 1.9.1,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release and binary convenience releases to be
> deployed to dist.apache.org [2], which are signed with the key with
> fingerprint E2C45417BED5C104154F341085BACB5AEFAE3202 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "release-1.9.1-rc1" [5],
> * website pull request listing the new release and adding announcement blog
> post [6].
>
> The vote will be open for at least 72 hours.
> Please cast your votes before *Oct. 3th 2019, 08:00 UTC*.
>
> It is adopted by majority approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Jark
>
> [1]
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12346003
> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.9.1-rc1/
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4]
> https://repository.apache.org/content/repositories/orgapacheflink-1272/
> [5]
>
> https://github.com/apache/flink/commit/4d56de81cb692c68a7d1dbfff13087a5079a8252
> [6] https://github.com/apache/flink-web/pull/274
>

Re: [CODE STYLE] Parameters of method are always final

[Zili Chen]

Thanks for your thoughts Arvid & Piotr,

I check out the effect of ParameterAssignment[1] and it does
prevent codes from modifying parameter which I argued above
the most value introduced by `final` modifier of parameter.

So firstly, I think it's good to enable ParameterAssignment in our
codebase.

Besides, there is no rule to forbid `final` modifier of parameter.
Instead, there is a rule to enforce `final` modifier[2] but given [1]
enabled it is actually redundant.

The main purpose for, given enforced not modify parameters, reducing
`final` modifiers of parameter is to remove redundant modifier so that we
don't see have declaration like

T fn(
  final ArgumentType1 argument1,
  final ArgumentType2 argument2,
  ...)

because we actually don't mark final everywhere so it might make some
confusions.

Given [1] is enforced these `final` are indeed redundant so that we can
add a convention to reduce `final` modifiers of parameters, which is a net
win.

Best,
tison.

[1] https://checkstyle.sourceforge.io/config_coding.html#ParameterAssignment
[2] https://checkstyle.sourceforge.io/config_misc.html#FinalParameters



Piotr Nowojski  于2019年10月7日周一 下午3:49写道：

> Hi,
>
> Couple of arguments to counter the proposal of making the `final` keyword
> obligatory. Can we prepare a code style/IDE settings to add it
> automatically? If not, I would be strongly against it, since:
>
> - Intellij’s automatic refactor actions will not work properly.
> - I don’t think it’s a big deal. I don’t remember having any issues with
> the lack or presence of the `final` keyword.
> - `final` is pretty much useless in most of the cases (it’s not `const`
> and it works only for the primitive types).
> - I don’t like the extra overhead of doing something of very little extra
> value. Especially the extra hassle of going back & forth during the reviews
> (both as a contributor & as a reviewer).
> - If missing `final` keyword caused some confusion, because surprisingly a
> parameter was modified somewhere in the function and it wasn’t obviously
> visible, the function is doing probably too many things and it’s too
> long/too complicated…
>
> Generally speaking, I’m against adding minor things to our codestyle that
> can not be enforced and added automatically.
>
> Piotrek
>
> > On 7 Oct 2019, at 09:13, Arvid Heise  wrote:
> >
> > Hi guys,
> >
> > I'm a bit torn. In general, +1 for making parameters effectively final.
> >
> > The question is how to enforce it. We can make it explicit (like Aljoscha
> > said). All IDEs will show immediately warnings/errors for violations. It
> > would allow to softly migrate code.
> >
> > Another option is to use a checkstyle rule [1]. Then, we could omit the
> > final keyword and rely on checkstyle checks as we do for quite a few
> other
> > things. A hard checkstyle rule would probably fail on a good portion of
> the
> > current code base. But we would also remove reassignment to parameters
> > (which I consider an anti-pattern).
> >
> > If we opt not to enforce it, then -1 for removing final keywords from my
> > side.
> >
> > Best,
> >
> > Arvid
> >
> > [1]
> >
> https://checkstyle.sourceforge.io/apidocs/com/puppycrawl/tools/checkstyle/checks/coding/ParameterAssignmentCheck.html
> >
> > On Fri, Oct 4, 2019 at 1:22 PM Zili Chen  wrote:
> >
> >> Hi Aljoscha,
> >>
> >> I totally agree with you on field topic. Of course it makes significant
> >> difference whether
> >> or not a field is final and IDE/compiler can help on checking.
> >>
> >> Here are several thoughts about final modifier on parameters and why I
> >> propose this one:
> >>
> >> 1. parameters should be always final
> >>
> >> As described above, there is no reason a parameter to be non-final. So
> >> different with field,
> >> a field can be final or non-final based on whether or not it is
> immutable.
> >> Thus with such a
> >> code style guide in our community, we can work towards a codebase every
> >> parameter is
> >> effectively final.
> >>
> >> 2. parameter final cannot be inherited
> >>
> >> Unfortunately Java doesn't keep final modifier of method parameter when
> >> inheritance. So
> >> even you mark a parameter as final in an interface or super class, you
> have
> >> to re-mark it
> >> as final in its implementation or subclass. From another perspective,
> final
> >> modifier of
> >> parameter is a local attribute of parameter so that we can narrow
> possible
> >> effect during
> >> rev

[VOTE] FLIP-74: Flink JobClient API

[Zili Chen]

Hi all,

I would like to start the vote for FLIP-74[1], which is discussed and
reached a consensus in the discussion thread[2].

The vote will be open util Oct. 9th(72h starting on Oct.7th), unless
there is an objection or not  enough votes.

Best,
tison.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-74%3A+Flink+JobClient+API
[2]
https://lists.apache.org/x/thread.html/b2e22a45aeb94a8d06b50c4de078f7b23d9ff08b8226918a1a903768@%3Cdev.flink.apache.org%3E