Re: [DISCUSS] FLINK-16194: Refactor the Kubernetes architecture design

[Canbin Zheng]

Hi, everyone,

I have pushed a PR  for this
issue, looking forward to your feedback.


Cheers,
Canbin Zheng

Canbin Zheng  于2020年2月26日周三 上午10:39写道：

> Thanks for the detailed PR advice, I would separate the commits as clear
> as possible to help the code reviewing.
>
>
> Cheers,
> Canbin Zheng
>
> tison  于2020年2月25日周二 下午10:11写道：
>
>> Thanks for your clarification Yang! We're on the same page.
>>
>> Best,
>> tison.
>>
>>
>> Yang Wang  于2020年2月25日周二 下午10:07写道：
>>
>>> Hi tison,
>>>
>>> I do not mean to keep two decorator at the same. Since the two
>>> decorators are
>>> not api compatible, it is meaningless. I am just thinking how
>>> to organize the
>>> commits/PRs to make the review easier. The reviewers may need some
>>> context
>>> to get the point.
>>>
>>>
>>>
>>> Best,
>>> Yang
>>>
>>> tison  于2020年2月25日周二 下午8:23写道：
>>>
 The process in my mind is somehow like this commit[1] which belongs to
 this pr[2]
 that we firstly introduce the new implementation and then replace it
 with the original
 one. The difference is that these two versions of decorators are not
 api compatible
 while adding a switch for such an internal abstraction or extracting a
 clumsy
 "common" interface doesn't benefit.

 Best,
 tison.

 [1]
 https://github.com/apache/flink/commit/1f2969357c441e24b71daef83d21563da9a93bb4
 [2] https://github.com/apache/flink/pull/9832




 tison  于2020年2月25日周二 下午8:08写道：

> I agree for separating commits we can have multiple commits that
> firstly add the new parameters
> and decorators,  and later replace current decorators with new
> decorators which are well
> unit tested.
>
> However, it makes no sense we have two codepaths from FlinkKubeClient
> to decorators
> since these two version of decorators are not api compatible and there
> is no reason we keep both
> of them.
>
> Best,
> tison.
>
>
> Yang Wang  于2020年2月25日周二 下午7:50写道：
>
>> I think if we could, splitting into as many PRs as possible is good.
>> Maybe we could
>> introduce the new designed decorators and parameter parser first, and
>> leave the existing
>> decorators as legacy. Once all the new decorators is ready and well
>> tested, we could
>> remove the legacy codes and use the new decorators in the kube client
>> implementation.
>>
>>
>> Best,
>> Yang
>>
>> Canbin Zheng  于2020年2月25日周二 下午6:16写道：
>>
>>> Hi, Till,
>>>
>>> Great thanks for your advice, I totally agree with you to split the
>>> changes up in as many PRs as possible. The part of "Parameter Parser" is
>>> trivial so that we prefer to make one PR to avoid adapting a lot of 
>>> pieces
>>> of code that would be deleted immediately with the following decorator
>>> refactoring PR. Actually I won't insist on one PR, could it be possible
>>> that I first try out with one PR and let the committers help assess 
>>> whether
>>> it is necessary to split the changes into several PRs?  Kindly expect to
>>> see your reply.
>>>
>>

[jira] [Created] (FLINK-16301) Annoying "Cannot find FunctionDefinition" messages with SQL for f_proctime or =

[Nico Kruber (Jira)]

Nico Kruber created FLINK-16301:
---

 Summary: Annoying "Cannot find FunctionDefinition" messages with 
SQL for f_proctime or =
 Key: FLINK-16301
 URL: https://issues.apache.org/jira/browse/FLINK-16301
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.10.0
Reporter: Nico Kruber


When running the following SQL query
{code}
SELECT
D1.col1 AS A,
D1.col2 AS B,
D1.col3 AS C,
D1.col4 AS D,
D1.col5 AS E,
D2.col1 AS F,
D2.col2 AS G,
D2.col3 AS H,
D2.col4 AS I,
D2.col5 AS J,
D3.col1 AS K,
D3.col2 AS L,
D3.col3 AS M,
D3.col4 AS N,
D3.col5 AS O,
D4.col1 AS P,
D4.col2 AS Q,
D4.col3 AS R,
D4.col4 AS S,
D4.col5 AS T,
D5.col1 AS U,
D5.col2 AS V,
D5.col3 AS W,
D5.col4 AS X,
D5.col5 AS Y
FROM
fact_table,
LATERAL TABLE (dimension_table1(f_proctime)) AS D1,
LATERAL TABLE (dimension_table2(f_proctime)) AS D2,
LATERAL TABLE (dimension_table3(f_proctime)) AS D3,
LATERAL TABLE (dimension_table4(f_proctime)) AS D4,
LATERAL TABLE (dimension_table5(f_proctime)) AS D5
WHERE
fact_table.dim1 = D1.id
AND fact_table.dim2 = D2.id
AND fact_table.dim3 = D3.id
AND fact_table.dim4 = D4.id
AND fact_table.dim5 = D5.id
{code}

with the Blink planner, it prints a log of bogus warnings about unknown 
functions for things like {{f_proctime}} or {{=}} at INFO level which should be 
DEBUG level at least in order not to bother the users with it. The messages I 
got are:

{code}
13:33:59,590 INFO  org.apache.flink.table.module.ModuleManager  
 - Cannot find FunctionDefinition f_proctime from any loaded modules
13:33:59,641 INFO  org.apache.flink.table.module.ModuleManager  
 - Cannot find FunctionDefinition f_proctime from any loaded modules
13:33:59,644 INFO  org.apache.flink.table.module.ModuleManager  
 - Cannot find FunctionDefinition f_proctime from any loaded modules
13:33:59,647 INFO  org.apache.flink.table.module.ModuleManager  
 - Cannot find FunctionDefinition f_proctime from any loaded modules
13:33:59,650 INFO  org.apache.flink.table.module.ModuleManager  
 - Cannot find FunctionDefinition f_proctime from any loaded modules
13:33:59,662 INFO  org.apache.flink.table.module.ModuleManager  
 - Cannot find FunctionDefinition = from any loaded modules
13:33:59,665 INFO  org.apache.flink.table.module.ModuleManager  
 - Cannot find FunctionDefinition = from any loaded modules
13:33:59,666 INFO  org.apache.flink.table.module.ModuleManager  
 - Got FunctionDefinition and from module core
13:33:59,667 INFO  org.apache.flink.table.module.ModuleManager  
 - Cannot find FunctionDefinition = from any loaded modules
13:33:59,668 INFO  org.apache.flink.table.module.ModuleManager  
 - Got FunctionDefinition and from module core
13:33:59,669 INFO  org.apache.flink.table.module.ModuleManager  
 - Cannot find FunctionDefinition = from any loaded modules
13:33:59,670 INFO  org.apache.flink.table.module.ModuleManager  
 - Got FunctionDefinition and from module core
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16302) add log list and read log by name for taskmanager

[lining (Jira)]

lining created FLINK-16302:
--

 Summary: add log list and read log by name for taskmanager
 Key: FLINK-16302
 URL: https://issues.apache.org/jira/browse/FLINK-16302
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / REST
Reporter: lining






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16303) add log list and read log by name for jobmanager

[lining (Jira)]

lining created FLINK-16303:
--

 Summary: add log list and read log by name for jobmanager
 Key: FLINK-16303
 URL: https://issues.apache.org/jira/browse/FLINK-16303
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / REST
Reporter: lining






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16304) Remove python packages bundled in the flink-python jar

[Wei Zhong (Jira)]

Wei Zhong created FLINK-16304:
-

 Summary: Remove python packages bundled in the flink-python jar
 Key: FLINK-16304
 URL: https://issues.apache.org/jira/browse/FLINK-16304
 Project: Flink
  Issue Type: Improvement
  Components: API / Python
Affects Versions: 1.11.0
Reporter: Wei Zhong
 Fix For: 1.11.0


As the Python Dependency Management API is sufficient to control the python 
environment, and current design that bundles python packages to the 
flink-python jar and extracts when running Python UDF has been a blocker of the 
Python UDF performance optimization. The python packages bundled in the 
flink-python jar need to be removed now.

It means that users need to ensure that the PyFlink and its dependencies are 
pre-installed in the client and cluster environment when submitting jobs. Of 
course the cluster environment can be configured via Python Dependency 
Management API.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16305) FlinkYarnSessionClI ignores target executor and uses yarn-session if YARN properties file is present

[Daniel Laszlo Magyar (Jira)]

Daniel Laszlo Magyar created FLINK-16305:


 Summary: FlinkYarnSessionClI ignores target executor and uses 
yarn-session if YARN properties file is present
 Key: FLINK-16305
 URL: https://issues.apache.org/jira/browse/FLINK-16305
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.10.0
Reporter: Daniel Laszlo Magyar


The presence of the hidden YARN property file (which contains the last started 
YARN session’s application id), causes the cli to ignore the 
{{execution.target}} property set in the {{conf/flink-conf.yaml}} configuration 
file, which leads to unexpected behaviour at the time of job submission via 
cli, e.g. when using {{flink run}} or SQL client.
 The code that ignores the execution target if the YARN application id is set 
in the hidden property file is at 
[https://github.com/apache/flink/blob/release-1.10/flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java#L337-L351].
 
Reproduction steps:
 • start flink yarn session via {{./bin/yarn-session.sh -d}}, this writes the 
application id to {{/tmp/.yarn-properties-root}}
 • set {{execution.target: yarn-per-job}} in {{/etc/flink/conf/flink-conf.yaml}}
 • enable debug logging
 • run a flink job e.g. {{flink run -d -p 2 examples/streaming/WordCount.jar 
--input README.txt}}
 • the logs below show that even though the {{execution.target}} property is 
read properly, {{FlinkYarnSessionCli}} is chosen and the execution.target is 
reset to yarn-session
{code:java}
20/02/26 12:14:24 INFO configuration.GlobalConfiguration: Loading configuration 
property: execution.target, yarn-per-job
...
20/02/26 12:14:24 INFO cli.FlinkYarnSessionCli: Found Yarn properties file 
under /tmp/.yarn-properties-root.
20/02/26 12:14:24 DEBUG fs.FileSystem: Loading extension file systems via 
services
20/02/26 12:14:24 DEBUG cli.CliFrontend: Custom commandlines: 
[org.apache.flink.yarn.cli.FlinkYarnSessionCli@43df23d3, 
org.apache.flink.client.cli.ExecutorCLI@6d60fe40, 
org.apache.flink.client.cli.DefaultCLI@792b749c]
20/02/26 12:14:24 DEBUG cli.CliFrontend: Checking custom commandline 
org.apache.flink.yarn.cli.FlinkYarnSessionCli@43df23d3, isActive: true
...
20/02/26 12:14:25 DEBUG cli.CliFrontend: Effective executor configuration: 
{...execution.target=yarn-session, }
20/02/26 12:14:25 INFO client.ClientUtils: Starting program (detached: true)

{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16306) Validate YARN session state before job submission

[Daniel Laszlo Magyar (Jira)]

Daniel Laszlo Magyar created FLINK-16306:


 Summary: Validate YARN session state before job submission
 Key: FLINK-16306
 URL: https://issues.apache.org/jira/browse/FLINK-16306
 Project: Flink
  Issue Type: Task
  Components: Client / Job Submission
Affects Versions: 1.10.0
Reporter: Daniel Laszlo Magyar


To better handle not properly stopped yarn sessions, state of the session 
should be validated before job submission.
 Currently if {{execution.target: yarn-session}} is set in 
{{conf/flink-conf.yaml}} and the hidden YARN property file 
{{/tmp/.yarn-properties-root}} is present, FlinkSessionCli tries to submit the 
job regardless of the session’s state. 
Apparently, the property file cannot get cleaned up automatically when the 
session is killed e.g. via {{yarn app -kill }} and this behaviour is 
pointed out in the logs upon running via yarn-session.sh, but the contained 
application state could be checked before submitting to it. The current 
behaviour feels inconsistent with the scenario when the YARN property file 
actually does get cleaned up e.g. by manually deleting the file, in which case 
a per-job cluster is spun up before submitting to it.
 
Replication steps:
 • start flink yarn session via {{./bin/yarn-session.sh -d}}, this writes the 
application id to {{/tmp/.yarn-properties-root}}
 • set {{execution.target: yarn-session}} in {{/etc/flink/conf/flink-conf.yaml}}
 • kill session via {{yarn app -kill }}
 • try to submit job, e.g.: {{flink run -d -p 2 
examples/streaming/WordCount.jar}}

The logs clearly state that the FlinkYarnSessionCli tries to submit the job to 
the killed application:
{code:java}
20/02/26 13:34:26 ERROR yarn.YarnClusterDescriptor: The application 
application_1582646904843_0021 doesn't run anymore. It has previously completed 
with final status: KILLED
...
20/02/26 13:34:26 ERROR cli.CliFrontend: Error while running the command.
org.apache.flink.client.program.ProgramInvocationException: The main method 
caused an error: Couldn't retrieve Yarn cluster
at 
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:335)
at 
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:205)
at 
org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:138)
at 
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:709)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:258)
at 
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:940)
at 
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1014)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
at 
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1014)
Caused by: org.apache.flink.client.deployment.ClusterRetrieveException: 
Couldn't retrieve Yarn cluster
at 
org.apache.flink.yarn.YarnClusterDescriptor.retrieve(YarnClusterDescriptor.java:365)
at 
org.apache.flink.yarn.YarnClusterDescriptor.retrieve(YarnClusterDescriptor.java:122)
at 
org.apache.flink.client.deployment.executors.AbstractSessionClusterExecutor.execute(AbstractSessionClusterExecutor.java:63)
at 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1750)
at 
org.apache.flink.streaming.api.environment.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:94)
at 
org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:63)
at 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1637)
at 
org.apache.flink.streaming.examples.wordcount.WordCount.main(WordCount.java:96)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:321)
... 11 more
Caused by: java.lang.RuntimeException: The Yarn application 
application_1582646904843_0021 doesn't run anymore.
at 
org.apache.flink.yarn.YarnClusterDescriptor.retrieve(YarnClusterDescriptor.java:352)

[jira] [Created] (FLINK-16307) LocalStandaloneFlinkResource should use Dist#startFlinkCluster

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-16307:


 Summary: LocalStandaloneFlinkResource should use 
Dist#startFlinkCluster
 Key: FLINK-16307
 URL: https://issues.apache.org/jira/browse/FLINK-16307
 Project: Flink
  Issue Type: Improvement
  Components: Test Infrastructure
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.11.0


The {{LocalStandaloneFlinkResource}} currently manually calls the 
jobmanager.sj/taskmanager.sh scripts.

Since the documentation instructs users to call call the start-cluster script 
we should do this too.

This also entails exposing a method on the distribution for modifying the 
{{slaves}} file.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16308) SQL connector download links are broken

[Fabian Hueske (Jira)]

Fabian Hueske created FLINK-16308:
-

 Summary: SQL connector download links are broken
 Key: FLINK-16308
 URL: https://issues.apache.org/jira/browse/FLINK-16308
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Reporter: Fabian Hueske


The download links for the SQL connectors on 
https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/connect.html
 are broken because central.maven.org is down.

The URLs should be updated to 
https://repo.maven.apache.org/maven2/org/apache/flink/



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16309) ElasticSearch 7 connector is missing in SQL connector list

[Fabian Hueske (Jira)]

Fabian Hueske created FLINK-16309:
-

 Summary: ElasticSearch 7 connector is missing in SQL connector list
 Key: FLINK-16309
 URL: https://issues.apache.org/jira/browse/FLINK-16309
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Reporter: Fabian Hueske


The ES7 connector is not listed on 
https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/connect.html



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] FLINK-16194: Refactor the Kubernetes architecture design

[Yang Wang]

Great work! I could help to review and test.

Best,
Yang

Canbin Zheng  于2020年2月27日周四 下午4:24写道：

> Hi, everyone,
>
> I have pushed a PR  for this
> issue, looking forward to your feedback.
>
>
> Cheers,
> Canbin Zheng
>
> Canbin Zheng  于2020年2月26日周三 上午10:39写道：
>
>> Thanks for the detailed PR advice, I would separate the commits as clear
>> as possible to help the code reviewing.
>>
>>
>> Cheers,
>> Canbin Zheng
>>
>> tison  于2020年2月25日周二 下午10:11写道：
>>
>>> Thanks for your clarification Yang! We're on the same page.
>>>
>>> Best,
>>> tison.
>>>
>>>
>>> Yang Wang  于2020年2月25日周二 下午10:07写道：
>>>
 Hi tison,

 I do not mean to keep two decorator at the same. Since the two
 decorators are
 not api compatible, it is meaningless. I am just thinking how
 to organize the
 commits/PRs to make the review easier. The reviewers may need some
 context
 to get the point.



 Best,
 Yang

 tison  于2020年2月25日周二 下午8:23写道：

> The process in my mind is somehow like this commit[1] which belongs to
> this pr[2]
> that we firstly introduce the new implementation and then replace it
> with the original
> one. The difference is that these two versions of decorators are not
> api compatible
> while adding a switch for such an internal abstraction or extracting a
> clumsy
> "common" interface doesn't benefit.
>
> Best,
> tison.
>
> [1]
> https://github.com/apache/flink/commit/1f2969357c441e24b71daef83d21563da9a93bb4
> [2] https://github.com/apache/flink/pull/9832
>
>
>
>
> tison  于2020年2月25日周二 下午8:08写道：
>
>> I agree for separating commits we can have multiple commits that
>> firstly add the new parameters
>> and decorators,  and later replace current decorators with new
>> decorators which are well
>> unit tested.
>>
>> However, it makes no sense we have two codepaths from FlinkKubeClient
>> to decorators
>> since these two version of decorators are not api compatible and
>> there is no reason we keep both
>> of them.
>>
>> Best,
>> tison.
>>
>>
>> Yang Wang  于2020年2月25日周二 下午7:50写道：
>>
>>> I think if we could, splitting into as many PRs as possible is good.
>>> Maybe we could
>>> introduce the new designed decorators and parameter parser first,
>>> and leave the existing
>>> decorators as legacy. Once all the new decorators is ready and well
>>> tested, we could
>>> remove the legacy codes and use the new decorators in the kube
>>> client implementation.
>>>
>>>
>>> Best,
>>> Yang
>>>
>>> Canbin Zheng  于2020年2月25日周二 下午6:16写道：
>>>
 Hi, Till,

 Great thanks for your advice, I totally agree with you to split the
 changes up in as many PRs as possible. The part of "Parameter Parser" 
 is
 trivial so that we prefer to make one PR to avoid adapting a lot of 
 pieces
 of code that would be deleted immediately with the following decorator
 refactoring PR. Actually I won't insist on one PR, could it be possible
 that I first try out with one PR and let the committers help assess 
 whether
 it is necessary to split the changes into several PRs?  Kindly expect 
 to
 see your reply.

>>>

[jira] [Created] (FLINK-16310) Javadoc of AppendingState.get() contradicts with behavior of UserFacingListState

[Tzu-Li (Gordon) Tai (Jira)]

Tzu-Li (Gordon) Tai created FLINK-16310:
---

 Summary: Javadoc of AppendingState.get() contradicts with behavior 
of UserFacingListState
 Key: FLINK-16310
 URL: https://issues.apache.org/jira/browse/FLINK-16310
 Project: Flink
  Issue Type: Improvement
  Components: Documentation, Runtime / State Backends
Affects Versions: 1.10.0, 1.9.2, 1.8.3, 1.7.2, 1.6.4
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai


The {{get()}} method on the user-facing state handle interface 
{{AppendingState}} states that:
{code}
@return The operator state value corresponding to the current input or NULL if 
the state is empty.
{code}

This behavior, is not true for the user-facing list state handles, as the 
behavior has always been that if the list state does not have elements, an 
empty list is returned (see FLINK-4307 / {{UserFacingListState}}).

The fix would be to only mention the null-returning behavior on the internal 
list state interface, i.e. {{InternalListState}}, and fix the Javadoc for 
{{AppendingState}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16311) PersistedAppendingBuffer state primitive has incorrect contracts for view() method

[Tzu-Li (Gordon) Tai (Jira)]

Tzu-Li (Gordon) Tai created FLINK-16311:
---

 Summary: PersistedAppendingBuffer state primitive has incorrect 
contracts for view() method
 Key: FLINK-16311
 URL: https://issues.apache.org/jira/browse/FLINK-16311
 Project: Flink
  Issue Type: Bug
  Components: Stateful Functions
Affects Versions: statefun-1.1
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai


The {{PersistedAppendingBuffer}} state primitive in Stateful Functions, states 
that {{view()}} returns {{null}} if the buffer wasn't accessed before, or after 
a {{clear()}}.

This actually is not possible. The {{PersistedAppendingBuffer}} primitive is 
based on Flink's {{ListState}} state handle, which always returns an empty list 
if the elements of the list is empty, and never a {{null}}.

This means that the {{PersistedAppendingBuffer}} won't be able to differentiate 
the cases (e.g. if the state wasn't accessed or cleared v.s. an empty buffer).
We suggest to just change the contract so that 
{{PersistedAppendingBuffer#view()}} follows the behavior of Flink's 
{{ListState}} to never return null.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16312) Stateful Function's HttpFunction falls into endless loop resending 0-sized batch requests

[Tzu-Li (Gordon) Tai (Jira)]

Tzu-Li (Gordon) Tai created FLINK-16312:
---

 Summary: Stateful Function's HttpFunction falls into endless loop 
resending 0-sized batch requests
 Key: FLINK-16312
 URL: https://issues.apache.org/jira/browse/FLINK-16312
 Project: Flink
  Issue Type: Bug
  Components: Stateful Functions
Affects Versions: statefun-1.1
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai


This is caused by FLINK-16311, because the {{HttpFunction}} was relying on 
contracts of {{PersistedAppendingBuffer}} that were not correct.

Specifically, the culprit is here:
https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/httpfn/HttpFunction.java#L152

This branch checks if the obtained view is {{null}}, if not, resends a batch 
request.
However, that would never be {{null}} due to FLINK-16311.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-100: Add Attempt Information

[Till Rohrmann]

I think your proposal is better because then we don't have to bump the REST
API version.

Cheers,
Till

On Thu, Feb 27, 2020 at 8:02 AM Yadong Xie  wrote:

> Hi Till
>
> We keep response a flattened SubtaskTimeInfo and an array of
> SubtaskTimeInfo to keep the restAPI backward compatible, since Flink users
> may still need the API('/jobs/{jobId}/vertices/{vertexId}/subtasktimes') to
> get the flattened SubtaskTimeInfo.
>
> If we want to get an array of SubtaskTimeInfo in the API, a new URL needs
> to be created other than reuse the old one.
>
> Both solutions are ok for me. What do you think about it?
>
> Till Rohrmann  于2020年2月26日周三 下午10:53写道：
>
> > Fair enough. If this should become a problem we could introduce it later
> as
> > well.
> >
> > What about changing the SubtasksTimeInfo response type into an array of
> > SubtaskTimeInfo? At the moment SubtasksTimeInfo contains a
> > flattened SubtaskTimeInfo and and array of SubtaskTimeInfo for the
> previous
> > attempts.
> >
> > Cheers,
> > Till
> >
> > On Wed, Feb 26, 2020 at 1:16 PM Yadong Xie  wrote:
> >
> > > Hi Till
> > >
> > > Thanks for your comments.
> > >
> > > > I have a comment concerning the SubtasksTimesHandler
> > >
> > > It would be much easier for the frontend to handle a large amount of
> data
> > > if we have a rest API parameter filter, but in my opinion, the attempt
> > list
> > > data is not large enough that we have to rely on the rest API
> parameters
> > > paging, we still can handle them all in the frontend.
> > >
> > > Users can filter the attempt list by the
> > status(scheduled/created/deploying
> > > and so on) and other keywords(attempt_id and so on) directly in the
> > > frontend since all data are listed from the rest API.
> > > If we move some of the filter parameters to the rest API path
> parameter,
> > > all the other filter parameters need to be moved too.
> > >
> > > I suggest adding an attempt id filter in the UI to help users filter
> the
> > > desired attempt, and all the filtering process is running inside the
> > > browser side, what do you think about this?
> > >
> > >
> > >
> > >
> > > Till Rohrmann  于2020年2月25日周二 下午11:40写道：
> > >
> > > > Hi Yadong,
> > > >
> > > > thanks for creating this FLIP. I like the idea to make the web-ui
> > > > information richer wrt to subtask attempt information.
> > > >
> > > > I have a comment concerning the SubtasksTimesHandler: Should we
> change
> > > the
> > > > response type SubtasksTimeInfo so that it simply contains an
> > > > array of SubtaskTimeInfo? One could add an attempt range path
> parameter
> > > to
> > > > the SubtasksTimesHandler to be able to control which attempts will be
> > > > returned.
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Tue, Feb 25, 2020 at 9:57 AM Benchao Li 
> > wrote:
> > > >
> > > > > Hi Yadong,
> > > > >
> > > > > Thanks for the updating.  LGTM now.
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Yadong Xie  于2020年2月25日周二 下午4:41写道：
> > > > >
> > > > > > Hi Kurt
> > > > > >
> > > > > > There will be no differences between batch jobs and stream jobs
> in
> > > > > > subtask-attempt level in the UI
> > > > > > The only differences are in the vertex timeline, I have added a
> > > > > screenshot
> > > > > > of the batch job in the FLIP-100 since the batch job will
> disappear
> > > > from
> > > > > > the list after it finished soon.
> > > > > > here is the link:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-100%3A+Add+Attempt+Information
> > > > > >
> > > > > >
> > > > > > Kurt Young  于2020年2月21日周五 上午11:51写道：
> > > > > >
> > > > > > > Hi Yadong,
> > > > > > >
> > > > > > > Thanks for the proposal, it's a useful feature, especially for
> > > batch
> > > > > > jobs.
> > > > > > > But according
> > > > > > > to the examples you gave, I can't tell whether i got required
> > > > > information
> > > > > > > from that.
> > > > > > > Can you replace the demo job to a more complex batch job and
> then
> > > we
> > > > > can
> > > > > > > see some
> > > > > > > differences of start/stop time of different tasks and attempts?
> > > > > > >
> > > > > > > Best,
> > > > > > > Kurt
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Feb 20, 2020 at 5:46 PM Yadong Xie <
> vthink...@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi all
> > > > > > > >
> > > > > > > > I want to start the vote for FLIP-100, which proposes to add
> > > > attempt
> > > > > > > > information inside subtask and timeline in web UI.
> > > > > > > >
> > > > > > > > To help everyone better understand the proposal, we spent
> some
> > > > > efforts
> > > > > > on
> > > > > > > > making an online POC
> > > > > > > >
> > > > > > > > Timeline Attempt (click the vertex timeline to see the
> > > > differences):
> > > > > > > > previous web:
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://101.132.122.69:8081/#/job/9d651769488466d33e7a607e85203543/timeline
> > > > > > > > POC

[jira] [Created] (FLINK-16313) flink-state-processor-api: surefire execution unstable on Azure

[Robert Metzger (Jira)]

Robert Metzger created FLINK-16313:
--

 Summary: flink-state-processor-api: surefire execution unstable on 
Azure
 Key: FLINK-16313
 URL: https://issues.apache.org/jira/browse/FLINK-16313
 Project: Flink
  Issue Type: Bug
  Components: API / State Processor, Tests
Reporter: Robert Metzger


Log file: 
https://dev.azure.com/rmetzger/Flink/_build/results?buildId=5686&view=logs&j=41cba0bb-1271-5adb-01cc-4768f26a8311&t=44574c85-1cd0-5978-cccf-f0cf7e87a36a

{code}
2020-02-27T12:36:35.2860111Z [INFO] flink-table-planner 
 SUCCESS [01:47 min]
2020-02-27T12:36:35.2860966Z [INFO] flink-cep-scala 
 SUCCESS [  5.041 s]
2020-02-27T12:36:35.2861740Z [INFO] flink-sql-client 
... SUCCESS [03:00 min]
2020-02-27T12:36:35.2862503Z [INFO] flink-state-processor-api 
.. FAILURE [ 15.394 s]
2020-02-27T12:36:35.2863237Z [INFO] 

2020-02-27T12:36:35.2863587Z [INFO] BUILD FAILURE
2020-02-27T12:36:35.2864071Z [INFO] 

2020-02-27T12:36:35.2864428Z [INFO] Total time: 05:38 min
2020-02-27T12:36:35.2866349Z [INFO] Finished at: 2020-02-27T12:36:35+00:00
2020-02-27T12:36:35.9345815Z [INFO] Final Memory: 147M/2914M
2020-02-27T12:36:35.9347238Z [INFO] 

2020-02-27T12:36:35.9355362Z [WARNING] The requested profile "skip-webui-build" 
could not be activated because it does not exist.
2020-02-27T12:36:35.9367919Z [ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.22.1:test (integration-tests) 
on project flink-state-processor-api_2.11: There are test failures.
2020-02-27T12:36:35.9368804Z [ERROR] 
2020-02-27T12:36:35.9369489Z [ERROR] Please refer to 
/__w/2/s/flink-libraries/flink-state-processing-api/target/surefire-reports for 
the individual test results.
2020-02-27T12:36:35.9370249Z [ERROR] Please refer to dump files (if any exist) 
[date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
2020-02-27T12:36:35.9370713Z [ERROR] ExecutionException Error occurred in 
starting fork, check output in log
2020-02-27T12:36:35.9371279Z [ERROR] 
org.apache.maven.surefire.booter.SurefireBooterForkException: 
ExecutionException Error occurred in starting fork, check output in log
2020-02-27T12:36:35.9372275Z [ERROR] at 
org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:510)
2020-02-27T12:36:35.9372917Z [ERROR] at 
org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:457)
2020-02-27T12:36:35.9373498Z [ERROR] at 
org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:298)
2020-02-27T12:36:35.9374064Z [ERROR] at 
org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:246)
2020-02-27T12:36:35.9374636Z [ERROR] at 
org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1183)
2020-02-27T12:36:35.9375344Z [ERROR] at 
org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1011)
2020-02-27T12:36:35.9376194Z [ERROR] at 
org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:857)
2020-02-27T12:36:35.9376791Z [ERROR] at 
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
2020-02-27T12:36:35.9377375Z [ERROR] at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
2020-02-27T12:36:35.9377898Z [ERROR] at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
2020-02-27T12:36:35.9378435Z [ERROR] at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
2020-02-27T12:36:35.9379063Z [ERROR] at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
2020-02-27T12:36:35.9379709Z [ERROR] at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
2020-02-27T12:36:35.9380367Z [ERROR] at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
2020-02-27T12:36:35.9381007Z [ERROR] at 
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
2020-02-27T12:36:35.9381510Z [ERROR] at 
org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:355)
2020-02-27T12:36:35.9381973Z [ERROR] at 
org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
2020-02-27T12:36:35.9382404Z [ERROR] at 
org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
2020-02-27T12:36:35.9382839Z [ERROR] at 
org.apache.maven.cli.MavenCli.doMain(Mave

[jira] [Created] (FLINK-16314) Revisit FlinkTypeFactory

[Timo Walther (Jira)]

Timo Walther created FLINK-16314:


 Summary: Revisit FlinkTypeFactory
 Key: FLINK-16314
 URL: https://issues.apache.org/jira/browse/FLINK-16314
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Planner
Reporter: Timo Walther
Assignee: Timo Walther


While trying to fix FLINK-14042, FlinkTypeFactory causes a couple of problems 
that can not be fixed easily. Some issues:

- Inconsistent caching: FlinkTypeFactory still maintains its own seenTypes 
whereas the legacy planner already removed it due to inconsistent equality 
issues. The same problem still exists in the Blink planner regarding time 
attributes. Let's investigate if we can implement a TimestampType.equals 
including this metadata.

- Mutation of nullability: {{createTypeWithNullability}} removes NOT NULL 
information when copying fields

- Handling of RAW types: According to FLIP-37, raw types should work with type 
serializer. Currently, type information is used.

- Relation to JavaTypeFactoryImpl: There is no concrete reason why we use 
JavaTypeFactoryImpl as far I can see. SqlTypeFactoryImpl should do the job.

I suggest to reimplement the FlinkTypeFactory in Java and perform a code 
review, code simplification at the same time.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-102: Add More Metrics to TaskManager

[Till Rohrmann]

Thinking a bit more about the problem whether to report the aggregated
memory statistics or the individual slot statistics, I think reporting it
on a per slot basis won't work nicely together with FLIP-56 (dynamic slot
allocation). The problem is that with FLIP-56, we will no longer have
dedicated slots. The number of slots might change over the lifetime of a
TaskExecutor. Hence, it won't be easy to generate a metric path for every
slot which are furthermore also ephemeral. So maybe, the more general and
easier solution would be to report the overall memory usage of a
TaskExecutor even though it means to do some aggregation on the
TaskExecutor.

Concerning the JVM limit: Isn't it mainly the code cache? If we display
this value, then we should explain what exactly it means. I fear that most
users won't understand what JVM limit actually means.

Cheers,
Till

On Wed, Feb 26, 2020 at 11:15 AM Yadong Xie  wrote:

> Hi Till
>
> Thanks a lot for your response
>
> > 2. I'm not entirely sure whether I would split the memory ...
>
> Split the memory display comes from the 'ancient' design of the web, it is
> ok for me to change it following total/heap/managed/network/direct/jvm
> overhead/mapped sequence
>
> > 3. Displaying the memory configurations...
>
> I agree with you that it is not a very nice way, but the hierarchical
> relationship of configurations is too complex and hard to display in the
> other ways (I have tried)
>
> if anyone has a better idea, please feels no hesitates to help me
>
>
> > 4. What does JVM limit mean in Non-heap.JVM-Overhead?
>
> JVM limit is "non-heap max metric minus metaspace configuration" as
> @Xintong
> Song  replyed in this mail thread
>
>
> Till Rohrmann  于2020年2月25日周二 下午6:58写道：
>
> > Thanks for creating this FLIP Yadong. I think your proposal makes it much
> > easier for the user to understand what's happening on Flink
> TaskManager's.
> >
> > I have some comments:
> >
> > 1. Some of the newly introduced metrics involve computations on the
> > TaskManager. I would like to avoid additional computations introduced by
> > metrics as much as possible because metrics should not affect the system.
> > In particular, total memory sizes which are configured should not be
> > derived computationally (getManagedMemoryTotal, getTotalMemorySize). For
> > the currently available memory sizes (e.g. getManagedMemoryUsed), one
> could
> > think about reporting them on a per slot basis and to do the aggregation
> on
> > the client side. Of course, this would increase the size of the response
> > payload.
> >
> > 2. I'm not entirely sure whether I would split the memory display into
> JVM
> > memory and non JVM memory as you've done it int the POC. From a user's
> > perspective, one could start displaying the total process memory. The
> next
> > three most important metrics are the heap, managed memory and network
> > buffer usage, I guess. If one is interested in more details, one could
> then
> > display the remaining direct memory usage, the JVM overhead (I'm not sure
> > whether I would call this non-heap though) and the mapped memory.
> >
> > 3. Displaying the memory configurations in three nested boxes does not
> look
> > so nice to me. I'm not sure how else one could display it, though.
> >
> > 4. What does JVM limit mean in Non-heap.JVM-Overhead?
> >
> > Cheers,
> > Till
> >
> > On Tue, Feb 25, 2020 at 8:19 AM Yadong Xie  wrote:
> >
> > > Hi Xintong
> > > thanks for your advice, the POC web and the FLIP doc was updated now
> > > here is the new link:
> > >
> > >
> >
> http://101.132.122.69:8081/web/#/task-manager/7e7cf0293645c8537caab915c829aa73/metrics
> > >
> > >
> > > Xintong Song  于2020年2月21日周五 下午12:00写道：
> > >
> > > > >
> > > > > 1. Should the managed memory be part of direct memory?
> > > > >
> > > > The answer is no. Managed memory is currently allocated by accessing
> to
> > > > private field of Unsafe. It is not accounted for in JVM's direct
> memory
> > > > limit and corresponding metrics. To that end, it is equivalent to
> > > > native memory.
> > > >
> > > >
> > > > > 2. Should the shuffle memory also be part of the managed memory?
> > > >
> > > > I don't think so. Shuffle (Network) memory is allocated with direct
> > > > buffers, and accounted for in JVM's direct memory limit and
> > corresponding
> > > > metrics. Moreover, the FLIP-49 memory model expose network memory and
> > > > managed memory as two independent components of the overall memory
> > > > footprint.
> > > >
> > > >
> > > > Thank you~
> > > >
> > > > Xintong Song
> > > >
> > > >
> > > >
> > > > On Fri, Feb 21, 2020 at 11:45 AM Kurt Young 
> wrote:
> > > >
> > > > > Some questions related to "managed memory":
> > > > >
> > > > > 1. Should the managed memory be part of direct memory?
> > > > > 2. Should the shuffle memory also be part of the managed memory?
> > > > >
> > > > > Best,
> > > > > Kurt
> > > > >
> > > > >
> > > > > On Fri, Feb 21, 2020 at 10:41 AM Xintong Song <
> tonysong...@gmail.com
> > >
> > > 

[jira] [Created] (FLINK-16315) throw JsonMappingException when using BatchTableEnvironment#explain to get the plan of sql with constant string

[godfrey he (Jira)]

godfrey he created FLINK-16315:
--

 Summary: throw JsonMappingException when using 
BatchTableEnvironment#explain to get the plan of sql with constant string  
 Key: FLINK-16315
 URL: https://issues.apache.org/jira/browse/FLINK-16315
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Legacy Planner
Reporter: godfrey he



{code:java}
ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
BatchTableEnvironment tEnv = BatchTableEnvironment.create(env);

tEnv.registerTableSource("MyTable", CommonTestData.getCsvTableSource());

Table table = tEnv.sqlQuery("select * from MyTable where first = '274' ");

System.out.println(tEnv.explain(table));
{code}

when executing the above code, the following exception will occur.

{panel:title=exception}
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonMappingException:
 Unexpected character ('U' (code 85)): was expecting comma to separate Object 
entries
 at [Source: (String)"{
"nodes": [

{
"id": 2,
"type": "source",
"pact": "Data Source",
"contents": "CsvTableSource(read fields: first, id, score, 
last)",
"parallelism": "8",
"global_properties": [
{ "name": "Partitioning", "value": "RANDOM_PARTITIONED" 
},
{ "name": "Partitioning Order", "value": "(none)" },
{ "name": "Uniqueness", "value": "not unique" }
],
"local_properties": [
{ "name": "Order", "value": "(none)" },
{ "name": "Grouping", "value": "not grouped" },
{ "name": "Uniq"[truncated 3501 chars]; line: 41, 
column: 15] (through reference chain: 
org.apache.flink.table.explain.PlanTree["nodes"]->java.util.ArrayList[1])

at 
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:394)
at 
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:365)
at 
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:302)
at 
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:245)
at 
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:27)
at 
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138)
at 
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:288)
at 
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151)
at 
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4202)
at 
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3205)
at 
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3173)
at 
org.apache.flink.table.explain.PlanJsonParser.getSqlExecutionPlan(PlanJsonParser.java:42)
at 
org.apache.flink.table.api.internal.BatchTableEnvImpl.explain(BatchTableEnvImpl.scala:208)
at 
org.apache.flink.table.api.internal.BatchTableEnvImpl.explain(BatchTableEnvImpl.scala:223)
{panel}





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16316) Provide new base class for StreamOperators

[Piotr Nowojski (Jira)]

Piotr Nowojski created FLINK-16316:
--

 Summary: Provide new base class for StreamOperators
 Key: FLINK-16316
 URL: https://issues.apache.org/jira/browse/FLINK-16316
 Project: Flink
  Issue Type: Sub-task
  Components: API / DataStream
Affects Versions: 1.10.0
Reporter: Piotr Nowojski
Assignee: Piotr Nowojski
 Fix For: 1.11.0


Current {{AbstractStreamOperator}} has at least couple of issues:
# it's using deprecated {{SetupableStreamOperator}} interface, instead of 
relaying on {{StreamOperatorFactory}}.
# it depends on {{OneInputStreamOperator}} and {{TwoInputStreamOperator}} 
interfaces, by providing indirectly implementation of methods like 
{{processLatencyMarker}}, {{processLatencyMarker1}} and 
{{processLatencyMarker2}}

Both of those things makes {{AbstractStreamOperator}} an unsuitable base 
implementation for {{MultipleInputStreamOperator}}. We should provide a new 
base class (or hierarchy of base classes), without those two limitations.

CC [~aljoscha]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16317) Implement missing features for MultipleInputStreamOperator

[Piotr Nowojski (Jira)]

Piotr Nowojski created FLINK-16317:
--

 Summary: Implement missing features for MultipleInputStreamOperator
 Key: FLINK-16317
 URL: https://issues.apache.org/jira/browse/FLINK-16317
 Project: Flink
  Issue Type: Sub-task
  Components: API / DataStream, Runtime / Task
Affects Versions: 1.10.0
Reporter: Piotr Nowojski
 Fix For: 1.11.0


After https://issues.apache.org/jira/browse/FLINK-16060 support for 
{{MultipleInputStreamOperator}} is incomplete. After defining new base class 
for the {{StreamOperator}} (FLINK-15687) that would be suitable to use with 
{{MultipleInputStreamOperator}}, we can provide support for missing features in 
the {{MultipleInputStreamOperator}}, like:
* keyed state support
* processing {{Watermark}}
* processing {{LatencyMarker}}
* {{StreamStatus}}
* input selection



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16318) PersistedTable and AppendingBufferAccessor use different type system then Persisted Value

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-16318:


 Summary: PersistedTable and AppendingBufferAccessor use different 
type system then Persisted Value
 Key: FLINK-16318
 URL: https://issues.apache.org/jira/browse/FLINK-16318
 Project: Flink
  Issue Type: Bug
Reporter: Seth Wiesman


In FlinkState PersisteValue is backed by a value state whose type information 
is determined by DynamicallyRegisteredTypes. AppendingBufferAccessor and 
PersistedTable use List and Map states whose state descriptors use classes 
which may return a different type information. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] FLIP-76: Unaligned checkpoints

[Piotr Nowojski]

Hi Yu,

Re 4.

Dynamic switching between unaligned and aligned checkpoints based on some kind 
of thresholds (timeout, or checkpoint size) is definitely one of the very first 
improvement that we want to tackle after implementing the MVP. Depending on the 
time constraints, dynamic switching can make to 1.11 or not. It’s hard to tell 
for me at this point of time.

Piotrek

> On 26 Feb 2020, at 15:59, Zhijiang  wrote:
> 
> Thanks for the further explanations, Yu!
> 
> 1. The inflight buffer spilling process is indeed handled asynchronously. 
> While the buffer is not finished spilling, it would not be recycled to reuse 
> again.
> Your understanding is right. I guess I misunderstood your previous concern of 
> additional memory consumption from the perspective of buffer usage.
> My point of no additional memory consumption is from the perspective of total 
> network memory size which would not be increased as a result.
> 
> 2. We treat the inflight buffers as input&output states which are equivalent 
> with existing operator states, and try to make use of all the existing 
> mechanisms for
> state handle and assignment during recovery. So i guess for the local 
> recovery it should be the similar case. I would think through whether it has 
> some special
> work to do around with local recovery, and then clarify it in FLIP after we 
> reach an agreement internally. BTW, this FLIP has not finalized yet.
> 
> 3. Yes, the previous proposal is for measuring how many inflight buffers to 
> be spilled which refers to the data size if really taking this way. I think 
> the proposed option
> in FLIP are the initial thoughts for various of possibilities. Which way we 
> decide to take for the first version, I guess we need to further finalize 
> before voting.
> 
> 4. I think there probably exists the requirements or scenarios from users as 
> you mentioned. Actually we have not finalized the way of switching to 
> unaligned checkpoint yet.
> Anyway we could provide an option for users to try out this feature at the 
> beginning, although it might be not the most ideal one. Another input is that 
> we know the motivation
> of unaligned checkpoint is from the scenarios of backpressure, but it might 
> also performs well in the case of non backpressure, even shorten the 
> checkpoint duration without
> obvious performance regression in our previous POC testing. So the 
> backpressure might not be the only factor to switch to the unaligned way in 
> practice I guess. Anyway your
> inputs are valuable for us to make the final decision.
> 
> Best,
> Zhijiang
> 
> 
> 
> 
> --
> From:Yu Li 
> Send Time:2020 Feb. 26 (Wed.) 15:59
> To:dev ; Zhijiang 
> Subject:Re: [DISCUSS] FLIP-76: Unaligned checkpoints
> 
> Hi Zhijiang,
> 
> Thanks for the quick reply!
> 
> For the 1st question, please allow me to confirm, that when doing 
> asynchronous checkpointing, disk spilling should happen in background in 
> parallel with receiving/sending new data, or else it would become 
> synchronous, right? Based on such assumption, some copy-on-write like 
> mechanism would be necessary to make sure the new updates won't modify the 
> to-be-checkpointed data, and this is where the additional memory consumption 
> comes from.
> 
> About point #2, I suggest we write it down in the FLIP document about local 
> recovery support (if reach a consensus here), to make sure it won't be 
> neglected in later implementation (I believe there're still some work to do 
> following existing local recovery mechanism). What do you think?
> 
> For the 3rd topic, do you mean UNALIGNED_WITH_MAX_INFLIGHT_DATA would set 
> some kind of threshold about "how much in-flight data to checkpoint"? If so, 
> could you further clarify the measurement (data size? record number? others?) 
> since there seems to be no description in the current FLIP doc? This is 
> somewhat different from my understanding after reading the FLIP...
> 
> Regarding question #4, I have no doubt that the new unaligned checkpoint 
> mechanism could make fast checkpoint possible, at the cost of more memory, 
> network bandwidth and disk space consumption. However, (correct me if I'm 
> wrong) for users who are satisfied with the existing aligned checkpoint 
> interval, taking the constant cost to prevent delayed checkpoint during back 
> pressure - a relatively low frequency event - may not be that pragmatic.
> 
> Best Regards,
> Yu
> 
> On Wed, 26 Feb 2020 at 15:07, Zhijiang  
> wrote:
> Hi Yu,
> 
> Thanks for concerning of this FLIP and sharing your thoughts! Let me try to 
> answer some below questions.
> 
> 1. Yes, the asynchronous checkpointing should be part of whole process and be 
> supported naturally. As for the network memory concern, 
> the inflight-buffers would be spilled into persistent storage while 
> triggering checkpoint, and are recycled to receive/send data after finish 
> spilling.
> We still reuse the current net

Re: [VOTE] FLIP-99: Make Max Exception Configurable

[Gary Yao]

+1 (binding)

Best,
Gary

On Thu, Feb 20, 2020 at 10:39 AM Yadong Xie  wrote:

> Hi all
>
> I want to start the vote for FLIP-99, which proposes to make the max
> exception configurable in web UI.
>
> To help everyone better understand the proposal, we spent some efforts on
> making an online POC
>
> previous web:
>
> http://101.132.122.69:8081/#/job/543e9dc0cb2cca4433116007f0931d1a/exceptions
> POC web:
>
> http://101.132.122.69:8081/web/#/job/543e9dc0cb2cca4433116007f0931d1a/exceptions
>
> The vote will last for at least 72 hours, following the consensus voting
> process.
>
> FLIP wiki:
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-99%3A+Make+Max+Exception+Configurable
>
> Discussion thread:
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
>
> Thanks,
>
> Yadong
>

[jira] [Created] (FLINK-16319) Pubsub-/Broadcast implementation

[Dan Pettersson (Jira)]

Dan Pettersson created FLINK-16319:
--

 Summary: Pubsub-/Broadcast implementation
 Key: FLINK-16319
 URL: https://issues.apache.org/jira/browse/FLINK-16319
 Project: Flink
  Issue Type: Improvement
  Components: Stateful Functions
Reporter: Dan Pettersson


Hi everyone,
 
I have a use case where the id of the functions are brokerId + instrumentId 
that receives trades and orders. The instrument has state changes (Open, 
halted, closed etc) that almost all the functions are interested in. Some 
functions only wants for example the Close message whereas other functions 
wants all state changes for the specific instrument. 
 
I've built a statefun pubsub module that exposes two interfaces, Subscriber and 
Publisher, with these two methods:
 
default void subscribe(Context context, Subscription... subscriptions)
  
default void publish(Context context, PublishMessage publishMessage)
 
Behind the interfaces is a hidden StatefulPubSubFunction that keeps track of 
which partition the subscriber is located in and to which topic it listens to.
 
Code is located under 
[https://github.com/danp11/flink-statefun/tree/master/statefun-pubsub] if 
anyone is interested.
 
This code is a "classic pub sub" pattern and I think that this kind of 
functionality would be a great addition to Stateful functions. I create this 
Jira to see if there is an interest to discuss how a optimal pubsub-/broadcast 
solution would look like in SF? Igal has previously mentioned that Broadcast 
could be a good fit for this kind of flow.
At the moment I don't know the internals of SF and-/or Flink good enough to 
come up with a proposal myself unfortunately.
 
I know you are very busy at the moment (Its impressive how much you have 
produced only the last couple of weeks!:-) but if someone, on a high level, has 
any ideas on where and how a pub sub pattern could be implemented I'd really 
appreciate it. In the future I hope we can come up with a proposal together as 
I need your help here.  If you think that a pubsub-/broadcast solution would 
make SF better that is :-)
 
Hope to hear your thoughts on this!
 
Thanks,
 
 /Dan



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16320) Can not use sub-queries in the VALUES clause

[Dawid Wysakowicz (Jira)]

Dawid Wysakowicz created FLINK-16320:


 Summary: Can not use sub-queries in the VALUES clause 
 Key: FLINK-16320
 URL: https://issues.apache.org/jira/browse/FLINK-16320
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Legacy Planner, Table SQL / Planner
Affects Versions: 1.11.0
Reporter: Dawid Wysakowicz


{code}
StreamExecutionEnvironment sEnv = 
StreamExecutionEnvironment.getExecutionEnvironment();
StreamTableEnvironment tableEnvironment = StreamTableEnvironment.create(
sEnv,
EnvironmentSettings.newInstance().useBlinkPlanner().build());

Table table = tableEnvironment.sqlQuery("SELECT * FROM (VALUES(1), (SELECT 
1))");

tableEnvironment.toRetractStream(table, Row.class).print();

System.out.println(tableEnvironment.explain(table));
{code}

Produces:

{code}
== Optimized Logical Plan ==
Union(all=[true], union=[EXPR$0])
:- Calc(select=[CAST(1) AS EXPR$0])
:  +- Values(type=[RecordType(INTEGER ZERO)], tuples=[[{ 0 }]])
+- Calc(select=[$f0 AS EXPR$0])
   +- Join(joinType=[LeftOuterJoin], where=[true], select=[ZERO, $f0], 
leftInputSpec=[NoUniqueKey], rightInputSpec=[JoinKeyContainsUniqueKey])
  :- Exchange(distribution=[single])
  :  +- Values(type=[RecordType(INTEGER ZERO)], tuples=[[{ 0 }]], 
reuse_id=[1])
  +- Exchange(distribution=[single])
 +- GroupAggregate(select=[SINGLE_VALUE(EXPR$0) AS $f0])
+- Exchange(distribution=[single])
   +- Calc(select=[1 AS EXPR$0])
  +- Reused(reference_id=[1])
{code}

which is wrong.

Legacy planner fails with:

{code}
validated type:
RecordType(INTEGER EXPR$0) NOT NULL
converted type:
RecordType(INTEGER NOT NULL EXPR$0) NOT NULL
rel:
LogicalProject(EXPR$0=[$0])
  LogicalUnion(all=[true])
LogicalProject(EXPR$0=[1])
  LogicalValues(tuples=[[{ 0 }]])
LogicalAggregate(group=[{}], agg#0=[SINGLE_VALUE($0)])
  LogicalProject(EXPR$0=[1])
LogicalValues(tuples=[[{ 0 }]])
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[VOTE] FLIP-93: JDBC catalog and Postgres catalog

[Bowen Li]

Hi all,

I'd like to kick off the vote for FLIP-93 [1] to add JDBC catalog and
Postgres catalog.

The vote will last for at least 72 hours, following the consensus voting
protocol.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-93%3A+JDBC+catalog+and+Postgres+catalog

Discussion thread:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-92-JDBC-catalog-and-Postgres-catalog-td36505.html

[jira] [Created] (FLINK-16321) Refactor HttpFunction to improve testability

[Igal Shilman (Jira)]

Igal Shilman created FLINK-16321:


 Summary: Refactor HttpFunction to improve testability
 Key: FLINK-16321
 URL: https://issues.apache.org/jira/browse/FLINK-16321
 Project: Flink
  Issue Type: Improvement
  Components: Stateful Functions
Reporter: Igal Shilman
 Fix For: statefun-1.1


The HTTP specific parts (okhttp client) can be extracted out of the 
HttpFunction,
into a ReuestReply function that is oblivious to the underlying transport, and 
only captures the protocol.
This would improve testability and would allow other transports, 
(like gRPC)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-84: Improve & Refactor API of TableEnvironment

[Terry Wang]

I look through the whole design and it’s a big improvement of usability on 
TableEnvironment’s api.

+1 (non-binding)

Best,
Terry Wang



> 2020年2月27日 14:59，godfrey he  写道：
> 
> Hi everyone,
> 
> I'd like to start the vote of FLIP-84[1], which proposes to deprecate some
> old APIs and introduce some new APIs in TableEnvironment. This FLIP is
> discussed and reached consensus in the discussion thread[2].
> 
> The vote will be open for at least 72 hours. Unless there is an objection,
> I will try to close it by Mar 1, 2020 07:00 UTC if we have received
> sufficient votes.
> 
> 
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-84%3A+Improve+%26+Refactor+API+of+TableEnvironment
> 
> [2]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-84-Improve-amp-Refactor-API-of-Table-Module-td34537.html
> 
> 
> Bests,
> Godfrey

Re: [DISCUSS] FLINK-16194: Refactor the Kubernetes architecture design

[Canbin Zheng]

Thanks very much Yang Wang.

Cheers,
Canbin Zheng

Yang Wang  于2020年2月27日周四 下午9:19写道：

> Great work! I could help to review and test.
>
> Best,
> Yang
>
> Canbin Zheng  于2020年2月27日周四 下午4:24写道：
>
>> Hi, everyone,
>>
>> I have pushed a PR  for this
>> issue, looking forward to your feedback.
>>
>>
>> Cheers,
>> Canbin Zheng
>>
>> Canbin Zheng  于2020年2月26日周三 上午10:39写道：
>>
>>> Thanks for the detailed PR advice, I would separate the commits as clear
>>> as possible to help the code reviewing.
>>>
>>>
>>> Cheers,
>>> Canbin Zheng
>>>
>>> tison  于2020年2月25日周二 下午10:11写道：
>>>
 Thanks for your clarification Yang! We're on the same page.

 Best,
 tison.


 Yang Wang  于2020年2月25日周二 下午10:07写道：

> Hi tison,
>
> I do not mean to keep two decorator at the same. Since the two
> decorators are
> not api compatible, it is meaningless. I am just thinking how
> to organize the
> commits/PRs to make the review easier. The reviewers may need some
> context
> to get the point.
>
>
>
> Best,
> Yang
>
> tison  于2020年2月25日周二 下午8:23写道：
>
>> The process in my mind is somehow like this commit[1] which belongs
>> to this pr[2]
>> that we firstly introduce the new implementation and then replace it
>> with the original
>> one. The difference is that these two versions of decorators are not
>> api compatible
>> while adding a switch for such an internal abstraction or extracting
>> a clumsy
>> "common" interface doesn't benefit.
>>
>> Best,
>> tison.
>>
>> [1]
>> https://github.com/apache/flink/commit/1f2969357c441e24b71daef83d21563da9a93bb4
>> [2] https://github.com/apache/flink/pull/9832
>>
>>
>>
>>
>> tison  于2020年2月25日周二 下午8:08写道：
>>
>>> I agree for separating commits we can have multiple commits that
>>> firstly add the new parameters
>>> and decorators,  and later replace current decorators with new
>>> decorators which are well
>>> unit tested.
>>>
>>> However, it makes no sense we have two codepaths from
>>> FlinkKubeClient to decorators
>>> since these two version of decorators are not api compatible and
>>> there is no reason we keep both
>>> of them.
>>>
>>> Best,
>>> tison.
>>>
>>>
>>> Yang Wang  于2020年2月25日周二 下午7:50写道：
>>>
 I think if we could, splitting into as many PRs as possible is
 good. Maybe we could
 introduce the new designed decorators and parameter parser first,
 and leave the existing
 decorators as legacy. Once all the new decorators is ready and well
 tested, we could
 remove the legacy codes and use the new decorators in the kube
 client implementation.


 Best,
 Yang

 Canbin Zheng  于2020年2月25日周二 下午6:16写道：

> Hi, Till,
>
> Great thanks for your advice, I totally agree with you to split
> the changes up in as many PRs as possible. The part of "Parameter 
> Parser"
> is trivial so that we prefer to make one PR to avoid adapting a lot of
> pieces of code that would be deleted immediately with the following
> decorator refactoring PR. Actually I won't insist on one PR, could it 
> be
> possible that I first try out with one PR and let the committers help
> assess whether it is necessary to split the changes into several PRs?
> Kindly expect to see your reply.
>

Re: [VOTE] FLIP-84: Improve & Refactor API of TableEnvironment

[Kurt Young]

 +1 (binding)

Best,
Kurt


On Fri, Feb 28, 2020 at 9:15 AM Terry Wang  wrote:

> I look through the whole design and it’s a big improvement of usability on
> TableEnvironment’s api.
>
> +1 (non-binding)
>
> Best,
> Terry Wang
>
>
>
> > 2020年2月27日 14:59，godfrey he  写道：
> >
> > Hi everyone,
> >
> > I'd like to start the vote of FLIP-84[1], which proposes to deprecate
> some
> > old APIs and introduce some new APIs in TableEnvironment. This FLIP is
> > discussed and reached consensus in the discussion thread[2].
> >
> > The vote will be open for at least 72 hours. Unless there is an
> objection,
> > I will try to close it by Mar 1, 2020 07:00 UTC if we have received
> > sufficient votes.
> >
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-84%3A+Improve+%26+Refactor+API+of+TableEnvironment
> >
> > [2]
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-84-Improve-amp-Refactor-API-of-Table-Module-td34537.html
> >
> >
> > Bests,
> > Godfrey
>
>

[jira] [Created] (FLINK-16322) wrong result after filter push down in parquet table source

[godfrey he (Jira)]

godfrey he created FLINK-16322:
--

 Summary: wrong result after filter push down in parquet table 
source
 Key: FLINK-16322
 URL: https://issues.apache.org/jira/browse/FLINK-16322
 Project: Flink
  Issue Type: Bug
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Reporter: godfrey he
 Attachments: parquet-1-1.parquet

I get the wrong result when run the following query:

source schema:
first VARCHAR
id INT
score DOUBLE 
last VARCHAR

data: (parquet file is in the attachment)
("Mike", 1, 12.3d, "Smith"),
("Bob", 2, 45.6d, "Taylor"),
("Sam", 3, 7.89d, "Miller"),
("Peter", 4, 0.12d, "Smith"),
("Liz", 5, 34.5d, "Williams"),
("Sally", 6, 6.78d, "Miller"),
("Alice", 7, 90.1d, "Smith"),
("Kelly", 8, 2.34d, "Williams")

query:
SELECT id, `first`, `last`, score FROM ParquetTable WHERE score < 3

the expected result size is 2, however the actual result size is 0. 








--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-84: Improve & Refactor API of TableEnvironment

[kant kodali]

Is this targeted towards Flink 1.11?

On Thu, Feb 27, 2020 at 6:32 PM Kurt Young  wrote:

>  +1 (binding)
>
> Best,
> Kurt
>
>
> On Fri, Feb 28, 2020 at 9:15 AM Terry Wang  wrote:
>
> > I look through the whole design and it’s a big improvement of usability
> on
> > TableEnvironment’s api.
> >
> > +1 (non-binding)
> >
> > Best,
> > Terry Wang
> >
> >
> >
> > > 2020年2月27日 14:59，godfrey he  写道：
> > >
> > > Hi everyone,
> > >
> > > I'd like to start the vote of FLIP-84[1], which proposes to deprecate
> > some
> > > old APIs and introduce some new APIs in TableEnvironment. This FLIP is
> > > discussed and reached consensus in the discussion thread[2].
> > >
> > > The vote will be open for at least 72 hours. Unless there is an
> > objection,
> > > I will try to close it by Mar 1, 2020 07:00 UTC if we have received
> > > sufficient votes.
> > >
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-84%3A+Improve+%26+Refactor+API+of+TableEnvironment
> > >
> > > [2]
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-84-Improve-amp-Refactor-API-of-Table-Module-td34537.html
> > >
> > >
> > > Bests,
> > > Godfrey
> >
> >
>

Re: [VOTE] FLIP-84: Improve & Refactor API of TableEnvironment

[godfrey he]

Hi kant, yes. We hope to deprecate the methods which confuse users ASAP.

Bests,
godfrey

kant kodali  于2020年2月28日周五 上午11:17写道：

> Is this targeted towards Flink 1.11?
>
> On Thu, Feb 27, 2020 at 6:32 PM Kurt Young  wrote:
>
> >  +1 (binding)
> >
> > Best,
> > Kurt
> >
> >
> > On Fri, Feb 28, 2020 at 9:15 AM Terry Wang  wrote:
> >
> > > I look through the whole design and it’s a big improvement of usability
> > on
> > > TableEnvironment’s api.
> > >
> > > +1 (non-binding)
> > >
> > > Best,
> > > Terry Wang
> > >
> > >
> > >
> > > > 2020年2月27日 14:59，godfrey he  写道：
> > > >
> > > > Hi everyone,
> > > >
> > > > I'd like to start the vote of FLIP-84[1], which proposes to deprecate
> > > some
> > > > old APIs and introduce some new APIs in TableEnvironment. This FLIP
> is
> > > > discussed and reached consensus in the discussion thread[2].
> > > >
> > > > The vote will be open for at least 72 hours. Unless there is an
> > > objection,
> > > > I will try to close it by Mar 1, 2020 07:00 UTC if we have received
> > > > sufficient votes.
> > > >
> > > >
> > > > [1]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-84%3A+Improve+%26+Refactor+API+of+TableEnvironment
> > > >
> > > > [2]
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-84-Improve-amp-Refactor-API-of-Table-Module-td34537.html
> > > >
> > > >
> > > > Bests,
> > > > Godfrey
> > >
> > >
> >
>

[jira] [Created] (FLINK-16323) Support to join a static table in streaming mode

[Jark Wu (Jira)]

Jark Wu created FLINK-16323:
---

 Summary: Support to join a static table in streaming mode
 Key: FLINK-16323
 URL: https://issues.apache.org/jira/browse/FLINK-16323
 Project: Flink
  Issue Type: New Feature
  Components: Table SQL / API, Table SQL / Planner
Reporter: Jark Wu


Currently, we already support to join a stream and a bounded stream using 
reguar join. However, this will be tranlsated into stream-stream join which is 
not efficient, because it will output early results and maybe retracted 
afterwards. 

A better and native support will be using a special temporal join operator 
which will block the streaming side until all the static table data is loaded. 

This can help users to join a huge table, e.g. MySQL table with billion records 
but changes little.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16324) Checkpoint tasks progress would display 100% in web UI even not all tasks finished

[Yun Tang (Jira)]

Yun Tang created FLINK-16324:


 Summary: Checkpoint tasks progress would display 100% in web UI 
even not all tasks finished
 Key: FLINK-16324
 URL: https://issues.apache.org/jira/browse/FLINK-16324
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Web Frontend
Reporter: Yun Tang
 Fix For: 1.10.1, 1.11.0


The checkpoint progress details would display 100% even not all tasks finished 
in web UI. e.g {{96/100}} would display as {{100%}} .



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16325) A connection check is required, and it needs to be reopened when the JDBC connection is interrupted

[renjianxu (Jira)]

renjianxu created FLINK-16325:
-

 Summary:  A connection check is required, and it needs to be 
reopened when the JDBC connection is interrupted
 Key: FLINK-16325
 URL: https://issues.apache.org/jira/browse/FLINK-16325
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / JDBC
Affects Versions: 1.10.0
Reporter: renjianxu


JDBCOutputFormat#writeRecord.

When writing data, if the JDBC connection has been disconnected, the data will 
be lost.Therefore, a connectivity judgment is required in the writeRecord 
method.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-99: Make Max Exception Configurable

[Yadong Xie]

Thanks all for the votes.

So far, we have

- 3 binding +1 votes (Jark, Till, Gary)
- 3 non-binding +1 votes (CongXian BenChao, zoudan)
- No -1 votes

The voting time has past and there are enough +1 votes to consider the
FLIP-99 approved.
Thank you all.

Gary Yao  于2020年2月28日周五 上午12:03写道：

> +1 (binding)
>
> Best,
> Gary
>
> On Thu, Feb 20, 2020 at 10:39 AM Yadong Xie  wrote:
>
> > Hi all
> >
> > I want to start the vote for FLIP-99, which proposes to make the max
> > exception configurable in web UI.
> >
> > To help everyone better understand the proposal, we spent some efforts on
> > making an online POC
> >
> > previous web:
> >
> >
> http://101.132.122.69:8081/#/job/543e9dc0cb2cca4433116007f0931d1a/exceptions
> > POC web:
> >
> >
> http://101.132.122.69:8081/web/#/job/543e9dc0cb2cca4433116007f0931d1a/exceptions
> >
> > The vote will last for at least 72 hours, following the consensus voting
> > process.
> >
> > FLIP wiki:
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-99%3A+Make+Max+Exception+Configurable
> >
> > Discussion thread:
> >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> >
> > Thanks,
> >
> > Yadong
> >
>

Re: [VOTE] FLIP-84: Improve & Refactor API of TableEnvironment

[kant kodali]

Nice!!

Sent from my iPhone

> On Feb 27, 2020, at 9:03 PM, godfrey he  wrote:
> 
> Hi kant, yes. We hope to deprecate the methods which confuse users ASAP.
> 
> Bests,
> godfrey
> 
> kant kodali  于2020年2月28日周五 上午11:17写道：
> 
>> Is this targeted towards Flink 1.11?
>> 
>>> On Thu, Feb 27, 2020 at 6:32 PM Kurt Young  wrote:
>>> 
>>> +1 (binding)
>>> 
>>> Best,
>>> Kurt
>>> 
>>> 
 On Fri, Feb 28, 2020 at 9:15 AM Terry Wang  wrote:
>>> 
 I look through the whole design and it’s a big improvement of usability
>>> on
 TableEnvironment’s api.
 
 +1 (non-binding)
 
 Best,
 Terry Wang
 
 
 
> 2020年2月27日 14:59，godfrey he  写道：
> 
> Hi everyone,
> 
> I'd like to start the vote of FLIP-84[1], which proposes to deprecate
 some
> old APIs and introduce some new APIs in TableEnvironment. This FLIP
>> is
> discussed and reached consensus in the discussion thread[2].
> 
> The vote will be open for at least 72 hours. Unless there is an
 objection,
> I will try to close it by Mar 1, 2020 07:00 UTC if we have received
> sufficient votes.
> 
> 
> [1]
> 
 
>>> 
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-84%3A+Improve+%26+Refactor+API+of+TableEnvironment
> 
> [2]
> 
 
>>> 
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-84-Improve-amp-Refactor-API-of-Table-Module-td34537.html
> 
> 
> Bests,
> Godfrey
 
 
>>> 
>> 

Flink 1.10 exception : Unable to instantiate java compiler

[LakeShen]

Hi community,
  now  I am using the flink 1.10 to run the flink task ,cluster
type is yarn . I use commandline to submit my flink job , the commandline
just like this :

flink run  -m yarn-cluster  --allowNonRestoredState  -c xxx.xxx.xx
 flink-stream-xxx.jar

Bug there is a exception to throw,the exception info is :

*org.apache.flink.client.program.ProgramInvocationException: The main
method caused an error: Unable to instantiate java compiler*
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:335)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:205)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:138)
at
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:664)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
at
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:895)
at
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:968)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
at
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:968)
Caused by: java.lang.IllegalStateException: Unable to instantiate java
compiler
at
org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.compile(JaninoRelMetadataProvider.java:434)
at
org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.load3(JaninoRelMetadataProvider.java:375)
at
org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.lambda$static$0(JaninoRelMetadataProvider.java:109)
at
org.apache.flink.calcite.shaded.com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:149)
at
org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3542)
at
org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2323)
at
org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2286)
at
org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
at
org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache.get(LocalCache.java:3953)
at
org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3957)
at
org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4875)
at
org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.create(JaninoRelMetadataProvider.java:475)
at
org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.revise(JaninoRelMetadataProvider.java:488)
at
org.apache.calcite.rel.metadata.RelMetadataQuery.revise(RelMetadataQuery.java:193)
at
org.apache.calcite.rel.metadata.RelMetadataQuery.getPulledUpPredicates(RelMetadataQuery.java:797)
at
org.apache.calcite.rel.rules.ReduceExpressionsRule$ProjectReduceExpressionsRule.onMatch(ReduceExpressionsRule.java:298)
at
org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:319)
at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:560)
at org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:419)
at
org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:256)
at
org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:127)
at
org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:215)
at org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:202)
at
org.apache.flink.table.planner.plan.optimize.program.FlinkHepProgram.optimize(FlinkHepProgram.scala:69)
at
org.apache.flink.table.planner.plan.optimize.program.FlinkHepRuleSetProgram.optimize(FlinkHepRuleSetProgram.scala:87)
at
org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram$$anonfun$optimize$1.apply(FlinkChainedProgram.scala:62)
at
org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram$$anonfun$optimize$1.apply(FlinkChainedProgram.scala:58)
at
scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
at
scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at
scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:104)
at
org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram.optimize(FlinkChainedProgram.scala:57)
at
org

[jira] [Created] (FLINK-16326) Eagerly validate strictly required Flink configurations for Stateful Functions

[Tzu-Li (Gordon) Tai (Jira)]

Tzu-Li (Gordon) Tai created FLINK-16326:
---

 Summary: Eagerly validate strictly required Flink configurations 
for Stateful Functions
 Key: FLINK-16326
 URL: https://issues.apache.org/jira/browse/FLINK-16326
 Project: Flink
  Issue Type: Improvement
  Components: Stateful Functions
Affects Versions: statefun-1.1
Reporter: Tzu-Li (Gordon) Tai


Currently, when Stateful Functions users want to set their own Flink 
configurations, they are required to build on top of a base template 
{{flink-conf.yaml}} which has some strictly required configurations predefined, 
such as parent-first classloading and state backend settings.

These Flink settings should never (as of now) be changed by the user, but there 
is no validation of that in place. We should do that eagerly pre-submission of 
the translated job, probably in {{StatefulFunctionsConfig}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Flink 1.10 exception : Unable to instantiate java compiler

[kant kodali]

Same problem!

On Thu, Feb 27, 2020 at 11:10 PM LakeShen  wrote:

> Hi community,
>   now  I am using the flink 1.10 to run the flink task
> ,cluster type is yarn . I use commandline to submit my flink job , the
> commandline just like this :
>
> flink run  -m yarn-cluster  --allowNonRestoredState  -c xxx.xxx.xx
>  flink-stream-xxx.jar
>
> Bug there is a exception to throw,the exception info is :
>
> *org.apache.flink.client.program.ProgramInvocationException: The main
> method caused an error: Unable to instantiate java compiler*
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:335)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:205)
> at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:138)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:664)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
> at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:895)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:968)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
> at
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:968)
> Caused by: java.lang.IllegalStateException: Unable to instantiate java
> compiler
> at
> org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.compile(JaninoRelMetadataProvider.java:434)
> at
> org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.load3(JaninoRelMetadataProvider.java:375)
> at
> org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.lambda$static$0(JaninoRelMetadataProvider.java:109)
> at
> org.apache.flink.calcite.shaded.com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:149)
> at
> org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3542)
> at
> org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2323)
> at
> org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2286)
> at
> org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
> at
> org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache.get(LocalCache.java:3953)
> at
> org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3957)
> at
> org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4875)
> at
> org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.create(JaninoRelMetadataProvider.java:475)
> at
> org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.revise(JaninoRelMetadataProvider.java:488)
> at
> org.apache.calcite.rel.metadata.RelMetadataQuery.revise(RelMetadataQuery.java:193)
> at
> org.apache.calcite.rel.metadata.RelMetadataQuery.getPulledUpPredicates(RelMetadataQuery.java:797)
> at
> org.apache.calcite.rel.rules.ReduceExpressionsRule$ProjectReduceExpressionsRule.onMatch(ReduceExpressionsRule.java:298)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:319)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:560)
> at org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:419)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:256)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:127)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:215)
> at org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:202)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkHepProgram.optimize(FlinkHepProgram.scala:69)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkHepRuleSetProgram.optimize(FlinkHepRuleSetProgram.scala:87)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram$$anonfun$optimize$1.apply(FlinkChainedProgram.scala:62)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram$$anonfun$optimize$1.apply(FlinkChainedProgram.scala:58)
> at
> scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
> at
> scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
> at scala.collection.Iterator$class.foreach(Iterator.scala:891)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at
> scala.collection.