[jira] [Created] (FLINK-22082) Nested projection push down doesn't work for data: row(array(row))

[Dian Fu (Jira)]

Dian Fu created FLINK-22082:
---

 Summary: Nested projection push down doesn't work for data: 
row(array(row))
 Key: FLINK-22082
 URL: https://issues.apache.org/jira/browse/FLINK-22082
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.12.0, 1.13.0
Reporter: Dian Fu


For the following job:

{code}
from pyflink.datastream import StreamExecutionEnvironment
from pyflink.table import TableConfig, StreamTableEnvironment

config = TableConfig()
env = StreamExecutionEnvironment.get_execution_environment()
t_env = StreamTableEnvironment.create(env, config)

source_ddl = """
CREATE TABLE InTable (
`ID` STRING,
`Timestamp` TIMESTAMP(3),
`Result` ROW(
`data` ROW(`value` BIGINT) ARRAY),
WATERMARK FOR `Timestamp` AS `Timestamp`
) WITH (
'connector' = 'filesystem',
'format' = 'json',
'path' = '/tmp/1.txt'
)
"""

sink_ddl = """
CREATE TABLE OutTable (
`ID` STRING,
`value` BIGINT
) WITH (
'connector' = 'print'
)
"""

t_env.execute_sql(source_ddl)
t_env.execute_sql(sink_ddl)

table = t_env.from_path('InTable')
table \
.select(
table.ID,
table.Result.data.at(1).value) \
.execute_insert('OutTable') \
.wait()
{code}

It will thrown the following exception:
{code}
: scala.MatchError: ITEM($2.data, 1) (of class org.apache.calcite.rex.RexCall)
at 
org.apache.flink.table.planner.plan.utils.NestedSchemaExtractor.internalVisit$1(NestedProjectionUtil.scala:273)
at 
org.apache.flink.table.planner.plan.utils.NestedSchemaExtractor.visitFieldAccess(NestedProjectionUtil.scala:283)
at 
org.apache.flink.table.planner.plan.utils.NestedSchemaExtractor.visitFieldAccess(NestedProjectionUtil.scala:269)
at org.apache.calcite.rex.RexFieldAccess.accept(RexFieldAccess.java:92)
at 
org.apache.flink.table.planner.plan.utils.NestedProjectionUtil$$anonfun$build$1.apply(NestedProjectionUtil.scala:112)
at 
org.apache.flink.table.planner.plan.utils.NestedProjectionUtil$$anonfun$build$1.apply(NestedProjectionUtil.scala:111)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at 
org.apache.flink.table.planner.plan.utils.NestedProjectionUtil$.build(NestedProjectionUtil.scala:111)
at 
org.apache.flink.table.planner.plan.utils.NestedProjectionUtil.build(NestedProjectionUtil.scala)
at 
org.apache.flink.table.planner.plan.rules.logical.ProjectWatermarkAssignerTransposeRule.getUsedFieldsInTopLevelProjectAndWatermarkAssigner(ProjectWatermarkAssignerTransposeRule.java:155)
at 
org.apache.flink.table.planner.plan.rules.logical.ProjectWatermarkAssignerTransposeRule.matches(ProjectWatermarkAssignerTransposeRule.java:65)
{code}

See 
https://stackoverflow.com/questions/66888486/pyflink-extract-nested-fields-from-json-array
 for more details



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22081) Entropy key not resolved if flink-s3-fs-hadoop is added as a plugin

[Chen Qin (Jira)]

Chen Qin created FLINK-22081:


 Summary: Entropy key not resolved if flink-s3-fs-hadoop is added 
as a plugin
 Key: FLINK-22081
 URL: https://issues.apache.org/jira/browse/FLINK-22081
 Project: Flink
  Issue Type: Bug
  Components: FileSystems
Reporter: Chen Qin
Assignee: Prem Santosh
 Fix For: 1.10.1, 1.11.0


Using flink 1.10

I added the flink-s3-fs-hadoop jar in plugins dir but I am seeing the 
checkpoints paths like 
{{s3://my_app/__ENTROPY__/app_name-staging/flink/checkpoints/e10f47968ae74934bd833108d2272419/chk-3071}}
 which means the entropy injection key is not being resolved. After some 
debugging I found that in the 
[EntropyInjector|https://github.com/apache/flink/blob/release-1.10.0/flink-core/src/main/java/org/apache/flink/core/fs/EntropyInjector.java#L97]
 we check if the given fileSystem is of type {{SafetyNetWrapperFileSystem}} and 
if so we check if the delegate is of type {{EntropyInjectingFileSystem}} but 
don't check for {{[ClassLoaderFixingFileSystem 
|https://github.com/apache/flink/blob/release-1.10.0/flink-core/src/main/java/org/apache/flink/core/fs/PluginFileSystemFactory.java#L65]}}
 which would be the type if S3 file system dependencies are added as a plugin.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Apache Flink Stateful Functions 3.0.0, release candidate #1

[Tzu-Li (Gordon) Tai]

+1 (binding)

- verified signatures and hashes
- NOTICE and LICENSE files in statefun-flink-distribution,
statefun-protobuf-shaded, and statefun-sdk-java looks sane
- maven clean install -Prun-e2e-tests (java 8) from source
- ran all examples and tutorials in apache/flink-statefun-playground with
the new artifacts
- Ran my SDK verifier utility [1] against the new Java and Python SDKs.

Cheers,
Gordon

[1] https://github.com/tzulitai/statefun-sdk-verifier

On Wed, Mar 31, 2021 at 8:50 PM Igal Shilman  wrote:

> Thanks Gordon for managing the release!
>
> +1 (non binding) from my side:
>
> Here are the results of my testing:
> - verified the signatures
> - verified that the source distribution doesn't contain any binary files
> - ran mvn clean install -Prun-e2e-tests with java8
> - ran the smoke test that sends 100 million messages locally.
> - extended the smoke test to include the remote sdks (1 function in the
> Java SDK, 1 function in the Python SDK), and it passes.
> - deployed to kubernetes with minio as an S3 replacement.
>
>
> On Tue, Mar 30, 2021 at 12:29 PM Tzu-Li (Gordon) Tai 
> wrote:
>
> > Hi everyone,
> >
> > Please review and vote on the release candidate #1 for the version 3.0.0
> of
> > Apache Flink Stateful Functions, as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > **Testing Guideline**
> >
> > You can find here [1] a page in the project wiki on instructions for
> > testing.
> > To cast a vote, it is not necessary to perform all listed checks,
> > but please mention which checks you have performed when voting.
> >
> > **Release Overview**
> >
> > As an overview, the release consists of the following:
> > a) Stateful Functions canonical source distribution, to be deployed to
> the
> > release repository at dist.apache.org
> > b) Stateful Functions Python SDK distributions to be deployed to PyPI
> > c) Maven artifacts to be deployed to the Maven Central Repository
> > d) New Dockerfiles for the release
> >
> > **Staging Areas to Review**
> >
> > The staging areas containing the above mentioned artifacts are as
> follows,
> > for your review:
> > * All artifacts for a) and b) can be found in the corresponding dev
> > repository at dist.apache.org [2]
> > * All artifacts for c) can be found at the Apache Nexus Repository [3]
> >
> > All artifacts are signed with the key
> > 1C1E2394D3194E1944613488F320986D35C33D6A [4]
> >
> > Other links for your review:
> > * JIRA release notes [5]
> > * source code tag “release-3.0.0-rc1” [6]
> > * PR for the new Dockerfiles [7]
> >
> > **Vote Duration**
> >
> > The voting time will run for at least 72 hours. I’m targeting this vote
> to
> > last until April. 2nd, 12pm CET.
> > It is adopted by majority approval, with at least 3 PMC affirmative
> votes.
> >
> > Thanks,
> > Gordon
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Stateful+Functions+Release
> > [2]
> https://dist.apache.org/repos/dist/dev/flink/flink-statefun-3.0.0-rc1/
> > [3]
> > https://repository.apache.org/content/repositories/orgapacheflink-1415/
> > [4] https://dist.apache.org/repos/dist/release/flink/KEYS
> > [5]
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12348822
> > [6] https://github.com/apache/flink-statefun/tree/release-3.0.0-rc1
> > [7] https://github.com/apache/flink-statefun-docker/pull/13
> >
>

[jira] [Created] (FLINK-22080) Refactor SqlClient for better testing

[Jark Wu (Jira)]

Jark Wu created FLINK-22080:
---

 Summary: Refactor SqlClient for better testing
 Key: FLINK-22080
 URL: https://issues.apache.org/jira/browse/FLINK-22080
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Client
Reporter: Jark Wu
 Fix For: 1.13.0


Currently, we added a JUnit Rule {{TerminalStreamsResource}} to replace 
{{System.in}} and {{System.out}} stream to get the output of SqlClient. 
However, this is not safe, especially used by multiple tests. 

We should refactor {{SqlClient}} to expose a new testing purpose {{main}} 
method which can pass in custom {{InputStream}} and {{OutputStream}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Feature freeze date for 1.13

[Kurt Young]

Hi Guowei and Dawid,

I want to request the permission to merge this feature [1], it's a useful
improvement to sql client and won't affect
other components too much. We were plan to merge it yesterday but met some
tricky multi-process issue which
has a very high possibility hanging the tests. It took us a while to find
out the root cause and fix it.

Since it's not too far away from feature freeze and RC0 also not created
yet, thus I would like to include this
in 1.13.

[1] https://issues.apache.org/jira/browse/FLINK-20320

Best,
Kurt


On Wed, Mar 31, 2021 at 5:55 PM Guowei Ma  wrote:

> Hi, community:
>
> Friendly reminder that today (3.31) is the last day of feature
> development. Under normal circumstances, you will not be able to submit new
> features from tomorrow (4.1). Tomorrow we will create 1.13.0-rc0 for
> testing, welcome to help test together.
> After the test is relatively stable, we will cut the release-1.13 branch.
>
> Best,
> Dawid & Guowei
>
>
> On Mon, Mar 29, 2021 at 5:17 PM Till Rohrmann 
> wrote:
>
>> +1 for the 31st of March for the feature freeze.
>>
>> Cheers,
>> Till
>>
>> On Mon, Mar 29, 2021 at 10:12 AM Robert Metzger 
>> wrote:
>>
>> > +1 for March 31st for the feature freeze.
>> >
>> >
>> >
>> > On Fri, Mar 26, 2021 at 3:39 PM Dawid Wysakowicz <
>> dwysakow...@apache.org>
>> > wrote:
>> >
>> > > Thank you Thomas! I'll definitely check the issue you linked.
>> > >
>> > > Best,
>> > >
>> > > Dawid
>> > >
>> > > On 23/03/2021 20:35, Thomas Weise wrote:
>> > > > Hi Dawid,
>> > > >
>> > > > Thanks for the heads up.
>> > > >
>> > > > Regarding the "Rebase and merge" button. I find that merge option
>> > useful,
>> > > > especially for small simple changes and for backports. The following
>> > > should
>> > > > help to safeguard from the issue encountered previously:
>> > > > https://github.com/jazzband/pip-tools/issues/1085
>> > > >
>> > > > Thanks,
>> > > > Thomas
>> > > >
>> > > >
>> > > > On Tue, Mar 23, 2021 at 4:58 AM Dawid Wysakowicz <
>> > dwysakow...@apache.org
>> > > >
>> > > > wrote:
>> > > >
>> > > >> Hi devs, users!
>> > > >>
>> > > >> 1. *Feature freeze date*
>> > > >>
>> > > >> We are approaching the end of March which we agreed would be the
>> time
>> > > for
>> > > >> a Feature Freeze. From the knowledge I've gather so far it still
>> seems
>> > > to
>> > > >> be a viable plan. I think it is a good time to agree on a
>> particular
>> > > date,
>> > > >> when it should happen. We suggest *(end of day CEST) March 31st*
>> > > >> (Wednesday next week) as the feature freeze time.
>> > > >>
>> > > >> Similarly as last time, we want to create RC0 on the day after the
>> > > feature
>> > > >> freeze, to make sure the RC creation process is running smoothly,
>> and
>> > to
>> > > >> have a common testing reference point.
>> > > >>
>> > > >> Having said that let us remind after Robert & Dian from the
>> previous
>> > > >> release what it a Feature Freeze means:
>> > > >>
>> > > >> *B) What does feature freeze mean?*After the feature freeze, no new
>> > > >> features are allowed to be merged to master. Only bug fixes and
>> > > >> documentation improvements.
>> > > >> The release managers will revert new feature commits after the
>> feature
>> > > >> freeze.
>> > > >> Rational: The goal of the feature freeze phase is to improve the
>> > system
>> > > >> stability by addressing known bugs. New features tend to introduce
>> new
>> > > >> instabilities, which would prolong the release process.
>> > > >> If you need to merge a new feature after the freeze, please open a
>> > > >> discussion on the dev@ list. If there are no objections by a PMC
>> > member
>> > > >> within 48 (workday)hours, the feature can be merged.
>> > > >>
>> > > >> 2. *Merge PRs from the command line*
>> > > >>
>> > > >> In the past releases it was quite frequent around the Feature
>> Freeze
>> > > date
>> > > >> that we ended up with a broken main branch that either did not
>> compile
>> > > or
>> > > >> there were failing tests. It was often due to concurrent merges to
>> the
>> > > main
>> > > >> branch via the "Rebase and merge" button. To overcome the problem
>> we
>> > > would
>> > > >> like to suggest only ever merging PRs from a command line. Thank
>> you
>> > > >> Stephan for the idea! The suggested workflow would look as follows:
>> > > >>
>> > > >>1. Pull the change and rebase on the current main branch
>> > > >>2. Build the project (e.g. from IDE, which should be faster than
>> > > >>building entire project from cmd) -> this should ensure the
>> project
>> > > compiles
>> > > >>3. Run the tests in the module that the change affects -> this
>> > should
>> > > >>greatly minimize the chances of failling tests
>> > > >>4. Push the change to the main branch
>> > > >>
>> > > >> Let us know what you think!
>> > > >>
>> > > >> Best,
>> > > >>
>> > > >> Guowei & Dawid
>> > > >>
>> > > >>
>> > > >>
>> > >
>> > >
>> >
>>
>

[jira] [Created] (FLINK-22079) ReactiveModeITCase stack with fine grained resource manager.

[Xintong Song (Jira)]

Xintong Song created FLINK-22079:


 Summary: ReactiveModeITCase stack with fine grained resource 
manager.
 Key: FLINK-22079
 URL: https://issues.apache.org/jira/browse/FLINK-22079
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.13.0
Reporter: Xintong Song


CI stage for fine grained resource management hangs. According to the maven 
logs, it is {{ReactiveModeITCase}} that never finish.

[https://dev.azure.com/sewen0794/Flink/_build/results?buildId=249=logs=cc649950-03e9-5fae-8326-2f1ad744b536=51cab6ca-669f-5dc0-221d-1e4f7dc4fc85=9962]

Not sure if this is related to {{FineGrainedSlotManager}} or not. Although we 
say reactive mode does not support fine grained resource management, the 
{{FineGrainedSlotManager}} itself is expected to work with both fine-grained 
and coarse-grained resource requirements.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22078) Introduce a Printer interface to rederict the output to the desired stream

[Shengkai Fang (Jira)]

Shengkai Fang created FLINK-22078:
-

 Summary: Introduce a Printer interface to rederict the output to 
the desired stream
 Key: FLINK-22078
 URL: https://issues.apache.org/jira/browse/FLINK-22078
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Client
Reporter: Shengkai Fang


The reason why we introduce this is the {{Terminal}} is not allow to modify the 
output stream after the {{Terminal}} is built. 

Therefore, we can introduce a Printer to control the behaviour.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22077) Wrong way to calculate cross-region ConsumedPartitionGroups in PipelinedRegionSchedulingStrategy

[Zhilong Hong (Jira)]

Zhilong Hong created FLINK-22077:


 Summary: Wrong way to calculate cross-region 
ConsumedPartitionGroups in PipelinedRegionSchedulingStrategy
 Key: FLINK-22077
 URL: https://issues.apache.org/jira/browse/FLINK-22077
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.13.0
Reporter: Zhilong Hong
 Fix For: 1.13.0


h3. Introduction

We implement a wrong way to calculate cross-region ConsumedPartitionGroups in 
{{PipelinedRegionSchedulingStrategy}}, it slows down the procedure of 
{{onExecutionStateChange}}, make the complexity from O(N) to O(N^2). Also the 
semantic of cross-region is totally wrong.
h3. Details

In {{PipelinedRegionSchedulingStrategy#startScheduling}}, as expected, we need 
to schedule all region with no external blocking edges, i.e., source regions. 
To decrease the complexity, we choose to schedule all the regions that has no 
external BLOCKING ConsumedPartitionGroups.

However, for the case illustrated in FLINK-22017, the region 2 has a 
ConsumedPartitionGroup, which has both internal and external blocking 
IntermediateResultPartitions. If we choose one to represent the entire 
ConsumedPartitionGroup, it may choose the internal one, and make the entire 
group internal. Region 2 will be scheduled.

As Region 1 is not finished, Region 2 cannot transition to running. A deadlock 
may happen if resource is not enough for both two regions.

To make it right, we introduced cross-region ConsumedPartitionGroups in 
FLINK-21330. The regions which has ConsumedPartitionGroups with both internal 
and external blocking IntermediateResultPartitions will be recorded. When we 
call {{startScheduling}}, these ConsumedPartitionGroups will be treated as 
external, and region 2 will not be scheduled.

But we have to admit that the implementation of cross-region is wrong. The 
ConsumedPartitionGroups that has multiple producer regions will be treated as 
cross-region groups. It is not the same logic as we mentioned above. The 
semantic is totally wrong. Also all the ALL-TO-ALL BLOCKING 
ConsumedPartitionGroups will be treated as cross-region groups, since their 
producers are in different regions. (Each producer has its own region.) This 
slows down the complexity from O(N) to O(N^2) for ALL-TO-ALL BLOCKING edges.
h3. Solution

To correctly calculate the cross-region ConsumedPartitionGroups, we can just 
calculate the producer regions for all ConsumedPartitionGroups, and then 
iterate all the regions and its ConsumedPartitionGroups. If the 
ConsumedPartitionGroup has two or more producer regions, and the regions 
contains current region, it is a cross-region ConsumedPartitionGroup. This 
meets the correct semantics, and make sure ALL-TO-ALL BLOCKING 
ConsumedPartitionGroups will not be treated as cross-region one. This fix will 
also decreases the complexity from O(N) to O(N^2). I prefer it's necessary to 
add this bug-fix to release 1.13.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22076) Python Test failed with "OSError: [Errno 12] Cannot allocate memory"

[Huang Xingbo (Jira)]

Huang Xingbo created FLINK-22076:


 Summary: Python Test failed with "OSError: [Errno 12] Cannot 
allocate memory"
 Key: FLINK-22076
 URL: https://issues.apache.org/jira/browse/FLINK-22076
 Project: Flink
  Issue Type: Bug
  Components: API / Python
Affects Versions: 1.13.0
Reporter: Huang Xingbo


https://dev.azure.com/sewen0794/Flink/_build/results?buildId=249=logs=fba17979-6d2e-591d-72f1-97cf42797c11=443dc6bf-b240-56df-6acf-c882d4b238da=21533

Python Test failed with "OSError: [Errno 12] Cannot allocate memory" in Azure 
Pipeline. I am not sure if it is caused by insufficient machine memory on Azure.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Apply external 3 days for FLINK-20387 after feature freeze

[Jark Wu]

+1 for more days for this FLIP.

Changing data type is a massive work, and supporting windowing on
timestamp_ltz is also
 a very complex job when considering DST. Would be great if we can have
more days to
finalize this work.

Best,
Jark

On Thu, 1 Apr 2021 at 03:46, Timo Walther  wrote:

> Hi everyone,
>
> I support Leonard's request. It was foreseeable that the changes of
> FLIP-162 will be massive and will take some time. By looking at PRs such as
>
> https://github.com/apache/flink/pull/15280
>
> I would also vote for giving a bit more time for proper reviews and
> finalizing this story for consistency.
>
> Regards,
> Timo
>
>
>
> On 31.03.21 17:56, Leonard Xu wrote:
> > Hi,  Dawid & Guowei
> >
> > Sorry to apply the extension, I want to apply 3 days for ticket
> [FLINK-20387] Support column of TIMESTAMP WITH LOCAL ZONE TIME type as
> rowtime[1],  it is the last ticket of FLIP-162[2] which aims to solve
> various time zone issues and offer a consistent time function behavior.  We
> experienced a long discussion for this FLIP and I took some efforts to
> resolve the  tricky daylight saving time problem. In fact, I have been
> working hard to promote this FLIP recently.
> >
> > But I really hope that this feature can join 1.13. The motivation is not
> because I want to complete this FLIP personally, but from the user's
> perspective that we can provide them a consistent time behavior and
> resolves the time zone issues naturally, we believe this will greatly
> improve the user experience, thus I’m asking to apply 3 days for only this
> ticket FLINK-20387.
> >
> > Best,
> > Leonard
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-20387
> > [2]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-162%3A+Consistent+Flink+SQL+time+function+behavior
> >
> >
> >> 在 2021年3月31日，17:54，Guowei Ma  写道：
> >>
> >> Hi, community:
> >>
> >> Friendly reminder that today (3.31) is the last day of feature
> development. Under normal circumstances, you will not be able to submit new
> features from tomorrow (4.1). Tomorrow we will create 1.13.0-rc0 for
> testing, welcome to help test together.
> >> After the test is relatively stable, we will cut the release-1.13
> branch.
> >>
> >> Best,
> >> Dawid & Guowei
> >>
> >>
> >> On Mon, Mar 29, 2021 at 5:17 PM Till Rohrmann  > wrote:
> >> +1 for the 31st of March for the feature freeze.
> >>
> >> Cheers,
> >> Till
> >>
> >> On Mon, Mar 29, 2021 at 10:12 AM Robert Metzger  > wrote:
> >>
> >>> +1 for March 31st for the feature freeze.
> >>>
> >>>
> >>>
> >>> On Fri, Mar 26, 2021 at 3:39 PM Dawid Wysakowicz <
> dwysakow...@apache.org >
> >>> wrote:
> >>>
>  Thank you Thomas! I'll definitely check the issue you linked.
> 
>  Best,
> 
>  Dawid
> 
>  On 23/03/2021 20:35, Thomas Weise wrote:
> > Hi Dawid,
> >
> > Thanks for the heads up.
> >
> > Regarding the "Rebase and merge" button. I find that merge option
> >>> useful,
> > especially for small simple changes and for backports. The following
>  should
> > help to safeguard from the issue encountered previously:
> > https://github.com/jazzband/pip-tools/issues/1085 <
> https://github.com/jazzband/pip-tools/issues/1085>
> >
> > Thanks,
> > Thomas
> >
> >
> > On Tue, Mar 23, 2021 at 4:58 AM Dawid Wysakowicz <
> >>> dwysakow...@apache.org 
> >
> > wrote:
> >
> >> Hi devs, users!
> >>
> >> 1. *Feature freeze date*
> >>
> >> We are approaching the end of March which we agreed would be the
> time
>  for
> >> a Feature Freeze. From the knowledge I've gather so far it still
> seems
>  to
> >> be a viable plan. I think it is a good time to agree on a particular
>  date,
> >> when it should happen. We suggest *(end of day CEST) March 31st*
> >> (Wednesday next week) as the feature freeze time.
> >>
> >> Similarly as last time, we want to create RC0 on the day after the
>  feature
> >> freeze, to make sure the RC creation process is running smoothly,
> and
> >>> to
> >> have a common testing reference point.
> >>
> >> Having said that let us remind after Robert & Dian from the previous
> >> release what it a Feature Freeze means:
> >>
> >> *B) What does feature freeze mean?*After the feature freeze, no new
> >> features are allowed to be merged to master. Only bug fixes and
> >> documentation improvements.
> >> The release managers will revert new feature commits after the
> feature
> >> freeze.
> >> Rational: The goal of the feature freeze phase is to improve the
> >>> system
> >> stability by addressing known bugs. New features tend to introduce
> new
> >> instabilities, which would prolong the release process.
> >> If you need to merge a new feature after the freeze, please open a
> >> 

1

[醉时无梦生]

1

Re: Apply external 3 days for FLINK-20387 after feature freeze

[Timo Walther]

Hi everyone,

I support Leonard's request. It was foreseeable that the changes of 
FLIP-162 will be massive and will take some time. By looking at PRs such as


https://github.com/apache/flink/pull/15280

I would also vote for giving a bit more time for proper reviews and 
finalizing this story for consistency.


Regards,
Timo



On 31.03.21 17:56, Leonard Xu wrote:

Hi,  Dawid & Guowei

Sorry to apply the extension, I want to apply 3 days for ticket  [FLINK-20387] 
Support column of TIMESTAMP WITH LOCAL ZONE TIME type as rowtime[1],  it is the 
last ticket of FLIP-162[2] which aims to solve various time zone issues and 
offer a consistent time function behavior.  We experienced a long discussion 
for this FLIP and I took some efforts to resolve the  tricky daylight saving 
time problem. In fact, I have been working hard to promote this FLIP recently.

But I really hope that this feature can join 1.13. The motivation is not 
because I want to complete this FLIP personally, but from the user's 
perspective that we can provide them a consistent time behavior and resolves 
the time zone issues naturally, we believe this will greatly improve the user 
experience, thus I’m asking to apply 3 days for only this ticket FLINK-20387.

Best,
Leonard

[1] https://issues.apache.org/jira/browse/FLINK-20387
[2] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-162%3A+Consistent+Flink+SQL+time+function+behavior



在 2021年3月31日，17:54，Guowei Ma  写道：

Hi, community:

Friendly reminder that today (3.31) is the last day of feature development. 
Under normal circumstances, you will not be able to submit new features from 
tomorrow (4.1). Tomorrow we will create 1.13.0-rc0 for testing, welcome to help 
test together.
After the test is relatively stable, we will cut the release-1.13 branch.

Best,
Dawid & Guowei


On Mon, Mar 29, 2021 at 5:17 PM Till Rohrmann mailto:trohrm...@apache.org>> wrote:
+1 for the 31st of March for the feature freeze.

Cheers,
Till

On Mon, Mar 29, 2021 at 10:12 AM Robert Metzger mailto:rmetz...@apache.org>> wrote:


+1 for March 31st for the feature freeze.



On Fri, Mar 26, 2021 at 3:39 PM Dawid Wysakowicz mailto:dwysakow...@apache.org>>
wrote:


Thank you Thomas! I'll definitely check the issue you linked.

Best,

Dawid

On 23/03/2021 20:35, Thomas Weise wrote:

Hi Dawid,

Thanks for the heads up.

Regarding the "Rebase and merge" button. I find that merge option

useful,

especially for small simple changes and for backports. The following

should

help to safeguard from the issue encountered previously:
https://github.com/jazzband/pip-tools/issues/1085 


Thanks,
Thomas


On Tue, Mar 23, 2021 at 4:58 AM Dawid Wysakowicz <

dwysakow...@apache.org 


wrote:


Hi devs, users!

1. *Feature freeze date*

We are approaching the end of March which we agreed would be the time

for

a Feature Freeze. From the knowledge I've gather so far it still seems

to

be a viable plan. I think it is a good time to agree on a particular

date,

when it should happen. We suggest *(end of day CEST) March 31st*
(Wednesday next week) as the feature freeze time.

Similarly as last time, we want to create RC0 on the day after the

feature

freeze, to make sure the RC creation process is running smoothly, and

to

have a common testing reference point.

Having said that let us remind after Robert & Dian from the previous
release what it a Feature Freeze means:

*B) What does feature freeze mean?*After the feature freeze, no new
features are allowed to be merged to master. Only bug fixes and
documentation improvements.
The release managers will revert new feature commits after the feature
freeze.
Rational: The goal of the feature freeze phase is to improve the

system

stability by addressing known bugs. New features tend to introduce new
instabilities, which would prolong the release process.
If you need to merge a new feature after the freeze, please open a
discussion on the dev@ list. If there are no objections by a PMC

member

within 48 (workday)hours, the feature can be merged.

2. *Merge PRs from the command line*

In the past releases it was quite frequent around the Feature Freeze

date

that we ended up with a broken main branch that either did not compile

or

there were failing tests. It was often due to concurrent merges to the

main

branch via the "Rebase and merge" button. To overcome the problem we

would

like to suggest only ever merging PRs from a command line. Thank you
Stephan for the idea! The suggested workflow would look as follows:

1. Pull the change and rebase on the current main branch
2. Build the project (e.g. from IDE, which should be faster than
building entire project from cmd) -> this should ensure the project

compiles

3. Run the tests in the module that the change affects -> this

should

greatly minimize the chances of failling tests
4. Push the change to the main branch

Let 

[jira] [Created] (FLINK-22075) Incorrect null outputs in left join

[Jamie Brandon (Jira)]

Jamie Brandon created FLINK-22075:
-

 Summary: Incorrect null outputs in left join
 Key: FLINK-22075
 URL: https://issues.apache.org/jira/browse/FLINK-22075
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / API
Affects Versions: 1.12.2
 Environment: 
https://github.com/jamii/streaming-consistency/blob/4e5d144dacf85e512bdc7afd77d031b5974d733e/pkgs.nix#L25-L46

```
[nix-shell:~/streaming-consistency/flink]$ java -version
openjdk version "1.8.0_265"
OpenJDK Runtime Environment (build 1.8.0_265-ga)
OpenJDK 64-Bit Server VM (build 25.265-bga, mixed mode)

[nix-shell:~/streaming-consistency/flink]$ flink --version
Version: 1.12.2, Commit ID: 4dedee0

[nix-shell:~/streaming-consistency/flink]$ nix-info
system: "x86_64-linux", multi-user?: yes, version: nix-env (Nix) 2.3.10, 
channels(jamie): "", channels(root): "nixos-20.09.3554.f8929dce13e", nixpkgs: 
/nix/var/nix/profiles/per-user/root/channels/nixos
```
Reporter: Jamie Brandon


I'm left joining a table with itself 
[here](https://github.com/jamii/streaming-consistency/blob/4e5d144dacf85e512bdc7afd77d031b5974d733e/flink/src/main/java/Demo.java#L55-L66).
 The output should have no nulls, or at least emit nulls and then retract them. 
Instead I see:

```
jamie@machine:~/streaming-consistency/flink$ wc -l tmp/outer_join_with_time
10 tmp/outer_join_with_time
jamie@machine:~/streaming-consistency/flink$ grep -c insert 
tmp/outer_join_with_time
10
jamie@machine:~/streaming-consistency/flink$ grep -c 'null' 
tmp/outer_join_with_time
16943
```

~17% of the outputs are incorrect and never retracted.

[Full 
output](https://gist.githubusercontent.com/jamii/983fee41609b1425fe7fa59d3249b249/raw/069b9dcd4faf9f6113114381bc7028c6642ca787/gistfile1.txt)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22074) testRequirementCheckOnlyTriggeredOnce(org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTest) failed

[Leonard Xu (Jira)]

Leonard Xu created FLINK-22074:
--

 Summary: 
testRequirementCheckOnlyTriggeredOnce(org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTest)
 failed
 Key: FLINK-22074
 URL: https://issues.apache.org/jira/browse/FLINK-22074
 Project: Flink
  Issue Type: Bug
  Components: Tests
Affects Versions: 1.13.0
Reporter: Leonard Xu
 Fix For: 1.13.0


[ERROR] Tests run: 14, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.654 
s <<< FAILURE! - in 
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTest 
[ERROR] 
testRequirementCheckOnlyTriggeredOnce(org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTest)
 Time elapsed: 0.059 s <<< FAILURE! 
java.lang.AssertionError: Expected to fail with a timeout. 
 at org.junit.Assert.fail(Assert.java:88) 
 at 
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTestBase.assertFutureNotComplete(FineGrainedSlotManagerTestBase.java:126)
 
 at 
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTest$10.lambda$new$3(FineGrainedSlotManagerTest.java:605)
 
 at 
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTestBase$Context.runTest(FineGrainedSlotManagerTestBase.java:197)
 
 at 
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTest$10.(FineGrainedSlotManagerTest.java:581)
 
 at 
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTest.testRequirementCheckOnlyTriggeredOnce(FineGrainedSlotManagerTest.java:565)
 
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
 at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 
 at java.lang.reflect.Method.invoke(Method.java:498) 
 at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
 
 at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
 
 at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
 
 at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
 
 at org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45) 
 at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) 
 at org.junit.rules.RunRules.evaluate(RunRules.java:20) 
 at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) 
 at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
 
 at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
 
 at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) 
 at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) 
 at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) 
 at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) 
 at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) 
 at org.junit.runners.ParentRunner.run(ParentRunner.java:363) 
 at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
 
 at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
 
 at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
 
 at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) 
 at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
 
 at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
 
 at 
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) 
 at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Apply external 3 days for FLINK-20387 after feature freeze

[Leonard Xu]

Hi,  Dawid & Guowei

Sorry to apply the extension, I want to apply 3 days for ticket  [FLINK-20387] 
Support column of TIMESTAMP WITH LOCAL ZONE TIME type as rowtime[1],  it is the 
last ticket of FLIP-162[2] which aims to solve various time zone issues and 
offer a consistent time function behavior.  We experienced a long discussion 
for this FLIP and I took some efforts to resolve the  tricky daylight saving 
time problem. In fact, I have been working hard to promote this FLIP recently.

But I really hope that this feature can join 1.13. The motivation is not 
because I want to complete this FLIP personally, but from the user's 
perspective that we can provide them a consistent time behavior and resolves 
the time zone issues naturally, we believe this will greatly improve the user 
experience, thus I’m asking to apply 3 days for only this ticket FLINK-20387.

Best,
Leonard

[1] https://issues.apache.org/jira/browse/FLINK-20387
[2] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-162%3A+Consistent+Flink+SQL+time+function+behavior


> 在 2021年3月31日，17:54，Guowei Ma  写道：
> 
> Hi, community:
> 
> Friendly reminder that today (3.31) is the last day of feature development. 
> Under normal circumstances, you will not be able to submit new features from 
> tomorrow (4.1). Tomorrow we will create 1.13.0-rc0 for testing, welcome to 
> help test together.
> After the test is relatively stable, we will cut the release-1.13 branch.
> 
> Best,
> Dawid & Guowei
> 
> 
> On Mon, Mar 29, 2021 at 5:17 PM Till Rohrmann  > wrote:
> +1 for the 31st of March for the feature freeze.
> 
> Cheers,
> Till
> 
> On Mon, Mar 29, 2021 at 10:12 AM Robert Metzger  > wrote:
> 
> > +1 for March 31st for the feature freeze.
> >
> >
> >
> > On Fri, Mar 26, 2021 at 3:39 PM Dawid Wysakowicz  > >
> > wrote:
> >
> > > Thank you Thomas! I'll definitely check the issue you linked.
> > >
> > > Best,
> > >
> > > Dawid
> > >
> > > On 23/03/2021 20:35, Thomas Weise wrote:
> > > > Hi Dawid,
> > > >
> > > > Thanks for the heads up.
> > > >
> > > > Regarding the "Rebase and merge" button. I find that merge option
> > useful,
> > > > especially for small simple changes and for backports. The following
> > > should
> > > > help to safeguard from the issue encountered previously:
> > > > https://github.com/jazzband/pip-tools/issues/1085 
> > > > 
> > > >
> > > > Thanks,
> > > > Thomas
> > > >
> > > >
> > > > On Tue, Mar 23, 2021 at 4:58 AM Dawid Wysakowicz <
> > dwysakow...@apache.org 
> > > >
> > > > wrote:
> > > >
> > > >> Hi devs, users!
> > > >>
> > > >> 1. *Feature freeze date*
> > > >>
> > > >> We are approaching the end of March which we agreed would be the time
> > > for
> > > >> a Feature Freeze. From the knowledge I've gather so far it still seems
> > > to
> > > >> be a viable plan. I think it is a good time to agree on a particular
> > > date,
> > > >> when it should happen. We suggest *(end of day CEST) March 31st*
> > > >> (Wednesday next week) as the feature freeze time.
> > > >>
> > > >> Similarly as last time, we want to create RC0 on the day after the
> > > feature
> > > >> freeze, to make sure the RC creation process is running smoothly, and
> > to
> > > >> have a common testing reference point.
> > > >>
> > > >> Having said that let us remind after Robert & Dian from the previous
> > > >> release what it a Feature Freeze means:
> > > >>
> > > >> *B) What does feature freeze mean?*After the feature freeze, no new
> > > >> features are allowed to be merged to master. Only bug fixes and
> > > >> documentation improvements.
> > > >> The release managers will revert new feature commits after the feature
> > > >> freeze.
> > > >> Rational: The goal of the feature freeze phase is to improve the
> > system
> > > >> stability by addressing known bugs. New features tend to introduce new
> > > >> instabilities, which would prolong the release process.
> > > >> If you need to merge a new feature after the freeze, please open a
> > > >> discussion on the dev@ list. If there are no objections by a PMC
> > member
> > > >> within 48 (workday)hours, the feature can be merged.
> > > >>
> > > >> 2. *Merge PRs from the command line*
> > > >>
> > > >> In the past releases it was quite frequent around the Feature Freeze
> > > date
> > > >> that we ended up with a broken main branch that either did not compile
> > > or
> > > >> there were failing tests. It was often due to concurrent merges to the
> > > main
> > > >> branch via the "Rebase and merge" button. To overcome the problem we
> > > would
> > > >> like to suggest only ever merging PRs from a command line. Thank you
> > > >> Stephan for the idea! The suggested workflow would look as follows:
> > > >>
> > > >>1. Pull the change and rebase on the current main branch
> > > >>2. Build the project (e.g. from IDE, which should 

[jira] [Created] (FLINK-22073) ParquetFileCompactionITCase fails on Azure

[Dawid Wysakowicz (Jira)]

Dawid Wysakowicz created FLINK-22073:


 Summary: ParquetFileCompactionITCase fails on Azure
 Key: FLINK-22073
 URL: https://issues.apache.org/jira/browse/FLINK-22073
 Project: Flink
  Issue Type: Bug
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.13.0
Reporter: Dawid Wysakowicz
 Fix For: 1.13.0


{code}
[INFO] Running org.apache.flink.formats.parquet.ParquetFsStreamingSinkITCase
[ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 17.612 
s <<< FAILURE! - in org.apache.flink.formats.parquet.ParquetFileCompactionITCase
[ERROR] 
testNonPartition(org.apache.flink.formats.parquet.ParquetFileCompactionITCase)  
Time elapsed: 1.509 s  <<< FAILURE!
java.lang.AssertionError: expected:<[+I[0, 0, 0], +I[0, 0, 0], +I[1, 1, 1], 
+I[1, 1, 1], +I[2, 2, 2], +I[2, 2, 2], +I[3, 3, 3], +I[3, 3, 3], +I[4, 4, 4], 
+I[4, 4, 4], +I[5, 5, 5], +I[5, 5, 5], +I[6, 6, 6], +I[6, 6, 6], +I[7, 7, 7], 
+I[7, 7, 7], +I[8, 8, 8], +I[8, 8, 8], +I[9, 9, 9], +I[9, 9, 9], +I[10, 0, 0], 
+I[10, 0, 0], +I[11, 1, 1], +I[11, 1, 1], +I[12, 2, 2], +I[12, 2, 2], +I[13, 3, 
3], +I[13, 3, 3], +I[14, 4, 4], +I[14, 4, 4], +I[15, 5, 5], +I[15, 5, 5], 
+I[16, 6, 6], +I[16, 6, 6], +I[17, 7, 7], +I[17, 7, 7], +I[18, 8, 8], +I[18, 8, 
8], +I[19, 9, 9], +I[19, 9, 9], +I[20, 0, 0], +I[20, 0, 0], +I[21, 1, 1], 
+I[21, 1, 1], +I[22, 2, 2], +I[22, 2, 2], +I[23, 3, 3], +I[23, 3, 3], +I[24, 4, 
4], +I[24, 4, 4], +I[25, 5, 5], +I[25, 5, 5], +I[26, 6, 6], +I[26, 6, 6], 
+I[27, 7, 7], +I[27, 7, 7], +I[28, 8, 8], +I[28, 8, 8], +I[29, 9, 9], +I[29, 9, 
9], +I[30, 0, 0], +I[30, 0, 0], +I[31, 1, 1], +I[31, 1, 1], +I[32, 2, 2], 
+I[32, 2, 2], +I[33, 3, 3], +I[33, 3, 3], +I[34, 4, 4], +I[34, 4, 4], +I[35, 5, 
5], +I[35, 5, 5], +I[36, 6, 6], +I[36, 6, 6], +I[37, 7, 7], +I[37, 7, 7], 
+I[38, 8, 8], +I[38, 8, 8], +I[39, 9, 9], +I[39, 9, 9], +I[40, 0, 0], +I[40, 0, 
0], +I[41, 1, 1], +I[41, 1, 1], +I[42, 2, 2], +I[42, 2, 2], +I[43, 3, 3], 
+I[43, 3, 3], +I[44, 4, 4], +I[44, 4, 4], +I[45, 5, 5], +I[45, 5, 5], +I[46, 6, 
6], +I[46, 6, 6], +I[47, 7, 7], +I[47, 7, 7], +I[48, 8, 8], +I[48, 8, 8], 
+I[49, 9, 9], +I[49, 9, 9], +I[50, 0, 0], +I[50, 0, 0], +I[51, 1, 1], +I[51, 1, 
1], +I[52, 2, 2], +I[52, 2, 2], +I[53, 3, 3], +I[53, 3, 3], +I[54, 4, 4], 
+I[54, 4, 4], +I[55, 5, 5], +I[55, 5, 5], +I[56, 6, 6], +I[56, 6, 6], +I[57, 7, 
7], +I[57, 7, 7], +I[58, 8, 8], +I[58, 8, 8], +I[59, 9, 9], +I[59, 9, 9], 
+I[60, 0, 0], +I[60, 0, 0], +I[61, 1, 1], +I[61, 1, 1], +I[62, 2, 2], +I[62, 2, 
2], +I[63, 3, 3], +I[63, 3, 3], +I[64, 4, 4], +I[64, 4, 4], +I[65, 5, 5], 
+I[65, 5, 5], +I[66, 6, 6], +I[66, 6, 6], +I[67, 7, 7], +I[67, 7, 7], +I[68, 8, 
8], +I[68, 8, 8], +I[69, 9, 9], +I[69, 9, 9], +I[70, 0, 0], +I[70, 0, 0], 
+I[71, 1, 1], +I[71, 1, 1], +I[72, 2, 2], +I[72, 2, 2], +I[73, 3, 3], +I[73, 3, 
3], +I[74, 4, 4], +I[74, 4, 4], +I[75, 5, 5], +I[75, 5, 5], +I[76, 6, 6], 
+I[76, 6, 6], +I[77, 7, 7], +I[77, 7, 7], +I[78, 8, 8], +I[78, 8, 8], +I[79, 9, 
9], +I[79, 9, 9], +I[80, 0, 0], +I[80, 0, 0], +I[81, 1, 1], +I[81, 1, 1], 
+I[82, 2, 2], +I[82, 2, 2], +I[83, 3, 3], +I[83, 3, 3], +I[84, 4, 4], +I[84, 4, 
4], +I[85, 5, 5], +I[85, 5, 5], +I[86, 6, 6], +I[86, 6, 6], +I[87, 7, 7], 
+I[87, 7, 7], +I[88, 8, 8], +I[88, 8, 8], +I[89, 9, 9], +I[89, 9, 9], +I[90, 0, 
0], +I[90, 0, 0], +I[91, 1, 1], +I[91, 1, 1], +I[92, 2, 2], +I[92, 2, 2], 
+I[93, 3, 3], +I[93, 3, 3], +I[94, 4, 4], +I[94, 4, 4], +I[95, 5, 5], +I[95, 5, 
5], +I[96, 6, 6], +I[96, 6, 6], +I[97, 7, 7], +I[97, 7, 7], +I[98, 8, 8], 
+I[98, 8, 8], +I[99, 9, 9], +I[99, 9, 9]]> but was:<[+I[0, 0, 0], +I[0, 0, 0], 
+I[1, 1, 1], +I[2, 2, 2], +I[2, 2, 2], +I[3, 3, 3], +I[3, 3, 3], +I[4, 4, 4], 
+I[5, 5, 5], +I[5, 5, 5], +I[6, 6, 6], +I[6, 6, 6], +I[7, 7, 7], +I[8, 8, 8], 
+I[8, 8, 8], +I[9, 9, 9], +I[9, 9, 9], +I[10, 0, 0], +I[11, 1, 1], +I[11, 1, 
1], +I[12, 2, 2], +I[12, 2, 2], +I[13, 3, 3], +I[14, 4, 4], +I[14, 4, 4], 
+I[15, 5, 5], +I[15, 5, 5], +I[16, 6, 6], +I[17, 7, 7], +I[17, 7, 7], +I[18, 8, 
8], +I[18, 8, 8], +I[19, 9, 9], +I[20, 0, 0], +I[20, 0, 0], +I[21, 1, 1], 
+I[21, 1, 1], +I[22, 2, 2], +I[23, 3, 3], +I[23, 3, 3], +I[24, 4, 4], +I[24, 4, 
4], +I[25, 5, 5], +I[26, 6, 6], +I[26, 6, 6], +I[27, 7, 7], +I[27, 7, 7], 
+I[28, 8, 8], +I[29, 9, 9], +I[29, 9, 9], +I[30, 0, 0], +I[30, 0, 0], +I[31, 1, 
1], +I[32, 2, 2], +I[32, 2, 2], +I[33, 3, 3], +I[33, 3, 3], +I[34, 4, 4], 
+I[35, 5, 5], +I[35, 5, 5], +I[36, 6, 6], +I[36, 6, 6], +I[37, 7, 7], +I[38, 8, 
8], +I[38, 8, 8], +I[39, 9, 9], +I[39, 9, 9], +I[40, 0, 0], +I[41, 1, 1], 
+I[41, 1, 1], +I[42, 2, 2], +I[42, 2, 2], +I[43, 3, 3], +I[44, 4, 4], +I[44, 4, 
4], +I[45, 5, 5], +I[45, 5, 5], +I[46, 6, 6], +I[47, 7, 7], +I[47, 7, 7], 
+I[48, 8, 8], +I[48, 8, 8], +I[49, 9, 9], +I[50, 0, 0], +I[50, 0, 0], +I[51, 1, 
1], +I[51, 1, 1], +I[52, 2, 2], +I[53, 3, 3], +I[53, 3, 3], +I[54, 4, 4], 
+I[54, 4, 4], 

[jira] [Created] (FLINK-22072) Rethink UnfulfillableSlotRequestException class hierarchy

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-22072:


 Summary: Rethink UnfulfillableSlotRequestException class hierarchy
 Key: FLINK-22072
 URL: https://issues.apache.org/jira/browse/FLINK-22072
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Reporter: Chesnay Schepler
 Fix For: 1.14.0


The ResourceManager currently can throw 2 different exceptions if not enough 
slots are available:
UnfulfillableSlotRequestException, if a slot could not be acquired, and no TM 
in the cluster could even theoretically provide it (due to the resource 
profiles not matching)
NoResourceAvailableException, if a slot could be acquired, because none are 
free or a task executor, that could theoretically provide it, could not be 
allocated

Despite being fairly similar in terms of semantics, hierarchy wise these are 
completely separate.

{code}
UnfulfillableSlotRequestException < ResourceManagerException < FlinkException
NoResourceAvailableException < JobException
{code}





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22071) LookupTableSource.LookupRuntimeProvider customizes parallelism

[Wong Mulan (Jira)]

Wong Mulan created FLINK-22071:
--

 Summary: LookupTableSource.LookupRuntimeProvider customizes 
parallelism
 Key: FLINK-22071
 URL: https://issues.apache.org/jira/browse/FLINK-22071
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Planner
Affects Versions: 1.13.0
Reporter: Wong Mulan


Now, sink table can customize parallelism. LookupTable is not supported. Could 
we add this capability.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Apache Flink Stateful Functions 3.0.0, release candidate #1

[Igal Shilman]

Thanks Gordon for managing the release!

+1 (non binding) from my side:

Here are the results of my testing:
- verified the signatures
- verified that the source distribution doesn't contain any binary files
- ran mvn clean install -Prun-e2e-tests with java8
- ran the smoke test that sends 100 million messages locally.
- extended the smoke test to include the remote sdks (1 function in the
Java SDK, 1 function in the Python SDK), and it passes.
- deployed to kubernetes with minio as an S3 replacement.


On Tue, Mar 30, 2021 at 12:29 PM Tzu-Li (Gordon) Tai 
wrote:

> Hi everyone,
>
> Please review and vote on the release candidate #1 for the version 3.0.0 of
> Apache Flink Stateful Functions, as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> **Testing Guideline**
>
> You can find here [1] a page in the project wiki on instructions for
> testing.
> To cast a vote, it is not necessary to perform all listed checks,
> but please mention which checks you have performed when voting.
>
> **Release Overview**
>
> As an overview, the release consists of the following:
> a) Stateful Functions canonical source distribution, to be deployed to the
> release repository at dist.apache.org
> b) Stateful Functions Python SDK distributions to be deployed to PyPI
> c) Maven artifacts to be deployed to the Maven Central Repository
> d) New Dockerfiles for the release
>
> **Staging Areas to Review**
>
> The staging areas containing the above mentioned artifacts are as follows,
> for your review:
> * All artifacts for a) and b) can be found in the corresponding dev
> repository at dist.apache.org [2]
> * All artifacts for c) can be found at the Apache Nexus Repository [3]
>
> All artifacts are signed with the key
> 1C1E2394D3194E1944613488F320986D35C33D6A [4]
>
> Other links for your review:
> * JIRA release notes [5]
> * source code tag “release-3.0.0-rc1” [6]
> * PR for the new Dockerfiles [7]
>
> **Vote Duration**
>
> The voting time will run for at least 72 hours. I’m targeting this vote to
> last until April. 2nd, 12pm CET.
> It is adopted by majority approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Gordon
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Stateful+Functions+Release
> [2] https://dist.apache.org/repos/dist/dev/flink/flink-statefun-3.0.0-rc1/
> [3]
> https://repository.apache.org/content/repositories/orgapacheflink-1415/
> [4] https://dist.apache.org/repos/dist/release/flink/KEYS
> [5]
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12348822
> [6] https://github.com/apache/flink-statefun/tree/release-3.0.0-rc1
> [7] https://github.com/apache/flink-statefun-docker/pull/13
>

[jira] [Created] (FLINK-22070) Support FileSink in PyFlink DataStream API

[Dian Fu (Jira)]

Dian Fu created FLINK-22070:
---

 Summary: Support FileSink in PyFlink DataStream API
 Key: FLINK-22070
 URL: https://issues.apache.org/jira/browse/FLINK-22070
 Project: Flink
  Issue Type: Sub-task
  Components: API / Python
Reporter: Dian Fu
Assignee: Dian Fu
 Fix For: 1.13.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Release flink-shaded 13.0, release candidate #0

[Dawid Wysakowicz]

+1

- Checked diffs between 1.12 and 1.11:
https://github.com/apache/flink-shaded/compare/release-12.0...release-13.0-rc1


- reviewed the website PR

- verified release notes and that the release contain the changes

- verified signatures and checksums and license files

Best,

Dawid

On 31/03/2021 14:07, Chesnay Schepler wrote:
> Hi everyone,
> Please review and vote on the release candidate #1 for the version
> 13.0, as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org
> [2], which are signed with the key with fingerprint C2EED7B111D464BA [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "release-13.0-rc1" [5],
> * website pull request listing the new release [6].
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Release Manager
>
> [1]
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12349618
> [2] https://dist.apache.org/repos/dist/dev/flink/flink-shaded-13.0-rc1/
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4]
> https://repository.apache.org/content/repositories/orgapacheflink-1416/
> [5] https://github.com/apache/flink-shaded/tree/release-13.0-rc1
> [6] https://github.com/apache/flink-web/pull/428
>


OpenPGP_signature
Description: OpenPGP digital signature

[VOTE] Release flink-shaded 13.0, release candidate #0

[Chesnay Schepler]

Hi everyone,
Please review and vote on the release candidate #1 for the version 13.0, 
as follows:

[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release to be deployed to dist.apache.org 
[2], which are signed with the key with fingerprint C2EED7B111D464BA [3],

* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "release-13.0-rc1" [5],
* website pull request listing the new release [6].

The vote will be open for at least 72 hours. It is adopted by majority 
approval, with at least 3 PMC affirmative votes.


Thanks,
Release Manager

[1] 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12349618

[2] https://dist.apache.org/repos/dist/dev/flink/flink-shaded-13.0-rc1/
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4] https://repository.apache.org/content/repositories/orgapacheflink-1416/
[5] https://github.com/apache/flink-shaded/tree/release-13.0-rc1
[6] https://github.com/apache/flink-web/pull/428

[jira] [Created] (FLINK-22069) Check Log Pollution for 1.13 release

[Stephan Ewen (Jira)]

Stephan Ewen created FLINK-22069:


 Summary: Check Log Pollution for 1.13 release
 Key: FLINK-22069
 URL: https://issues.apache.org/jira/browse/FLINK-22069
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Reporter: Stephan Ewen
 Fix For: 1.13.0


We should check for log pollution and confusing log lines before the release.
Below are some lines I stumbled over while using Flink during testing.

-

These lines show up on any execution of a local job and make me think I forgot 
to configure something I probably should have, wondering whether this might 
cause problems later?

These have been in Flink for a few releases now, might be worth rephrasing, 
though.

{code}
2021-03-30 17:57:22,483 INFO  
org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils [] - The 
configuration option taskmanager.cpu.cores required for local execution is not 
set, setting it to the maximal possible value.
2021-03-30 17:57:22,483 INFO  
org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils [] - The 
configuration option taskmanager.memory.task.heap.size required for local 
execution is not set, setting it to the maximal possible value.
2021-03-30 17:57:22,483 INFO  
org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils [] - The 
configuration option taskmanager.memory.task.off-heap.size required for local 
execution is not set, setting it to the maximal possible value.
2021-03-30 17:57:22,483 INFO  
org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils [] - The 
configuration option taskmanager.memory.network.min required for local 
execution is not set, setting it to its default value 64 mb.
2021-03-30 17:57:22,483 INFO  
org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils [] - The 
configuration option taskmanager.memory.network.max required for local 
execution is not set, setting it to its default value 64 mb.
2021-03-30 17:57:22,483 INFO  
org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils [] - The 
configuration option taskmanager.memory.managed.size required for local 
execution is not set, setting it to its default value 128 mb.
{code}

-

These lines show up on every job start, even if there is no recovery but just a 
plain job start. They are not particularly problematic, but also not helping.

{code}
2021-03-30 17:57:27,839 INFO  
org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate [] - 
Converting recovered input channels (8 channels)
2021-03-30 17:57:27,839 INFO  
org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate [] - 
Converting recovered input channels (8 channels)
2021-03-30 17:57:27,839 INFO  
org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate [] - 
Converting recovered input channels (8 channels)
2021-03-30 17:57:27,839 INFO  
org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate [] - 
Converting recovered input channels (8 channels)
2021-03-30 17:57:27,839 INFO  
org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate [] - 
Converting recovered input channels (8 channels)
2021-03-30 17:57:27,839 INFO  
org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate [] - 
Converting recovered input channels (8 channels)
2021-03-30 17:57:27,839 INFO  
org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate [] - 
Converting recovered input channels (8 channels)
2021-03-30 17:57:27,855 INFO  
org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate [] - 
Converting recovered input channels (8 channels)
{code}



When using {{DataStream.collect()}} we always have an excpetion in the log for 
the first fetch attempt, before the JM is ready.
The loop retries and the program succeeds, but the exception in the log raises 
confusion about whether there is a swallowed but impactful error.

{code}
7199 [main] WARN  
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher [] - An 
exception occurs when fetching query results
java.util.concurrent.ExecutionException: 
org.apache.flink.runtime.dispatcher.UnavailableDispatcherOperationException: 
Unable to get JobMasterGateway for initializing job. The requested operation is 
not available while the JobManager is initializing.
at 
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) 
~[?:?]
at 
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999) ~[?:?]
at 
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.sendRequest(CollectResultFetcher.java:155)
 ~[classes/:?]
at 
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:126)
 [classes/:?]
at 

[jira] [Created] (FLINK-22068) FlinkKinesisConsumerTest.testPeriodicWatermark fails on azure

[Dawid Wysakowicz (Jira)]

Dawid Wysakowicz created FLINK-22068:


 Summary: FlinkKinesisConsumerTest.testPeriodicWatermark fails on 
azure
 Key: FLINK-22068
 URL: https://issues.apache.org/jira/browse/FLINK-22068
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kinesis
Affects Versions: 1.13.0
Reporter: Dawid Wysakowicz
 Fix For: 1.13.0


{code}
[ERROR] Tests run: 11, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.567 
s <<< FAILURE! - in 
org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumerTest
[ERROR] 
testPeriodicWatermark(org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumerTest)
  Time elapsed: 0.845 s  <<< FAILURE!
java.lang.AssertionError: 

Expected: iterable containing [, ]
 but: item 0: was 
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.junit.Assert.assertThat(Assert.java:956)
at org.junit.Assert.assertThat(Assert.java:923)
at 
org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumerTest.testPeriodicWatermark(FlinkKinesisConsumerTest.java:988)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:68)
at 
org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.runTestMethod(PowerMockJUnit44RunnerDelegateImpl.java:326)
at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:89)
at 
org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:97)
at 
org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.executeTest(PowerMockJUnit44RunnerDelegateImpl.java:310)
at 
org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.executeTestInSuper(PowerMockJUnit47RunnerDelegateImpl.java:131)
at 
org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.access$100(PowerMockJUnit47RunnerDelegateImpl.java:59)
at 
org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner$TestExecutorStatement.evaluate(PowerMockJUnit47RunnerDelegateImpl.java:147)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at 
org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
at 
org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.evaluateStatement(PowerMockJUnit47RunnerDelegateImpl.java:107)
at 
org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.executeTest(PowerMockJUnit47RunnerDelegateImpl.java:82)
at 
org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.runBeforesThenTestThenAfters(PowerMockJUnit44RunnerDelegateImpl.java:298)
at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:87)
at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:50)
at 
org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.invokeTestMethod(PowerMockJUnit44RunnerDelegateImpl.java:218)
at 
org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.runMethods(PowerMockJUnit44RunnerDelegateImpl.java:160)
at 
org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$1.run(PowerMockJUnit44RunnerDelegateImpl.java:134)
at 
org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:34)
at 
org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:44)
at 
org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.run(PowerMockJUnit44RunnerDelegateImpl.java:136)
at 
org.powermock.modules.junit4.common.internal.impl.JUnit4TestSuiteChunkerImpl.run(JUnit4TestSuiteChunkerImpl.java:117)
at 
org.powermock.modules.junit4.common.internal.impl.AbstractCommonPowerMockRunner.run(AbstractCommonPowerMockRunner.java:57)
at 
org.powermock.modules.junit4.PowerMockRunner.run(PowerMockRunner.java:59)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at 

[jira] [Created] (FLINK-22067) SavepointWindowReaderITCase.testApplyEvictorWindowStateReader

[Till Rohrmann (Jira)]

Till Rohrmann created FLINK-22067:
-

 Summary: 
SavepointWindowReaderITCase.testApplyEvictorWindowStateReader
 Key: FLINK-22067
 URL: https://issues.apache.org/jira/browse/FLINK-22067
 Project: Flink
  Issue Type: Bug
  Components: API / State Processor
Affects Versions: 1.13.0
Reporter: Till Rohrmann


The test case {{SavepointWindowReaderITCase.testApplyEvictorWindowStateReader}} 
failed on AZP with:

{code}

at 
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
at 
org.apache.flink.state.api.utils.SavepointTestBase.takeSavepoint(SavepointTestBase.java:69)
... 33 more
Caused by: java.util.concurrent.TimeoutException: Invocation of public default 
java.util.concurrent.CompletableFuture 
org.apache.flink.runtime.webmonitor.RestfulGateway.triggerSavepoint(org.apache.flink.api.common.JobID,java.lang.String,boolean,org.apache.flink.api.common.time.Time)
 timed out.
at com.sun.proxy.$Proxy32.triggerSavepoint(Unknown Source)
at 
org.apache.flink.runtime.minicluster.MiniCluster.lambda$triggerSavepoint$8(MiniCluster.java:716)
at 
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
at 
java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:628)
at 
java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:1996)
at 
org.apache.flink.runtime.minicluster.MiniCluster.runDispatcherCommand(MiniCluster.java:751)
at 
org.apache.flink.runtime.minicluster.MiniCluster.triggerSavepoint(MiniCluster.java:714)
at 
org.apache.flink.client.program.MiniClusterClient.triggerSavepoint(MiniClusterClient.java:101)
at 
org.apache.flink.state.api.utils.SavepointTestBase.triggerSavepoint(SavepointTestBase.java:93)
at 
org.apache.flink.state.api.utils.SavepointTestBase.lambda$takeSavepoint$0(SavepointTestBase.java:68)
at 
java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:966)
at 
java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:940)
at 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at 
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1646)
at 
java.util.concurrent.CompletableFuture$AsyncRun.exec(CompletableFuture.java:1632)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at 
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at 
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Caused by: akka.pattern.AskTimeoutException: Ask timed out on 
[Actor[akka://flink/user/rpc/dispatcher_2#-390276455]] after [1 ms]. 
Message of type [org.apache.flink.runtime.rpc.messages.LocalFencedMessage]. A 
typical reason for `AskTimeoutException` is that the recipient actor didn't 
send a reply.
at akka.pattern.PromiseActorRef$$anonfun$2.apply(AskSupport.scala:635)
at akka.pattern.PromiseActorRef$$anonfun$2.apply(AskSupport.scala:635)
at 
akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:648)
at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205)
at 
scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
at 
scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
at 
scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
at 
akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
at 
akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:279)
at 
akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283)
at 
akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
at java.lang.Thread.run(Thread.java:748)
{code}

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=15809=logs=b2f046ab-ae17-5406-acdc-240be7e870e4=93e5ae06-d194-513d-ba8d-150ef6da1d7c=9197



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Feature freeze date for 1.13

[Guowei Ma]

Hi, community:

Friendly reminder that today (3.31) is the last day of feature development.
Under normal circumstances, you will not be able to submit new features
from tomorrow (4.1). Tomorrow we will create 1.13.0-rc0 for testing,
welcome to help test together.
After the test is relatively stable, we will cut the release-1.13 branch.

Best,
Dawid & Guowei


On Mon, Mar 29, 2021 at 5:17 PM Till Rohrmann  wrote:

> +1 for the 31st of March for the feature freeze.
>
> Cheers,
> Till
>
> On Mon, Mar 29, 2021 at 10:12 AM Robert Metzger 
> wrote:
>
> > +1 for March 31st for the feature freeze.
> >
> >
> >
> > On Fri, Mar 26, 2021 at 3:39 PM Dawid Wysakowicz  >
> > wrote:
> >
> > > Thank you Thomas! I'll definitely check the issue you linked.
> > >
> > > Best,
> > >
> > > Dawid
> > >
> > > On 23/03/2021 20:35, Thomas Weise wrote:
> > > > Hi Dawid,
> > > >
> > > > Thanks for the heads up.
> > > >
> > > > Regarding the "Rebase and merge" button. I find that merge option
> > useful,
> > > > especially for small simple changes and for backports. The following
> > > should
> > > > help to safeguard from the issue encountered previously:
> > > > https://github.com/jazzband/pip-tools/issues/1085
> > > >
> > > > Thanks,
> > > > Thomas
> > > >
> > > >
> > > > On Tue, Mar 23, 2021 at 4:58 AM Dawid Wysakowicz <
> > dwysakow...@apache.org
> > > >
> > > > wrote:
> > > >
> > > >> Hi devs, users!
> > > >>
> > > >> 1. *Feature freeze date*
> > > >>
> > > >> We are approaching the end of March which we agreed would be the
> time
> > > for
> > > >> a Feature Freeze. From the knowledge I've gather so far it still
> seems
> > > to
> > > >> be a viable plan. I think it is a good time to agree on a particular
> > > date,
> > > >> when it should happen. We suggest *(end of day CEST) March 31st*
> > > >> (Wednesday next week) as the feature freeze time.
> > > >>
> > > >> Similarly as last time, we want to create RC0 on the day after the
> > > feature
> > > >> freeze, to make sure the RC creation process is running smoothly,
> and
> > to
> > > >> have a common testing reference point.
> > > >>
> > > >> Having said that let us remind after Robert & Dian from the previous
> > > >> release what it a Feature Freeze means:
> > > >>
> > > >> *B) What does feature freeze mean?*After the feature freeze, no new
> > > >> features are allowed to be merged to master. Only bug fixes and
> > > >> documentation improvements.
> > > >> The release managers will revert new feature commits after the
> feature
> > > >> freeze.
> > > >> Rational: The goal of the feature freeze phase is to improve the
> > system
> > > >> stability by addressing known bugs. New features tend to introduce
> new
> > > >> instabilities, which would prolong the release process.
> > > >> If you need to merge a new feature after the freeze, please open a
> > > >> discussion on the dev@ list. If there are no objections by a PMC
> > member
> > > >> within 48 (workday)hours, the feature can be merged.
> > > >>
> > > >> 2. *Merge PRs from the command line*
> > > >>
> > > >> In the past releases it was quite frequent around the Feature Freeze
> > > date
> > > >> that we ended up with a broken main branch that either did not
> compile
> > > or
> > > >> there were failing tests. It was often due to concurrent merges to
> the
> > > main
> > > >> branch via the "Rebase and merge" button. To overcome the problem we
> > > would
> > > >> like to suggest only ever merging PRs from a command line. Thank you
> > > >> Stephan for the idea! The suggested workflow would look as follows:
> > > >>
> > > >>1. Pull the change and rebase on the current main branch
> > > >>2. Build the project (e.g. from IDE, which should be faster than
> > > >>building entire project from cmd) -> this should ensure the
> project
> > > compiles
> > > >>3. Run the tests in the module that the change affects -> this
> > should
> > > >>greatly minimize the chances of failling tests
> > > >>4. Push the change to the main branch
> > > >>
> > > >> Let us know what you think!
> > > >>
> > > >> Best,
> > > >>
> > > >> Guowei & Dawid
> > > >>
> > > >>
> > > >>
> > >
> > >
> >
>

[jira] [Created] (FLINK-22066) Improve Error Message on unknown Ingress Type

[Konstantin Knauf (Jira)]

Konstantin Knauf created FLINK-22066:


 Summary: Improve Error Message on unknown Ingress Type
 Key: FLINK-22066
 URL: https://issues.apache.org/jira/browse/FLINK-22066
 Project: Flink
  Issue Type: Improvement
  Components: Stateful Functions
Reporter: Konstantin Knauf


When using an unknown ingress type the following exception is thrown on 
startup: 
```
2021-03-31 09:45:48,417 ERROR 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - Could not 
start cluster entrypoint StatefulFunctionsClusterEntryPoint.
org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to 
initialize the cluster entrypoint StatefulFunctionsClusterEntryPoint.
at 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:200)
 ~[flink-dist_2.12-1.12.1.jar:1.12.1]
at 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:569)
 [flink-dist_2.12-1.12.1.jar:1.12.1]
at 
org.apache.flink.statefun.flink.launcher.StatefulFunctionsClusterEntryPoint.main(StatefulFunctionsClusterEntryPoint.java:99)
 [statefun-flink-distribution.jar:3.0.0]
Caused by: org.apache.flink.util.FlinkException: Could not create the 
DispatcherResourceManagerComponent.
at 
org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory.create(DefaultDispatcherResourceManagerComponentFactory.java:275)
 ~[flink-dist_2.12-1.12.1.jar:1.12.1]
at 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:234)
 ~[flink-dist_2.12-1.12.1.jar:1.12.1]
at 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:178)
 ~[flink-dist_2.12-1.12.1.jar:1.12.1]
at 
org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
 ~[flink-dist_2.12-1.12.1.jar:1.12.1]
at 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:175)
 ~[flink-dist_2.12-1.12.1.jar:1.12.1]
... 2 more
Caused by: org.apache.flink.util.FlinkRuntimeException: Could not retrieve the 
JobGraph.
at 
org.apache.flink.runtime.dispatcher.runner.JobDispatcherLeaderProcessFactoryFactory.createFactory(JobDispatcherLeaderProcessFactoryFactory.java:58)
 ~[flink-dist_2.12-1.12.1.jar:1.12.1]
at 
org.apache.flink.runtime.dispatcher.runner.DefaultDispatcherRunnerFactory.createDispatcherRunner(DefaultDispatcherRunnerFactory.java:54)
 ~[flink-dist_2.12-1.12.1.jar:1.12.1]
at 
org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory.create(DefaultDispatcherResourceManagerComponentFactory.java:212)
 ~[flink-dist_2.12-1.12.1.jar:1.12.1]
at 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:234)
 ~[flink-dist_2.12-1.12.1.jar:1.12.1]
at 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:178)
 ~[flink-dist_2.12-1.12.1.jar:1.12.1]
at 
org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
 ~[flink-dist_2.12-1.12.1.jar:1.12.1]
at 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:175)
 ~[flink-dist_2.12-1.12.1.jar:1.12.1]
... 2 more
Caused by: org.apache.flink.util.FlinkException: Could not create the JobGraph 
from the provided user code jar.
at 
org.apache.flink.statefun.flink.launcher.StatefulFunctionsJobGraphRetriever.retrieveJobGraph(StatefulFunctionsJobGraphRetriever.java:107)
 ~[statefun-flink-distribution.jar:3.0.0]
at 
org.apache.flink.runtime.dispatcher.runner.JobDispatcherLeaderProcessFactoryFactory.createFactory(JobDispatcherLeaderProcessFactoryFactory.java:55)
 ~[flink-dist_2.12-1.12.1.jar:1.12.1]
at 
org.apache.flink.runtime.dispatcher.runner.DefaultDispatcherRunnerFactory.createDispatcherRunner(DefaultDispatcherRunnerFactory.java:54)
 ~[flink-dist_2.12-1.12.1.jar:1.12.1]
at 
org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory.create(DefaultDispatcherResourceManagerComponentFactory.java:212)
 ~[flink-dist_2.12-1.12.1.jar:1.12.1]
at 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:234)
 ~[flink-dist_2.12-1.12.1.jar:1.12.1]
at 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:178)
 ~[flink-dist_2.12-1.12.1.jar:1.12.1]
at 
org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
 ~[flink-dist_2.12-1.12.1.jar:1.12.1]
at 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:175)
 

[jira] [Created] (FLINK-22065) Beautify the parse error exception when input invalid command in the sql client

[Shengkai Fang (Jira)]

Shengkai Fang created FLINK-22065:
-

 Summary: Beautify the parse error exception when input invalid 
command in the sql client
 Key: FLINK-22065
 URL: https://issues.apache.org/jira/browse/FLINK-22065
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Client
Affects Versions: 1.13.0
Reporter: Shengkai Fang
 Fix For: 1.13.0


!https://static.dingtalk.com/media/lALPD26eOprT2ztwzQWg_1440_112.png_720x720g.jpg?renderWidth=1440=112=1=0=im!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22064) Don't submit statement set when no insert is added in the sql client

[Shengkai Fang (Jira)]

Shengkai Fang created FLINK-22064:
-

 Summary: Don't submit statement set when no insert  is added in 
the sql client
 Key: FLINK-22064
 URL: https://issues.apache.org/jira/browse/FLINK-22064
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Client
Affects Versions: 1.13.0
Reporter: Shengkai Fang
 Fix For: 1.13.0


!https://static.dingtalk.com/media/lALPD4Bhs9KtV3jM4s0F2A_1496_226.png_720x720g.jpg?renderWidth=1496=226=1=0=im!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Flink job cannot find recover path after using entropy injection for s3 file systems

[Till Rohrmann]

Thanks for creating this PR. I think it would be good to re-open the issue
and post your analysis there together with the proposal for the fix.

Cheers,
Till

On Wed, Mar 31, 2021 at 3:41 AM chenqin  wrote:

> link fix pr here https://github.com/apache/flink/pull/15442
> we might need someone help review and merge meanwhile.
>
>
>
> --
> Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
>

Re: flink 1.11 class loading question

[Till Rohrmann]

Hi Chen,

with version 1.10 Flink introduced that RocksDB uses Flink's managed memory
[1]. This shall prevent RocksDB from exceeding the memory limits of a
process/container. Unfortunately, this is not yet perfect due to a problem
in RocksDB [2]. Due to this fact, RocksDB can still exceed the managed
memory budget. What you could do is to configure a higher off-heap size for
your tasks via taskmanager.memory.task.off-heap.size to compensate for this.

I also pull in Yu Li who can tell you more about the current limitations of
the memory limitation for RocksDB.

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.12/ops/state/state_backends.html#memory-management
[2] https://issues.apache.org/jira/browse/FLINK-15532

Cheers,
Till

On Tue, Mar 30, 2021 at 7:36 PM chenqin  wrote:

> Hi Till,
>
> We did some investigation and found this memory usage point to
> rocksdbstatebackend running on managed memory. So far we have seen this bug
> in rocksdbstatebackend on managed memory. we followed suggestion [1] and
> disabled managed memory management so far not seeing issue.
>
> I felt this might be a major bug since we run flink 1.11.2 with managed
> RocksDBstatebackend in mulitple large production jobs and consistency repo
> yarn kill after job runs a period of time.
>
> [1]
>
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Debugging-quot-Container-is-running-beyond-physical-memory-limits-quot-on-YARN-for-a-long-running-stb-td38227.html
>
>
>
>
>
> --
> Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
>

[jira] [Created] (FLINK-22063) Lookup Join outputs wrong results for some scenario

[Shuo Cheng (Jira)]

Shuo Cheng created FLINK-22063:
--

 Summary: Lookup Join outputs wrong results for some scenario
 Key: FLINK-22063
 URL: https://issues.apache.org/jira/browse/FLINK-22063
 Project: Flink
  Issue Type: New Feature
  Components: Table SQL / API
Affects Versions: 1.12.2
Reporter: Shuo Cheng


Reproduce the bug as following:

In LookupJoinITCase, given the sql 

{code:sql}
SELECT 
T.id, T.len, D.id, T.content, D.name 
FROM src AS T JOIN user_table for system_time as of T.proctime AS D 
ON T.id = D.id and cast(T.len as bigint) = D.id
{code}

the following execution plan is generated:

{code:java}
LegacySink(name=[DataStreamTableSink], fields=[id, len, id0, content, name])
+- Calc(select=[id, len, id0, content, name])
   +- LookupJoin(table=[**], joinType=[InnerJoin], async=[false], 
lookup=[id=len0], select=[id, len, content, len0, id, name])
  +- Calc(select=[id, len, content, CAST(len) AS len0])
 +- TableSourceScan(table=[[**]], fields=[id, len, content])
{code}

As we can see, the condition `T.id = D.id` is lost, so a wrong result may be 
produced.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22062) Add "Data Sources" (FLIP-27 sources overview page) page from 1.12x to master

[Robert Metzger (Jira)]

Robert Metzger created FLINK-22062:
--

 Summary: Add "Data Sources" (FLIP-27 sources overview page) page 
from 1.12x to master
 Key: FLINK-22062
 URL: https://issues.apache.org/jira/browse/FLINK-22062
 Project: Flink
  Issue Type: Bug
  Components: API / DataStream, Documentation
Affects Versions: 1.13.0
Reporter: Robert Metzger
 Fix For: 1.13.0


This page 
https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/stream/sources.html
 is not available in the latest docs on master.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22061) The DEFAULT_NON_SPLITTABLE_FILE_ENUMERATOR defined in FileSource should points to NonSplittingRecursiveEnumerator

[Dian Fu (Jira)]

Dian Fu created FLINK-22061:
---

 Summary: The DEFAULT_NON_SPLITTABLE_FILE_ENUMERATOR defined in 
FileSource should points to NonSplittingRecursiveEnumerator
 Key: FLINK-22061
 URL: https://issues.apache.org/jira/browse/FLINK-22061
 Project: Flink
  Issue Type: Bug
  Components: Connectors / FileSystem
Affects Versions: 1.12.0
Reporter: Dian Fu
Assignee: Dian Fu
 Fix For: 1.13.0, 1.12.3






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22060) Move null handling from ErrorInfo into Task.notifyFinalState

[Matthias (Jira)]

Matthias created FLINK-22060:


 Summary: Move null handling from ErrorInfo into 
Task.notifyFinalState
 Key: FLINK-22060
 URL: https://issues.apache.org/jira/browse/FLINK-22060
 Project: Flink
  Issue Type: Bug
Reporter: Matthias


Ideally, we would want to have this {{null}} check since a failure should 
always have a cause. We didn't introduce a null check so far because of 
FLINK-21376. There is [ErrorInfo. 
createErrorInfoWithNullableCause|https://github.com/apache/flink/blob/c77a686c195d1742c276f4a9e75899c8b85377bb/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ErrorInfo.java#L50]
 for handling this. But it feels to be handled in the wrong place. Instead, we 
should substitute the {{null}} value by the {{FlinkException}} in 
[Task.notifyFinalState|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java#L1003].
 This way, we make sure that {{null}} never ends up in the 
{{ExecutionGraph}}-related code which enables us to make the invariants 
stricter on our end.

This would serve as a preparation task



--
This message was sent by Atlassian Jira
(v8.3.4#803005)