[jira] [Created] (FLINK-20824) BlockingShuffleITCase. testSortMergeBlockingShuffle test failed with "Inconsistent availability: expected true"

2020-12-30 Thread Huang Xingbo (Jira) 
Huang Xingbo created FLINK-20824:


 Summary: BlockingShuffleITCase. testSortMergeBlockingShuffle test 
failed with "Inconsistent availability: expected true"
 Key: FLINK-20824
 URL: https://issues.apache.org/jira/browse/FLINK-20824
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Network
Affects Versions: 1.13.0
Reporter: Huang Xingbo


[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=11516&view=logs&j=219e462f-e75e-506c-3671-5017d866ccf6&t=4c5dc768-5c82-5ab0-660d-086cb90b76a0]
{code:java}
2020-12-30T22:45:42.5933715Z [ERROR] 
testSortMergeBlockingShuffle(org.apache.flink.test.runtime.BlockingShuffleITCase)
  Time elapsed: 3.153 s  <<< FAILURE!
2020-12-30T22:45:42.5934985Z java.lang.AssertionError: 
org.apache.flink.runtime.JobException: Recovery is suppressed by 
NoRestartBackoffTimeStrategy
2020-12-30T22:45:42.5935943Zat 
org.apache.flink.test.runtime.JobGraphRunningUtil.execute(JobGraphRunningUtil.java:60)
2020-12-30T22:45:42.5936979Zat 
org.apache.flink.test.runtime.BlockingShuffleITCase.testSortMergeBlockingShuffle(BlockingShuffleITCase.java:70)
2020-12-30T22:45:42.5937885Zat 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2020-12-30T22:45:42.5938572Zat 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2020-12-30T22:45:42.5939448Zat 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2020-12-30T22:45:42.5940142Zat 
java.lang.reflect.Method.invoke(Method.java:498)
2020-12-30T22:45:42.5940944Zat 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2020-12-30T22:45:42.5942210Zat 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2020-12-30T22:45:42.5943167Zat 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2020-12-30T22:45:42.5943971Zat 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2020-12-30T22:45:42.5944703Zat 
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
2020-12-30T22:45:42.5945422Zat 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
2020-12-30T22:45:42.5946138Zat 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
2020-12-30T22:45:42.5946615Zat 
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2020-12-30T22:45:42.5947160Zat 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2020-12-30T22:45:42.5947764Zat 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2020-12-30T22:45:42.5948475Zat 
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2020-12-30T22:45:42.5948925Zat 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2020-12-30T22:45:42.5949328Zat 
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2020-12-30T22:45:42.5949787Zat 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
2020-12-30T22:45:42.5950558Zat 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
2020-12-30T22:45:42.5951084Zat 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
2020-12-30T22:45:42.5951608Zat 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
2020-12-30T22:45:42.5952153Zat 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
2020-12-30T22:45:42.5952809Zat 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
2020-12-30T22:45:42.5953323Zat 
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
2020-12-30T22:45:42.5953806Zat 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
2020-12-30T22:45:42.5954306Z Caused by: org.apache.flink.runtime.JobException: 
Recovery is suppressed by NoRestartBackoffTimeStrategy
2020-12-30T22:45:42.5954931Zat 
org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:118)
2020-12-30T22:45:42.5955641Zat 
org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:80)
2020-12-30T22:45:42.5956279Zat 
org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:230)
2020-12-30T22:45:42.5956925Zat 
org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:221)
2020-12-30T22:45:42.5957624Zat 
org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:212)
2020-12-30T22:45:42.5958234Zat 
org.apache.flink.runtime.scheduler.SchedulerBase.upd

[jira] [Created] (FLINK-20823) Update documentation to mention Table/SQL API doesn't provide cross-major-version state compatibility

2020-12-30 Thread Jark Wu (Jira) 
Jark Wu created FLINK-20823:
---

 Summary: Update documentation to mention Table/SQL API doesn't 
provide cross-major-version state compatibility
 Key: FLINK-20823
 URL: https://issues.apache.org/jira/browse/FLINK-20823
 Project: Flink
  Issue Type: Improvement
  Components: Documentation, Table SQL / API
Reporter: Jark Wu


As discussed in the mailing list: 
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Did-Flink-1-11-break-backwards-compatibility-for-the-table-environment-tp47472p47492.html

Flink Table/SQL API doesn't provide cross-major-version state compatibility, 
however, this is not documented in anywhere. We should update the 
documentation. Besides, we should also mention that we provide state 
compatibility across minor versions. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Did Flink 1.11 break backwards compatibility for the table environment?

2020-12-30 Thread Jark Wu 
Hi Guenther,

I think it's safe to use legacy mode in your case,
because the RowKind is never used in the old planner.

Hi Till,

It seems that the cross-major-version state incompatibility is
not documented.
I created FLINK-20823 to update the documentation.

Best,
Jark



On Thu, 31 Dec 2020 at 08:14, Guenther Starnberger 
wrote:

> On Wed, Dec 30, 2020 at 11:21 AM Jark Wu  wrote:
>
> Jark,
>
> > If you are using the old planner in 1.9, and using the old planner in
> 1.11,
> > then a state migration is
> > needed because of the new added RowKind field. This is documented in the
> > 1.11 release note [1].
>
> Yes - that's exactly the setup that we're using. Both versions
> currently use the old planner and state migration fails due to the
> "The new key serializer must be compatible" error.
>
> For testing I tried to patch the Flink source and to force usage of
> (only) the legacy mode in the RowSerializer (so that the state doesn't
> need to be migrated) and this seems to work. However, I'm not sure if
> this has any unintended side-effects so it seems that rebuilding the
> state is a much safer and more maintainable approach.
>
> - Guenther
>

[jira] [Created] (FLINK-20822) Don't check whether a function is generic in hive catalog

2020-12-30 Thread Rui Li (Jira) 
Rui Li created FLINK-20822:
--

 Summary: Don't check whether a function is generic in hive catalog
 Key: FLINK-20822
 URL: https://issues.apache.org/jira/browse/FLINK-20822
 Project: Flink
  Issue Type: Task
  Components: Connectors / Hive
Reporter: Rui Li






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20821) `select row(map[1,2],'ab')` parses failed

2020-12-30 Thread godfrey he (Jira) 
godfrey he created FLINK-20821:
--

 Summary: `select row(map[1,2],'ab')` parses failed
 Key: FLINK-20821
 URL: https://issues.apache.org/jira/browse/FLINK-20821
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Reporter: godfrey he


when executing {{select row(map[1,2],'ab')}}, we encounter the following error:

{code:text}
org.apache.flink.table.api.SqlParserException: SQL parse failed. Encountered 
"[" at line 1, column 15.
Was expecting one of:
")" ...
"," ...


at 
org.apache.flink.table.planner.calcite.CalciteParser.parse(CalciteParser.java:56)
at 
org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:74)
at 
org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlQuery(TableEnvironmentImpl.java:640)
{code}

while, the similar statement {{select row('ab',map[1,2])}} can be parsed 
successfully.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Did Flink 1.11 break backwards compatibility for the table environment?

2020-12-30 Thread Guenther Starnberger 
On Wed, Dec 30, 2020 at 11:21 AM Jark Wu  wrote:

Jark,

> If you are using the old planner in 1.9, and using the old planner in 1.11,
> then a state migration is
> needed because of the new added RowKind field. This is documented in the
> 1.11 release note [1].

Yes - that's exactly the setup that we're using. Both versions
currently use the old planner and state migration fails due to the
"The new key serializer must be compatible" error.

For testing I tried to patch the Flink source and to force usage of
(only) the legacy mode in the RowSerializer (so that the state doesn't
need to be migrated) and this seems to work. However, I'm not sure if
this has any unintended side-effects so it seems that rebuilding the
state is a much safer and more maintainable approach.

- Guenther

Re: Did Flink 1.11 break backwards compatibility for the table environment?

2020-12-30 Thread Guenther Starnberger 
On Wed, Dec 30, 2020 at 10:47 AM Till Rohrmann  wrote:

Till,

> sorry for overlooking your colleague's email.

Not a problem at all! Thanks for the response to my email.

> A bit confusing is that FLINK-16998 explicitly
> states that this change is not breaking backwards compatibility.

The comments regarding backwards compatibility are the reason why I
assumed that this behavior might be unintended. If it's not a bug
we're probably going to try rebuilding the state of the affected jobs.
Just wanted to reach out in case this is not the intended behavior.

> What you could try to use as a workaround is Flink's state processor API
> [2] which allows you to rewrite savepoints.

I briefly looked into the state processor API, but I wasn't sure if
the version in 1.11 would work for us as it seems that support for
reading window state was only added in 1.12 (upgrading to 1.12 would
require larger changes as we're still using some deprecated APIs that
have been removed in 1.12).

- Guenther

[jira] [Created] (FLINK-20820) Rename o.a.f.table.runtime.generated package in blink runtime

2020-12-30 Thread Chesnay Schepler (Jira) 
Chesnay Schepler created FLINK-20820:


 Summary: Rename o.a.f.table.runtime.generated package in blink 
runtime
 Key: FLINK-20820
 URL: https://issues.apache.org/jira/browse/FLINK-20820
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Runtime
Reporter: Chesnay Schepler


The {{org/apache/flink/table/runtime/generated}} in flink-table-runtime-blink 
does not contain generate code and hence should be renamed to avoid confusion.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Support local aggregate push down for Blink batch planner

2020-12-30 Thread Sebastian Liu 
Hi Jark, Thx a lot for your quick reply and valuable suggestions.
For (1): Agree: Since we are in the period of upgrading the new table
source api,
we really should consider the new interface for the new optimize rule. If
the new rule
doesn't use the new api, we'll have to upgrade it sooner or later. I have
change to use
the ability interface for the SupportsAggregatePushDown definition in above
proposal.

For (2): Agree: Change to use CallExpression is a better choice, and have
resolved this
comment in the proposal.

For (3): I suggest we first support the JDBC connector, as we don't have
Druid connector
and ES connector just has sink api at present.

But perhaps the biggest question may be whether we should use idea 1 or
idea 2 in proposal.

What do you think?  After we reach the agreement on the proposal, our team
can drive to
complete this feature.

Jark Wu  于2020年12月29日周二 下午2:58写道：

> Hi Sebastian,
>
> Thanks for the proposal. I think this is a great improvement for Flink SQL.
> I went through the design doc and have following thoughts:
>
> 1) Flink has deprecated the legacy TableSource in 1.11 and proposed a new
>  set of DynamicTableSource interfaces. Could you update your proposal to
> use the new interfaces?
>  Follow the existing ability interfaces, e.g.
> SupportsFilterPushDown, SupportsProjectionPushDown.
>
> 2) Personally, I think CallExpression would be a better representation than
> separate `FunctionDefinition` and args. Because, it would be easier to know
> what's the index and type of the arguments.
>
> 3) It would be better to list which connectors will be supported in the
> plan?
>
> Best,
> Jark
>
>
> On Tue, 29 Dec 2020 at 00:49, Sebastian Liu  wrote:
>
> > Hi all,
> >
> > I'd like to discuss a new feature for the Blink Planner.
> > Aggregate operator of Flink SQL is currently fully done at Flink layer.
> > With the developing of storage, many downstream storage of Flink SQL has
> > the ability to deal with Aggregation operator.
> > Pushing down Aggregate to data source layer will improve performance from
> > the perspective of the network IO and computation overhead.
> >
> > I have drafted a design doc for this new feature.
> >
> >
> https://docs.google.com/document/d/1kGwC_h4qBNxF2eMEz6T6arByOB8yilrPLqDN0QBQXW4/edit?usp=sharing
> >
> > Any comment or discussion is welcome.
> >
> > --
> >
> > *With kind regards
> > 
> > Sebastian Liu 刘洋
> > Institute of Computing Technology, Chinese Academy of Science
> > Mobile\WeChat: +86—15201613655
> > E-mail: liuyang0...@gmail.com 
> > QQ: 3239559*
> >
>


-- 

*With kind regards

Sebastian Liu 刘洋
Institute of Computing Technology, Chinese Academy of Science
Mobile\WeChat: +86—15201613655
E-mail: liuyang0...@gmail.com 
QQ: 3239559*

Re: Did Flink 1.11 break backwards compatibility for the table environment?

2020-12-30 Thread Till Rohrmann 
Are these limitations documented somewhere @Jark? I couldn't find it on the
quick. If not, then we should update the documentation accordingly. In
particular the problem with using the RowData as a key makes FLINK-16998
not easy to work around.

Cheers,
Till

On Wed, Dec 30, 2020 at 11:20 AM Jark Wu  wrote:

> Hi Guenther,
>
> If you are using the old planner in 1.9, and using the old planner in
> 1.11, then a state migration is
> needed because of the new added RowKind field. This is documented in the
> 1.11 release note [1].
>
> If you are using the old planner in 1.9, and using the blink planner in
> 1.11, the state is not compatible.
> Because blink planner uses a different serializer for the keys and fields,
> i.e. RowData vs Row.
>
> Actually, Flink Table/SQL API doesn't provide state compatibility across
> major versions (i.e. 1.9, 1.10, 1.11).
> This is because it's quite difficult to keep state compatible for SQL
> queries as the physical plan may change
> when we introduce even a minor optimization, and we may also change the
> state structure to have better performance for operators.
>
> Best,
> Jark
>
> [1]:
> https://ci.apache.org/projects/flink/flink-docs-release-1.11/release-notes/flink-1.11.html#added-a-changeflag-to-row-type-flink-16998
>
> On Wed, 30 Dec 2020 at 17:47, Till Rohrmann  wrote:
>
>> Hi Guenther,
>>
>> sorry for overlooking your colleague's email.
>>
>> I think the answer to your problem is twofold. The underlying problem is
>> that your query seems to use `RowData` as a key for some keyed operation.
>> Since changing the key format might entail that keys need to be differently
>> partitioned, Flink does not support changing the key format. That is why
>> Flink fails also if the key format is compatible after migration. There is
>> a small warning about this on the state evolution page [1].
>>
>> The other part of the answer is that Flink does not support strict
>> backwards compatibility for SQL queries if I am not mistaken (please chime
>> in if this is no longer correct @Timo Walther  and
>> @j...@apache.org ). The problem is that queries might
>> result in different execution plans after a version upgrade which then can
>> not be mapped to the old state. Admittedly, in this case, it should have
>> been possible but changing the `RowData` type which is used as a key breaks
>> backwards compatibility. A bit confusing is that FLINK-16998 explicitly
>> states that this change is not breaking backwards compatibility.
>>
>> What you could try to use as a workaround is Flink's state processor API
>> [2] which allows you to rewrite savepoints.
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/schema_evolution.html
>> [2]
>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
>>
>> Cheers,
>> Till
>>
>> On Wed, Dec 30, 2020 at 3:30 AM Guenther Starnberger 
>> wrote:
>>
>>> Hello,
>>>
>>> We're currently running into an issue upgrading the state of an
>>> application to Flink 1.11 and I think this could be caused by a
>>> potential backwards incompatibility that was introduced with Flink
>>> 1.11. A colleague of mine recently posted about it on the users list
>>> (without a response), but I'd like to bring this up here on the dev
>>> list in order to figure out if that incompatibility is intended
>>> behavior and/or a known issue.
>>>
>>> We're seeing that issue when trying to load the RocksDB state from
>>> Flink 1.9 into Flink 1.11 with an application that uses the Flink
>>> table environment. Immediately after startup
>>> RocksDBFullRestoreOperation#restoreKVStateMetaData raises an "The new
>>> key serializer must be compatible" exception.
>>>
>>> The reason for that seems to be that FLINK-16998 changed the row
>>> serialization format by adding a new Row#getKind field. There's a
>>> legacy mode of the row serializer but that's only used for reading the
>>> existing snapshot. As a consequence
>>> RowSerializerSnapshot#resolveOuterSchemaCompatibility will always
>>> return COMPATIBLE_AFTER_MIGRATION when reading a Flink 1.9 savepoint
>>> with Flink 1.11.
>>>
>>> The problem is that AbstractRocksDBRestoreOperation#readMetaData
>>> treats "compatible after migration" the same as "incompatible" and
>>> throws a "The new key serializer must be compatible." exception if it
>>> encounters that result.
>>>
>>> Is it expected that the introduction of Row#getKind breaks existing
>>> older state or is that a bug? So far I only reproduced this issue in a
>>> somewhat more complex codebase, but in case this is an unknown issue
>>> or not the intended behavior I can try to provide a small testcase (to
>>> rule out that anything in our own code triggers that issue).
>>>
>>> Example of a query that triggers that issue:
>>> https://gist.github.com/gstarnberger/f19a30df179c72a622490cbb041adb21
>>>
>>> Full stacktrace:
>>> https://gist.github.com/gstarnberger/336d94e723b3ec599d09e17dd7d0ee00
>>>
>>> - Guenther
>>>
>>

[jira] [Created] (FLINK-20819) flink : Connectors : JDBC test failing on Red Hat 8.x PowerPC Linux

2020-12-30 Thread Priya (Jira) 
Priya created FLINK-20819:
-

 Summary: flink : Connectors : JDBC test failing on Red Hat 8.x 
PowerPC Linux
 Key: FLINK-20819
 URL: https://issues.apache.org/jira/browse/FLINK-20819
 Project: Flink
  Issue Type: Bug
  Components: Tests
Affects Versions: 1.11.3
 Environment: NAME="Red Hat Enterprise Linux"
VERSION="8.3 (Ootpa)"

Java 

**openjdk version "1.8.0_275"
OpenJDK Runtime Environment (build 1.8.0_275-b01)
OpenJDK 64-Bit Server VM (build 25.275-b01, mixed mode)
Reporter: Priya


git clone [https://github.com/apache/flink.git]
cd flink
mvn clean package

 

[ERROR] org.apache.flink.connector.jdbc.catalog.factory.JdbcCatalogFactoryTest  
Time elapsed: 1.028 s  <<< ERROR!
java.lang.IllegalStateException: No Postgres binary found for Linux / ppc64le



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Did Flink 1.11 break backwards compatibility for the table environment?

2020-12-30 Thread Jark Wu 
Hi Guenther,

If you are using the old planner in 1.9, and using the old planner in 1.11,
then a state migration is
needed because of the new added RowKind field. This is documented in the
1.11 release note [1].

If you are using the old planner in 1.9, and using the blink planner in
1.11, the state is not compatible.
Because blink planner uses a different serializer for the keys and fields,
i.e. RowData vs Row.

Actually, Flink Table/SQL API doesn't provide state compatibility across
major versions (i.e. 1.9, 1.10, 1.11).
This is because it's quite difficult to keep state compatible for SQL
queries as the physical plan may change
when we introduce even a minor optimization, and we may also change the
state structure to have better performance for operators.

Best,
Jark

[1]:
https://ci.apache.org/projects/flink/flink-docs-release-1.11/release-notes/flink-1.11.html#added-a-changeflag-to-row-type-flink-16998

On Wed, 30 Dec 2020 at 17:47, Till Rohrmann  wrote:

> Hi Guenther,
>
> sorry for overlooking your colleague's email.
>
> I think the answer to your problem is twofold. The underlying problem is
> that your query seems to use `RowData` as a key for some keyed operation.
> Since changing the key format might entail that keys need to be differently
> partitioned, Flink does not support changing the key format. That is why
> Flink fails also if the key format is compatible after migration. There is
> a small warning about this on the state evolution page [1].
>
> The other part of the answer is that Flink does not support strict
> backwards compatibility for SQL queries if I am not mistaken (please chime
> in if this is no longer correct @Timo Walther  and
> @j...@apache.org ). The problem is that queries might
> result in different execution plans after a version upgrade which then can
> not be mapped to the old state. Admittedly, in this case, it should have
> been possible but changing the `RowData` type which is used as a key breaks
> backwards compatibility. A bit confusing is that FLINK-16998 explicitly
> states that this change is not breaking backwards compatibility.
>
> What you could try to use as a workaround is Flink's state processor API
> [2] which allows you to rewrite savepoints.
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/schema_evolution.html
> [2]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
>
> Cheers,
> Till
>
> On Wed, Dec 30, 2020 at 3:30 AM Guenther Starnberger 
> wrote:
>
>> Hello,
>>
>> We're currently running into an issue upgrading the state of an
>> application to Flink 1.11 and I think this could be caused by a
>> potential backwards incompatibility that was introduced with Flink
>> 1.11. A colleague of mine recently posted about it on the users list
>> (without a response), but I'd like to bring this up here on the dev
>> list in order to figure out if that incompatibility is intended
>> behavior and/or a known issue.
>>
>> We're seeing that issue when trying to load the RocksDB state from
>> Flink 1.9 into Flink 1.11 with an application that uses the Flink
>> table environment. Immediately after startup
>> RocksDBFullRestoreOperation#restoreKVStateMetaData raises an "The new
>> key serializer must be compatible" exception.
>>
>> The reason for that seems to be that FLINK-16998 changed the row
>> serialization format by adding a new Row#getKind field. There's a
>> legacy mode of the row serializer but that's only used for reading the
>> existing snapshot. As a consequence
>> RowSerializerSnapshot#resolveOuterSchemaCompatibility will always
>> return COMPATIBLE_AFTER_MIGRATION when reading a Flink 1.9 savepoint
>> with Flink 1.11.
>>
>> The problem is that AbstractRocksDBRestoreOperation#readMetaData
>> treats "compatible after migration" the same as "incompatible" and
>> throws a "The new key serializer must be compatible." exception if it
>> encounters that result.
>>
>> Is it expected that the introduction of Row#getKind breaks existing
>> older state or is that a bug? So far I only reproduced this issue in a
>> somewhat more complex codebase, but in case this is an unknown issue
>> or not the intended behavior I can try to provide a small testcase (to
>> rule out that anything in our own code triggers that issue).
>>
>> Example of a query that triggers that issue:
>> https://gist.github.com/gstarnberger/f19a30df179c72a622490cbb041adb21
>>
>> Full stacktrace:
>> https://gist.github.com/gstarnberger/336d94e723b3ec599d09e17dd7d0ee00
>>
>> - Guenther
>>
>

[jira] [Created] (FLINK-20818) End to end test produce excessive amount of logs

2020-12-30 Thread Till Rohrmann (Jira) 
Till Rohrmann created FLINK-20818:
-

 Summary: End to end test produce excessive amount of logs
 Key: FLINK-20818
 URL: https://issues.apache.org/jira/browse/FLINK-20818
 Project: Flink
  Issue Type: Bug
  Components: Test Infrastructure
Affects Versions: 1.13.0
Reporter: Till Rohrmann
 Fix For: 1.13.0


The end to end test produce an excessive amount of logs. For example in this 
run [1] the log file is roughly 57 MB and it is no longer possible to properly 
scroll in this file when using the web interface. I think there should not be a 
reason for producing almost 60 MB of log output.

[1] 
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=11467&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee&t=ff888d9b-cd34-53cc-d90f-3e446d355529



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New formatting rules are now in effect

2020-12-30 Thread Chesnay Schepler 

1) No, it is not possible to exclude certain code blocks from formatting.
There is a workaround though for this case; you can add en empty comment 
(//) to the end of a line to prevent subsequent lines from being added 
to it.

https://github.com/google/google-java-format/issues/137

Note that any spotless-specific features would like not help us anyway, 
unless we'd be fine with not using the IntelliJ plugin.


2) The code style has not been updated yet.
The indent choices with the google-java-format is either 2 spaces for 
everything, or 4 spaces + 8 spaces for continuations.

In other words, 4 spaces is simply not an option.

On 12/30/2020 9:09 AM, Jark Wu wrote:

Hi,

I have played with the format plugin these days and found some problems.
Maybe some of them are personal taste.

1) Is it possible to disable auto-format for some code blocks?
For example, the format of code generation
StructuredObjectConverter#generateCode is manually
  adjusted for readability. However, the format plugin breaks it and it's
hard to read now.
See before[1] and after[2]. cc @Timo Walther 

2) Using 4 spaces or 8 spaces for continuation indent?
AOSP uses 8 spaces for continuation indent.
However, Flink Code Style suggests "Each new line should have one extra
indentation relative to
the line of the called entity" which means 4 spaces.
Personally, I think 4 spaces may be more friendly for Java lambdas.  An
example:

8 spaces:

wrapClassLoader(
 () ->
 environment
 .getModules()
 .forEach(
 (name, entry) ->
 modules.put(
 name,
 createModule(
 entry.asMap(),
classLoader;

4 spaces:

wrapClassLoader(
 () ->
 environment
 .getModules()
 .forEach(
 (name, entry) ->
 modules.put(name, createModule(entry.asMap(),
classLoader;



Best,
Jark

[1]:
https://github.com/apache/flink/blob/8fc6eaead00d6e98535874b0a137bc59716d260a/flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/data/conversion/StructuredObjectConverter.java#L169
[2]:
https://github.com/apache/flink/blob/master/flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/data/conversion/StructuredObjectConverter.java#L170
[3]:
https://flink.apache.org/contributing/code-style-and-quality-formatting.html#breaking-the-lines-of-too-long-statements

On Tue, 29 Dec 2020 at 19:19, Chesnay Schepler  wrote:


I have filed FLINK-20803
 for the issue that
Till raised.

google-java-format 1.8 and above require Java 11 to /run/, so we'll
stick with 1.7 for the time being.
This requires a manual installation of the google-java-format plugin,
and remembering to not update the plugin (urgh...).

Long term we may want to think about requiring Java 11 for /development/
(*not* usage) of Flink.

On 12/29/2020 12:13 PM, Till Rohrmann wrote:

I might have found a problem:

In my current setup I am using the google-java-format plugin version
1.9.0.0 which uses google-java-format 1.9.0. In our spotless

configuration

we are using google-java-format 1.7.0, however. The result is that

spotless

formats my code differently than IntelliJ. The following snippet shows

the

difference:

IntelliJ formatting with google-java-format 1.9.0:

return


Optional.ofNullable(archivedExecutionGraph.getAllVertices().get(jobVertexId))

 .map(
 accessExecutionJobVertex ->

   Arrays.asList(accessExecutionJobVertex.getTaskVertices()))
 .orElse(Collections.emptyList())
 .stream()
 .map(AccessExecutionVertex::getCurrentExecutionAttempt)
 .collect(Collectors.toList());

Spotless formatting with google-java-format 1.7.0:

return


Optional.ofNullable(archivedExecutionGraph.getAllVertices().get(jobVertexId))

.map(
accessExecutionJobVertex ->

Arrays.asList(accessExecutionJobVertex.getTaskVertices()))
.orElse(Collections.emptyList()).stream()
.map(AccessExecutionVertex::getCurrentExecutionAttempt)
.collect(Collectors.toList());

Note that the .stream() method is in the same line as .orElse().

I think this raises a bit the question which versions do we want to use?
Manually installing google-java-format plugin version 1.7.0.5 solved the
problem for me.

Cheers,
Till

On Tue, Dec 29, 2020 at 11:58 AM Flavio Pompermaier <

pomperma...@okkam.it>

wrote:


Thanks Aljoscha and Chesnay for this small but important improvement!
In the new year writing new Flink features will be funnier than ever ;)

On Tue, Dec 29, 2020 at 9:58 AM Till Rohrmann 
wrote:


Thanks a lot for this effort Aljoscha and Chesnay! Finally we have a

common

code style :-)

Cheers,
Till

On Tu

Re: Did Flink 1.11 break backwards compatibility for the table environment?

2020-12-30 Thread Till Rohrmann 
Hi Guenther,

sorry for overlooking your colleague's email.

I think the answer to your problem is twofold. The underlying problem is
that your query seems to use `RowData` as a key for some keyed operation.
Since changing the key format might entail that keys need to be differently
partitioned, Flink does not support changing the key format. That is why
Flink fails also if the key format is compatible after migration. There is
a small warning about this on the state evolution page [1].

The other part of the answer is that Flink does not support strict
backwards compatibility for SQL queries if I am not mistaken (please chime
in if this is no longer correct @Timo Walther  and
@j...@apache.org ). The problem is that queries might
result in different execution plans after a version upgrade which then can
not be mapped to the old state. Admittedly, in this case, it should have
been possible but changing the `RowData` type which is used as a key breaks
backwards compatibility. A bit confusing is that FLINK-16998 explicitly
states that this change is not breaking backwards compatibility.

What you could try to use as a workaround is Flink's state processor API
[2] which allows you to rewrite savepoints.

[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/schema_evolution.html
[2]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html

Cheers,
Till

On Wed, Dec 30, 2020 at 3:30 AM Guenther Starnberger 
wrote:

> Hello,
>
> We're currently running into an issue upgrading the state of an
> application to Flink 1.11 and I think this could be caused by a
> potential backwards incompatibility that was introduced with Flink
> 1.11. A colleague of mine recently posted about it on the users list
> (without a response), but I'd like to bring this up here on the dev
> list in order to figure out if that incompatibility is intended
> behavior and/or a known issue.
>
> We're seeing that issue when trying to load the RocksDB state from
> Flink 1.9 into Flink 1.11 with an application that uses the Flink
> table environment. Immediately after startup
> RocksDBFullRestoreOperation#restoreKVStateMetaData raises an "The new
> key serializer must be compatible" exception.
>
> The reason for that seems to be that FLINK-16998 changed the row
> serialization format by adding a new Row#getKind field. There's a
> legacy mode of the row serializer but that's only used for reading the
> existing snapshot. As a consequence
> RowSerializerSnapshot#resolveOuterSchemaCompatibility will always
> return COMPATIBLE_AFTER_MIGRATION when reading a Flink 1.9 savepoint
> with Flink 1.11.
>
> The problem is that AbstractRocksDBRestoreOperation#readMetaData
> treats "compatible after migration" the same as "incompatible" and
> throws a "The new key serializer must be compatible." exception if it
> encounters that result.
>
> Is it expected that the introduction of Row#getKind breaks existing
> older state or is that a bug? So far I only reproduced this issue in a
> somewhat more complex codebase, but in case this is an unknown issue
> or not the intended behavior I can try to provide a small testcase (to
> rule out that anything in our own code triggers that issue).
>
> Example of a query that triggers that issue:
> https://gist.github.com/gstarnberger/f19a30df179c72a622490cbb041adb21
>
> Full stacktrace:
> https://gist.github.com/gstarnberger/336d94e723b3ec599d09e17dd7d0ee00
>
> - Guenther
>

[jira] [Created] (FLINK-20817) kafka source have 200 field,Causes the class to be generated failed

2020-12-30 Thread WeiNan Zhao (Jira) 
WeiNan Zhao created FLINK-20817:
---

 Summary: kafka source have 200 field,Causes the class to be 
generated failed
 Key: FLINK-20817
 URL: https://issues.apache.org/jira/browse/FLINK-20817
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / API, Table SQL / Planner
Affects Versions: 1.10.2
Reporter: WeiNan Zhao


Too many fields of flink source cause the generated code to be too long, and a 
codehaus error is reported.

Is there a way to bypass this restriction instead of changing your own SQL, 
because we have many situations where the source table has many fields



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New formatting rules are now in effect

2020-12-30 Thread Jark Wu 
Hi,

I have played with the format plugin these days and found some problems.
Maybe some of them are personal taste.

1) Is it possible to disable auto-format for some code blocks?
For example, the format of code generation
StructuredObjectConverter#generateCode is manually
 adjusted for readability. However, the format plugin breaks it and it's
hard to read now.
See before[1] and after[2]. cc @Timo Walther 

2) Using 4 spaces or 8 spaces for continuation indent?
AOSP uses 8 spaces for continuation indent.
However, Flink Code Style suggests "Each new line should have one extra
indentation relative to
the line of the called entity" which means 4 spaces.
Personally, I think 4 spaces may be more friendly for Java lambdas.  An
example:

8 spaces:

wrapClassLoader(
() ->
environment
.getModules()
.forEach(
(name, entry) ->
modules.put(
name,
createModule(
entry.asMap(),
classLoader;

4 spaces:

wrapClassLoader(
() ->
environment
.getModules()
.forEach(
(name, entry) ->
modules.put(name, createModule(entry.asMap(),
classLoader;



Best,
Jark

[1]:
https://github.com/apache/flink/blob/8fc6eaead00d6e98535874b0a137bc59716d260a/flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/data/conversion/StructuredObjectConverter.java#L169
[2]:
https://github.com/apache/flink/blob/master/flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/data/conversion/StructuredObjectConverter.java#L170
[3]:
https://flink.apache.org/contributing/code-style-and-quality-formatting.html#breaking-the-lines-of-too-long-statements

On Tue, 29 Dec 2020 at 19:19, Chesnay Schepler  wrote:

> I have filed FLINK-20803
>  for the issue that
> Till raised.
>
> google-java-format 1.8 and above require Java 11 to /run/, so we'll
> stick with 1.7 for the time being.
> This requires a manual installation of the google-java-format plugin,
> and remembering to not update the plugin (urgh...).
>
> Long term we may want to think about requiring Java 11 for /development/
> (*not* usage) of Flink.
>
> On 12/29/2020 12:13 PM, Till Rohrmann wrote:
> > I might have found a problem:
> >
> > In my current setup I am using the google-java-format plugin version
> > 1.9.0.0 which uses google-java-format 1.9.0. In our spotless
> configuration
> > we are using google-java-format 1.7.0, however. The result is that
> spotless
> > formats my code differently than IntelliJ. The following snippet shows
> the
> > difference:
> >
> > IntelliJ formatting with google-java-format 1.9.0:
> >
> > return
> >
> Optional.ofNullable(archivedExecutionGraph.getAllVertices().get(jobVertexId))
> > .map(
> > accessExecutionJobVertex ->
> >
> >   Arrays.asList(accessExecutionJobVertex.getTaskVertices()))
> > .orElse(Collections.emptyList())
> > .stream()
> > .map(AccessExecutionVertex::getCurrentExecutionAttempt)
> > .collect(Collectors.toList());
> >
> > Spotless formatting with google-java-format 1.7.0:
> >
> > return
> >
> Optional.ofNullable(archivedExecutionGraph.getAllVertices().get(jobVertexId))
> >.map(
> >accessExecutionJobVertex ->
> >
> > Arrays.asList(accessExecutionJobVertex.getTaskVertices()))
> >.orElse(Collections.emptyList()).stream()
> >.map(AccessExecutionVertex::getCurrentExecutionAttempt)
> >.collect(Collectors.toList());
> >
> > Note that the .stream() method is in the same line as .orElse().
> >
> > I think this raises a bit the question which versions do we want to use?
> > Manually installing google-java-format plugin version 1.7.0.5 solved the
> > problem for me.
> >
> > Cheers,
> > Till
> >
> > On Tue, Dec 29, 2020 at 11:58 AM Flavio Pompermaier <
> pomperma...@okkam.it>
> > wrote:
> >
> >> Thanks Aljoscha and Chesnay for this small but important improvement!
> >> In the new year writing new Flink features will be funnier than ever ;)
> >>
> >> On Tue, Dec 29, 2020 at 9:58 AM Till Rohrmann 
> >> wrote:
> >>
> >>> Thanks a lot for this effort Aljoscha and Chesnay! Finally we have a
> >> common
> >>> code style :-)
> >>>
> >>> Cheers,
> >>> Till
> >>>
> >>> On Tue, Dec 29, 2020 at 7:32 AM Matthias Pohl 
> >>> wrote:
> >>>
>  Yes, thanks for driving this, Aljoscha. ...and Chesnay as well for
> >>> helping
>  to finalize it.
>  Good job.
> 
>  Matthias
> 
>  On Tue, Dec 29, 2020 at 5:23 AM Jark Wu  wrote:
> 
> > Thanks Aljoscha and Chesnay for the great work!
> >
> > Best,
> > Jark
> >
> > On Tue, 29 Dec 2020 at 11:11, Xintong Song 
>  wrote:
> >