[jira] [Created] (FLINK-20744) org.apache.flink.test.checkpointing.UnalignedCheckpointITCase fails due to java.lang.ArrayIndexOutOfBoundsException

[Matthias (Jira)]

Matthias created FLINK-20744:


 Summary: 
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase fails due to 
java.lang.ArrayIndexOutOfBoundsException
 Key: FLINK-20744
 URL: https://issues.apache.org/jira/browse/FLINK-20744
 Project: Flink
  Issue Type: Test
  Components: Runtime / Checkpointing
Affects Versions: 1.13.0
Reporter: Matthias


[Build|https://dev.azure.com/mapohl/flink/_build/results?buildId=140=logs=6e55a443-5252-5db5-c632-109baf464772=9df6efca-61d0-513a-97ad-edb76d85786a=8819]
 failed due to {{UnalignedCheckpointITCase}} failure:
{code:java}
at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
at akka.actor.ActorCell.invoke(ActorCell.scala:561)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
at akka.dispatch.Mailbox.run(Mailbox.scala:225)
at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
... 4 more
Caused by: java.lang.RuntimeException: -4
at 
org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:106)
at 
org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:89)
at 
org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:45)
at 
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52)
at 
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30)
at 
org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:53)
at 
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase$MinEmittingFunction.flatMap1(UnalignedCheckpointITCase.java:368)
at 
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase$MinEmittingFunction.flatMap1(UnalignedCheckpointITCase.java:348)
at 
org.apache.flink.streaming.api.operators.co.CoStreamFlatMap.processElement1(CoStreamFlatMap.java:53)
at 
org.apache.flink.streaming.runtime.io.StreamTwoInputProcessorFactory.processRecord1(StreamTwoInputProcessorFactory.java:166)
at 
org.apache.flink.streaming.runtime.io.StreamTwoInputProcessorFactory.lambda$create$0(StreamTwoInputProcessorFactory.java:123)
at 
org.apache.flink.streaming.runtime.io.StreamTwoInputProcessorFactory$StreamTaskNetworkOutput.emitRecord(StreamTwoInputProcessorFactory.java:243)
at 
org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:179)
at 
org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:152)
at 
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:67)
at 
org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.processInput(StreamTwoInputProcessor.java:92)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:372)
at 
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:186)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:575)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:539)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ArrayIndexOutOfBoundsException: -4
at 
org.apache.flink.runtime.io.network.partition.BufferWritingResultPartition.appendUnicastDataForNewRecord(BufferWritingResultPartition.java:238)
at 
org.apache.flink.runtime.io.network.partition.BufferWritingResultPartition.emitRecord(BufferWritingResultPartition.java:140)
at 
org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:100)
at 
org.apache.flink.runtime.io.network.api.writer.ChannelSelectorRecordWriter.emit(ChannelSelectorRecordWriter.java:53)
at 
org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:103)
... 22 more
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Apache Flink Stateful Functions 2.2.2, release candidate #2

[Igal Shilman]

Thanks for driving the release Gordon!


here is a +1 (non binding) from my side:


- verified checksums, signatures etc’.

- verified that there are no binary files in the source distribution.

- built successfully with jdk11

- built successfully with jdk8

- e2e passes with jdk11

- e2e passes with jdk8

- I've ran a version of the SmokeTest that we have in master, that is based
on [1]

against this RC, and it passes successfully.

- taken a save point on the verification job [1] taken at StateFun 2.2.0
and successfully restored with StateFun 2.2.2.

- taken a save point with StateFun 2.2.1, and restored successfully with
2.2.2



[1] https://github.com/igalshilman/statefun-verifier/tree/ng



On Wednesday, December 23, 2020, Tzu-Li (Gordon) Tai 
wrote:

> +1 (binding)
>
> - Verified hashes and signatures
> - Verified NOTICE files in statefun-flink-datastream and
> statefun-flink-distribution
> - Built locally: mvn clean install -Prun-e2e-tests
> - Successfully restored from a 2.2.0 savepoint that contains feedback
> events
> - Verified that the proposed new Dockerfiles work properly
>
> On Wed, Dec 23, 2020 at 1:58 PM Tzu-Li (Gordon) Tai 
> wrote:
>
> > Hi everyone,
> >
> > Please review and vote on the release candidate #2 for the version 2.2.2
> > of Apache Flink Stateful Functions, as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > ***Testing Guideline***
> >
> > You can find here [1] a page in the project wiki on instructions for
> > testing.
> > To cast a vote, it is not necessary to perform all listed checks,
> > but please mention which checks you have performed when voting.
> >
> > ***Release Overview***
> >
> > As an overview, the release consists of the following:
> > a) Stateful Functions canonical source distribution, to be deployed to
> the
> > release repository at dist.apache.org
> > b) Stateful Functions Python SDK distributions to be deployed to PyPI
> > c) Maven artifacts to be deployed to the Maven Central Repository
> > d) New Dockerfiles for the release
> > e) Release announcement to be published to the Flink website
> >
> > ***Staging Areas to Review***
> >
> > The staging areas containing the above mentioned artifacts are as
> follows,
> > for your review:
> > * All artifacts for a) and b) can be found in the corresponding dev
> > repository at dist.apache.org [2]
> > * All artifacts for c) can be found at the Apache Nexus Repository [3]
> >
> > All artifacts are signed with the key
> > 1C1E2394D3194E1944613488F320986D35C33D6A [4]
> >
> > Other links for your review:
> > * JIRA release notes [5]
> > * source code tag "release-2.2.2-rc2" [6]
> > * PR for the new Dockerfiles [7]
> > * PR for the release announcement blog post [8]
> >
> > ***Vote Duration***
> >
> >  Taking into account the Christmas holidays, I propose the voting time
> > to run for a week.
> > I'm targeting this vote to last until Dec. 29th, 10am CET.
> > It is adopted by majority approval, with at least 3 PMC affirmative
> votes.
> >
> > Thanks,
> > Gordon
> >
> > [1]
> > https://cwiki.apache.org/confluence/display/FLINK/
> Verifying+a+Flink+Stateful+Functions+Release
> > [2] https://dist.apache.org/repos/dist/dev/flink/flink-statefun-
> 2.2.2-rc2/
> > [3] https://repository.apache.org/content/repositories/org apache
> > flink-1409/
> >  >
> > [4] https://dist.apache.org/repos/dist/release/flink/KEYS
> > [5]
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12315522=12349366
> > [6] https://github.com/apache/flink-statefun/tree/release-2.2.2-rc2
> > [7] https://github.com/apache/flink-statefun-docker/pull/12
> > [8] https://github.com/apache/flink-web/pull/404
> >
>

[jira] [Created] (FLINK-20743) Print ContainerId For RemoteTransportException

[yang gang (Jira)]

yang gang created FLINK-20743:
-

 Summary: Print ContainerId For RemoteTransportException
 Key: FLINK-20743
 URL: https://issues.apache.org/jira/browse/FLINK-20743
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Network
Affects Versions: 1.11.1, 1.10.0, 1.12.1
Reporter: yang gang
 Attachments: image-2020-12-23-14-48-13-489.png

!image-2020-12-23-14-48-13-489.png!
RemoteTransportException, this exception reminds the user which service has a 
problem by means of Ip/Port.
When we troubleshoot the problem, the information is not accurate enough. 
Usually at this time we need to look at the running log of the container that 
has the problem, but when we see this log, it also shows that the container has 
died, so pass Ip/ The port method can no longer quickly locate a specific 
container.
So I hope that when such an exception occurs, I hope to print the containerId。

E.g:
Connection unexpectedly closed by remote task manager 
'hostName/ip:port/containerId'. This might indicate that the remote task 
manager was lost.
 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20742) Skip deployment of StateFun example artifacts

[Tzu-Li (Gordon) Tai (Jira)]

Tzu-Li (Gordon) Tai created FLINK-20742:
---

 Summary: Skip deployment of StateFun example artifacts
 Key: FLINK-20742
 URL: https://issues.apache.org/jira/browse/FLINK-20742
 Project: Flink
  Issue Type: Bug
  Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
 Fix For: statefun-3.0.0


Starting from the next Stateful Functions release, we'd like to stop publishing 
Maven artifacts for the examples.

We never expect users to be trying out examples through this artifacts, and 
therefore releasing them is not required.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Apache Flink Stateful Functions 2.2.2, release candidate #2

[Tzu-Li (Gordon) Tai]

+1 (binding)

- Verified hashes and signatures
- Verified NOTICE files in statefun-flink-datastream and
statefun-flink-distribution
- Built locally: mvn clean install -Prun-e2e-tests
- Successfully restored from a 2.2.0 savepoint that contains feedback events
- Verified that the proposed new Dockerfiles work properly

On Wed, Dec 23, 2020 at 1:58 PM Tzu-Li (Gordon) Tai 
wrote:

> Hi everyone,
>
> Please review and vote on the release candidate #2 for the version 2.2.2
> of Apache Flink Stateful Functions, as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> ***Testing Guideline***
>
> You can find here [1] a page in the project wiki on instructions for
> testing.
> To cast a vote, it is not necessary to perform all listed checks,
> but please mention which checks you have performed when voting.
>
> ***Release Overview***
>
> As an overview, the release consists of the following:
> a) Stateful Functions canonical source distribution, to be deployed to the
> release repository at dist.apache.org
> b) Stateful Functions Python SDK distributions to be deployed to PyPI
> c) Maven artifacts to be deployed to the Maven Central Repository
> d) New Dockerfiles for the release
> e) Release announcement to be published to the Flink website
>
> ***Staging Areas to Review***
>
> The staging areas containing the above mentioned artifacts are as follows,
> for your review:
> * All artifacts for a) and b) can be found in the corresponding dev
> repository at dist.apache.org [2]
> * All artifacts for c) can be found at the Apache Nexus Repository [3]
>
> All artifacts are signed with the key
> 1C1E2394D3194E1944613488F320986D35C33D6A [4]
>
> Other links for your review:
> * JIRA release notes [5]
> * source code tag "release-2.2.2-rc2" [6]
> * PR for the new Dockerfiles [7]
> * PR for the release announcement blog post [8]
>
> ***Vote Duration***
>
>  Taking into account the Christmas holidays, I propose the voting time
> to run for a week.
> I'm targeting this vote to last until Dec. 29th, 10am CET.
> It is adopted by majority approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Gordon
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Stateful+Functions+Release
> [2] https://dist.apache.org/repos/dist/dev/flink/flink-statefun-2.2.2-rc2/
> [3] https://repository.apache.org/content/repositories/org apache
> flink-1409/
> 
> [4] https://dist.apache.org/repos/dist/release/flink/KEYS
> [5]
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12349366
> [6] https://github.com/apache/flink-statefun/tree/release-2.2.2-rc2
> [7] https://github.com/apache/flink-statefun-docker/pull/12
> [8] https://github.com/apache/flink-web/pull/404
>

[VOTE] Apache Flink Stateful Functions 2.2.2, release candidate #2

[Tzu-Li (Gordon) Tai]

Hi everyone,

Please review and vote on the release candidate #2 for the version 2.2.2 of
Apache Flink Stateful Functions, as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

***Testing Guideline***

You can find here [1] a page in the project wiki on instructions for
testing.
To cast a vote, it is not necessary to perform all listed checks,
but please mention which checks you have performed when voting.

***Release Overview***

As an overview, the release consists of the following:
a) Stateful Functions canonical source distribution, to be deployed to the
release repository at dist.apache.org
b) Stateful Functions Python SDK distributions to be deployed to PyPI
c) Maven artifacts to be deployed to the Maven Central Repository
d) New Dockerfiles for the release
e) Release announcement to be published to the Flink website

***Staging Areas to Review***

The staging areas containing the above mentioned artifacts are as follows,
for your review:
* All artifacts for a) and b) can be found in the corresponding dev
repository at dist.apache.org [2]
* All artifacts for c) can be found at the Apache Nexus Repository [3]

All artifacts are signed with the key
1C1E2394D3194E1944613488F320986D35C33D6A [4]

Other links for your review:
* JIRA release notes [5]
* source code tag "release-2.2.2-rc2" [6]
* PR for the new Dockerfiles [7]
* PR for the release announcement blog post [8]

***Vote Duration***

 Taking into account the Christmas holidays, I propose the voting time
to run for a week.
I'm targeting this vote to last until Dec. 29th, 10am CET.
It is adopted by majority approval, with at least 3 PMC affirmative votes.

Thanks,
Gordon

[1]
https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Stateful+Functions+Release
[2] https://dist.apache.org/repos/dist/dev/flink/flink-statefun-2.2.2-rc2/
[3] https://repository.apache.org/content/repositories/org apache
flink-1409/

[4] https://dist.apache.org/repos/dist/release/flink/KEYS
[5]
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12349366
[6] https://github.com/apache/flink-statefun/tree/release-2.2.2-rc2
[7] https://github.com/apache/flink-statefun-docker/pull/12
[8] https://github.com/apache/flink-web/pull/404

[jira] [Created] (FLINK-20741) StateFun's DataStream interop SDK module is missing a valid NOTICE file

[Tzu-Li (Gordon) Tai (Jira)]

Tzu-Li (Gordon) Tai created FLINK-20741:
---

 Summary: StateFun's DataStream interop SDK module is missing a 
valid NOTICE file
 Key: FLINK-20741
 URL: https://issues.apache.org/jira/browse/FLINK-20741
 Project: Flink
  Issue Type: Bug
  Components: Stateful Functions
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
 Fix For: statefun-2.2.2


The {{statefun-flink-datastream}} module bundles a fat jar, just like the 
{{statefun-flink-distribution}} module. Likewise, for legal purposes, all 
bundled dependencies need to be acknowledged in the NOTICE file.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20740) Use managed memory to avoid direct memory OOM error

[Yingjie Cao (Jira)]

Yingjie Cao created FLINK-20740:
---

 Summary: Use managed memory to avoid direct memory OOM error
 Key: FLINK-20740
 URL: https://issues.apache.org/jira/browse/FLINK-20740
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Network
Affects Versions: 1.12.0
Reporter: Yingjie Cao
 Fix For: 1.13.0


Currently, sort-merge blocking shuffle uses some unmanaged memory for data 
writing and reading, which means users must increase the size of direct memory, 
otherwise, one may encounter direct memory OOM error, which is really bad for 
usability.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20739) Ban `if` from HiveModule

[jinfeng (Jira)]

jinfeng created FLINK-20739:
---

 Summary: Ban  `if`  from HiveModule
 Key: FLINK-20739
 URL: https://issues.apache.org/jira/browse/FLINK-20739
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Hive
Affects Versions: 1.10.0
Reporter: jinfeng


When using hiveModule, the if function is treated as a normal function.
If I have a SQL like this: 
 
{code:java}
 insert into Sink select  if(size(split(`test`, '-')) > 1, split(`test`, 
'-')[10], 'error') from Source   {code}
 
It will throw arrayIndexOutOfBoundsException in Flink1.10,  becase 
size(split(`test`, '-')) > 1 ， split(`test`, '-')[10]， ‘error’   will be 
calculated first, and then if function will be  calculated



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20738) Separate the implementation of batch aggregate nodes

[godfrey he (Jira)]

godfrey he created FLINK-20738:
--

 Summary: Separate the implementation of batch aggregate nodes
 Key: FLINK-20738
 URL: https://issues.apache.org/jira/browse/FLINK-20738
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: godfrey he
 Fix For: 1.13.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20737) Separate the implementation of stream aggregate nodes

[godfrey he (Jira)]

godfrey he created FLINK-20737:
--

 Summary: Separate the implementation of stream aggregate nodes 
 Key: FLINK-20737
 URL: https://issues.apache.org/jira/browse/FLINK-20737
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: godfrey he
 Fix For: 1.13.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Make Temporal Join syntax easier to use

[Danny Chan]

Thanks Shuo Cheng for driving this discussion ~

Generally i have the same feeling that our temporal table syntax is a
little verbose compared to KSQL or other RDBMS. (they just use the normal
join syntax for the temporal join).

>I think maybe we can add a pseudo-column `PROCTIME` (or `_PROCTIME` to
avoid conflict

I don't think add a pseudo-column is a good solution because of these
reasons:

- The normal pseudo-column or system column like _rowID_ has a
underneath storage, user can select the column from a table [1] , and each
row has a deterministic value bind to it for the pseudo-column (although it
may change when the row is deleted and inserted again), but the PROCTIME
for Flink behaves more like a row level runtime attribute which is
different for different queries and even different SQL contexts.

- The pseudo-column make the table schema more complex but they are only
useful when we want to use the time-attributes.

> Actually, we have another simpler solution, i.e., enrich the syntax for
temporal table join to support 'FOR SYSTEM_TIME AS OF PROCTIME()'.

Maybe our biggest concern is that the syntax does not make it clear where
the who triggers the PROCTIME() computation, similar with
`current_timestamp`, from my understanding, the `PROCTIME()` is computed in
row level by the system, when a record from the LHS is used to join the RHS
table. So generally i'm +1 for 'FOR SYSTEM_TIME AS OF PROCTIME()'.

BTW, in the long term, we should think how to simplifies the `FOR
SYSTEM_TIME AS OF ` syntax, because the proc time temporal table join is
the most common case and we should make the temporal table join default to
be 'PROCTIME'. Ideally a normal `A JOIN B` can describe a PROCTIME temporal
table join. The solution to add pseudo-column seems deviate further and
further from this path.

[1]
https://docs.oracle.com/cd/B19306_01/server.102/b14200/pseudocolumns008.htm

Shuo Cheng  于2020年12月21日周一 上午10:16写道：

> Hi community,
>
> Temporal table join in Flink is a powerful feature, especially processing
> time temporal table join, which is very commonly used in production.
> However, the syntax of temporal table join only supports 'FOR SYSTEM_TIME
> AS OF' left table's time attribute field currently. e.g.,
>
> ```
> SELECT *
> FROM Orders AS o
> JOIN Products FOR SYSTEM_TIME AS OF o.proctime AS p
> ON o.productId = p.productId
> ```
>
> That means the users have to explicitly define a the proctime field either
> in the left table DDL or the left temporary view, which is inconvenient to
> use.
>
> >> Preferred solution
> I think maybe we can add a pseudo-column `PROCTIME` (or `_PROCTIME` to
> avoid conflict) to the table by default, just like the pseudo-column of
> classic database, e.g., `ROWID` in Oracle. In this way, we can use Temporal
> table join very conveniently.
>
> >> Less elegant solution
> Actually, we have another simpler solution, i.e., enrich the syntax for
> temporal table join to support 'FOR SYSTEM_TIME AS OF PROCTIME()'. It is
> also very convenient, However, the `PROCTIME()` in 'FOR SYSTEM_TIME AS OF
> PROCTIME()' is ambiguous, because it cannot tell where the version time of
> temporal table comes from, left table or right table? The former is what we
> want. So I think this solution is not preferred.
>
> Looking forward to your feedback~
>
> Best,
> Shuo
>

[jira] [Created] (FLINK-20736) Separate the implementation of BatchExecLimit and StreamExecLimit

[Wenlong Lyu (Jira)]

Wenlong Lyu created FLINK-20736:
---

 Summary: Separate the implementation of BatchExecLimit and 
StreamExecLimit
 Key: FLINK-20736
 URL: https://issues.apache.org/jira/browse/FLINK-20736
 Project: Flink
  Issue Type: Sub-task
Reporter: Wenlong Lyu






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20735) Support configurable log level

[zlzhang0122 (Jira)]

zlzhang0122 created FLINK-20735:
---

 Summary: Support configurable log level
 Key: FLINK-20735
 URL: https://issues.apache.org/jira/browse/FLINK-20735
 Project: Flink
  Issue Type: Improvement
Affects Versions: 1.12.0
Reporter: zlzhang0122
 Fix For: 1.13.0


Now the default log level of flink is "INFO" and if we want to change the 
default log level, we have to change that in many log file and this may affect 
all the jobs using these config, I think we can expose a log level config for 
users to change the default log level for every single job.And if users don't 
provide the config value, the default log level won't be changed. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20734) SQL client error when running joining query

[owr (Jira)]

owr created FLINK-20734:
---

 Summary: SQL client error when running joining query
 Key: FLINK-20734
 URL: https://issues.apache.org/jira/browse/FLINK-20734
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Client
Affects Versions: 1.12.0
Reporter: owr


Run into the following errors when performing join query in SQL client.

01:52:46.189 [main] ERROR org.apache.flink.table.client.SqlClient - SQL Client 
must stop. Unexpected exception. This is a bug. Please consider filing an 
issue.01:52:46.189 [main] ERROR org.apache.flink.table.client.SqlClient - SQL 
Client must stop. Unexpected exception. This is a bug. Please consider filing 
an issue.java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 at 
java.util.ArrayList.rangeCheck(ArrayList.java:659) ~[?:1.8.0_272] at 
java.util.ArrayList.get(ArrayList.java:435) ~[?:1.8.0_272] at 
org.apache.flink.table.client.cli.CliResultView.openRow(CliResultView.java:149) 
~[flink-sql-client_2.11-1.12.0.jar:1.12.0] at 
org.apache.flink.table.client.cli.CliTableResultView.evaluate(CliTableResultView.java:154)
 ~[flink-sql-client_2.11-1.12.0.jar:1.12.0] at 
org.apache.flink.table.client.cli.CliTableResultView.evaluate(CliTableResultView.java:52)
 ~[flink-sql-client_2.11-1.12.0.jar:1.12.0] at 
org.apache.flink.table.client.cli.CliView.open(CliView.java:125) 
~[flink-sql-client_2.11-1.12.0.jar:1.12.0] at 
org.apache.flink.table.client.cli.CliClient.callSelect(CliClient.java:661) 
~[flink-sql-client_2.11-1.12.0.jar:1.12.0] at 
org.apache.flink.table.client.cli.CliClient.callCommand(CliClient.java:324) 
~[flink-sql-client_2.11-1.12.0.jar:1.12.0] at 
java.util.Optional.ifPresent(Optional.java:159) ~[?:1.8.0_272] at 
org.apache.flink.table.client.cli.CliClient.open(CliClient.java:216) 
~[flink-sql-client_2.11-1.12.0.jar:1.12.0] at 
org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:141) 
~[flink-sql-client_2.11-1.12.0.jar:1.12.0] at 
org.apache.flink.table.client.SqlClient.start(SqlClient.java:114) 
~[flink-sql-client_2.11-1.12.0.jar:1.12.0] at 
org.apache.flink.table.client.SqlClient.main(SqlClient.java:196) 
[flink-sql-client_2.11-1.12.0.jar:1.12.0]Exception in thread "main" 
org.apache.flink.table.client.SqlClientException: Unexpected exception. This is 
a bug. Please consider filing an issue. at 
org.apache.flink.table.client.SqlClient.main(SqlClient.java:208)Caused by: 
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 at 
java.util.ArrayList.rangeCheck(ArrayList.java:659) at 
java.util.ArrayList.get(ArrayList.java:435) at 
org.apache.flink.table.client.cli.CliResultView.openRow(CliResultView.java:149) 
at 
org.apache.flink.table.client.cli.CliTableResultView.evaluate(CliTableResultView.java:154)
 at 
org.apache.flink.table.client.cli.CliTableResultView.evaluate(CliTableResultView.java:52)
 at org.apache.flink.table.client.cli.CliView.open(CliView.java:125) at 
org.apache.flink.table.client.cli.CliClient.callSelect(CliClient.java:661) at 
org.apache.flink.table.client.cli.CliClient.callCommand(CliClient.java:324) at 
java.util.Optional.ifPresent(Optional.java:159) at 
org.apache.flink.table.client.cli.CliClient.open(CliClient.java:216) at 
org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:141) at 
org.apache.flink.table.client.SqlClient.start(SqlClient.java:114) at 
org.apache.flink.table.client.SqlClient.main(SqlClient.java:196)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20733) Introduce the upsert-pulsar Connector

[Jianyun Zhao (Jira)]

Jianyun Zhao created FLINK-20733:


 Summary: Introduce the upsert-pulsar Connector
 Key: FLINK-20733
 URL: https://issues.apache.org/jira/browse/FLINK-20733
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / Common
Affects Versions: 1.13.0
Reporter: Jianyun Zhao






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20732) Pulsar Sink

[Jianyun Zhao (Jira)]

Jianyun Zhao created FLINK-20732:


 Summary: Pulsar Sink
 Key: FLINK-20732
 URL: https://issues.apache.org/jira/browse/FLINK-20732
 Project: Flink
  Issue Type: Sub-task
Reporter: Jianyun Zhao






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20731) Pulsar Source

[Jianyun Zhao (Jira)]

Jianyun Zhao created FLINK-20731:


 Summary: Pulsar Source
 Key: FLINK-20731
 URL: https://issues.apache.org/jira/browse/FLINK-20731
 Project: Flink
  Issue Type: Sub-task
Reporter: Jianyun Zhao






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20730) Pulsar Catalog

[Jianyun Zhao (Jira)]

Jianyun Zhao created FLINK-20730:


 Summary: Pulsar Catalog
 Key: FLINK-20730
 URL: https://issues.apache.org/jira/browse/FLINK-20730
 Project: Flink
  Issue Type: Sub-task
Reporter: Jianyun Zhao






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20729) Pulsar DynamicTable

[Jianyun Zhao (Jira)]

Jianyun Zhao created FLINK-20729:


 Summary: Pulsar DynamicTable
 Key: FLINK-20729
 URL: https://issues.apache.org/jira/browse/FLINK-20729
 Project: Flink
  Issue Type: Sub-task
Reporter: Jianyun Zhao






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20728) Pulsar SinkFunction

[Jianyun Zhao (Jira)]

Jianyun Zhao created FLINK-20728:


 Summary: Pulsar SinkFunction
 Key: FLINK-20728
 URL: https://issues.apache.org/jira/browse/FLINK-20728
 Project: Flink
  Issue Type: Sub-task
Reporter: Jianyun Zhao






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20727) Pulsar SourceFunction

[Jianyun Zhao (Jira)]

Jianyun Zhao created FLINK-20727:


 Summary: Pulsar SourceFunction
 Key: FLINK-20727
 URL: https://issues.apache.org/jira/browse/FLINK-20727
 Project: Flink
  Issue Type: Sub-task
Reporter: Jianyun Zhao






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20726) Support Pulsar connector

[Jianyun Zhao (Jira)]

Jianyun Zhao created FLINK-20726:


 Summary: Support Pulsar connector
 Key: FLINK-20726
 URL: https://issues.apache.org/jira/browse/FLINK-20726
 Project: Flink
  Issue Type: New Feature
  Components: Connectors / Common
Affects Versions: 1.13.0
Reporter: Jianyun Zhao


Pulsar is an important player in messaging middleware, and it is essential for 
Flink to support Pulsar.

Our existing code is maintained at 
[streamnative/pulsar-flink|https://github.com/streamnative/pulsar-flink] , next 
we will split it into several pr merges back to the community.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20725) TableFunction#getTypeInference is not work well for JdbcRowDataLookupFunction, it's always throws Exception

[Lijie Wang (Jira)]

Lijie Wang created FLINK-20725:
--

 Summary: TableFunction#getTypeInference is not work well for 
JdbcRowDataLookupFunction, it's always throws Exception
 Key: FLINK-20725
 URL: https://issues.apache.org/jira/browse/FLINK-20725
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / API
Affects Versions: 1.13.0
Reporter: Lijie Wang


I find that TableFunction#getTypeInference  is not work well for 
JdbcRowDataLookupFunction , it's always throws Exception.

 

When I add following test in JdbcLookupFunctionTest and run it:
{code:java}
@Test
public void testTypeInference() {
   final TableEnvironment env = TableEnvironment.create(EnvironmentSettings
  .newInstance().build());

   final DataTypeFactory dataTypeFactory = ((TableEnvironmentInternal) env)
  .getCatalogManager()
  .getDataTypeFactory();

   JdbcLookupFunction lookupFunction = buildLookupFunction();

   lookupFunction.getTypeInference(dataTypeFactory);
}
{code}
 

it always throws following exception:
{code:java}
Test testTypeInference(org.apache.flink.connector.jdbc.JdbcLookupFunctionTest) 
failed with:Test 
testTypeInference(org.apache.flink.connector.jdbc.JdbcLookupFunctionTest) 
failed with:org.apache.flink.table.api.ValidationException: Could not extract a 
valid type inference for function class 
'org.apache.flink.connector.jdbc.table.JdbcLookupFunction'. Please check for 
implementation mistakes and/or provide a corresponding hint. at 
org.apache.flink.table.types.extraction.ExtractionUtils.extractionError(ExtractionUtils.java:359)
 at 
org.apache.flink.table.types.extraction.TypeInferenceExtractor.extractTypeInference(TypeInferenceExtractor.java:160)
 at 
org.apache.flink.table.types.extraction.TypeInferenceExtractor.forTableFunction(TypeInferenceExtractor.java:119)
 at 
org.apache.flink.table.functions.TableFunction.getTypeInference(TableFunction.java:212)
 at 
org.apache.flink.connector.jdbc.JdbcLookupFunctionTest.testTypeInference(JdbcLookupFunctionTest.java:103)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498) at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
 at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
 at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
 at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
 at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) 
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) 
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at 
org.junit.rules.RunRules.evaluate(RunRules.java:20) at 
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
 at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
 at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at 
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at 
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) at 
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) at 
org.junit.rules.RunRules.evaluate(RunRules.java:20) at 
org.junit.runners.ParentRunner.run(ParentRunner.java:363) at 
org.junit.runner.JUnitCore.run(JUnitCore.java:137) at 
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
 at 
com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
 at 
com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220)
 at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53)Caused by: 
org.apache.flink.table.api.ValidationException: Error in extracting a signature 
to output mapping. at 
org.apache.flink.table.types.extraction.ExtractionUtils.extractionError(ExtractionUtils.java:359)
 at 
org.apache.flink.table.types.extraction.FunctionMappingExtractor.extractOutputMapping(FunctionMappingExtractor.java:118)
 at 
org.apache.flink.table.types.extraction.TypeInferenceExtractor.extractTypeInferenceOrError(TypeInferenceExtractor.java:170)
 at 
org.apache.flink.table.types.extraction.TypeInferenceExtractor.extractTypeInference(TypeInferenceExtractor.java:158)
 ... 32 moreCaused by: org.apache.flink.table.api.ValidationException: Unable 
to extract a type inference from method:public void 

[jira] [Created] (FLINK-20724) Create a http handler for aggregating metrics from whole job

[Piotr Nowojski (Jira)]

Piotr Nowojski created FLINK-20724:
--

 Summary: Create a http handler for aggregating metrics from whole 
job
 Key: FLINK-20724
 URL: https://issues.apache.org/jira/browse/FLINK-20724
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Metrics
Reporter: Piotr Nowojski
 Fix For: 1.13.0


Create a similar http handler to {{AggregatingSubtasksMetricsHandler}}, but one 
that would aggregate metrics per task, from all of the job vertices. The new 
handler would only take {{JobID}} as a parameter. So that Web UI can in one RPC 
obtain {{max(isBackPressureRatio)}} / {{max(isCausingBackPressureRatio)}} per 
each task in the job graph.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20723) testCassandraPojoNoAnnotatedKeyspaceAtLeastOnceSink failed due to NoHostAvailableException

[Matthias (Jira)]

Matthias created FLINK-20723:


 Summary: testCassandraPojoNoAnnotatedKeyspaceAtLeastOnceSink 
failed due to NoHostAvailableException
 Key: FLINK-20723
 URL: https://issues.apache.org/jira/browse/FLINK-20723
 Project: Flink
  Issue Type: Test
  Components: Connectors / Cassandra
Affects Versions: 1.11.3
 Environment: 
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=11137=results
Reporter: Matthias


[Build 
20201221.17|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=11137=results]
 failed due to {{NoHostAvailableException}}:
{code}
[ERROR] Tests run: 17, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 
167.927 s <<< FAILURE! - in 
org.apache.flink.streaming.connectors.cassandra.CassandraConnectorITCase
[ERROR] 
testCassandraPojoNoAnnotatedKeyspaceAtLeastOnceSink(org.apache.flink.streaming.connectors.cassandra.CassandraConnectorITCase)
  Time elapsed: 12.234 s  <<< ERROR!
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried 
for query failed (tried: /127.0.0.1:9042 
(com.datastax.driver.core.exceptions.OperationTimedOutException: [/127.0.0.1] 
Timed out waiting for server response))
at 
com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:84)
at 
com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:37)
at 
com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
at 
com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
at 
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
at 
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:39)
at 
org.apache.flink.streaming.connectors.cassandra.CassandraConnectorITCase.createTable(CassandraConnectorITCase.java:221)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at 
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All 
host(s) tried for query failed (tried: /127.0.0.1:9042 
(com.datastax.driver.core.exceptions.OperationTimedOutException: [/127.0.0.1] 
Timed out waiting for server response))
at 

[jira] [Created] (FLINK-20722) Fail to insert into hive table due to ClassCastException

[Rui Li (Jira)]

Rui Li created FLINK-20722:
--

 Summary: Fail to insert into hive table due to ClassCastException
 Key: FLINK-20722
 URL: https://issues.apache.org/jira/browse/FLINK-20722
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Runtime
Reporter: Rui Li
 Fix For: 1.13.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20721) Remove unknown input channels and process to update partitions

[Zhu Zhu (Jira)]

Zhu Zhu created FLINK-20721:
---

 Summary: Remove unknown input channels and process to update 
partitions
 Key: FLINK-20721
 URL: https://issues.apache.org/jira/browse/FLINK-20721
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination, Runtime / Network
Affects Versions: 1.13.0
Reporter: Zhu Zhu
 Fix For: 1.13.0


With the latest pipelined region scheduling, Flink no longer launch a task 
before knowing the locations of all the partitions it consumes. 
`scheduleOrUpdateConsumers` is no longer needed so we removed it in FLINK-20439.

Unknown input channels and the process to update it is also no longer needed. 
I'd propose to remove them and the benefits are:
1. simplifying the code of both scheduler and shuffle components
2. simplifying interfaces of ShuffleEnvironment and ShuffleDescriptor 
3. ensure the assumptions in InputGate#resumeConsumption() implementations
4. allow to remove ScheduleMode#allowLazyDeployment() and later completely 
remove ScheduleMode





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20720) Add documentation for ProcessFunction in Python DataStream API

[Shuiqiang Chen (Jira)]

Shuiqiang Chen created FLINK-20720:
--

 Summary: Add documentation for ProcessFunction in Python 
DataStream API
 Key: FLINK-20720
 URL: https://issues.apache.org/jira/browse/FLINK-20720
 Project: Flink
  Issue Type: Improvement
  Components: API / Python, Documentation
Reporter: Shuiqiang Chen
 Fix For: 1.13.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20719) Change BatchExecNode & StreamExecNode to interface and make each node extended from ExecNodeBase directly

[godfrey he (Jira)]

godfrey he created FLINK-20719:
--

 Summary: Change BatchExecNode & StreamExecNode to interface and 
make each node extended from ExecNodeBase directly
 Key: FLINK-20719
 URL: https://issues.apache.org/jira/browse/FLINK-20719
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: godfrey he
 Fix For: 1.13.0


Currently the inheritance structure of exec nodes is:

{code:java}
ExecNode (interface)
   |_ ExecNodeBase (abstract class)
 |_ BatchExecNode (abstract class)
   |_ BatchExecCalc
   |_ ...
 |_ StreamExecNode (abstract class)
   |_ StreamExecCalc
   |_ ...
{code}
the advantage is: each specific node only needs to inherit one base class, and 
BatchExecNode and StreamExecNode can provide some default implementation
the disadvantage is: common class of specific batch node and stream node must 
be interface which only can provide some utility method through default 
implementation and its sub-classes also have some deduplicate code.  if the 
common class can be abstract class, its sub-classes is cleaner.  such as: Calc 
classes

the structure after adjustment is:
{code:java}
ExecNode (interface)
   |_ ExecNodeBase (abstract class)
   |_ BatchExecNode (interface)
   |_ StreamExecNode (interface)

class BatchExecCalc extends ExecNodeBase implements BatchExecNode
...

class StreamExecCalc extends ExecNodeBase implements StreamExecNode
...
{code}
the advantage is: common class can be abstract class, its sub-classes will be 
cleaner; we can add some specific method definition in BatchExecNode and 
StreamExecNode, which is more extendable.
the disadvantage is: each specific node must extend from ExecNodeBase and 
Batch(/Stream)ExecNode.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20718) Create isCausingBackPressureRatio and isCausingBackPressure metrics

[Piotr Nowojski (Jira)]

Piotr Nowojski created FLINK-20718:
--

 Summary: Create isCausingBackPressureRatio and 
isCausingBackPressure metrics
 Key: FLINK-20718
 URL: https://issues.apache.org/jira/browse/FLINK-20718
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Metrics
Reporter: Piotr Nowojski
 Fix For: 1.13.0


create {{isCausingBackPressureRatio}} and {{isCausingBackPressure}} metrics. 
Based on either {{!isBackPressured && inPoolUsage > 0.5}} or {{!isBackPressured 
&& idleTimeMsPerSecond < 1ms}} . Probably the latter version is better, as it 
would neatly ignore the problem that {{inPoolUsage}} metric doesn't work with 
{{LocalInputChannel}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20717) Create isBackPressuredRatio metric

[Piotr Nowojski (Jira)]

Piotr Nowojski created FLINK-20717:
--

 Summary: Create isBackPressuredRatio metric
 Key: FLINK-20717
 URL: https://issues.apache.org/jira/browse/FLINK-20717
 Project: Flink
  Issue Type: Sub-task
Reporter: Piotr Nowojski
Assignee: Piotr Nowojski
 Fix For: 1.13.0


Create {{isBackPressuredRatio}} metric, based on a new {{GaugeView}} 
(equivalent of {{MeterView}} for {{Gauge}} that {{implements Meter, View}})



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20716) Pluggable shuffle service public interface IndexedInputGate mixed with Netty shuffle service implementation.

[Jin Xing (Jira)]

Jin Xing created FLINK-20716:


 Summary: Pluggable shuffle service public interface 
IndexedInputGate mixed with Netty shuffle service implementation.
 Key: FLINK-20716
 URL: https://issues.apache.org/jira/browse/FLINK-20716
 Project: Flink
  Issue Type: Bug
Reporter: Jin Xing






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Apache Flink Stateful Functions 2.2.2, release candidate #1

[Tzu-Li (Gordon) Tai]

It seems like the statefun-flink-datastream artifact is missing a valid
NOTICE file, since it bundles dependencies and builds a fat jar.
Any bundled dependencies in released artifacts need to be acknowledged in
the NOTICE file of the artifact.

-1

I'm hereby cancelling this RC due to this, and will create RC2 tomorrow
with the fix.
For people already testing this RC, this should not affect functionality,
so you may choose to continue testing as is.

Cheers,
Gordon

On Tue, Dec 22, 2020 at 2:45 PM Tzu-Li (Gordon) Tai 
wrote:

> Hi everyone,
>
> Please review and vote on the release candidate #1 for the version 2.2.2
> of Apache Flink Stateful Functions, as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> ***Testing Guideline***
>
> You can find here [1] a page in the project wiki on instructions for
> testing.
> To cast a vote, it is not necessary to perform all listed checks,
> but please mention which checks you have performed when voting.
>
> ***Release Overview***
>
> As an overview, the release consists of the following:
> a) Stateful Functions canonical source distribution, to be deployed to the
> release repository at dist.apache.org
> b) Stateful Functions Python SDK distributions to be deployed to PyPI
> c) Maven artifacts to be deployed to the Maven Central Repository
> d) New Dockerfiles for the release
> e) Release announcement to be published to the Flink website
>
> ***Staging Areas to Review***
>
> The staging areas containing the above mentioned artifacts are as follows,
> for your review:
> * All artifacts for a) and b) can be found in the corresponding dev
> repository at dist.apache.org [2]
> * All artifacts for c) can be found at the Apache Nexus Repository [3]
>
> All artifacts are signed with the key
> 1C1E2394D3194E1944613488F320986D35C33D6A [4]
>
> Other links for your review:
> * JIRA release notes [5]
> * source code tag "release-2.2.2-rc1" [6]
> * PR for the new Dockerfiles [7]
> * PR for the release announcement blog post [8]
>
> ***Vote Duration***
>
> The voting time will run for 72 hours, and will also take into account the
> Christmas holidays.
> I'm targeting this vote to last until Dec. 29th, 10am CET.
> It is adopted by majority approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Gordon
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Stateful+Functions+Release
> [2] https://dist.apache.org/repos/dist/dev/flink/flink-statefun-2.2.2-rc1/
> [3] https://repository.apache.org/content/repositories/org apache
> flink-1408/
> 
> [4] https://dist.apache.org/repos/dist/release/flink/KEYS
> [5]
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12349366
> [6] https://github.com/apache/flink-statefun/tree/release-2.2.2-rc1
> [7] https://github.com/apache/flink-statefun-docker/pull/12
> [8] https://github.com/apache/flink-web/pull/404
>

[jira] [Created] (FLINK-20715) pyflink DataStream filter error.

[Enter (Jira)]

Enter created FLINK-20715:
-

 Summary: pyflink DataStream filter error.
 Key: FLINK-20715
 URL: https://issues.apache.org/jira/browse/FLINK-20715
 Project: Flink
  Issue Type: Bug
  Components: API / DataStream, API / Python
Affects Versions: 1.12.0
 Environment: Flink 1.12
Reporter: Enter


```
 class MyFilterFunction(FilterFunction):

def filter(self, value):
 return value[0] % 2 == 0

def demo_stream():
 see = StreamExecutionEnvironment.get_execution_environment()
 see.set_parallelism(1)
 ds = see.from_collection([(1, 'Hi', 'Hello'), (2, 'Hello', 'Hi')],
 type_info=Types.ROW(
 [Types.INT(), Types.STRING(), Types.STRING()])
 )
 ds.filter(MyFilterFunction()).print()
 ds.print()
 # 执行任务;
 see.execute('job1')

if __name__ == '__main__':
 demo_stream()
 ```

 

    raise Py4JError(    raise Py4JError(py4j.protocol.Py4JError: An error 
occurred while calling o0.__getstate__. 
Trace:org.apache.flink.api.python.shaded.py4j.Py4JException: Method 
__getstate__([]) does not exist at 
org.apache.flink.api.python.shaded.py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)
 at 
org.apache.flink.api.python.shaded.py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326)
 at org.apache.flink.api.python.shaded.py4j.Gateway.invoke(Gateway.java:274) at 
org.apache.flink.api.python.shaded.py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
 at 
org.apache.flink.api.python.shaded.py4j.commands.CallCommand.execute(CallCommand.java:79)
 at 
org.apache.flink.api.python.shaded.py4j.GatewayConnection.run(GatewayConnection.java:238)
 at java.lang.Thread.run(Thread.java:748)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[DISCUSS] Allow streaming operators to use managed memory

[Jark Wu]

Hi all,

I found that currently the managed memory can only be used in 3 workloads
[1]:
- state backends for streaming jobs
- sorting, hash tables for batch jobs
- python UDFs

And the configuration option `taskmanager.memory.managed.consumer-weights`
only allows values: PYTHON and DATAPROC (state in streaming or algorithms
in batch).
I'm confused why it doesn't allow streaming operators to use managed memory
for purposes other than state backends.

The background is that we are planning to use some batch algorithms
(sorting & bytes hash table) to improve the performance of streaming SQL
operators, especially for the mini-batch operators.
Currently, the mini-batch operators are buffering input records and
accumulators in heap (i.e. Java HashMap) which is not efficient and there
are potential risks of full GC and OOM.
With the managed memory, we can fully use the memory to buffer more data
without worrying about OOM and improve the performance a lot.

What do you think about allowing streaming operators to use managed memory
and exposing it in configuration.

Best,
Jark

[1]:
https://ci.apache.org/projects/flink/flink-docs-master/deployment/memory/mem_setup_tm.html#managed-memory