[jira] [Created] (FLINK-21840) can't submit flink k8s session job with kubernetes.rest-service.exposed.type=NodePort

2021-03-16 Thread hayden zhou (Jira)
hayden zhou created FLINK-21840: --- Summary: can't submit flink k8s session job with kubernetes.rest-service.exposed.type=NodePort Key: FLINK-21840 URL: https://issues.apache.org/jira/browse/FLINK-21840 P

[jira] [Created] (FLINK-21839) SinkFunction snapshotState don't snapshot all data when trigger a stop-with-drain savepoint

2021-03-16 Thread Darcy Lin (Jira)
Darcy Lin created FLINK-21839: - Summary: SinkFunction snapshotState don't snapshot all data when trigger a stop-with-drain savepoint Key: FLINK-21839 URL: https://issues.apache.org/jira/browse/FLINK-21839

[jira] [Created] (FLINK-21838) Retranslate "JDBC SQL Connector" page into Chinese

2021-03-16 Thread jjiey (Jira)
jjiey created FLINK-21838: - Summary: Retranslate "JDBC SQL Connector" page into Chinese Key: FLINK-21838 URL: https://issues.apache.org/jira/browse/FLINK-21838 Project: Flink Issue Type: Task

[jira] [Created] (FLINK-21837) Support StreamExecIntervalJoin/StreamExecLookupJoin/StreamExecTemporalJoin json ser/des

2021-03-16 Thread Terry Wang (Jira)
Terry Wang created FLINK-21837: -- Summary: Support StreamExecIntervalJoin/StreamExecLookupJoin/StreamExecTemporalJoin json ser/des Key: FLINK-21837 URL: https://issues.apache.org/jira/browse/FLINK-21837 P

[jira] [Created] (FLINK-21836) Introduce RegexOperationConverter

2021-03-16 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-21836: - Summary: Introduce RegexOperationConverter Key: FLINK-21836 URL: https://issues.apache.org/jira/browse/FLINK-21836 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-21835) RocksDBStateBackendReaderKeyedStateITCase fail

2021-03-16 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-21835: - Summary: RocksDBStateBackendReaderKeyedStateITCase fail Key: FLINK-21835 URL: https://issues.apache.org/jira/browse/FLINK-21835 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-21834) org.apache.flink.core.fs.AbstractRecoverableWriterTest.testResumeWithWrongOffset fail

2021-03-16 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-21834: - Summary: org.apache.flink.core.fs.AbstractRecoverableWriterTest.testResumeWithWrongOffset fail Key: FLINK-21834 URL: https://issues.apache.org/jira/browse/FLINK-21834 Proj

[jira] [Created] (FLINK-21833) TemporalRowTimeJoinOperator State Leak Although configure idle.state.retention.time

2021-03-16 Thread lynn1.zhang (Jira)
lynn1.zhang created FLINK-21833: --- Summary: TemporalRowTimeJoinOperator State Leak Although configure idle.state.retention.time Key: FLINK-21833 URL: https://issues.apache.org/jira/browse/FLINK-21833 Pro

[jira] [Created] (FLINK-21832) Avro Confluent Schema Registry nightly end-to-end fail

2021-03-16 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-21832: - Summary: Avro Confluent Schema Registry nightly end-to-end fail Key: FLINK-21832 URL: https://issues.apache.org/jira/browse/FLINK-21832 Project: Flink Issue Type

Re: [DISCUSS] Split PyFlink packages into two packages: apache-flink and apache-flink-libraries

2021-03-16 Thread Xingbo Huang
Hi Chesnay, Yes, in most cases, we can indeed download the required jars in `setup.py`, which is also the solution I originally thought of reducing the size of wheel packages. However, I'm afraid that it will not work in scenarios when accessing the external network is not possible which is very c

[jira] [Created] (FLINK-21831) CLONE - Add ChangelogStateBackend (proxy-everything)

2021-03-16 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-21831: - Summary: CLONE - Add ChangelogStateBackend (proxy-everything) Key: FLINK-21831 URL: https://issues.apache.org/jira/browse/FLINK-21831 Project: Flink

Flink job cannot find recover path after using entropy injection for s3 file systems

2021-03-16 Thread Rainie Li
Hi Flink Developers. We enabled entropy injection for s3, here is our setting on Yarn Cluster. s3.entropy.key: _entropy_ s3.entropy.length: 1 state.checkpoints.dir: 's3a://{bucket name}/dev/checkpoints/_entropy_' I have two questions: 1. After enabling entropy, job's checkpoint path changed to: *

[jira] [Created] (FLINK-21830) [kinesis][auth] Add support for Session Token

2021-03-16 Thread Danny Cranmer (Jira)
Danny Cranmer created FLINK-21830: - Summary: [kinesis][auth] Add support for Session Token Key: FLINK-21830 URL: https://issues.apache.org/jira/browse/FLINK-21830 Project: Flink Issue Type: I

[jira] [Created] (FLINK-21829) create HiveCatalog with custom hadoopconfdir first

2021-03-16 Thread hehuiyuan (Jira)
hehuiyuan created FLINK-21829: - Summary: create HiveCatalog with custom hadoopconfdir first Key: FLINK-21829 URL: https://issues.apache.org/jira/browse/FLINK-21829 Project: Flink Issue Type: Wish

[jira] [Created] (FLINK-21828) Add an AWS-λ example to stateful-playground

2021-03-16 Thread Igal Shilman (Jira)
Igal Shilman created FLINK-21828: Summary: Add an AWS-λ example to stateful-playground Key: FLINK-21828 URL: https://issues.apache.org/jira/browse/FLINK-21828 Project: Flink Issue Type: Task

[jira] [Created] (FLINK-21827) Migrate the Python examples to the new SDK

2021-03-16 Thread Igal Shilman (Jira)
Igal Shilman created FLINK-21827: Summary: Migrate the Python examples to the new SDK Key: FLINK-21827 URL: https://issues.apache.org/jira/browse/FLINK-21827 Project: Flink Issue Type: Task

[jira] [Created] (FLINK-21826) Migrate the shopping-cart-example to the new SDK.

2021-03-16 Thread Igal Shilman (Jira)
Igal Shilman created FLINK-21826: Summary: Migrate the shopping-cart-example to the new SDK. Key: FLINK-21826 URL: https://issues.apache.org/jira/browse/FLINK-21826 Project: Flink Issue Type:

[jira] [Created] (FLINK-21825) AvroInputFormat doesn't honor -1 length on FileInputSplits

2021-03-16 Thread Nicolas Ferrario (Jira)
Nicolas Ferrario created FLINK-21825: Summary: AvroInputFormat doesn't honor -1 length on FileInputSplits Key: FLINK-21825 URL: https://issues.apache.org/jira/browse/FLINK-21825 Project: Flink

[jira] [Created] (FLINK-21824) Add a simple Json Kafka producer script

2021-03-16 Thread Igal Shilman (Jira)
Igal Shilman created FLINK-21824: Summary: Add a simple Json Kafka producer script Key: FLINK-21824 URL: https://issues.apache.org/jira/browse/FLINK-21824 Project: Flink Issue Type: Task

Re: [Parquet support]

2021-03-16 Thread Etienne Chauchot
Hi all, The fix (https://issues.apache.org/jira/browse/FLINK-21388) is now also available for flink 1.12 also (thanks Jingsong for merging the cherrypick PR) But before releasing 1.12 branch, I'd like this other PR to be merged: https://github.com/apache/flink/pull/15156 that introduces Par

[jira] [Created] (FLINK-21823) RocksDB on FreeBSD

2021-03-16 Thread Jira
Mårten Lindblad created FLINK-21823: --- Summary: RocksDB on FreeBSD Key: FLINK-21823 URL: https://issues.apache.org/jira/browse/FLINK-21823 Project: Flink Issue Type: Bug Components

[jira] [Created] (FLINK-21822) Migrate CatalogFactory to the new Factory stack

2021-03-16 Thread Jira
Ingo Bürk created FLINK-21822: - Summary: Migrate CatalogFactory to the new Factory stack Key: FLINK-21822 URL: https://issues.apache.org/jira/browse/FLINK-21822 Project: Flink Issue Type: Improve

[jira] [Created] (FLINK-21821) StreamTask.cancelables was closed concurrent between task canceler and task thread

2021-03-16 Thread Kezhu Wang (Jira)
Kezhu Wang created FLINK-21821: -- Summary: StreamTask.cancelables was closed concurrent between task canceler and task thread Key: FLINK-21821 URL: https://issues.apache.org/jira/browse/FLINK-21821 Projec

Re: [DISCUSS] Split PyFlink packages into two packages: apache-flink and apache-flink-libraries

2021-03-16 Thread Chesnay Schepler
This proposed apache-flink-libraries package would just contain the binary, right? And effectively be unusable to the python audience on it's own. Essentially we are just abusing Pypi for shipping a java binary. Is there no way for us to download the jars when the python package is being inst

Re: [DISCUSS] Split PyFlink packages into two packages: apache-flink and apache-flink-libraries

2021-03-16 Thread Dian Fu
Yes, the size of .whl file in PyFlink will also be about 3MB if we split the package. Currently the package is big because we bundled the jar files in it. > 2021年3月16日 下午8:13,Chesnay Schepler 写道: > > key difference being that the beam .whl files are 3mb large, aka 60x smaller. > > On 3/16/2021

Re: [DISCUSS] Split PyFlink packages into two packages: apache-flink and apache-flink-libraries

2021-03-16 Thread Chesnay Schepler
key difference being that the beam .whl files are 3mb large, aka 60x smaller. On 3/16/2021 1:06 PM, Dian Fu wrote: Hi Chesnay, We will publish binary packages separately for: 1) Python 3.5 / 3.6 / 3.7 / 3.8 (since 1.12) separately 2) Linux / Mac separately Besides, there is also a source pack

Re: [DISCUSS] Split PyFlink packages into two packages: apache-flink and apache-flink-libraries

2021-03-16 Thread Dian Fu
Hi Chesnay, We will publish binary packages separately for: 1) Python 3.5 / 3.6 / 3.7 / 3.8 (since 1.12) separately 2) Linux / Mac separately Besides, there is also a source package which is used when none of the above binary packages is usable, e.g. for Window users. PS: publishing multiple bi

[jira] [Created] (FLINK-21820) JDBC connector shouldn't read all rows in per statement by default

2021-03-16 Thread Leonard Xu (Jira)
Leonard Xu created FLINK-21820: -- Summary: JDBC connector shouldn't read all rows in per statement by default Key: FLINK-21820 URL: https://issues.apache.org/jira/browse/FLINK-21820 Project: Flink

Re: [DISCUSS] Split PyFlink packages into two packages: apache-flink and apache-flink-libraries

2021-03-16 Thread Xintong Song
And it's not only uploaded to PyPI, but the ASF mirrors as well. https://dist.apache.org/repos/dist/release/flink/flink-1.12.2/python/ Thank you~ Xintong Song On Tue, Mar 16, 2021 at 7:41 PM Xintong Song wrote: > Actually, I think it's 9 packages, not 7. > > Check here for the 1.12.2 packag

Re: [DISCUSS] Split PyFlink packages into two packages: apache-flink and apache-flink-libraries

2021-03-16 Thread Xintong Song
Actually, I think it's 9 packages, not 7. Check here for the 1.12.2 packages. https://pypi.org/project/apache-flink/#files Thank you~ Xintong Song On Tue, Mar 16, 2021 at 7:08 PM Chesnay Schepler wrote: > Am I reading this correctly that we publish 7 different artifacts just > for python? >

Re: [DISCUSS] Split PyFlink packages into two packages: apache-flink and apache-flink-libraries

2021-03-16 Thread Chesnay Schepler
Am I reading this correctly that we publish 7 different artifacts just for python? What does the release matrix look like? On 3/16/2021 3:45 AM, Dian Fu wrote: Hi Xingbo, Thanks a lot for bringing up this discussion. Actually the size limit already becomes an issue during releasing 1.11.3 an

Re: [DISCUSS] Removal of flink-swift-fs-hadoop module

2021-03-16 Thread Robert Metzger
Thanks a lot for your feedback! It seems that there was no objection to this proposal. I filed a ticket for removing the filesystem: https://issues.apache.org/jira/browse/FLINK-21819 On Wed, Jan 27, 2021 at 2:28 PM Till Rohrmann wrote: > +1, > > Cheers, > Till > > On Wed, Jan 27, 2021 at 11:03

[jira] [Created] (FLINK-21819) Remove swift FS filesystem

2021-03-16 Thread Robert Metzger (Jira)
Robert Metzger created FLINK-21819: -- Summary: Remove swift FS filesystem Key: FLINK-21819 URL: https://issues.apache.org/jira/browse/FLINK-21819 Project: Flink Issue Type: Task Com

[jira] [Created] (FLINK-21818) Refactor SlicingWindowAggOperatorBuilder to accept serializer instead of LogicalType

2021-03-16 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-21818: Summary: Refactor SlicingWindowAggOperatorBuilder to accept serializer instead of LogicalType Key: FLINK-21818 URL: https://issues.apache.org/jira/browse/FLINK-21818

[jira] [Created] (FLINK-21817) FLIP-27 Source might break subtask and split assignment upon rescale

2021-03-16 Thread Kezhu Wang (Jira)
Kezhu Wang created FLINK-21817: -- Summary: FLIP-27 Source might break subtask and split assignment upon rescale Key: FLINK-21817 URL: https://issues.apache.org/jira/browse/FLINK-21817 Project: Flink

[jira] [Created] (FLINK-21816) Support StreamExecGroupTableAggregate json ser/de

2021-03-16 Thread Wenlong Lyu (Jira)
Wenlong Lyu created FLINK-21816: --- Summary: Support StreamExecGroupTableAggregate json ser/de Key: FLINK-21816 URL: https://issues.apache.org/jira/browse/FLINK-21816 Project: Flink Issue Type: S

[jira] [Created] (FLINK-21815) Support StreamExecUnion json ser/de

2021-03-16 Thread Wenlong Lyu (Jira)
Wenlong Lyu created FLINK-21815: --- Summary: Support StreamExecUnion json ser/de Key: FLINK-21815 URL: https://issues.apache.org/jira/browse/FLINK-21815 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-21814) flink CollectSink error info unfriendly

2021-03-16 Thread Peihui He (Jira)
Peihui He created FLINK-21814: - Summary: flink CollectSink error info unfriendly Key: FLINK-21814 URL: https://issues.apache.org/jira/browse/FLINK-21814 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-21813) Support StreamExecOverAggregate json ser/de

2021-03-16 Thread Wenlong Lyu (Jira)
Wenlong Lyu created FLINK-21813: --- Summary: Support StreamExecOverAggregate json ser/de Key: FLINK-21813 URL: https://issues.apache.org/jira/browse/FLINK-21813 Project: Flink Issue Type: Sub-tas

[jira] [Created] (FLINK-21812) How to read data from kafka and sink to Postgresql Database

2021-03-16 Thread Bhagi (Jira)
Bhagi created FLINK-21812: - Summary: How to read data from kafka and sink to Postgresql Database Key: FLINK-21812 URL: https://issues.apache.org/jira/browse/FLINK-21812 Project: Flink Issue Type: Ta

[jira] [Created] (FLINK-21811) Support StreamExecJoin json serialization/deserialization

2021-03-16 Thread Terry Wang (Jira)
Terry Wang created FLINK-21811: -- Summary: Support StreamExecJoin json serialization/deserialization Key: FLINK-21811 URL: https://issues.apache.org/jira/browse/FLINK-21811 Project: Flink Issue T

[jira] [Created] (FLINK-21810) PyFlink end-to-end fails with "sorted[*]: unbound variable"

2021-03-16 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-21810: Summary: PyFlink end-to-end fails with "sorted[*]: unbound variable" Key: FLINK-21810 URL: https://issues.apache.org/jira/browse/FLINK-21810 Project: Flink

[jira] [Created] (FLINK-21809) PyFlink end-to-end test fail

2021-03-16 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-21809: - Summary: PyFlink end-to-end test fail Key: FLINK-21809 URL: https://issues.apache.org/jira/browse/FLINK-21809 Project: Flink Issue Type: Bug Components: