[jira] [Created] (FLINK-15533) Writing DataStream as text file fails due to output path already exists

2020-01-08 Thread Rui Li (Jira)
Rui Li created FLINK-15533: -- Summary: Writing DataStream as text file fails due to output path already exists Key: FLINK-15533 URL: https://issues.apache.org/jira/browse/FLINK-15533 Project: Flink

[jira] [Created] (FLINK-15532) Enable strict capacity limit for memory usage for RocksDB

2020-01-08 Thread Yun Tang (Jira)
Yun Tang created FLINK-15532: Summary: Enable strict capacity limit for memory usage for RocksDB Key: FLINK-15532 URL: https://issues.apache.org/jira/browse/FLINK-15532 Project: Flink Issue

[jira] [Created] (FLINK-15531) Nightly build stalls in e2e-ha-scala-2.12 stage

2020-01-08 Thread Yu Li (Jira)
Yu Li created FLINK-15531: - Summary: Nightly build stalls in e2e-ha-scala-2.12 stage Key: FLINK-15531 URL: https://issues.apache.org/jira/browse/FLINK-15531 Project: Flink Issue Type: Bug

Re: [DISCUSS] Make AppendingState#add refuse to add null element

2020-01-08 Thread Yu Li
+1 for unifying the behavior to refusing adding null element. Nice catch and thanks for bringing up the discussion! Best Regards, Yu On Wed, 8 Jan 2020 at 22:50, Aljoscha Krettek wrote: > Hi, > > As I said in the discussion on the Jira issue, I’m in favour of this > change! > > This is the

[jira] [Created] (FLINK-15530) Replace process memory with flink memory for TMs in default flink-conf.yaml

2020-01-08 Thread Xintong Song (Jira)
Xintong Song created FLINK-15530: Summary: Replace process memory with flink memory for TMs in default flink-conf.yaml Key: FLINK-15530 URL: https://issues.apache.org/jira/browse/FLINK-15530 Project:

Re: [DISCUSS] Some feedback after trying out the new FLIP-49 memory configurations

2020-01-08 Thread Xintong Song
Thanks all for the discussion. I believe we have get consensus on all the open questions discussed in this thread. Since Andrey already create a jira ticket for renaming shuffle memory config keys with "taskmanager.memory.network.*", I'll create ticket for the other topic that puts flink.size in

Re: What is the suggested way to validate SQL?

2020-01-08 Thread godfrey he
hi kaibo, As we discuss offline, I think it's a clean way that flink-table provides an interface (or a tool) to do the sql validation for platform users. `tEnv.sqlUpdate` or `tEnv.explain(false)` is a temporary solution which contains too many unrelated logic (just consider the functionality

[jira] [Created] (FLINK-15529) Update upgrade compatibility table (docs/ops/upgrading.md) for 1.10.0

2020-01-08 Thread Yu Li (Jira)
Yu Li created FLINK-15529: - Summary: Update upgrade compatibility table (docs/ops/upgrading.md) for 1.10.0 Key: FLINK-15529 URL: https://issues.apache.org/jira/browse/FLINK-15529 Project: Flink

[jira] [Created] (FLINK-15528) Support higher precision of TIMESTAMP type as rowtime attribute

2020-01-08 Thread Jark Wu (Jira)
Jark Wu created FLINK-15528: --- Summary: Support higher precision of TIMESTAMP type as rowtime attribute Key: FLINK-15528 URL: https://issues.apache.org/jira/browse/FLINK-15528 Project: Flink Issue

[jira] [Created] (FLINK-15527) can not control the number of container on yarn single job module

2020-01-08 Thread chenchencc (Jira)
chenchencc created FLINK-15527: -- Summary: can not control the number of container on yarn single job module Key: FLINK-15527 URL: https://issues.apache.org/jira/browse/FLINK-15527 Project: Flink

Re: [DISCUSS] FLIP-92: JDBC catalog and Postgres catalog

2020-01-08 Thread Jingsong Li
Thanks Bowen for driving this, +1 for this, The DDL schema definition is a headache for users, and catalog is a solution to this problem. I have some questions and suggestions: - We can provide a Builder for Catalog, In my opinion, defaultDatabase, username, pwd can be included in JDBC DB url.

Re: [DISCUSS] FLIP-92: JDBC catalog and Postgres catalog

2020-01-08 Thread Jark Wu
Thanks Bowen for driving this. +1 to this feature. My concern is that why introducing a `PostgresJDBCCatalog`, not a generic `JDBCCatalog` (catalog.type = 'postgres' vs 'jdbc') ? >From my understanding, JDBC catalog is similar to JDBC source/sink. For JDBC source/sink, we have a generic

[jira] [Created] (FLINK-15526) Add RollingPolicy to PartitionWriters

2020-01-08 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-15526: Summary: Add RollingPolicy to PartitionWriters Key: FLINK-15526 URL: https://issues.apache.org/jira/browse/FLINK-15526 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-15525) HBase connector should use new type system to suppport precision/scale

2020-01-08 Thread Zhenghua Gao (Jira)
Zhenghua Gao created FLINK-15525: Summary: HBase connector should use new type system to suppport precision/scale Key: FLINK-15525 URL: https://issues.apache.org/jira/browse/FLINK-15525 Project:

Re: [DISCUSS] FLIP-72: Introduce Pulsar Connector

2020-01-08 Thread Bowen Li
Hi Yijie, There's just one more concern on the yaml configs. Otherwise, I think we should be good to go. Can you update your PR and ensure all tests pass? I can help review and merge in the next couple weeks. Thanks, Bowen On Mon, Dec 23, 2019 at 7:03 PM Yijie Shen wrote: > Hi Bowen, > >

[DISCUSS] FLIP-92: JDBC catalog and Postgres catalog

2020-01-08 Thread Bowen Li
Hi dev, I'd like to kick off a discussion on adding JDBC catalogs, specifically Postgres catalog in Flink [1]. Currently users have to manually create schemas in Flink source/sink mirroring tables in their relational databases in use cases like JDBC read/write and consuming CDC. Many users have

Re: [DISCUSS] FLIP-85: Delayed Job Graph Generation

2020-01-08 Thread tison
not always, Yang Wang is also not yet a committer but he can join the channel. I cannot find the id by clicking “Add new member in channel” so come to you and ask for try out the link. Possibly I will find other ways but the original purpose is that the slack channel is a public area we discuss

Re: [DISCUSS] FLIP-85: Delayed Job Graph Generation

2020-01-08 Thread Peter Huang
Hi Tison, I am not the committer of Flink yet. I think I can't join it also. Best Regards Peter Huang On Wed, Jan 8, 2020 at 9:39 AM tison wrote: > Hi Peter, > > Could you try out this link? https://the-asf.slack.com/messages/CNA3ADZPH > > Best, > tison. > > > Peter Huang 于2020年1月9日周四

[jira] [Created] (FLINK-15524) Some links in Chinese documents points to English documents by mistake

2020-01-08 Thread Yun Tang (Jira)
Yun Tang created FLINK-15524: Summary: Some links in Chinese documents points to English documents by mistake Key: FLINK-15524 URL: https://issues.apache.org/jira/browse/FLINK-15524 Project: Flink

Re: [DISCUSS] FLIP-85: Delayed Job Graph Generation

2020-01-08 Thread tison
Hi Peter, Could you try out this link? https://the-asf.slack.com/messages/CNA3ADZPH Best, tison. Peter Huang 于2020年1月9日周四 上午1:22写道: > Hi Tison, > > I can't join the group with shared link. Would you please add me into the > group? My slack account is huangzhenqiu0825. > Thank you in advance.

Re: [DISCUSS] FLIP-85: Delayed Job Graph Generation

2020-01-08 Thread Peter Huang
Hi Tison, I can't join the group with shared link. Would you please add me into the group? My slack account is huangzhenqiu0825. Thank you in advance. Best Regards Peter Huang On Wed, Jan 8, 2020 at 12:02 AM tison wrote: > Hi Peter, > > As described above, this effort should get attention

[jira] [Created] (FLINK-15523) ConfigConstants generally excluded from japicmp

2020-01-08 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-15523: Summary: ConfigConstants generally excluded from japicmp Key: FLINK-15523 URL: https://issues.apache.org/jira/browse/FLINK-15523 Project: Flink

Re: [DISCUSS] Make AppendingState#add refuse to add null element

2020-01-08 Thread Aljoscha Krettek
Hi, As I said in the discussion on the Jira issue, I’m in favour of this change! This is the Jira Issue, for reference: https://issues.apache.org/jira/browse/FLINK-15424 Best, Aljoscha > On 8. Jan 2020, at 15:16, Congxian Qiu wrote: > > Dear All > > > Currently, we found the

[jira] [Created] (FLINK-15522) Misleading root cause exception when cancelling the job

2020-01-08 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-15522: - Summary: Misleading root cause exception when cancelling the job Key: FLINK-15522 URL: https://issues.apache.org/jira/browse/FLINK-15522 Project: Flink

[jira] [Created] (FLINK-15521) TestUtils#copyDirectory should follow symbolic links

2020-01-08 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-15521: Summary: TestUtils#copyDirectory should follow symbolic links Key: FLINK-15521 URL: https://issues.apache.org/jira/browse/FLINK-15521 Project: Flink

[jira] [Created] (FLINK-15520) Prometheus E2E test should use DownloadCache

2020-01-08 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-15520: Summary: Prometheus E2E test should use DownloadCache Key: FLINK-15520 URL: https://issues.apache.org/jira/browse/FLINK-15520 Project: Flink Issue

[jira] [Created] (FLINK-15519) Preserve logs from BashJavaUtils and make them part of TM logs

2020-01-08 Thread Andrey Zagrebin (Jira)
Andrey Zagrebin created FLINK-15519: --- Summary: Preserve logs from BashJavaUtils and make them part of TM logs Key: FLINK-15519 URL: https://issues.apache.org/jira/browse/FLINK-15519 Project: Flink

[DISCUSS] Make AppendingState#add refuse to add null element

2020-01-08 Thread Congxian Qiu
Dear All Currently, we found the implementations of AppendingState#add are not the same, taking some as example: - HeapReducingState will clear state if add null element - RocksDBReducingState will add null element if serializer can serialize null - Both HeapListState and

[jira] [Created] (FLINK-15518) Don't hide web frontend side pane automatically

2020-01-08 Thread Aljoscha Krettek (Jira)
Aljoscha Krettek created FLINK-15518: Summary: Don't hide web frontend side pane automatically Key: FLINK-15518 URL: https://issues.apache.org/jira/browse/FLINK-15518 Project: Flink

Re: What is the suggested way to validate SQL?

2020-01-08 Thread Arvid Heise
A common approach is to add the connector jar as test dependencies and have a smoke test that just starts the job with a temporary external system spawned with docker. I usually use test containers [1]. Then you simply need to execute the integration tests in your IDE and usually can even debug

Re: [DISCUSS] Some feedback after trying out the new FLIP-49 memory configurations

2020-01-08 Thread Andrey Zagrebin
It also looks to me that we should only swap network and memory in the option names: 'taskmanager.memory.network.*'. There is no strong consensus towards using new 'shuffle' naming so we can just rename it to 'taskmanager.memory.network.*' as 'shuffle' naming has never been released. When we have

[jira] [Created] (FLINK-15517) Use back 'network' in 'shuffle' memory config option names

2020-01-08 Thread Andrey Zagrebin (Jira)
Andrey Zagrebin created FLINK-15517: --- Summary: Use back 'network' in 'shuffle' memory config option names Key: FLINK-15517 URL: https://issues.apache.org/jira/browse/FLINK-15517 Project: Flink

Re: [DISCUSS] Include flink-ml-api and flink-ml-lib in opt

2020-01-08 Thread Hequn Cheng
Hi Jincheng, Thanks a lot for your feedback! Yes, I agree with you. There are cases that multi jars need to be uploaded. I will prepare another discussion later. Maybe with a simple design doc. Best, Hequn On Wed, Jan 8, 2020 at 3:06 PM jincheng sun wrote: > Thanks for bring up this

[jira] [Created] (FLINK-15516) `ProcessingTimeSessionWindows` and `EventTimeSessionWindows` should be generic

2020-01-08 Thread zhihao zhang (Jira)
zhihao zhang created FLINK-15516: Summary: `ProcessingTimeSessionWindows` and `EventTimeSessionWindows` should be generic Key: FLINK-15516 URL: https://issues.apache.org/jira/browse/FLINK-15516

Re: [DISCUSS] FLIP-90: Support SQL 2016-2017 JSON functions in Flink SQL

2020-01-08 Thread Forward Xu
hi,Dann,Thanks you very much. Best, Forward Danny Chan 于2020年1月8日周三 下午2:25写道: > Thanks Forward ~ > > I would help to view code from the Calcite side. > > Best, > Danny Chan > 在 2020年1月7日 +0800 PM2:01,Forward Xu ,写道: > > Thanks Jark for checking the doc. hi,Timo, please help to check to see if

[jira] [Created] (FLINK-15515) Document that Hive connector should be used with blink planner

2020-01-08 Thread Rui Li (Jira)
Rui Li created FLINK-15515: -- Summary: Document that Hive connector should be used with blink planner Key: FLINK-15515 URL: https://issues.apache.org/jira/browse/FLINK-15515 Project: Flink Issue

[jira] [Created] (FLINK-15514) Bump ES6 connector version from 6.3.1 to 6.8.6

2020-01-08 Thread vinoyang (Jira)
vinoyang created FLINK-15514: Summary: Bump ES6 connector version from 6.3.1 to 6.8.6 Key: FLINK-15514 URL: https://issues.apache.org/jira/browse/FLINK-15514 Project: Flink Issue Type: Wish

[jira] [Created] (FLINK-15513) Error in SQL Reserved Keywords doc

2020-01-08 Thread Zou (Jira)
Zou created FLINK-15513: --- Summary: Error in SQL Reserved Keywords doc Key: FLINK-15513 URL: https://issues.apache.org/jira/browse/FLINK-15513 Project: Flink Issue Type: Improvement

Re: [DISCUSS] Some feedback after trying out the new FLIP-49 memory configurations

2020-01-08 Thread Stephan Ewen
About the name for the key "taskmanager.memory.network.*" or "taskmanager.memory.shuffle.*": This is a nice to have, but not super critical. If there ends up being no consensus, I can withdraw my concern. On Wed, Jan 8, 2020 at 4:01 AM Xintong Song wrote: > Thanks for the explanations @Stephan,

Re: [DISCUSS] FLIP-85: Delayed Job Graph Generation

2020-01-08 Thread tison
Hi Peter, As described above, this effort should get attention from people developing FLIP-73 a.k.a. Executor abstractions. I recommend you to join the public slack channel[1] for Flink Client API Enhancement and you can try to share you detailed thoughts there. It possibly gets more concrete