[jira] [Created] (FLINK-19121) Avoid access HDFS in HiveBulkWriterFactory

2020-09-01 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-19121: Summary: Avoid access HDFS in HiveBulkWriterFactory Key: FLINK-19121 URL: https://issues.apache.org/jira/browse/FLINK-19121 Project: Flink Issue Type: New

Re: [DISCUSS] FLIP-136: Improve interoperability between DataStream and Table API

2020-09-01 Thread Jingsong Li
> a Row has two modes represented by an internal boolean flag `hasFieldOrder` +1 confusion with Dawid that what's the result when index-based setters and name-based setters are mixed used. And name-based setters look like append instead of set. It reminds me of Avro's `GenericRecord`, We should

Re: [DISCUSS] FLIP-136: Improve interoperability between DataStream and Table API

2020-09-01 Thread Danny Chan
Timo, Thanks for the discussion I have only read the "Conversion of DataStream to Table" part so i would only put some objections there ~ > StreamTableEnvironment.fromInsertStream(DataStream): Table At first glance, from the perspective of a user, i'm confused by why we must dintinguish on

Re: [DISCUSS] FLIP-136: Improve interoperability between DataStream and Table API

2020-09-01 Thread Jark Wu
Jingsong raised a good point. We need to be more careful when deprecating APIs. For example, tEnv#createTemporaryView was introduced in release-1.10, users became familiar with this API in the previous release, but now we want to deprecate it in the next release. I also have some concerns about

Re: [DISCUSS] FLIP-136: Improve interoperability between DataStream and Table API

2020-09-01 Thread Jingsong Li
Thanks Timo for driving. My first impression is, can we not deprecate these API? - StreamTableEnvironment.fromDataStream(DataStream): Table - StreamTableEnvironment.fromDataStream(DataStream, Expression...): Table - StreamTableEnvironment.createTemporaryView(String, DataStream, Expression...):

[jira] [Created] (FLINK-19120) Failed to build Flink with HBase2.1 even using the latest Flink release version.

2020-09-01 Thread Tang Yan (Jira)
Tang Yan created FLINK-19120: Summary: Failed to build Flink with HBase2.1 even using the latest Flink release version. Key: FLINK-19120 URL: https://issues.apache.org/jira/browse/FLINK-19120 Project:

[jira] [Created] (FLINK-19119) Update the documentation to use Expression instead of strings in the Python Table API

2020-09-01 Thread Dian Fu (Jira)
Dian Fu created FLINK-19119: --- Summary: Update the documentation to use Expression instead of strings in the Python Table API Key: FLINK-19119 URL: https://issues.apache.org/jira/browse/FLINK-19119 Project:

[jira] [Created] (FLINK-19118) Support Expression in the operations of the Python Table API

2020-09-01 Thread Dian Fu (Jira)
Dian Fu created FLINK-19118: --- Summary: Support Expression in the operations of the Python Table API Key: FLINK-19118 URL: https://issues.apache.org/jira/browse/FLINK-19118 Project: Flink Issue

[jira] [Created] (FLINK-19117) FLINK-18620

2020-09-01 Thread Xintong Song (Jira)
Xintong Song created FLINK-19117: Summary: FLINK-18620 Key: FLINK-19117 URL: https://issues.apache.org/jira/browse/FLINK-19117 Project: Flink Issue Type: Improvement Reporter:

[jira] [Created] (FLINK-19115) Null value fields cannot be ignored in ElasticSearch Sink

2020-09-01 Thread McClone (Jira)
McClone created FLINK-19115: --- Summary: Null value fields cannot be ignored in ElasticSearch Sink Key: FLINK-19115 URL: https://issues.apache.org/jira/browse/FLINK-19115 Project: Flink Issue Type:

[jira] [Created] (FLINK-19114) Introduce Expression class in the Python Table API

2020-09-01 Thread Dian Fu (Jira)
Dian Fu created FLINK-19114: --- Summary: Introduce Expression class in the Python Table API Key: FLINK-19114 URL: https://issues.apache.org/jira/browse/FLINK-19114 Project: Flink Issue Type:

[jira] [Created] (FLINK-19113) Add support for checkpointing with selectable inputs

2020-09-01 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-19113: - Summary: Add support for checkpointing with selectable inputs Key: FLINK-19113 URL: https://issues.apache.org/jira/browse/FLINK-19113 Project: Flink

[jira] [Created] (FLINK-19112) No access to metric group in ScalarFunction when optimizing

2020-09-01 Thread Nico Kruber (Jira)
Nico Kruber created FLINK-19112: --- Summary: No access to metric group in ScalarFunction when optimizing Key: FLINK-19112 URL: https://issues.apache.org/jira/browse/FLINK-19112 Project: Flink

Re: [DISCUSS] FLIP-141: Intra-Slot Managed Memory Sharing

2020-09-01 Thread Till Rohrmann
Thanks for creating this FLIP Xintong. I agree with the previous comments that the memory configuration should be as easy as possible. Every new knob has the potential to confuse users and/or allows him to shoot himself in the foot. Consequently, I am +1 for the first proposal in the FLIP since

Re: [DISCUSS] FLIP-136: Improve interoperability between DataStream and Table API

2020-09-01 Thread Timo Walther
Thanks for the healthy discussion Jark and Dawid. 6. "Row#setFieldNames(@Nullable Map fieldNames)" Yes, I'm concerned about about the per-record performance. A converter or serializer should prepare an immutable Map instance before (stored in a member variable) that is simply passed to every

[jira] [Created] (FLINK-19111) Flink Docker image creates checkpoints as root user and hits permission denied afterwards

2020-09-01 Thread Tobias Kaymak (Jira)
Tobias Kaymak created FLINK-19111: - Summary: Flink Docker image creates checkpoints as root user and hits permission denied afterwards Key: FLINK-19111 URL: https://issues.apache.org/jira/browse/FLINK-19111

Re: [DISCUSS] FLIP-136: Improve interoperability between DataStream and Table API

2020-09-01 Thread Jark Wu
Hi Timo, Thanks for the quick response. 5. "StreamStatementSet#attachToStream()" Joining or using connect() with a different DataStream is a good case. cc @Godfrey , what do you think about the `attachToStream()` API? 6. "Row#setFieldNames(@Nullable Map fieldNames)" > We need a Map for constant

Re: [DISCUSS] FLIP-136: Improve interoperability between DataStream and Table API

2020-09-01 Thread Dawid Wysakowicz
Hi all, I really like the ideas of this FLIP. I think it improves user experience quite a bit. I wanted to add just two comments: 1. As for the StatementSet I like the approach described in the FLIP for its simplicity. Moreover the way I see it is that if a user wants to work with DataStream,

Re: [DISCUSS] FLIP-141: Intra-Slot Managed Memory Sharing

2020-09-01 Thread Aljoscha Krettek
Hi, playing devils advocate here: should we even make the memory weights configurable? We could go with weights that should make sense for most cases in the first version and only introduce configurable weights when (if) users need them. Regarding where/how things are configured, I think

Re: [DISCUSS] FLIP-134: DataStream Semantics for Bounded Input

2020-09-01 Thread Aljoscha Krettek
Hmm, it seems I left out the Dev ML in my mail. Looping that back in.. On 28.08.20 13:54, Dawid Wysakowicz wrote: @Aljoscha Let me bring back to the ML some of the points we discussed offline. Ad. 1 Yes I agree it's not just about scheduling. It includes more changes to the runtime. We might

Re: [DISCUSS] FLIP-136: Improve interoperability between DataStream and Table API

2020-09-01 Thread Timo Walther
Hi Jark, thanks for the detailed review. Let me answer your concerns: ## Conversion of DataStream to Table 1. "We limit the usage of `system_rowtime()/system_proctime` to the leaf of a QueryOperation tree in the validation phase." I'm fine with allowing `system_proctime` everywhere in the

Re: Flink Speedcenter worker machine replaced

2020-09-01 Thread Till Rohrmann
Thanks for letting the community know Roman! Cheers, Till On Tue, Sep 1, 2020 at 10:57 AM Khachatryan Roman < khachatryan.ro...@gmail.com> wrote: > Hello, > > Yesterday the machine executing Flink benchmarks was replaced due to > hardware problems. > The HW configuration is different, so the

Re: FileSystemHaServices and BlobStore

2020-09-01 Thread Yang Wang
Hi Alexey, Glad to hear that your are interested the K8s HA support. Roman's answer is just on point. "FileSystemBlobStore" is trying to store the user jars, job graph, etc. on the distributed storage(e.g. HDFS, S3, GFS). So when the JobManager failover, it could fetch the blob data from remote

Flink Speedcenter worker machine replaced

2020-09-01 Thread Khachatryan Roman
Hello, Yesterday the machine executing Flink benchmarks was replaced due to hardware problems. The HW configuration is different, so the results may differ from what we had previously. Regards, Roman

Re: Next Stateful Functions Release

2020-09-01 Thread Igal Shilman
@Seth - Possibly, it turns out to be more tricky than originally thought, we will have it as a stretch goal for this release. On Mon, Aug 31, 2020 at 3:12 PM Seth Wiesman wrote: > +1 for Sept 10. Do you think we'd be able to get a fix for FLINK-18894 by > then? > >

[jira] [Created] (FLINK-19109) Split Reader eats chained periodic watermarks

2020-09-01 Thread David Anderson (Jira)
David Anderson created FLINK-19109: -- Summary: Split Reader eats chained periodic watermarks Key: FLINK-19109 URL: https://issues.apache.org/jira/browse/FLINK-19109 Project: Flink Issue

[jira] [Created] (FLINK-19108) Stop expanding the identifiers with scope aliased by the system with 'EXPR$' prefix

2020-09-01 Thread Danny Chen (Jira)
Danny Chen created FLINK-19108: -- Summary: Stop expanding the identifiers with scope aliased by the system with 'EXPR$' prefix Key: FLINK-19108 URL: https://issues.apache.org/jira/browse/FLINK-19108

[DISCUSS] FLIP-140: Introduce bounded style execution for keyed streams

2020-09-01 Thread Dawid Wysakowicz
Hi devs, As described in the FLIP-131[1] we intend to deprecate and remove the DataSet API in the future in favour of the DataStream API for both bounded/batch and unbounded/streaming jobs. Ideally, we should be able to stay in the same performance ballpark with bounded DataStream programs as

[jira] [Created] (FLINK-19107) Add basic checkpoint and recovery config keys to template flink-conf.yaml

2020-09-01 Thread Tzu-Li (Gordon) Tai (Jira)
Tzu-Li (Gordon) Tai created FLINK-19107: --- Summary: Add basic checkpoint and recovery config keys to template flink-conf.yaml Key: FLINK-19107 URL: https://issues.apache.org/jira/browse/FLINK-19107