Re: Timestamp(timezone) conversion bug in non blink Table/SQL runtime

lem to the same > internal function as you. > > Med venlig hilsen / Best regards > Lasse Nedergaard > > > Den 22. jul. 2019 kl. 20.46 skrev Shuyi Chen : > > Hi all, > > Currently, in the non-blink table/SQL runtime, Flink used > SqlFunctions.internalToTimestamp(lo

Timestamp(timezone) conversion bug in non blink Table/SQL runtime

Hi all, Currently, in the non-blink table/SQL runtime, Flink used SqlFunctions.internalToTimestamp(long v) from Calcite to convert event time (in long) to java.sql.Timestamp. However, as discussed in the recent Calcite mailing list (Jul. 19, 2019), SqlFunctions.internalToTimestamp() assumes the

Re: [ANNOUNCE] Rong Rong becomes a Flink committer

Congratulations, Rong! On Thu, Jul 11, 2019 at 8:26 AM Yu Li wrote: > Congratulations Rong! > > Best Regards, > Yu > > > On Thu, 11 Jul 2019 at 22:54, zhijiang wrote: > >> Congratulations Rong! >> >> Best, >> Zhijiang >> >> -- >>

Re: Flink HA cluster on YARN is restarted more than yarn.application-attempts value

AFAIR, your manual kill won't count towards the max-attempt counter in hadoop's logic. Please see this post for more details: http://johnjianfang.blogspot.com/2015/04/the-number-of-maximum-attempts-of-yarn.html . On Sun, Jun 2, 2019 at 9:48 AM 新平和礼 wrote: > Hi all, > > > I'm Flink newbie, and

Re: Backoff strategies for async IO functions?

Hi Konstantin, (cc Till since he owns the code) For async-IO, IO failure and retry is a common & expected pattern. In most of the use cases, users will need to deal with IO failure and retry. Therefore, I think it's better to address the problem in Flink rather than user implementing its custom

Re: Flink - excessive amount of yarn container requests for versions > 1.4.0

We've seen similar issue in our production, you can refer to this JIRA ( https://issues.apache.org/jira/browse/FLINK-10848) for more detail. Shuyi On Sun, Dec 9, 2018 at 11:27 PM sohimankotia wrote: > Hi , > > While running Flink streaming job it is requesting more than specified > resources

Re: Flink cluster security conf.: keberos.keytab add to run yarn-cluster

Do you mean have these two options as the command line options? If so, AFAIK, I dont think it's supported now. What do you need it? Thanks. On Wed, Oct 31, 2018 at 11:43 AM Marke Builder wrote: > Hi, > > So far I have added my keytab and principal in the flink-conf.yaml: >

Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem

also to FLINK-10556[1]. Please kindly review and share > your feedback. > > > Thanks, > Xuefu > > [1] https://issues.apache.org/jira/browse/FLINK-10556 > > -- > Sender:Xuefu > Sent at:2018 Oct 25 (Thu)

Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem

Welcome to the community and thanks for the great proposal, Xuefu! I think the proposal can be divided into 2 stages: making Flink to support Hive features, and make Hive to work with Flink. I agreed with Timo that on starting with a smaller scope, so we can make progress faster. As for [6], a

Re: HDFS HA issue in Flink 1.4 caused by failing to use fs.hdfs.hadoopconf in Flink conf

I think the bug is introduced in FLINK-7643 (Rework FileSystem loading to use factories). In YarnApplicationMasterRunner, after the JIRA, FileSystem was not properly initialized with the correct flink configuration before calling runApplicationMaster(). W/o the initialization, a call of

Re: AM Delegation Token Regeneration

Hi Paul, currently, Flink intentionally disables DT and only use keytab. I am not aware that DT regeneration is part of FLIP-6 (@till, correct me if I am wrong). I've created a security improvement design

Re: Ask for SQL using kafka in Flink

Given the popularity of Flink SQL and Kafka as streaming source, I think we can add some examples of using Kafka[XXX]TableSource in flink-examples/flink-examples-table module. What do you guys think? Cheers Shuyi On Mon, Jun 4, 2018 at 12:57 AM, Timo Walther wrote: > Hi, > > as you can see in

Re: is it OK to PR just for code comment errors?

I think this is classified as "hotifx" in the contributor list, no JIRA should be needed. *" Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for

Re: Ask about convert KafkaTableSource to table

Please take a look at the table api/SQL doc [1] below on how to register a table/tablesource. Thanks. [1] https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/table/common.html#register-a-table On Thu, May 31, 2018 at 6:22 AM, Radhya Sahal wrote: > Hi, > > Could you help me which is

Re: env.execute() ?

by command line (or a script) ? Or is it possible to do > internally in sync with application ? > > > > Esa > > > > *From:* Shuyi Chen > *Sent:* Wednesday, May 30, 2018 8:18 AM > *To:* Esa Heikkinen > *Cc:* Fabian Hueske ; user@flink.apache.org > *Subject:* Re: e

Re: env.execute() ?

Hi Esa, I think having more than one env.execute() is anti-pattern in Flink. env.execute() behaves differently depending on the env. For local, it will generate the flink job graph, and start a local mini cluster in background to run the job graph directly. For remote case, it will generate the

Re: Insert data into Cassandra without Flink Cassandra connection

Maybe you can share a bit more about why you need only one connection to Cassandra across all TaskManagers, so we can better help? On Wed, May 2, 2018 at 4:08 AM, Piotr Nowojski wrote: > Hi, > > The only way that I can think of is if you keep your flatMap operator with

Re: Need Help/Code Examples with reading/writing Parquet File with Flink ?

AFA I remember, there is no ParquetInputFormat in Flink. But there is a JIRA logged and an attempt in this PR , but was never merged. We do have an internal implementation that is being used in our

Re: How to rebalance a table without converting to dataset

Hi Darshan, thanks for raising the problem. We do have similar use of rebalancing in Flink SQL, where we want to rebalance the Kafka input with more partitions to increase parallelism in streaming. As Fabian suggests, rebalancing is not relation algebra. The closest use of the operation I can

Re: Flink Client job submission through SSL

pports Kerberos authentication for job submission through > command line client then how should i pass the kerberos credentials (keytab > and principal) to the flink client? > > On Fri, Apr 6, 2018 at 12:56 PM, Shuyi Chen <suez1...@gmail.com> wrote: > >> Hi Sampath, >> &

Re: Flink Client job submission through SSL

Hi Sampath, Yes, Flink support Kerberos authentication for job submission. You can take a look at the document here for more detail (https://ci.apache.org/ projects/flink/flink-docs-release-1.4/ops/security-kerberos.html). Also, please make sure to use Flink release 1.4.1 or above, because there

Re: Issue in Flink/Zookeeper authentication via Kerberos

On Mon, Apr 2, 2018 at 3:44 PM, Shuyi Chen <suez1...@gmail.com> wrote: > Hi Sarthak, > > Happy to help. Could you please share the jobmanager/taskmanager log and > flink conf again? > > Also, Flink 1.4.0 has a regression on kerberos security (keytab path in > Tas