Re: `env.java.opts` not persisting after job canceled or failed and then restarted

Hey Aaron, I'm glad to hear that you resolved the issue. I think a docs contribution for this would be very helpful and could update this page: https://github.com/apache/flink/blob/master/docs/monitoring/debugging_classloading.md. If you want to create a separate JIRA ticket for this, ping me

Re: No resource available error while testing HA

Hi Gary, Thanks for the help. Gary Yao-3 wrote > You are writing that it takes YARN 10 minutes to restart the application > master (AM). However, in my experiments the AM container is restarted > within a > few seconds when after killing the process. If in your setup YARN actually > needs 10

Re: About KafkaConsumer and WM'ing and EventTime charactersitics

Hi Vishal May this doc[1] be helpful for you. [1] https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/connectors/kafka.html#kafka-consumers-and-timestamp-extractionwatermark-emission Best, Congxian Vishal Santoshi 于2019年1月30日周三 上午4:36写道： > It seems from >

Case When in Flink Table API

How can I use the correct way of *Case When *this example: myTlb.select( "o_orderdate.substring(0,4) as o_year, volume, (when(n_name.like('%BRAZIL%'),volume).else(0)) as case_volume" ) Flink errors on the line (when(n_name.like('%BRAZIL%'),volume).else(0)) as case_volume"

Re: `env.java.opts` not persisting after job canceled or failed and then restarted

Hi Ufuk, I'll answer your question, but first I'll give you an update on how we resolved the issue: * adding `org.apache.hadoop.io.compress.SnappyCodec` to `classloader.parent-first-patterns.additional` in `flink-conf.yaml` (though, putting `org.apache.hadoop.util.NativeCodeLoader` also worked)

How to save table with header

Hi, I can save tables in a CSV file like this: TableSink q6Sink = new CsvTableSink(SinkPath, ","); temp.writeToSink(q6Sink); but I want to save the table with the table header as the first line. Is it possible in Flink?

Re: How to load multiple same-format files with single batch job?

Hi, You can point a file-based input format to a directory and the input format should read all files in that directory. That works as well for TableSources that are internally use file-based input formats. Is that what you are looking for? Best, Fabian Am Mo., 28. Jan. 2019 um 17:22 Uhr

Re: About KafkaConsumer and WM'ing and EventTime charactersitics

It seems from https://ci.apache.org/projects/flink/flink-docs-stable/dev/event_time.html that iTimeCharacteristic.IngestionTime should do the trick. Just wanted to confirm that the ingestion time is the event time provided by the kafka producer. On Tue, Jan 29, 2019 at 3:21 PM Vishal Santoshi

About KafkaConsumer and WM'ing and EventTime charactersitics

In case where one needs t to use kafka event time ( ingestion time ) for watermark generation and timestamp extraction is setting EventTimeCharactersitic as EventTime enough ? Or is this explicit code required ? consumer.assignTimestampsAndWatermarks(new AssignerWithPunctuatedWatermarks() {

Re: Issue setting up Flink in Kubernetes

Hi Tim, There is an end-to-end test in the Flink repository that starts a job cluster in Kubernetes (minikube) [1]. If that does not help you, can you answer the questions below? What docker images are you using? Can you share the kubernetes resource definitions? Can you share the complete logs

Re: Select feilds in Table API

The problem is that the table "lineitem" does not have a field "l_returnflag". The field in "lineitem" are named [TMP_2, TMP_5, TMP_1, TMP_0, TMP_4, TMP_6, TMP_3]. I guess it depends on how you obtained lineitem. Best, Fabian Am Di., 29. Jan. 2019 um 16:38 Uhr schrieb Soheil Pourbafrani <

Re: Connector for IBM MQ

sorry for typo yep we developed few days back. On Tue, Jan 29, 2019 at 10:27 PM Puneet Kinra < puneet.ki...@customercentria.com> wrote: > Yep I did we days back. > > On Tue, Jan 29, 2019 at 10:13 PM wrote: > >> Hi all, >> >> >> >> I was wondering if anybody has anybody made a connector (Source)

Re: Connector for IBM MQ

Yep I did we days back. On Tue, Jan 29, 2019 at 10:13 PM wrote: > Hi all, > > > > I was wondering if anybody has anybody made a connector (Source) to be > used with IBM MQ? > > Also if somebody could point me to any doc on how to write a custom > connector, it would be appreciated. > > > > We

Connector for IBM MQ

Hi all, I was wondering if anybody has anybody made a connector (Source) to be used with IBM MQ? Also if somebody could point me to any doc on how to write a custom connector, it would be appreciated. We are using Scala 2.12 and Flink 1.7. Kind regards, Jacopo Gobbi Visit our website at

Re: Flink Yarn Cluster - Jobs Isolation

Run each job individually as described here: https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/deployment/yarn_setup.html#run-a-single-flink-job-on-yarn Yes they will run concurrently and be completely isolated from each other. -Jamie On Sun, Jan 27, 2019 at 6:08 AM Eran Twili

Re: Forking a stream with Flink

I think there is misunderstanding . I want to compare raw json and transformed record . Hence I need two consumer and merge the stream for comparison. I have pipeline defined . pipeline does source(kafka) ,transformation,dedup and persisting to DB . [image: image.png] Before reaching to DB task

Re: No resource available error while testing HA

Hi Averell, > Is there any way to avoid this? As if I run this as an AWS EMR job, the job > would be considered failed, while it is actually be restored automatically by > YARN after 10 minutes). You are writing that it takes YARN 10 minutes to restart the application master (AM). However, in my

Select feilds in Table API

Hi, I'm trying select some fields: lineitem .select( "l_returnflag," + "l_linestatus," + "l_quantity.sum as sum_qty," + "(l_extendedprice * (l_discount - 1)).sum as sum_disc_price," + "l_extendedprice.sum as

Filter Date type in Table API

Hi, I want to filter a field of type Date (Java.sql.Date) like the following: filter("f_date <= '1998-10-02'") and filter("f_date <= '1998/10/02'") Expression 'f_date <= 1998/10/02 failed on input check: Comparison is only supported for numeric types and comparable types of same type, got Date

Re: Forking a stream with Flink

Hi Selvaraj In your pojo add data member as status or something like that,now set it error in case it is invaild .pass the output of flatmap to split opertor there you can split the stream On Tue, Jan 29, 2019 at 6:39 PM Selvaraj chennappan < selvarajchennap...@gmail.com> wrote: > UseCase:- We

Flink Table API Sum method

Hi, How can I use the Flink Table API SUM function? For example something like this: table.agg(sum("feild1"))

Re: connecting two streams flink

I have pipeline defined . pipeline does source(kafka) ,transformation,dedup and persisting to DB . [image: image.png] Before reaching to DB task lots of transformation is applied on the pipeline Therefore want to validate the record with raw json message which is available in kafka with the

Re: connecting two streams flink

If c1 and c2 are listing to the same topic they will consume the same data . so i cant understand this > these two streams one(c2) is fast and other(c1) On Tue, Jan 29, 2019 at 2:44 PM Selvaraj chennappan < selvarajchennap...@gmail.com> wrote: > Team, > > I have two kafka consumer for

Re: Forking a stream with Flink

UseCase:- We have kafka consumer to read messages(json ) then it applies to flatmap for transformation based on the rules ( rules are complex ) and convert it to pojo . We want to verify the record(pojo) is valid by checking field by field of that record .if record is invalid due to

Re: Forking a stream with Flink

Im not sure if i got your question correctly, can you elaborate more on your use case

connecting two streams flink

Team, I have two kafka consumer for same topic and want to join second stream to first after couple of subtasks computation in the first stream then validate the record . KT - C1 ,C2 KT - C1 - Transformation(FlatMap) - Dedup - Validate --ifvalidsave it to DB -C2 - Process

Forking a stream with Flink

Hello Flink Community, Let's say I have multiple machines I get data from. I want to process the data of each machine seperately, but in the same way. Is it possible to "fork" a stream by some parameter and then processing the forks indepentently from each other, natively? Or do I need to do that

Re: Checkpoints failed with NullPointerException

Hi! Thanks for reporting this. This looks like a bug that we fixed in Flink 1.7.1 [1]. Would you be able to try with 1.7.1 and see if the issue is still happening for you? Cheers, Gordon [1] https://issues.apache.org/jira/browse/FLINK-11094 On Tue, Jan 29, 2019, 6:29 PM Averell I tried to

Re: Checkpoints failed with NullPointerException

I tried to create a savepoint on HDFS, and got the same exception: The program finished with the following exception: org.apache.flink.util.FlinkException: Triggering a savepoint for the job 028e392d02bd229ed08f50a2da5227e2 failed.

打包报scala版本问题 大佬们 怎么解决呢？

打包报scala版本问题 大佬们 怎么解决呢？ -- 发件人：Kurt Young 发送时间：2019年1月29日(星期二) 18:14 收件人：user-zh 抄　送：赵慧 主　题：Re: 欢迎来到 Apache Flink 社区 中文用户数量庞大呀~ Best, Kurt On Tue, Jan 29, 2019 at 6:10 PM Zhen Li wrote: 大家好， 从 Apache Flink

Re: KeyBy is not creating different keyed streams for different keys

Hi Harshith You can replace the GenericDataObject with Tuple3 and keyBy("A", "B") with keyBy(1, 2) then have a try. And you can see the doc[1] for reference also. [1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/api_concepts.html#define-keys-using-key-selector-functions Best,

Re: 欢迎来到 Apache Flink 社区

大家好， 从 Apache Flink 推特账号看到的消息，第一时间订阅了！✌️ 震 > 在 2019年1月29日，下午6:04，赵慧 写道： > > 开心～测试下是否成功进入中文邮件列表 > > > -- > 发件人：Shuai Xu > 发送时间：2019年1月29日(星期二) 18:02 > 收件人：user-zh > 主　题：Re: 欢迎来到 Apache Flink 社区 > > 欢迎大家 > >

Re: AssertionError: mismatched type $5 TIMESTAMP(3)

Thanks Timo, I didn't realise supplying Row could automatically apply the correct types. In this case your suggestion doesn't solve the problem though, I still get the exact same error. I assume that's because there isn't a time attribute type on the tradesByInstr table itself, but rather on

回复：欢迎来到 Apache Flink 社区

开心～测试下是否成功进入中文邮件列表 -- 发件人：Shuai Xu 发送时间：2019年1月29日(星期二) 18:02 收件人：user-zh 主　题：Re: 欢迎来到 Apache Flink 社区 欢迎大家 Congxian Qiu 于2019年1月29日周二 下午5:59写道： > 欢迎来到 Apache Flink 社区 > Best, > Congxian > > > Kurt Young 于2019年1月29日周二

Re: 欢迎来到 Apache Flink 社区

赞社区官方中文邮件列表。 Thanks, Zhuoluo  > 在 2019年1月29日，下午5:44，Kurt Young 写道： > > 欢迎来到 Apache Flink 社区 smime.p7s Description: S/MIME cryptographic signature

Re: 欢迎来到 Apache Flink 社区

欢迎大家 Congxian Qiu 于2019年1月29日周二 下午5:59写道： > 欢迎来到 Apache Flink 社区 > Best, > Congxian > > > Kurt Young 于2019年1月29日周二 下午5:57写道： > > > 继续测试一把 > > > > > > On Tue, Jan 29, 2019 at 5:44 PM Kurt Young wrote: > > > > > 欢迎来到 Apache Flink 社区 > > > > > >

Re: 欢迎来到 Apache Flink 社区

继续测试一把 On Tue, Jan 29, 2019 at 5:44 PM Kurt Young wrote: > 欢迎来到 Apache Flink 社区 >

Checkpoints failed with NullPointerException

Hi everyone, I am getting NullPointerException when the job is creating checkpoints. My configuration is: Flink 1.7.0 running on AWS EMR, using incremental RockDBStateBackEnd on S3. Sinks are parquet files on S3 and ElasticSearch (I'm not sure whether sinks are relevant to this error). There had

欢迎来到 Apache Flink 社区

欢迎来到 Apache Flink 社区

Re: AssertionError: mismatched type $5 TIMESTAMP(3)

Hi Chris, the exception message is a bit misleading. The time attribute (time indicator) type is an internal type and should not be used by users. The following line should solve your issue. Instead of: DataStream> tradesByInstrStream = tableEnv.toRetractStream(tradesByInstr, typeInfo);

Re: SQL Client (Streaming + Checkpoint)

Hi Vijay, in general Yun is right, the SQL Client is still in an early prototyping phase. Some configuration features are missing. You can track the progress of this feature here: https://issues.apache.org/jira/browse/FLINK-10265 It should be possible to use the global Flink configuration

大家好！

测试（test）

Re: date format in Flink SQL

Hi Soheil, the functions for date/time conversion are pretty limited so far. The full list of supported functions can be found here [1]. If you need more (which is usually the case), it is easy to implement a custom function [2]. We rely on Java's java.sql.Date as a data type. You can use

date format in Flink SQL

Hi, I want to convert a string in the format of 1996-8-01 to date and create Table from the dataset of Tuple3 at the end. Since I want to apply SQL queries on the date field of the table, for example, "date_column < 1996-8-01", which java format of date is supported in Flink?