Re: flink 1.4.2. java.lang.IllegalStateException: Could not initialize operator state backend

Thank You Andrey. Arity of the job has not changed. Here issue is that job will run for sometime (with checkpoint enabled) and then after some time will get into above exception. The job keeps restarting afterwards. One thing that I want point out here is that we have a custom *serialization

Re: Error While Initializing S3A FileSystem

Thanks, Ken. That makes sense! I'll start a new thread. On Wed, May 15, 2019 at 7:12 PM Ken Krugler wrote: > Hi Manish, > > It’s best to start a new thread if you have a new question - see > https://home.apache.org/~hossman/#threadhijack for reasons why… > > Regards, > > — Ken > > > On May 15,

Re: Error While Initializing S3A FileSystem

Hi Manish, It’s best to start a new thread if you have a new question - see https://home.apache.org/~hossman/#threadhijack for reasons why… Regards, — Ken > On May 15, 2019, at 4:46 PM, Manish Bellani wrote: > > Hi Ken, > > Thanks for the

Re: Error While Initializing S3A FileSystem

Hi Ken, Thanks for the quick response, you are actually right, the job seems to be running even after that error appears. It was crashing earlier (due to fs.s3a.multipart.size being too high) and I confused it with this error since that was the first one popping out and OOM wasn't apparent

Re: Error While Initializing S3A FileSystem

Hi Manish, Are you sure this is an exception that’s actually killing the job? Asking because https://issues.apache.org/jira/browse/BEANUTILS-477 talks about Commons Beanutils logging this exception, but it’s a warning vs. something being

Error While Initializing S3A FileSystem

hey Friends, Thanks for all the work you have been doing on flink, I have been trying to use BucketingSink (backed by S3AFileSystem) to write data to s3 and I'm running into some issues (which I suspect could be dependency/packaging related) that'd try to describe here. The data pipeline is

Applying multiple calculation on data aggregated on window

Hi, Im my environment I need to collect stream of messages into windows based on some fields as key and then I need to do multiple calculations that will apply on specaified messages. for example if i had the following messages on the window: {ts: 1, key: a, value: 10} {ts: 1, key: b, value: 0}

Re: Migrating Existing TTL State to 1.8

Hi Andrey, Thank you for the reply. We are using incremental checkpointing. Good to know that the incremental cleanup only applies to the heap state backend. Looks like taking some downtime to take a full savepoint and restore everything is inevitable. Thanks, -- Ning On Wed, 15 May 2019

Re: Getting java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError when stopping/canceling job.

So these are the two exceptions I see in the logs... Exception in thread "vert.x-worker-thread-0" Exception in thread "vert.x-internal-blocking-0" java.lang.NoClassDefFoundError: io/netty/util/concurrent/FastThreadLocal at

Flink with HBase

Hi, Does flink support exactly once processing using HBase as sink? I am not able to find any documentation supporting the same. Thanks Nikhil

User Interface not showing the actual count received and produced

Hi Flink team, I am developing a flow which uses · FlinkKafkaConsumer010 to consume message from Kafka and · FlinkKinesisProducer to produce the results to Kinesis. In the user interface, I always see Bytes and Record received from Kafka is zero even though it is receiving

Re: NPE in Flink 1.8.0

Hi We had the same issue. Make sure everything is using Flink 1.8 and not half on 1.7.2 and the other half on 1.8. Make sure to prune docker images and so on, even maven repo. Farouk Le mer. 15 mai 2019 à 18:08, Akshay Shinde a écrit : > Hi > > > We would highly appreciate any information

NPE in Flink 1.8.0

Hi We would highly appreciate any information on following stacktrace. Our flink job is simply writing data to Cassandra using cassandra sink and we are getting following exception - Caused by: java.lang.NullPointerException at

Re: Getting java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError when stopping/canceling job.

Hi John, could you share the full stack trace or better logs? It looks like something is trying to be executed in vertx.io code after the local task has been stopped and the class loader for the user code has been unloaded. Maybe from some daemon thread pool. Best, Andrey On Wed, May 15, 2019

Re: flink 1.4.2. java.lang.IllegalStateException: Could not initialize operator state backend

Hi, I am not sure that FLINK-8836 is related to the failure in the stack trace. You say you are using Flink in production, does it mean it always worked and has started to fail recently? >From the stack trace, it looks like the arity of some

Re: Table program cannot be compiled

Hi, I am looping in Timo and Dawid to look at the problem. On Tue, May 14, 2019 at 9:12 PM shkob1 wrote: > BTW looking at past posts on this issue[1] it should have been fixed? i'm > using version 1.7.2 > Also the recommendation was to use a custom function, though that's exactly > what im

Getting java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError when stopping/canceling job.

Hi, I'm using vertx.io as an async JDBC client for a RichAsyncFunction it works fine but when I stop the job I get... java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: io/vertx/core/impl/VertxImpl$SharedWorkerPool Is there a way to avoid/fix this?

FlinkSQL fails when rowtime meets dirty data

I use FlinkSQL to process Kafka data in the following format： | id | server_time | | 1 | 2019-05-15 10:00:00 | | 2 | 2019-05-15 10:00:00 | ... and I define rowtime from the server_time field: new Schema() .field("rowtime", Types.SQL_TIMESTAMP) .rowtime(new

答复: flink metrics的 Reporter 问题

好的，感谢 发件人: Xintong Song 发送时间: 2019年5月15日 21:17 收件人: user-zh@flink.apache.org 主题: Re: flink metrics的 Reporter 问题 取hostname的第一部分是为了和hdfs的用法保持一致，可以参考一下当时的issue，作者专门提到了为什么要这么做。

Re: Flink and Prometheus setup in K8s

Hi all, To answer my own questions I worked on the following solution: 1) Custom Docker image which pulls the Flink image and moves Prometheus jar to the correct folder [1, 2]. 2) I wrote manifests for Kubernetes with service discovery configuration for Kubernetes [3]. Besides the 'official'

答复: flink metrics的 Reporter 问题

Hi 唐云 我用的是flink1.8 感谢你的解答，我刚刚也找到了源码里面的[2]方式截取方式。目前来说，应该只能自己在report中，调用系统变量来获取主机名称了。 发件人: Yun Tang 发送时间: 2019年5月15日 21:11 收件人: user-zh@flink.apache.org 主题: Re: flink metrics的 Reporter 问题 Hi 嘉诚 不清楚你使用的Flink具体版本，不过这个显示host-name第一部分的逻辑是一直存在的，因为大部分场景下host-name只需要取第一部分即可表征。具体实现代码可以参阅 [1] 和 [2] 。

Re: flink metrics的 Reporter 问题

取hostname的第一部分是为了和hdfs的用法保持一致，可以参考一下当时的issue，作者专门提到了为什么要这么做。 https://issues.apache.org/jira/browse/FLINK-1170?focusedCommentId=14175285=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14175285 Thank you~ Xintong Song On Wed, May 15, 2019 at 9:11 PM Yun Tang wrote:

Re: flink metrics的 Reporter 问题

Hi 嘉诚 不清楚你使用的Flink具体版本，不过这个显示host-name第一部分的逻辑是一直存在的，因为大部分场景下host-name只需要取第一部分即可表征。具体实现代码可以参阅 [1] 和 [2] 。 受到你的启发，我创建了一个JIRA [3] 来追踪这个问题，解法是提供一个metrics options，使得你们场景下可以展示metrics的完整hostname 祝好 唐云 [1]

Checkpoints periodically fail with hdfs as the state backend - Could not flush and close the file system output stream

Hello, Every once in a while our checkpoints fail with the following exception: /AsynchronousException{java.lang.Exception: Could not materialize checkpoint 65912 for operator AGGREGATION-FILTER (2/2).} at

flink metrics的 Reporter 问题

大家好： 我按照官网的文档，调试了flink metrics 的 reporter ，加载了Slf4jReporter，这个Reporter运行是正常了，但是发现了个问题， 在taskManager中打印里面的信息的时候，打印出来的是： ambari.taskmanager.container_e31_1557826320302_0005_01_02.Status.JVM.ClassLoader.ClassesLoaded: 12044 这里的格式范围，我看了源码应该是.taskmanager..:

Re:Re: flink-kafka Trigger 无法触发问题

非常感谢您的答复。接下来按照您提供的思路进行排查 在 2019-05-15 16:23:04，"Terry Wang" 写道： >有可能是并行度设置大时，source的部分并发没有数据，导致eventTime未更新。可以排查下是否是这个问题 > >> 在 2019年5月15日，下午2:18，13341000780 <13341000...@163.com> 写道： >> >> hi, 各位大牛好！ >> 自定义了窗口触发器trigger，在onElement函数中注册了EventTimeTimer。出现了很离奇的问题，当并行度Parallelism >>

Re: flink-kafka Trigger 无法触发问题

有可能是并行度设置大时，source的部分并发没有数据，导致eventTime未更新。可以排查下是否是这个问题 > 在 2019年5月15日，下午2:18，13341000780 <13341000...@163.com> 写道： > > hi, 各位大牛好！ > 自定义了窗口触发器trigger，在onElement函数中注册了EventTimeTimer。出现了很离奇的问题，当并行度Parallelism >

Passing a custom SourceContext to a SourceFunction

Hi - I have a custom SourceFunction .. class MySourceFunction[T](data: Seq[T]) extends SourceFunction[T] { def run(ctx: SourceContext[T]): Unit = { data.foreach(d ⇒ ctx.collect(d)) } } When this function is run during job execution, the SourceContext that gets passed serializes the

flink-kafka Trigger 无法触发问题

hi, 各位大牛好！ 自定义了窗口触发器trigger，在onElement函数中注册了EventTimeTimer。出现了很离奇的问题，当并行度Parallelism 设置的比slots数和CPU核数小时，能成功触发onEventTime函数，当大于slots数或者大于CPU核数时，发现无法触发onEventTime，已确定元素能成功进入窗口，即onElement函数能成功触发。有人遇到过类似的问题吗，求解答。 非常感谢.

Re: Flink ML Use cases

Thanks a lot Rong and Sameer. Looks like this is what I wanted. I will try the above projects. *Regards,* *Abhishek Kumar Singh* *Search Engineer* *Mob :+91 7709735480 * *...* On Wed, May 15, 2019 at 8:00 AM Rong Rong wrote: > Hi Abhishek, > > Based on your description, I think this FLIP