Re: Is Flink:1.5 Docker image broken?

Well those are unofficial, so you might raise a correspoinding issue on github (since the images are there) for that. On 31 May 2018 at 08:09, Chirag Dewan wrote: > Hi, > > flink:latest docker image doesn't seem to work. I am not able to access > the Flink Dashboard after deploying it on Kuberne

Is Flink:1.5 Docker image broken?

Hi, flink:latest docker image doesn't seem to work. I am not able to access the Flink Dashboard after deploying it on Kubernetes.   Anyone else facing the issue? Thanks, Chirag 

Re: NPE in flink sql over-window

Thanks for the replay. Yes, it only happen if I config the idle state retention times. The error occurs the first time before the first recovery. I haven't run with proctime but rowtime in flink 1.4.x. I am not sure if it will cause problems with proctime in 1.4.x. I am adding some trace log

Gluster as file system for state backend

Hi, I am evaluating some File Systems as state backend. I can see that Flink currently supports S3, MAPRFS and HDFS as file systems.  However, I was hoping I can use Gluster as my state backend, since its already a part of existing eco system. Since I have stateful operators in my job and I am e

Re: JVM metrics disappearing after job crash, restart

We keep track of metrics by using the value of MetricGroup::getMetricIdentifier, which returns the fully qualified metric name. The query that we use to monitor metrics filters for metrics IDs that match '%Status.JVM.Memory%'. As long as the new metrics come online via the MetricReporter interface

Re: Flink and AWS S3 integration: java.lang.NullPointerException: null uri host

Did you run Flink on AWS EMR or somewhere else? Have you read and followed instructions on https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/aws.html#amazon-web-services-aws ? On Wed, May 30, 2018 at 7:08 AM, Fabian Wollert wrote: > Hi, I'm trying to set up Checkpoints

Re: JVM metrics disappearing after job crash, restart

How are your metrics dimensionalized/named? Task managers often have UIDs generated for them. The task id dimension will change on restart. If you name your metric based on this 'task_id' there would be a discontinuity with the old metric. On Wed, May 30, 2018 at 4:49 PM, Nikolas Davis wrote: >

JVM metrics disappearing after job crash, restart

Howdy, We are seeing our task manager JVM metrics disappear over time. This last time we correlated it to our job crashing and restarting. I wasn't able to grab the failing exception to share. Any thoughts? We track metrics through the MetricReporter interface. As far as I can tell this more or l

Re: TimerService/Watermarks and Checkpoints

Thanks for the explanation. I looked at this metric closely and noticed there are some events arriving in out of order. The hypothesis I have is, when the job is restarted, all of the small out of order chunks add up and show a significant number. The graph below shows the number of out of order ev

Re: env.execute() ?

I think you might be looking for the functionality provided by the clusterclient [1]. But I am not sure if I fully understand the meaning of "do internally in sync with application". Maybe you can give a concrete use case, so we can help better, if the ClusterClient is not what you want. [1] https

Multiple Task Slots support in Flink 1.5

Hi! I came across the following point in release notes of 1.5 version: "The allocation of TaskManagers with multiple slots is not fully supported yet." Does this mean the support for it will come as a patc

Re: TimerService/Watermarks and Checkpoints

Hi Nara and Sihua, That's indeed an unexpected behavior and it would be good to identify the reason for the late data. As Sihua said, watermarks are currently not checkpointed and reset to Long.MIN_VALUE upon restart. AFAIK, the main reason why WMs are not checkpointed is that the special type of

Re: TimerService/Watermarks and Checkpoints

Thanks Sihua. If it's reset to Long.MIN_VALUE I can't explain why outOfOrderEvents are reported. Because the event time on the data will always be greater than Long.MIN_VALUE. Following are the steps to reproduce this scenario. - A source to produce events with timestamps that is increasing for ev

Use element of the DataStream in parameter of RichMapFunction (open function not called)

Hello, I'm working with Flink 1.4.2 (Scala API) and I'm having some trouble with my custom RichMapFunction as I want the element in my Datastream to also be used for a parameter of this custom class. My RichMapFunction is a simple counter based on a MapState Let's say I have those classes -

Flink and AWS S3 integration: java.lang.NullPointerException: null uri host

Hi, I'm trying to set up Checkpoints for Flink Jobs with S3 as a filesystem backend. I configured the following: state.backend=filesystem state.backend.fs.checkpointdir=s3:///mybucket/ state.checkpoints.dir=s3:///mybucket/ state.checkpoints.num-retained=3 I also copied the flink-s3-fs-hadoop-1.5.

ML in Streaming API

Hello and thanks for the subscription! I am using Streaming API to develop a ML algorithm and i would like your opinions regarding the following issues: *1)* The input is read from a big size file with d-dimensional points, and i want to perform a parallel count window. In each parallel count wi

[ANNOUNCE] Weekly community update #22

Dear community, I know it has been quite some time but here is the weekly community update thread #22. Please post any news and updates you want to share with the community to this thread. # Release 1.5 After a lot of work and good amount of testing, the community has finally released Apache Fli

Re: Task did not exit gracefully and lost TaskManager

met the same problem in 1.4 when I cancel job, one of taskmanager keep logging the exception -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

RE: env.execute() ?

Hi Ok. Thanks for the clarification. But the controlling of savepoints is only possible by command line (or a script) ? Or is it possible to do internally in sync with application ? Esa From: Shuyi Chen Sent: Wednesday, May 30, 2018 8:18 AM To: Esa Heikkinen Cc: Fabian Hueske ; user@flink.ap

Re: NPE in flink sql over-window

Hi, Dawid's analysis is certainly correct, but looking at the code this should not happen. I have a few questions: - You said this only happens if you configure idle state retention times, right? - Does the error occur the first time without a previous recovery? - Did you run the same query on Fl

Re: Build Cassandra Connector with some customization

directory: flink-connectors/flink-connector-cassandra command: mvn package -Ddriver.version=3.1.4 -Dguava.version=16.0.1 On 29.05.2018 14:41, Soheil Pourbafrani wrote: I want to build Flink Cassandra connector against datastax version 3.1.4 guava 16.0.1 using what command I can do that? and in

Re: REST API "broken" on YARN because POST is not allowed via YARN proxy

Thanks, Till! On Wed, May 30, 2018 at 10:39 AM, Till Rohrmann wrote: > Hi Juho, > > I created a JIRA issue for the problem [1]. Yes, the solution would be to > use only GET requests. Either we wrap our requests in a GET request or we > change our handlers to accept GET requests only. One thing w

Re: REST API "broken" on YARN because POST is not allowed via YARN proxy

Hi Juho, I created a JIRA issue for the problem [1]. Yes, the solution would be to use only GET requests. Either we wrap our requests in a GET request or we change our handlers to accept GET requests only. One thing which we have to fix first is that also the jar file upload goes through REST. [1

Re: Batch job stuck in Canceled state in Flink 1.5

Great to hear :-) On Tue, May 29, 2018 at 4:56 PM, Amit Jain wrote: > Thanks Till. `taskmanager.network.request-backoff.max` option helped in > my case. We tried this on 1.5.0 and jobs are running fine. > > > -- > Thanks > Amit > > On Thu 24 May, 2018, 4:58 PM Amit Jain, wrote: > >> Thanks! Ti

Re: NPE in flink sql over-window

Hi Yan, I think it is a bug in the ProcTimeBoundedRangeOver. It tries to access a list of elements that was already cleared and does not check against null. Could you please file a JIRA for that? Best, Dawid On 30/05/18 08:27, Yan Zhou [FDS Science] wrote: > > I also get warnning that CodeCa

Re: REST API "broken" on YARN because POST is not allowed via YARN proxy

Hi, I tried to search Flink Jira for this but couldn't find a ticket to match. If there's no ticket yet, did I understand correctly though, that you would be open to support GET as an alternative method for all of Flink's REST operations? Now that 1.5 was released, it's a pity that this problem ki