Re: Error submitting stand-alone Flink job to EMR YARN cluster

at 11:28 AM, Hanson, Bruce <bruce.han...@here.com> wrote: > I’m trying to submit a stand-alone Flink job to a YARN cluster running on > EMR (Elastic MapReduce) nodes in AWS. When it tries to start a container > for the Job Manager, it fails. The error message from the conta

Re: Error submitting stand-alone Flink job to EMR YARN cluster

, Bruce <bruce.han...@here.com> wrote: > I’m trying to submit a stand-alone Flink job to a YARN cluster running on > EMR (Elastic MapReduce) nodes in AWS. When it tries to start a container > for the Job Manager, it fails. The error message from the container is > below. The

Re: How to avoid breaking states when upgrading Flink job?

;>>> >>>>> I'm afraid I don't know how to solve this one right now, except by >>>>> switching to Java. >>>>> >>>>> Cheers, >>>>> Aljoscha >>>>> >>>>> On Thu, 30 Jun 2016 at

Re: How to avoid breaking states when upgrading Flink job?

ache.org> wrote: >>>> >>>>> Hi Josh, >>>>> >>>>> You have to assign UIDs to all operators to change the topology. Plus, >>>>> you have to add dummy operators for all UIDs which you removed; this >>>>> is a limita

Re: How to avoid breaking states when upgrading Flink job?

ators for all UIDs which you removed; this >>>> is a limitation currently because Flink will attempt to find all UIDs >>>> of the old job. >>>> >>>> Cheers, >>>> Max >>>> >>>> On Wed, Jun 29, 2016 at 9:00 PM, Jo

Error submitting stand-alone Flink job to EMR YARN cluster

I’m trying to submit a stand-alone Flink job to a YARN cluster running on EMR (Elastic MapReduce) nodes in AWS. When it tries to start a container for the Job Manager, it fails. The error message from the container is below. The command I’m using is: $ flink run -m yarn-cluster -yn 1 -ynm

Re: How to avoid breaking states when upgrading Flink job?

change the topology. Plus, >>> you have to add dummy operators for all UIDs which you removed; this >>> is a limitation currently because Flink will attempt to find all UIDs >>> of the old job. >>> >>> Cheers, >>> Max >>&g

Re: How to avoid breaking states when upgrading Flink job?

;> Cheers, >> Max >> >> On Wed, Jun 29, 2016 at 9:00 PM, Josh <jof...@gmail.com> wrote: >> > Hi all, >> > Is there any information out there on how to avoid breaking saved >> > states/savepoints when making changes to a Flink job and redeploying it? &g

Re: How to avoid breaking states when upgrading Flink job?

on how to avoid breaking saved > > states/savepoints when making changes to a Flink job and redeploying it? > > > > I want to know how to avoid exceptions like this: > > > > java.lang.RuntimeException: Failed to deserialize state handle and setup > > ini

Re: How to avoid breaking states when upgrading Flink job?

jof...@gmail.com> wrote: > Hi all, > Is there any information out there on how to avoid breaking saved > states/savepoints when making changes to a Flink job and redeploying it? > > I want to know how to avoid exceptions like this: > > java.lang.RuntimeException: Failed to deseria

How to avoid breaking states when upgrading Flink job?

Hi all, Is there any information out there on how to avoid breaking saved states/savepoints when making changes to a Flink job and redeploying it? I want to know how to avoid exceptions like this: java.lang.RuntimeException: Failed to deserialize state handle and setup initial operator state

Re: ClassCastException when redeploying Flink job on running cluster

gt;>> >>>> I'm not sure how to change my AvroDeserializationSchema to prevent this >>>> from happening though! Do you have any ideas? >>>> >>>> Josh >>>> >>>> >>>> >>>> On Wed, Jun 8, 2016 at 11:23 AM, Till Roh

Re: ClassCastException when redeploying Flink job on running cluster

ser code class >>>> loader). I suspect that this class is now somewhere cached (e.g. the avro >>>> serializer) and when you run your program a second time, then there is a >>>> new user code class loader which has loaded the same class and now you want >>&

Re: ClassCastException when redeploying Flink job on running cluster

ich has loaded the same class and now you want >>> to convert an instance of the first class into the second class. However, >>> these two classes are not identical since they were loaded by different >>> class loaders. >>> >>> In order to find the culprit,

Re: ClassCastException when redeploying Flink job on running cluster

gt;> AvroDeserializationSchema. I suspect that something similar to >> https://issues.apache.org/jira/browse/FLINK-1390 is happening. >> >> Cheers, >> Till >> >> On Wed, Jun 8, 2016 at 10:38 AM, Josh <jof...@gmail.com> wrote: >> >>> Hi

Re: ClassCastException when redeploying Flink job on running cluster

> https://issues.apache.org/jira/browse/FLINK-1390 is happening. > > Cheers, > Till > > On Wed, Jun 8, 2016 at 10:38 AM, Josh <jof...@gmail.com> wrote: > >> Hi all, >> >> Currently I have to relaunch my Flink cluster eve

Re: ClassCastException when redeploying Flink job on running cluster

jof...@gmail.com> wrote: > Hi all, > > Currently I have to relaunch my Flink cluster every time I want to > upgrade/redeploy my Flink job, because otherwise I get a ClassCastException: > > java.lang.ClassCastException: com.me.avro.MyAvroType cannot be cast to > com.me.avro.

ClassCastException when redeploying Flink job on running cluster

Hi all, Currently I have to relaunch my Flink cluster every time I want to upgrade/redeploy my Flink job, because otherwise I get a ClassCastException: java.lang.ClassCastException: com.me.avro.MyAvroType cannot be cast to com.me.avro.MyAvroType It's related to MyAvroType which is an class

Re: Visualize result of Flink job

<user@flink.apache.org> > *Dato:* Man, 30. maj 2016 12:31 > > *Emne:* Re: Visualize result of Flink job > > Hi Palle, > I think there is currently no way of sending the data from a streaming > Flink job into Zeppelin. > What rate / amount of data do you expect to send

Re: Visualize result of Flink job

k.apache.org <user@flink.apache.org> > Dato: Man, 30. maj 2016 12:31 > Emne: Re: Visualize result of Flink job > > Hi Palle, I think there is currently no way of sending the data from > a streaming Flink job into Zeppelin.What rate / amount of data do you > expect to send e

Re: Visualize result of Flink job

Hi Palle, I think there is currently no way of sending the data from a streaming Flink job into Zeppelin. What rate / amount of data do you expect to send every 10 seconds to the visualization tool? People have used Flink -> ES -> Kibana for this purpose in the past [1], but I think y

Re: Visualize result of Flink job

gt; > Til: user@flink.apache.org > Dato: Man, 30. maj 2016 08:20 > Emne: Re: Visualize result of Flink job > > I know exactly what to visualize. As I wrote, it is the latest > result of the Flink job I would like to visualize. There is no need > to use elastic to find

Re: Visualize result of Flink job

tion tools that work well with Flink? > ...where it is easy to export the result of a Flink job to a user interface > (web). > > Thanks > Palle

Visualize result of Flink job

view. And therefore I may also experience difficulties implementing the view in Elastic. Does anyone know of any other visualization tools that work well with Flink? ...where it is easy to export the result of a Flink job to a user interface (web). Thanks Palle

Re: How to test serializability of a Flink job

Hi Simone, do you have a stack trace for the error? Usually the user code serialization is the same locally and on a cluster. On Tue, Apr 5, 2016 at 12:02 PM, Simone Robutti < simone.robu...@radicalbit.io> wrote: > Hello, > > last week I got a problem where my job worked in local mode but could

Re: Flink Job History Dump

Hey Robert! This is currently not possible :-(, but this is a feature that is on Flink's road map. A very inconvenient workaround could be to manually query the REST APIs [1] and dump the responses somewhere and query it there. – Ufuk [1]

Flink Job History Dump

Hi everyone, I'm using Flink 0.10.2 to run some benchmarks on my cluster and I would like to compare it to Spark 1.6.0. Spark has an eventLog property that I can use to have the history written to HDFS, and then later view it offline on the History Server. Does Flink have a similar Feature,

How to test serializability of a Flink job

Hello, last week I got a problem where my job worked in local mode but could not be serialized on the cluster. I assume that local mode does not really serialize all the operators (the problem was with a custom map function) and I need to enforce this behaviour in local mode or, better, be able

Re: Unable to submit flink job that uses Avro data

> On Wed, 23.03.2016 06:59, Chesnay Schepler wrote > Could you be missing the call to execute()? Yes, that was it. Can't believe I missed that ! Thank you Chesnay. Best, Tarandeep On 23.03.2016 01:25, Tarandeep Singh wrote: >> Hi, >> >> I wrote a simple Flink job tha

Re: Unable to submit flink job that uses Avro data

Could you be missing the call to execute()? On 23.03.2016 01:25, Tarandeep Singh wrote: Hi, I wrote a simple Flink job that uses Avro input format to read avro file and save the results in avro format. The job does not get submitted and job client exist out immediately. Same thing happens

Re: Flink job on secure Yarn fails after many hours

; Envoyé : mardi 15 mars 2016 16:51 > À : user@flink.apache.org > Cc : Niels Basjes > Objet : Re: Flink job on secure Yarn fails after many hours > > Hi Thomas, > > Nils (CC) and I found out that you need at least Hadoop version 2.6.1 > to properly run Kerberos applications on Hadoop clust

Re: Flink job on secure Yarn fails after many hours

0 AM, Thomas Lamirault > <thomas.lamira...@ericsson.com> wrote: >> >> Hi Max, >> >> I will try these workaround. >> Thanks >> >> Thomas >> >> >> De : Maximilian Michels [m...@apache.org] >

Re: Flink job on secure Yarn fails after many hours

homas > > > > > > De : ni...@basj.es [ni...@basj.es] de la part de Niels Basjes > [ni...@basjes.nl] > Envoyé : vendredi 4 décembre 2015 10:40 > À : user@flink.apache.org > Objet : Re: Flink job on secure Yarn fails after many hours > > Hi

Re: Flink job on secure Yarn fails after many hours

Hi Niels, Just got back from our CI. The build above would fail with a Checkstyle error. I corrected that. Also I have built the binaries for your Hadoop version 2.6.0. Binaries: https://drive.google.com/file/d/0BziY9U_qva1sZ1FVR3RWeVNrNzA/view?usp=sharing Source:

Re: Flink job on secure Yarn fails after many hours

I mentioned that the exception gets thrown when requesting container status information. We need this to send a heartbeat to YARN but it is not very crucial if this fails once for the running job. Possibly, we could work around this problem by retrying N times in case of an exception. Would it be

Re: Flink job on secure Yarn fails after many hours

Hi Niels, You mentioned you have the option to update Hadoop and redeploy the job. Would be great if you could do that and let us know how it turns out. Cheers, Max On Wed, Dec 2, 2015 at 3:45 PM, Niels Basjes wrote: > Hi, > > I posted the entire log from the first log line at

Re: Flink job on secure Yarn fails after many hours

Hi, I posted the entire log from the first log line at the moment of failure to the very end of the logfile. This is all I have. As far as I understand the Kerberos and Keytab mechanism in Hadoop Yarn is that it catches the "Invalid Token" and then (if keytab) gets a new Kerberos ticket (or

Re: Flink job on secure Yarn fails after many hours

Great. Here is the commit to try out: https://github.com/mxm/flink/commit/f49b9635bec703541f19cb8c615f302a07ea88b3 If you already have the Flink repository, check it out using git fetch https://github.com/mxm/flink/ f49b9635bec703541f19cb8c615f302a07ea88b3 && git checkout FETCH_HEAD

Re: Flink job on secure Yarn fails after many hours

Sure, just give me the git repo url to build and I'll give it a try. Niels On Wed, Dec 2, 2015 at 4:28 PM, Maximilian Michels wrote: > I mentioned that the exception gets thrown when requesting container > status information. We need this to send a heartbeat to YARN but it is

Re: Flink job on secure Yarn fails after many hours

I forgot you're using Flink 0.10.1. The above was for the master. So here's the commit for Flink 0.10.1: https://github.com/mxm/flink/commit/a41f3866f4097586a7b2262093088861b62930cd git fetch https://github.com/mxm/flink/ \ a41f3866f4097586a7b2262093088861b62930cd && git checkout FETCH_HEAD

Re: Flink job on secure Yarn fails after many hours

Hi Niels, Sorry for hear you experienced this exception. From a first glance, it looks like a bug in Hadoop to me. > "Not retrying because the invoked method is not idempotent, and unable to > determine whether it was invoked" That is nothing to worry about. This is Hadoop's internal retry

Re: starting flink job from bash script with maven

Thanks for letting us know! The problem with Java Serialization is that they often swallow exceptions and you only see a corrupted byte stream in the end. So far, I have found no workaround for that. Stephan On Fri, Jul 24, 2015 at 11:31 AM, Stefano Bortoli s.bort...@gmail.com wrote: It

Re: Monitoring memory usage of a Flink Job

memory usage. Cheers, Fabian 2015-06-15 14:34 GMT+02:00 Tamara Mendt tammyme...@gmail.com: Hi, I am running some experiments on Flink and was wondering if there is some way to monitor the memory usage of a Flink Job (running locally and on a cluster). I need to run multiple jobs and compare

Re: Monitoring memory usage of a Flink Job

logs, but I'm not sure about that. Besides logs, I doubt that Flink monitors memory usage. Cheers, Fabian 2015-06-15 14:34 GMT+02:00 Tamara Mendt tammyme...@gmail.com: Hi, I am running some experiments on Flink and was wondering if there is some way to monitor the memory usage of a Flink

Re: Monitoring memory usage of a Flink Job

tammyme...@gmail.com: Hi, I am running some experiments on Flink and was wondering if there is some way to monitor the memory usage of a Flink Job (running locally and on a cluster). I need to run multiple jobs and compare their memory usage. Cheers, Tamara ​ -- Tamara Mendt

Monitoring memory usage of a Flink Job

Hi, I am running some experiments on Flink and was wondering if there is some way to monitor the memory usage of a Flink Job (running locally and on a cluster). I need to run multiple jobs and compare their memory usage. Cheers, Tamara

Re: Monitoring memory usage of a Flink Job

Tamara Mendt tammyme...@gmail.com: Hi, I am running some experiments on Flink and was wondering if there is some way to monitor the memory usage of a Flink Job (running locally and on a cluster). I need to run multiple jobs and compare their memory usage. Cheers, Tamara

Re: JSON data source for Flink Job

the strings and parse them to JSON with your favorite library (like Jackson or so). Stephan On Thu, May 28, 2015 at 12:24 PM, Tamara Mendt tammyme...@gmail.com wrote: Hello, I have a JSON file containing multiple JSON objects and wish to use this as a data source for a Flink Job. What is the best

EOFException when running Flink job

the 0.9.0-milestone-1 release. Best, Stefan [1] Flink job: https://gist.github.com/knub/0bfec859a563009c1d57 [2] Job manager logs: https://gist.github.com/knub/01e3a4b0edb8cde66ff4 [3] One task manager's logs: https://gist.github.com/knub/8f2f953da95c8d7adefc

Re: EOFException when running Flink job

jobs. I already ran the Flink program that generated them several times: the error persists. You can find the logs at [2] and [3]. I am using the 0.9.0-milestone-1 release. Best, Stefan [1] Flink job: https://gist.github.com/knub/0bfec859a563009c1d57 [2] Job manager logs: https

<    4   5   6   7   8   9