Re: spark vs flink batch performance

Hi all, In the mean time i have three workers. Any thoughts about improving flink performance? Thank you... On Nov 17, 2016 00:38, "CPC" wrote: > Hi all, > > I am trying to compare spark and flink batch performance. In my test i am > using ratings.csv in http://files.grouplens.org/ > datasets/

Re: Why use Kafka after all?

Just to be clear, what I'm looking for is a way to serialize a POJO class for Kafka but also for Flink, I'm not sure the interface of both frameworks are compatible but it seems they aren't. For Kafka (producer) I need a Serializer and a Deserializer class, and for Flink (consumer) a Serialization

Re: Enforce an operation to run in an exact host in flink

Yeah, I need to process latency. Any suggestions? On Nov 18, 2016 12:47 AM, "Aljoscha Krettek" wrote: > Hi, > I'm afraid that it is currently not possible to specify the machine on > which a Flink operator should run. What do you need this for? Are you > trying to measure processing latency? > >

FLINK-2821 - Flink on Kubernetes

Hi Max, I am running into an issue on running flink on Kubernetes – basically during submission of a job. It looks like akka related. I then ran into this Jira FLINK-2821. I am not using flink directly , but I embed flink streaming and runtime jars as part a Java app. Would you have a Git PR whic

Re: Why use Kafka after all?

Tzu-Li Tai, thanks for your response. I've seen the example you mentioned before, TaxiRideSchema.java, but it's way too simplified. In a real POJO class you may have multiple fields such as integers, strings, doubles, etc. So serializing them as a string like in the example wouldn't work (you can

Regarding time window based on the values received in the stream

Hi, I have a requirement for my thesis project where I need to set the time window based on the value which I received in the event stream. I need to collect the values when the value starts at 0 and increments to a max value(which is unknown) till the value resets to 0 again on which a trigger i

Re: add FLINK_LIB_DIR to classpath on yarn -> add properties file to class path on yarn

The JVM only accepts Jar files in the classpath. You will have to load your custom files from the working directory of the node where the lib directory is shipped. By the way, the /lib directory is meant for Jar files. If you want to ship a custom file, it's better to add it to the ship files in t

Cross product of datastream and dataset

We're having trouble mapping our problem to Flink. - For each incoming item - Generate tuples of the item crossed with a data set - Filter the tuples based on a condition - Know the count of matching tuples This seems to be mashup of DataStream and DataSet, but it appears you can't operate with t

fileInpuFormat of type json

Hello, Does the Flink Scala api have a method for reading json files? If not, are there any examples on how this is accomplished? Thanks -daviD

Re: Type of TypeVariable 'K' in 'class <> could not be determined

The one that's currently in my github will give you the error, In my other file i made a really ugly workaround by adding the element in an ArrayList as a single item. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Type-of-TypeVariable-K-in

Re: Enforce an operation to run in an exact host in flink

Hi, I'm afraid that it is currently not possible to specify the machine on which a Flink operator should run. What do you need this for? Are you trying to measure processing latency? Cheers, Aljoscha On Thu, 17 Nov 2016 at 09:15 Eranga Heshan wrote: > Any help please? > > > Eranga Heshan > *Und

Re: Could not find job with id XXXXX

No that should not cause the job manager to fail. Do you have the complete job manager logs available to further look into this? – Ufuk On 17 November 2016 at 13:39:47, Sendoh (unicorn.bana...@gmail.com) wrote: > Thank you for your reply. > It sounds for me should not be the error that causing j

Re: Type of TypeVariable 'K' in 'class <> could not be determined

Sorry i already pushed a new update, But in testclass.java if you change line 266 to: tempgraphdoubles.run(new SingleSourceShortestTemporalPathEAT3(maxIterations)).print(); And then run the testclass.java You should get the error -- View this message in context: http://apache-flink-user-mail

Re: Type of TypeVariable 'K' in 'class <> could not be determined

I checked out your repository. Which test do I have to run in order to get the error? Am 17/11/16 um 14:33 schrieb otherwise777: Hey Vasia, I made this simple mapper to illustrate the problem, the file i'm working on is here: https://github.com/otherwise777/Temporal_Graph_library/blob/master/s

Re: Maintaining watermarks per key, instead of per operator instance

Hi Fabian, your proposed solution for: Multiple window aggregations You can construct a data flow of cascading window operators and fork off (to emit or further processing) the result after each window. Input -> window(15 secs) -> window(1 min) -> window(15 min) -> ...

Re: add FLINK_LIB_DIR to classpath on yarn -> add properties file to class path on yarn

Hi Robert, Sorry for the late response. I checked the TM and JM logs but am not able to see that file added in the classpath. I have deployed Flink on EMR, and placed the yaml file in /usr/lib/flink/lib folder. What am I doing wrong here ? Regards, Vinay Patil On Wed, Oct 26, 2016 at 1:25 PM,

Re: Type of TypeVariable 'K' in 'class <> could not be determined

Hey Vasia, I made this simple mapper to illustrate the problem, the file i'm working on is here: https://github.com/otherwise777/Temporal_Graph_library/blob/master/src/main/java/Tgraphs/SingleSourceShortestTemporalPathEATBetweenness.java

Re: Type of TypeVariable 'K' in 'class <> could not be determined

Hi Wouter, with InitVerticesMapper() are you trying to map the vertex value to a Tuple2 or to a Double? Your mapper is turning the vertex values into a Tuple2<> but your scatter-gather UDFs are defining Double vertex values. -Vasia. On 17 November 2016 at 14:03, otherwise777 wrote: > Hello tim

Re: Type of TypeVariable 'K' in 'class <> could not be determined

Hello timo, the whole project is on github: https://github.com/otherwise777/Temporal_Graph_library The Tgraphalgorithm is here: https://github.com/otherwise777/Temporal_Graph_library/blob/master/src/main/java/Tgraphs/TGraphAlgorithm.java

Re: Type of TypeVariable 'K' in 'class <> could not be determined

Hi, Flink uses the signature of the class to determine the return types of a function ("class MyClass implements MapFunction"). You always have to keep in mind that every generic in a "new ()" gets erasured by Java to "new ()". So variable K has to be determined by the input type (whic

Type of TypeVariable 'K' in 'class <> could not be determined

I get this error: *Exception in thread "main" org.apache.flink.api.common.functions.InvalidTypesException: Type of TypeVariable 'K' in 'class Tgraphs.SingleSourceShortestTemporalPathEAT3$InitVerticesMapper' could not be determined. This is most likely a type erasure problem. The type extraction cu

Re: Could not find job with id XXXXX

Thank you for your reply. It sounds for me should not be the error that causing job manager down? Or it can? Currently we use 1.1.3. Best, Sendoh -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Could-not-find-job-with-id-X-tp10170p101

Re: Could not find job with id XXXXX

That's most probably due to a job that has already terminated, but you still have a browser open querying the job manager for the job. The log level for this has been recently reduced to DEBUG (for both upcoming Flink 1.1.4 and 1.2.0). If you are not explicitly missing a job in the web UI that

Could not find job with id XXXXX

Hi Flink users, Suddenly I discovered this "Could not find job with id". What would be the possible causes for this? It would be good to know the Job name of that job id but I cannot neither go to web UI nor use ./bin/flink list 2016-11-16 16:26:21,276 WARN org.apache.flink.runtime.webmonitor.Ru

Docker

Hi, It would be really nice to have an official docker image (e.g. https://github.com/apache/flink/tree/master/flink-contrib/docker-flink ). Or a least to have a regular image/build on docker hub best, Stephan

Re: Error handling

Hi, Here is the code which triggers the error(part of sink): @Override public void invoke(KafkaLog value) throws Exception { .. if (arg instanceof String && "error".equals((String)arg)) { throw new IOException("search for error"); }

Re: Enforce an operation to run in an exact host in flink

Any help please? Eranga Heshan *Undergraduate* Computer Science & Engineering University of Moratuwa Mobile: +94 71 138 2686 <%2B94%2071%20552%202087> Email: era...@wso2.com