Re: Writing test for Flink streaming jobs

Thank you for your response. flink-spector looks really nice. I tried but got some errors regarding types, maybe because of the thing Alex mentioned. I am looking forward to the new version. Thanks, Hironori. 2016-05-30 16:45 GMT+09:00 lofifnc : > Hi, > > Flinkspector is indeed a good choice to

Re: S3 as streaming source

Hi all, I think we can use `readFile`, `readFileStream` methods in `StreamExecutionEnvironment` to create streaming source from S3 because data are stored as file in S3. But I haven’t test it. Regards, Chiwan Park > On Jun 3, 2016, at 2:37 PM, Tzu-Li (Gordon) Tai wrote: > > Hi Soumya, > > N

Re: S3 as streaming source

Hi Soumya, No, currently there is no Flink standard supported S3 streaming source. As far as I know, there isn't one out in the public yet either. The community is open to submissions for new connectors, so if you happen to be working on one for S3, you can file up a JIRA to let us know. Also, ar

S3 as streaming source

Is there a standard Flink S3 source yet? Thanks -Soumya

Re: Submitting job via Flink web interface giving Null pointer exception

Hi Ritesh, I tried to run a local flink using the same script and submitted a job successfully via Web interface and command line both. I suggest to try one of the flink examples. Streaming Job - https://ci.apache.org/projects/flink/flink-docs-release-1.0/quickstart/run_example_quickstart.html B

Re: Result comparison from 2 DataStream Sources

Hi again! Thanks for your tips and gists, those are being really helpful. However, I probably didn't express my idea properly and it has been a litle misunderstood. I have been thinking about how to do this during these days and I will try to put a concrete example of what I want and if my way is t

Re: Adding Context To Logs

the job name is straight up not accessible from within a function. Until recently it wasn't even accessible /anywhere/ on the TaskManager. To get the job name in a function you would have to pass it from the TaskManager to the task, to the operator and (probably) to the RuntimeContext. If you

Re: Adding Context To Logs

Hello Paul, If I understand correctly, your issues seem to be related to: https://issues.apache.org/jira/browse/FLINK-1502?jql=assignee%20in%20(Zentol)%20AND%20text%20~%20%22metrics%22

Re: Maintaining watermarks per key, instead of per operator instance

Hi Aljoscha! Is it possible somehow to use the RichXFunction in CEP? The task is pretty similar, but I have to ignore once the next triggered event for the same key. On Wed, Jun 1, 2016 at 2:54 PM, Aljoscha Krettek wrote: > Hi, > yeah, in that case per-key watermarks would be useful for you. I w

Adding Context To Logs

Hi, We are trying to rework the way our Flink jobs (and really it becomes how the operators within the jobs) log. Specifically we are trying to add contextual information to the log messages so that it is easier to filter and search for a specific jobs logs after they have been send through to

Re: Flink Kafka Consumer throwing Null Pointer Exception

I just wrote an answer for this on Stackoverflow: The problem is in this function: @Override public TypeInformation getProducedType() { // TODO Auto-generated method stub return null; } you cannot return null here. On Thu, 2 Jun 2016 at 17:57 Akshay Thaka Shingote wrote: > Hello, > >

Re: Exception in thread main: No such exception errpr

Yeah i will do that as a last resort. But the strange fact is in IDE it is working fine. Although I am not entirely aware of the mechanics on how flink is simulated on IDE. On Thu, 2 Jun 2016 17:35 Kostas Kloudas, wrote: > The reason seems to be that somehow an older version of Flink is still in

Re: Exception in thread main: No such exception errpr

The reason seems to be that somehow an older version of Flink is still in your dependencies. As you can see here: https://issues.apache.org/jira/browse/FLINK-3306 at 0.10 the method you cannot find had 2 arguments and after this commit it has 3

Re: Exception in thread main: No such exception errpr

Yeah I think so. Here is a part of log file. 2016-06-02 18:37:27,376 INFO org.apache.flink.runtime.jobmanager.JobManager- 2016-06-02 18:37:27,377 INFO org.apache.flink.runtime.jobmanager.JobManager

Re: Exception in thread main: No such exception errpr

Form the Flink logs you see that the version is the correct right (1.0.3)? > On Jun 2, 2016, at 4:19 PM, Debaditya Roy wrote: > > Hello, > > I updated the artifact Id and regenerated the jar, but it is giving the same > error. It is somehow trying to access the method which is not there in the

Re: Exception in thread main: No such exception errpr

Hello, I updated the artifact Id and regenerated the jar, but it is giving the same error. It is somehow trying to access the method which is not there in the updated version. Warm Regards, Debaditya On Thu, Jun 2, 2016 at 4:10 PM, Robert Metzger wrote: > The correct artifact id is flink-yarn-

Re: Exception in thread main: No such exception errpr

The correct artifact id is flink-yarn-tests_2.10. On Thu, Jun 2, 2016 at 3:54 PM, Debaditya Roy wrote: > Hi Kostas, > > Doing this throws an error in the pom. > > >org.apache.flink >flink-yarn-tests >${flink.version} > > > 'org.apache.flink:flink-yarn-tests:1.0.3' not found. > > Wa

Re: Exception in thread main: No such exception errpr

Hi Kostas, Doing this throws an error in the pom. org.apache.flink flink-yarn-tests ${flink.version} 'org.apache.flink:flink-yarn-tests:1.0.3' not found. Warm Regards, Debaditya On Thu, Jun 2, 2016 at 3:32 PM, Kostas Kloudas wrote: > Could you replace 0.10-SNAPSHOT with ${flink.ve

Re: Exception in thread main: No such exception errpr

Could you replace 0.10-SNAPSHOT with ${flink.version} in the pom? Thanks, Kostas > On Jun 2, 2016, at 3:13 PM, Debaditya Roy wrote: > > 0.10-SNAPSHOT

Re: Internal buffers

Dear Ufuk, the wiki entry is exactly what i was looking for. I found it quite complicated to understand on a first attempt but i will dedicate some more time for it in the future. Thanks. Regards Leon 1. Jun 2016 13:06 by u...@apache.org: > There is this in the Wiki: > https://cwiki.apache.

Re: Maintaining watermarks per key, instead of per operator instance

Hi again Aljoscha, understood. Thanks for the link. I really like the straightforward approach concerning storing state. It makes things very easy. The improvements are very interesting, particularly the composite triggers. That would significantly improve flexibility. Kind regards Leon 1. Ju

Re: Exception in thread main: No such exception errpr

Hello Kostas, Here is the pom http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> 4.0.0 org.myorg.quickstart quickstart 0.1 jar Flink

Re: Exception in thread main: No such exception errpr

Could you please post your pom file and which version of Flink you downloaded? The latter you can find it in the beginning of the log files. Kostas > On Jun 2, 2016, at 2:55 PM, Debaditya Roy wrote: > > Hi Kostas, > > I followed this > https://ci.apache.org/projects/flink/flink-docs-release-1

Re: Exception in thread main: No such exception errpr

Hi Kostas, I followed this https://ci.apache.org/projects/flink/flink-docs-release-1.0/quickstart/java_api_quickstart.html . Just for your information, in IDE the program is running fine, it is only in the command line when I try to submit it throws me the error. Warm Regards, Debaditya On Thu,

Re: Exception in thread main: No such exception errpr

Hi Debaditya, When creating your application, did you follow the steps described in: https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/local_execution.html (of course adjusted to the release yo

Re: Exception in thread main: No such exception errpr

Hello Kostas, Thanks for you reply. I don't think that is the case because I am running the application on the jvm of my local machine, not on a distributed cluster. Any other input? Warm Regards, Debaditya On Thu, Jun 2, 2016 at 1:27 PM, Kostas Kloudas wrote: > Hello Debaditya, > > From th

Re: Exception in thread main: No such exception errpr

Hello Debaditya, From the exception message you posted it seems that it is a linkage error. Could it be that you are combining different versions of Flink when running your application? E.g. you have version X running on your cluster and you create your jar against version Y on your local mach

Re: Submitting job via Flink web interface giving Null pointer exception

on command line I'm executing: $ flink run project.jar This is on a Flink standalone mode of deployment using the start-local.sh script. The web interface is running into Null pointer exception on an internal codebase. Weird thing is that it does not happen when submitting the job via commandline

Exception in thread main: No such exception errpr

Hi, I am trying to run a simple flink program, which reads an image from disk and does some image processing and stores it back to the disk. For the purpose I have a custom defined class, which I am using in dataset and passing it onwards to flatmap function. However the experiment encountered an

Re: No key found restore States

Hi In other words, what's the easiest way to clean up states in flink, if this key may never arrive again? --Thanks Simon > On 02 Jun 2016, at 10:16, simon peyer wrote: > > Hi Max > > Thanks for your answer. > We have some states, on some keys, which we would like to delete after a > certai

Re: API request to submit job takes over 1hr

Hi Shannon, thank you for further investigating the issue. Its fine to keep the discussion on the user@ list. Most devs are on the user list as well and we'll probably file some JIRAs. Regarding your suggestions: 1. Not sure if making the job submission non-blocking is a good idea. We would proba

Re: No key found restore States

Hi Max Thanks for your answer. We have some states, on some keys, which we would like to delete after a certain time. And since there is no option at the moment to put an "expiriece" date on it, I just use the snapshot function to test and verify if the current key is still in some threshold.