Re: Flink job on secure Yarn fails after many hours

Niels, are you still facing this issue? As far as I understood it, the security changes in Flink 1.2.0 use a new Kerberos mechanism that allows infinite token renewal. On Thu, Mar 17, 2016 at 7:30 AM, Maximilian Michels wrote: > Hi Niels, > > Thanks for the feedback. As far as

[ANNOUNCE] Flink Forward San Francisco 10-11 Apr 2017 community discount codes

Dear Flink community, I would like to bring Flink Forward San Francisco to your attention. After hosting Flink Forward for two years in Berlin, Germany, we decided to bring it to the US west coast as well. Check out this very nice video summary from last year's conference in Berlin:

Re: [E] Re: unable to add more servers in zookeeper quorum peers in flink 1.2

both before > and after the equals sign. > > > On Mar 23, 2017, at 10:12 AM, Robert Metzger <rmetz...@apache.org> wrote: > > Thank you for verifying. Fixed in master: http://git-wip-us. > apache.org/repos/asf/flink/commit/3e860b40 > > On Wed, Mar 22, 2017 at 9:25 PM

Re: Task manager number mismatch container number on mesos

Could you provide the logs of the task manager that still runs as a container but doesn't show up as a Taskmanager? On Thu, Mar 23, 2017 at 11:38 AM, Renjie Liu wrote: > Permanent. I've waited for several minutes and the task manager is still > lost. > > On Thu, Mar 23,

Re: Odd error

Hi, I assume the flatMap(new RecordSplit()) is emitting a RawRecord. Is it possible that you've also added an empty constructor to it while adding the compareTo() method? I think the problem is that one of your types (probably the schema) is recognized as a nested POJO. Check out this

Re: [E] Re: unable to add more servers in zookeeper quorum peers in flink 1.2

Thank you for verifying. Fixed in master: http://git-wip-us.apache.org/repos/asf/flink/commit/3e860b40 On Wed, Mar 22, 2017 at 9:25 PM, wrote: > That worked.. Thanks Chesnay, > > > > > [image: Verizon] > > Kanagaraj Vengidasamy > RTCI

Re: Flink AUR package is available

Amazing, thanks a lot! On Thu, Mar 23, 2017 at 10:36 AM, Tao Meng wrote: > Hi all, > > For arch linux users, I have created flink AUR package. > We can use the package manager to install flink and use the systemd > manager flink as service. > If you have any questions or

Re: RocksDB segfaults

Florian, can you post the log of the Taskmanager where the segfault happened ? On Wed, Mar 22, 2017 at 6:19 PM, Stefan Richter wrote: > Hi, > > for the first checkpoint, from the stacktrace I assume that the backend is > not accessed as part of processing an

Re: Threading issue

Hi, how many unique combinations of your key "partition","threadNumber","schemaId" exist? In my opinion, all sinks should receive data if there are enough different keys. On Wed, Mar 22, 2017 at 3:41 AM, Telco Phone wrote: > I am looking to get readers from kafka / keyBy and

Re: [POLL] Who still uses Java 7 with Flink ?

than near the end? There is > currently little cost to staying with Java 7 since no Flink code or pull > requests have been written for Java 8. > > Greg > > > > On Mar 23, 2017, at 6:37 AM, Robert Metzger <rmetz...@apache.org> wrote: > > Looks like 9% on twitte

Re: A question about iterations and prioritizing "control" over "data" inputs

To very quickly respond to Theo's question: No, it is not possible to have records overtake each other in the buffer. This could potentially void the exactly once processing guarantees, in the case when records overtake checkpoint barriers. On Wed, Mar 15, 2017 at 5:58 PM, Kathleen Sharp

Re: Telling if a job has caught up with Kafka

Sorry for joining this discussion late, but there is already a metric for the offset lag in our 0.9+ consumers. Its called the "records-lag-max": https://kafka.apache.org/documentation/#new_consumer_fetch_monitoring and its exposed via Flink's metrics system. The only issue is that it only shows

Re: Return of Flink shading problems in 1.2.0

Here is the JIRA: https://issues.apache.org/jira/browse/FLINK-6125 On Mon, Mar 20, 2017 at 10:27 AM, Robert Metzger <rmetz...@apache.org> wrote: > Hi Craig, > > I was able to reproduce the issue with maven 3.3 in Flink 1.2. I'll look > into it. > > On Fri, Mar 17, 2017 at

Re: Return of Flink shading problems in 1.2.0

let > me know of a new workaround for 3.3.x. > > > > Thanks! > > Craig > > > > *From: *"Foster, Craig" <foscr...@amazon.com> > *Reply-To: *"user@flink.apache.org" <user@flink.apache.org> > *Date: *Friday, March 17, 2017 at 7:23

Re: Flink 1.2 and Cassandra Connector

, 2017 at 6:08 PM, Robert Metzger <rmetz...@apache.org> wrote: > Yep, this is definitively a bug / misconfiguration in the build system. > > The cassandra client defines metrics-core as a dependency, but the shading > is dropping the dependency when building the dependency reduced

Re: Flink 1.2 and Cassandra Connector

;>> commons-io:commons-io >>> commons-cli:commons-cli >>> >>> >>> org.apache.flink:* >

Re: Checkpointing with RocksDB as statebackend

Yes, you can change the GC using the env.java.opts parameter. We are not setting any GC on YARN. On Thu, Mar 16, 2017 at 1:50 PM, Stephan Ewen wrote: > The only immediate workaround is to use windows with "reduce" or "fold" or > "aggregate" and not "apply". And to not use an

Re: [POLL] Who still uses Java 7 with Flink ?

I've put it also on our Twitter account: https://twitter.com/ApacheFlink/status/842015062667755521 On Wed, Mar 15, 2017 at 2:19 PM, Martin Neumann wrote: > I think this easier done in a straw poll than in an email conversation. > I created one at:

Re: Source and Sink Flink

Hi Alberto, It should be possible. The IBM MQ supports the JMS standard, and we have a JMS compatible connector for Flink in Apache Bahir: http://bahir.apache. org/docs/flink/current/flink-streaming-activemq/ For writing files to HDFS, we have the bucketing sink in Flink

Re: Frontend classpath issue

t as a default, otherwise I am in favor of reverting to the 1.1 > default. (My logic is that the user will only observe a difference in > behavior when the new setup actually causes problems) > > Gyula > > On Fri, Feb 24, 2017, 17:53 Robert Metzger <rmetz...@apache.org> wrote

Re: AWS exception serialization problem

erializer` as the > serializer for Throwables in Kryo, but I’m not sure if we should actually > do this, or just wait for the Kryo fix to be released. > > - Gordon > > > On March 11, 2017 at 9:54:03 AM, Shannon Carey (sca...@expedia.com) wrote: > > Here ya go (see at

Re: Streaming Exception

.java:260) > at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue. > runTask(ForkJoinPool.java:1339) > at scala.concurrent.forkjoin.ForkJoinPool.runWorker( > ForkJoinPool.java:1979) > at scala.concurrent.forkjoin.ForkJoinWorkerThread.run( > ForkJoinWorkerThread.java:107) > &

Re: Any good ideas for online/offline detection of devices that send events?

pro tip for debugging watermarks: They are exposed via a metric in Flink 1.2. On Tue, Mar 7, 2017 at 1:37 PM, Bruno Aranda wrote: > Hi Gordon, > > Many thanks for your helpful ideas. We tried yesterday the CEP approach, > but could not figure it out. The ProcessFunction

Re: Isolate Tasks - Run Distinct Tasks in Different Task Managers

There is currently no way in Flink to define such scheduling constraints. On Wed, Mar 8, 2017 at 5:00 PM, PedroMrChaves wrote: > Thanks for the response. > > I would like to assure that the map operator is not in the same task > manager > as the window/apply operator,

Re: Job completion or failure callback?

Hi Shannon, the web UI runs on the same JVM as the JobManager, so log outputs should go there. There is no way of running user code on the JobManager on job completion. We try to not allow users to execute code on the JobManager...bringing the JM down, will kill the entire cluster :) What you

Re: AWS exception serialization problem

Can one of you guys provide us with a minimal example to reproduce the issue? (Ideally locally, not using EMR?) I think once we can reproduce the issue its easy to fix. On Thu, Mar 9, 2017 at 1:24 AM, Bruno Aranda wrote: > Hi Stephan, we are running Flink 1.2.0 on Yarn

Re: Issues with Event Time and Kafka

Hi Ethan, how late elements (elements with event time after the watermark) are handled depends on the operator. Flink's window operators will trigger a single event window when they fall into the "allowed lateness" timeframe. Otherwise, they are dropped. On Thu, Mar 9, 2017 at 5:30 PM,

Re: Flink support for DeepLearning4j or other deep learning library

Hi, I'm not aware of any community efforts into that direction. This is the only thing google brought up: https://www.slideshare.net/FlinkForward/suneel-marthi-deep-learning-with-apache-flink-and-dl4j But in general, I don't think that it should be terribly hard to get started with something,

Re: Reference configs for HA / RocksDB / YARN / Zookeeper / HDFS

Hi Dave, Let me answer your questions: 1. The RocksDB state backend always stores the data on local disks for speed. The back up is done to HDFS or any other distributed file system. The local data directory is configured automatically by YARN. 2. You need to manually configure zookeeper in the

Re: Flink Standalone Service

Hi Daniel, if you install the flink RPMs (or DEBs) from Apache Bigtop, they should come with init.d or service files. On Thu, Mar 9, 2017 at 6:10 PM, Daniel Skates wrote: > Hi all, > > Is there a init.d or similar service script for Flink on Redhat (or > Centos) 7? Mostly

Re: Flink - Writing Test Case for the Datastream

Hi Mahesh, In the kafka tests, were using a pattern of killing a job by throwing a "SuccessException" after a certain number of messages have passed. Just check the Kafka tests to see how its done :) On Thu, Mar 9, 2017 at 10:09 PM, MAHESH KUMAR wrote: > Hi Team,

Re: questions on custom state with flink window

Hi Sai, 1) I think its okay to keep state in a RichWindowFunction. 2) I think it stays forever, yes 3) I'm including Nico, he can probably help you with the queryable state question. 4) I guess that's a queryable state question too. On Fri, Mar 10, 2017 at 1:04 AM, saiprasad mishra

Re: Streaming Exception

Hi, this error is only logged at WARN level. As Kaibo already said, its not a critical issue. Can you send some more messages from your log. Usually the Jobmanager logs why a taskmanager has failed. And the last few log messages of the failed TM itself are also often helpful. On Fri, Mar 10,

Re: Request jira permission

You'll have permissions in the next two minutes :) On Fri, Mar 10, 2017 at 2:19 PM, Mauro Cortellazzi < mauro.cortella...@radicalbit.io> wrote: > Hello Comunity, > > i would help and contribute into Flink and i'm already registered into > jira. > > I've created an issue [1] about documentation

Re: Integrate Flink with S3 on EMR cluster

Hi Vinay, using the HADOOP_CLASSPATH variable on the client machine is the recommended way to solve this problem. I'll update the documentation accordingly. On Wed, Mar 8, 2017 at 10:26 AM, vinay patil wrote: > Hi , > > @Shannon - I am not facing any issue while

Re: Performance tuning

gister my own Kryo > serializer? > > > Best regards, > Dmitry > > On Thu, Feb 23, 2017 at 8:59 PM, Robert Metzger <rmetz...@apache.org> > wrote: > >> Hi Dmitry, >> >> Cool! Looks like you've taken the right approach to analyze the >> performance

Re: [test][ignore] Sending an email to user@flink without being subscribed ...

Please keep ignoring my messages here. I'll talk to Infra again if the issue persists. On Wed, Mar 1, 2017 at 11:35 AM, Robert Metzger <metzg...@web.de> wrote: > As part of https://issues.apache.org/jira/browse/INFRA-13594, I'm sending > another message from my other email address,

Re: [test][ignore] Sending an email to user@flink without being subscribed ...

As part of https://issues.apache.org/jira/browse/INFRA-13594, I'm sending another message from my other email address, to check out the rejection message :) On Thu, Feb 23, 2017 at 4:58 PM, Robert Metzger <rmetz...@apache.org> wrote: > Please ignore these messages. > > I'll talk t

Re: Flink the right tool for the job ? Huge Data window lateness

Hi, sounds like a cool project. What's the size of one data point? If one datapoint is 2 kb, you'll have 100 800 000 * 2048 bytes = 206 gigabytes of state. That's something one or two machines (depending on the disk throughput) should be able to handle. If possible, I would recommend you to do

Re: Frontend classpath issue

d add a flag for adding the user jar to all > the classpaths. > > On Fri, 24 Feb 2017 at 14:50 Robert Metzger <rmetz...@apache.org> wrote: > >> I agree with you Gyula, this change is dangerous. I have seen another case >> from a user with Hadoop dependencies that crashed in F

Re: Frontend classpath issue

ncy conflict? > > Thanks, > Gyula > > Robert Metzger <rmetz...@apache.org> ezt írta (időpont: 2017. febr. 23., > Cs, 21:44): > >> Hi, >> Since Flink 1.2 "per job yarn applications" (when you do "-m >> yarn-cluster") include the job jar i

Re: Performance tuning

kpressure. > > I understand that it's not possible to keep the rate the same when adding > more components due to communication overhead. > I'm just trying to reduce it. > > > Best regards, > Dmitry > > On Thu, Feb 23, 2017 at 4:17 PM, Robert Metzger <rmetz...@apache.o

Re: Difference between partition and groupBy

Hi Patrick, I think (but I'm not 100% sure) its not a difference in what the engine does in the end, its more of an API thing. When you are grouping, you can perform operations such as reducing afterwards. On a partitioned dataset, you can do stuff like processing each partition in parallel, or

Re: Frontend classpath issue

Hi, Since Flink 1.2 "per job yarn applications" (when you do "-m yarn-cluster") include the job jar into the classpath as well. Does this change explain the behavior? On Thu, Feb 23, 2017 at 4:59 PM, Gyula Fóra wrote: > Hi, > > I have a problem that the frontend somehow seems

Re: Flink not reading from Kafka

ing Flink on a DC/OS cluster. > > AFAIR I checked the taskmanager log from the Flink UI in Mesos. It said > stdout was not available. But this may be due to the fact that Flink on > DC/OS is not yet very stable .. > > regards. > > On Fri, Feb 24, 2017 at 1:41 AM, Robert Metz

Re: Flink checkpointing gets stuck

Hi Shai, I think we don't have so many users running Flink on Azure. Maybe you are the first to put some heavy load onto that infrastructure using Flink. I would guess that your problems are caused by the same root cause, just the way the job is being cancelled is a bit different based on what is

Re: Flink not reading from Kafka

eaming/examples/kafka/WriteIntoKafka.java#L71 .. > > It worked for me. Is the stdout disabled somehow by default ? > > regards. > > On Thu, Feb 23, 2017 at 9:42 PM, Robert Metzger <rmetz...@apache.org> > wrote: > >> Hi Mohit, >> >> is there

Re: Performance tuning

Hi Dmitry, sorry for the late response. Where are you reading the data from? Did you check if one operator is causing backpressure? Are you using checkpointing? Serialization is often the cause for slow processing. However, its very hard to diagnose potential other causes without any details on

Re: Flink not reading from Kafka

Hi Mohit, is there new data being produced into the topic? The properties.setProperty("auto.offset.reset", "earliest"); setting only applies if you haven't consumed anything in this consumer group. So if you have read all the data in the topic before, you won't see anything new showing up. On

Re: How to achieve exactly once on node failure using Kafka

Hi, exactly. You have to make sure that you can write data for the same ID multiple times. Exactly once in Flink is only guaranteed for registered state. So if you have a flatMap() with a "counter" variable, that is held in a "ValueState", this counter will always be in sync with the number of

Re: flink on yarn ha

Hi, This looks like a shading issue. Can you post the classpath the JobManager / AppMaster is logging on startup on the mailing list? If seems that Hadoop loads an unshaded version of the SecurityProtos. Maybe there is some hadoop version mixup. Are you using a Hadoop distribution (like CDH or

Re: [test][ignore] Sending an email to user@flink without being subscribed ...

Please ignore these messages. I'll talk to the ASF infra how we can resolve the issue. On Thu, Feb 23, 2017 at 4:54 PM, Robert Metzger <metzg...@web.de> wrote: > I'm testing what happens if I'm sending an email to the user@flink list > without being subscribed. > > On the dev

[test][ignore] Sending an email to user@flink without being subscribed ...

I'm testing what happens if I'm sending an email to the user@flink list without being subscribed. On the dev@ list, moderators get an email in that case. I have the suspicion that you can post on the user@ list without subscribing first. We have often users that ask a question, we give an initial

Re: Unknown I/O error while extracting contained jar files

Hi, which JVM variant and version are you using? What's your operating system? This is a pretty generic issue. If you search for "ZipException: error in opening zip file" on google, you'll find plenty of people who have this issue as well. I think its unrelated to Flink and more of a general JVM

Re: Arrays values in keyBy

I've filed a JIRA for this issue: https://issues.apache.org/jira/browse/FLINK-5874 On Wed, Jul 20, 2016 at 4:32 PM, Stephan Ewen wrote: > I thing we can simply add this behavior when we use the TypeComparator in > the keyBy() function. It can implement the hashCode() as a

Re: Previously working job fails on Flink 1.2.0

I've filed a JIRA for the problem: https://issues.apache.org/jira/browse/FLINK-5874 On Tue, Feb 21, 2017 at 4:09 PM, Stephan Ewen wrote: > @Steffen > > Yes, you can currently not use arrays as keys. There is a check missing > that gives you a proper error message for that. > >

Re: Log4J

I've also (successfully) tried running Flink with log4j2 to connect it to greylog2. If I remember correctly, the biggest problem was "injecting" the log4j2 properties file into the classpath (when running Flink on YARN). Maybe you need to put the file into the lib/ folder, so that it is shipped

Re: JavaDoc 404

, 2017 at 10:49 AM, Yassine MARZOUGUI < y.marzou...@mindlytix.com> wrote: > Thanks Robert and Ufuk for the update. > > 2017-02-07 18:43 GMT+01:00 Robert Metzger <rmetz...@apache.org>: > >> I've filed a JIRA for the issue: https://issues.apache.o >> rg/jira/browse/

Re: Flink 1.2 Maven dependency

Hi Dominik, I hope the artifacts were distributed properly. did you get download errors for the 1.2.0 version from any official maven servers? Maybe mvnrepository.com is slow indexing new artifacts? Best, Robert On Fri, Feb 10, 2017 at 12:02 AM, Yassine MARZOUGUI < y.marzou...@mindlytix.com>

Re: Flink 1.2 and Cassandra Connector

Hi Nico, The cassandra connector should be available on Maven central: http://search.maven.org/#artifactdetails%7Corg.apache.flink%7Cflink-connector-cassandra_2.10%7C1.2.0%7Cjar Potentially, the issue you've mentioned is due to some shading issue. Is the "com/codahale/metrics/Metric" class in

Re: #kafka/#ssl: how to enable ssl authentication for a new kafka consumer?

I've added another answer on SO that explains how you can pass a custom configuration object to the execution environment. On Thu, Feb 9, 2017 at 11:09 AM, alex.decastro wrote: > I found a similar question and answer at #stackoverflow >

Re: #kafka/#ssl: how to enable ssl authentication for a new kafka consumer?

Check out the documentation: https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/connectors/kafka.html#enabling-kerberos-authentication-for-versions-09-and-above-only On Wed, Feb 8, 2017 at 4:40 PM, alex.decastro wrote: > Dear flinkers, > I'm consuming from

Re: 1.2 release date

uence/display/FLINK/List+of+contributors would be updated > > But after announcement it’s not necessary I think > > > > *From:* Robert Metzger [mailto:rmetz...@apache.org] > *Sent:* Tuesday, February 7, 2017 7:58 PM > > *To:* user@flink.apache.org > *Subject:* Re: 1.2 release

Re: JavaDoc 404

I've filed a JIRA for the issue: https://issues.apache.org/jira/browse/FLINK-5736 On Tue, Feb 7, 2017 at 5:00 PM, Robert Metzger <rmetz...@apache.org> wrote: > Yes, I'll try to fix it asap. Sorry for the inconvenience. > > On Mon, Feb 6, 2017 at 4:43 PM, Ufuk Celebi <u...

Re: Issue Faced: Not able to get the consumer offsets from Kafka when using Flink with Flink-Kafka Connector

Hi Mahesh, this is a known limitation of Apache Kafka: https://www.mail-archive.com/users@kafka.apache.org/msg22595.html You could implement a tool that is manually retrieving the latest offset for the group from the __offsets topic. On Tue, Feb 7, 2017 at 6:24 PM, MAHESH KUMAR

Re: Netty issues while deploying Flink with Yarn on MapR

Hi, cool! Yes, creating a JIRA for the problem is a good idea. Once you've found a way to fix the issue, you can open a pull request referencing the issue. Regards, Robert On Tue, Feb 7, 2017 at 6:20 PM, ani.desh1512 wrote: > Thanks Robert. > I would love to try to

Re: Netty issues while deploying Flink with Yarn on MapR

Hi Aniket, great analysis of the problem! Thank you for looking so well into it! Would you be interested in trying to solve the problem for Flink? We could try to provide a maven build profile that sets the correct versions and excludes. We could maybe also provide a MapR specific release of

Re: logback

Hi Dmitry, Did you also check out this documentation page? https://ci.apache.org/projects/flink/flink-docs-release-1.2/monitoring/best_practices.html#use-logback-when-running-flink-on-a-cluster On Tue, Feb 7, 2017 at 1:07 PM, Dmitry Golubets wrote: > Hi, > > documentation

Re: Dealing with latency in Sink

Hi Mohit, Flink doesn't allow dynamic up or downscaling of parallel operator instances at runtime. However, you can stop and restore from a savepoint with a different parallelism. This way, you can adopt to workload changes. Flink's handling of backpressure is very implicit. If you want to

Re: To get Schema for jdbc database in Flink

Currently, there is no streaming JDBC connector. Check out this thread from last year: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/JDBC-Streaming-Connector-td10508.html On Mon, Feb 6, 2017 at 5:00 PM, Ufuk Celebi wrote: > I'm not sure how well this works

Re: JavaDoc 404

Yes, I'll try to fix it asap. Sorry for the inconvenience. On Mon, Feb 6, 2017 at 4:43 PM, Ufuk Celebi wrote: > Thanks for reporting this. I think Robert (cc'd) is working in fixing > this, correct? > > On Sat, Feb 4, 2017 at 12:12 PM, Yassine MARZOUGUI >

Re: 1.2 release date

Hi Anton, which contributors list are you referring to? I've included all release contributors into the rel announcement. On Mon, Feb 6, 2017 at 11:50 AM, Anton Solovev wrote: > Hi, > > > > Could you update List of contributors after that? J > > > > *Anton Solovev* > >

Re: Regarding Flink as a web service

There's also a nice documentation page on the feature now: https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/stream/queryable_state.html On Tue, Jan 31, 2017 at 6:18 PM, Aljoscha Krettek wrote: > +u...@apache.org Because he implemented

Re: TaskManager randomly dies

Hi, which Flink version are you using? This issue occurred quite freqently in the 1.2.0 RC0 and should be fixed in later RCs. On Fri, Jan 27, 2017 at 4:13 PM, Malte Schwarzer wrote: > Hi all, > > when running a Flink batch job, from time to time a TaskManager dies >

Re: User configuration

Hi, Is this what you are looking for? https://ci.apache.org/projects/flink/flink-docs-release-1.2/monitoring/best_practices.html#parsing-command-line-arguments-and-passing-them-around-in-your-flink-application On Thu, Jan 26, 2017 at 5:38 PM, Dmitry Golubets wrote: > Hi, >

Re: Flink dependencies shading

Hi Dmitry, I think this issue is new. Where is the AWS SDK dependency coming from? Maybe you can resolve the issue on your side for now. I've filed a JIRA for this issue: https://issues.apache.org/jira/browse/FLINK-5661 On Wed, Jan 25, 2017 at 8:24 PM, Dmitry Golubets

Re: Debugging, logging and measuring operator subtask performance

Hi Dominik, You could measure the throughput at each task in your job to see if one operator is causing the slowdown (for example using Flink's metrics system) Maybe the backpressure view already helps finding the task that causes the issue. Did you check if there are enough resources available

Re: Issues while restarting a job on HA cluster

Hi Ani, This error is independent of cancel vs stop. Its an issue of loading the MapR classes from the classloaders. Do you user jars contain any MapR code (either mapr streams or maprdb)? If so, I would recommend you to put these MapR libraries into the "lib/" folder of Flink. They'll then be

Re: Kafka data not read in FlinkKafkaConsumer09 second time from command line

Hi, I would guess that the watermark generation does not work as expected. I would recommend to log the extracted timestamps + the watermarks to understand how time is progressing, and when watermarks are generated to trigger a window computation. On Tue, Jan 24, 2017 at 6:53 PM, Sujit Sakre

Re: Rate-limit processing

Hi Florian, you can rate-limit the Kafka consumer by implementing a custom DeserializationSchema that sleeps a bit from time to time (or at each deserialization step) On Tue, Jan 24, 2017 at 1:16 PM, Florian König wrote: > Hi Till, > > thank you for the very helpful

Re: Improving Flink Performance

Hi Jonas, The good news is that your job is completely parallelizable. So if you are running it on a cluster, you can scale it at least to the number of Kafka partitions you have (actually even further, because the Kafka consumers are not the issue). I don't think that the scala (=akka) worker

Re: Flink with Yarn on MapR

Hi, I think this is a re-post of a question I've already "answered": http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Flink-with-Yarn-on-MapR-td15448.html On Wed, Jan 25, 2017 at 12:12 AM, ani.desh1512 wrote: > Hi, > I am trying to setup flink with Yarn

Re: 答复: [DISCUSS] Apache Flink 1.2.0 RC0 (Non-voting testing release candidate)

RC1 creation is in progress ... On Mon, Jan 23, 2017 at 10:33 AM, Robert Metzger <rmetz...@apache.org> wrote: > Hi all, > > I would like to do a proper voting RC1 early this week. > From the issues mentioned here, most of them have pull requests or were > changed to a l

Re: 答复: [DISCUSS] Apache Flink 1.2.0 RC0 (Non-voting testing release candidate)

https, > you > > >> just > > >> > have to enable it as per the security documentation. > > >> > > > >> > > > >> > On 12.01.2017 16:20, Till Rohrmann wrote: > > >> > > > >> > I also found an issue: > > &g

Re: Reading compressed XML data

Hi Sebastian, I'm not aware of a better way of implementing this in Flink. You could implement your own XmlInputFormat using Flink's InputFormat abstractions, but you would end up with almost exactly the same code as Mahout / Hadoop. I wonder why the decompression with the XmlInputFormat doesn't

Re: 1.1.4 on YARN - vcores change?

Hi Shannon, Flink is reading the number of available vcores from the local YARN configuration. Is it possible that the YARN / Hadoop config on the machine where you are submitting your job from sets the number of vcores as 4 ? On Fri, Jan 13, 2017 at 12:51 AM, Shannon Carey

Re: 答复: [DISCUSS] Apache Flink 1.2.0 RC0 (Non-voting testing release candidate)

邮件原件- > 发件人: Robert Metzger [mailto:rmetz...@apache.org] > 发送时间: 2017年1月3日 20:44 > 收件人: d...@flink.apache.org > 抄送: user@flink.apache.org > 主题: [DISCUSS] Apache Flink 1.2.0 RC0 (Non-voting testing release candidate) > > Hi, > > First of all, I wish everybody a happy new y

Re: Problem with JodaTime

Hi Stephan, Can you post the list of fields in the POJO and the full exception (so that I can see which serializer is being used). In general, to fix such an issue, you have to implement a custom serializer for the field that is causing the issues. On Thu, Dec 22, 2016 at 3:44 PM, Stephan

Re: Can I see the kafka header information in the Flink connector?

Hi Ron, there is a KeyedDeserializationSchema for the Kafka connector, that exposes the source partition, offset and topic. Is that what you are looking for? On Thu, Dec 22, 2016 at 5:33 PM, Ron Crocker wrote: > Looking at the Kafka 0.8 connector API, my deserializer

Re: Standalone cluster layout

Hi Avihai, 1. As much as possible (I would leave the operating system at least 1 GB of memory). It depends also on the workload you have. For streaming workload with very small state, you can use Flink with 1-2 GB of heap space and still get very good performance. 2. Yes, I would recommend to run

Re: PartitionedState and watermark of Window coGroup()

Hi, elements are coGrouped on the specified key. So only elements with the same key in both streams end up in the same group. Yes, the watermark uses the minimum of both streams. On Tue, Dec 13, 2016 at 7:02 PM, Sendoh wrote: > Hi Flink users, > > I'm a bit

Re: checkpoint notifier not found?

Hi Abhishek, you can not push to the Flink repository directly. Only Flink committers are allowed to do that. But you can fork the Flink repository on github to your own GitHub account and then push the changes to your Github. Then, you can create a pull request to offer those changes to the main

Re: [DISCUSS] "Who's hiring on Flink" monthly thread in the mailing lists?

Exactly. On Tue, Dec 13, 2016 at 4:40 PM, Timur Shenkao <t...@timshenkao.su> wrote: > How to subscribe? > community-subscr...@flink.apache.org ? > > On Tue, Dec 13, 2016 at 6:32 PM, Robert Metzger <rmetz...@apache.org> > wrote: > >> The commun...@flink.apache.

Re: [DISCUSS] "Who's hiring on Flink" monthly thread in the mailing lists?

The commun...@flink.apache.org has been created :) On Tue, Dec 13, 2016 at 10:43 AM, Robert Metzger <rmetz...@apache.org> wrote: > +1. I've requested the community@ mailing list from infra. > > On Tue, Dec 13, 2016 at 10:40 AM, Kostas Tzoumas <ktzou...@apache.org> &

Re: [DISCUSS] "Who's hiring on Flink" monthly thread in the mailing lists?

te on the Web Page. This feels more > intuitive > >> to me and gives a better overview. > >> > >> Best regards, > >> Felix > >> > >> On Dec 9, 2016 14:20, "Ufuk Celebi" <u...@apache.org> wrote: > >> > >> > >

Re: [DISCUSS] "Who's hiring on Flink" monthly thread in the mailing lists?

hat subscribing to a new mailing list is an overhead. > As a temp solution, we could cc the dev and user list in the first few > (say, 3) threads and encourage folks in these threads to sign up for the > news@ list. > > On Thu, Dec 8, 2016 at 10:07 AM, Robert Metzger <rm

Re: [DISCUSS] "Who's hiring on Flink" monthly thread in the mailing lists?

Thank you for speaking up Kanstantsin. I really don't want to downgrade the experience on the user@ list. I wonder if jobs@flink would be a too narrowly-scoped mailing list. Maybe we could also start a community@flink (alternatively also general@) mailing list for everything relating to the

Re: Resource under-utilization when using RocksDb state backend

m not very experienced in sussing out > IO issues so perhaps there is something else I'm missing. > > I'll keep investigating. If I continue to come up empty then I guess my > next steps may be to stage some independent tests directly against RocksDb. > > -Cliff > > > On

Re: Flink 1.1.3 OOME Permgen

atic field is a cache, > which is filled by all classes, which were serialized/deserialized by > Jackson. > > Best, > > Konstantin > > On 05.12.2016 11:55, Robert Metzger wrote: > > I've submitted Wordcount 410 times to a testing cluster and a streaming > >

Re: flink-job-in-yarn,has max memory

Hi, The TaskManager reports a total memory usage of 3 GB. That's fine, given that you requested containers of size 4GB. Flink doesn't allocate all the memory assigned to the container to the heap. Are you running a batch or a streaming job? On Tue, Nov 29, 2016 at 12:43 PM,

<    1   2   3   4   5   6   7   8   9   10   >