Re: [ANNOUNCE] Welcome Stefan Richter as a new committer

Woohoo, Just saw this (was travelling). Congratulations, Stefan! Looking forward to the promising future of the backend state. On Mon, Feb 13, 2017 at 5:20 PM, Stefan Richter wrote: > Thanks a lot! I feel very happy and will try help the Flink community as > good

Re: rocksdb without checkpointing

Good to hear that. On which machine you are running your Flink Job, also what are the configurations you have used for RocksDB I am currently running on C3.4xlarge with predefined option set to FLASH_SSD_OPTIMIZED Regards, Vinay Patil On Thu, Feb 16, 2017 at 10:31 AM, abhishekrs [via Apache

Re: Trapping Streaming Errors

Hi Joe, you can also insert a MapFunction between the Kafka source and the keyBy to validate the IDs. The mapper will be chained and should not add only minimal overhead. If you want to keep the events which were incorrectly deserialized, you can use split() to move them somewhere. Validation in

Log4J

Is there a way to reload a log4j.properties file without stopping and starting the job server?

Trapping Streaming Errors

If I am processing a stream in the following manner: val stream = env.addSource(consumer).name("KafkaStream") .keyBy(x => (x.obj.ID1(),x.obj.ID2(),x.obj.ID3()) .flatMap(new FlatMapProcessor) and the IDs bomb out because of deserialization issues, my job crashes with a 'Could not extract

Re: rocksdb without checkpointing

Hi Abhishek, You can disable checkpointing by not commenting env.enableCheckpointing What do you mean by "We are trying to do application level checkpointing" Regards, Vinay Patil On Thu, Feb 16, 2017 at 12:42 AM, abhishekrs [via Apache Flink User Mailing List archive.]

Re: Flink jdbc

See the tutorial at the beginning of: flink-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCInputFormat.java Looks like plugging in "org.h2.Driver" should do. On Wed, Feb 15, 2017 at 4:59 PM, Punit Tandel wrote: > Hi All > > Does flink jdbc

Flink jdbc

Hi All Does flink jdbc support writing the data into H2 Database? Thanks Punit

Flink Job Exception

Hi All, I'm trying to run a streaming job with flink 1.2 version and there are 3 task managers with 12 task slots. Irrespective of the parallelism that I give it always fails with the below error and I found a JIRA link corresponding to this issue. Can I know by when this will be resolved since

Reliable Distributed FS support (HCFS)

Hello, Regarding the Filesystem abstraction support, we are planning to use a distributed file system which complies with Hadoop Compatible File System (HCFS) standard in place of standard HDFS. According to the documentation

rocksdb without checkpointing

Is it possible to set state backend as RocksDB without asking it to checkpoint? We are trying to do application level checkpointing (since it gives us better flexibility to upgrade our flink pipeline and also restore state in a application specific upgrade friendly way). So we don’t really need

Re: Checkpointing with RocksDB as statebackend

Hi Guys, Can anyone please help me with this issue Regards, Vinay Patil On Wed, Feb 15, 2017 at 6:17 PM, Vinay Patil wrote: > Hi Ted, > > I have 3 boxes in my pipeline , 1st and 2nd box containing source and s3 > sink and the 3rd box is window operator followed by

Re: Checkpointing with RocksDB as statebackend

Hi Ted, I have 3 boxes in my pipeline , 1st and 2nd box containing source and s3 sink and the 3rd box is window operator followed by chained operators and a s3 sink So in the details link section I can see that that S3 sink is taking time for the acknowledgement and it is not even going to the

Re: Checkpointing with RocksDB as statebackend

What did the More Details link say ? Thanks > On Feb 15, 2017, at 3:11 AM, vinay patil wrote: > > Hi, > > I have kept the checkpointing interval to 6secs and minimum pause between > checkpoints to 5secs, while testing the pipeline I have observed that that > for some

Checkpointing with RocksDB as statebackend

Hi, I have kept the checkpointing interval to 6secs and minimum pause between checkpoints to 5secs, while testing the pipeline I have observed that that for some checkpoints it is taking long time , as you can see in the attached snapshot checkpoint id 19 took the maximum time before it gets

Re: Unable to use Scala's BeanProperty with classes

Thanks Timo, removing `@BeanProperty` is giving no getters, no setters error On Wed, Feb 15, 2017 at 3:45 PM, Timo Walther wrote: > Forget what I said about omitting `var`, this would remove the field from > the POJO. I opened a PR for fixing the issue:

Re: A way to control redistribution of operator state?

Hi, I think the clean solution would be using raw keyed state once it becomes available. For the meantime, your solution could work. However, you should be aware that your approach does not rely on a contract but an implementation detail that *could* change between versions and break your code

Re: Unable to use Scala's BeanProperty with classes

Forget what I said about omitting `var`, this would remove the field from the POJO. I opened a PR for fixing the issue: https://github.com/apache/flink/pull/3318 As a workaround: If you just want to have a POJO for the Cassandra Sink you don't need to add the `@BeanProperty` annotation. Flink

Re: Unable to use Scala's BeanProperty with classes

Hello, There is an open PR about adding support for case classes to the cassandra sinks: https://github.com/apache/flink/pull/2633 You would have to checkout the branch and build it yourself. If this works for you it would be great if you could also give some feedback either here or in the

Re: Unable to use Scala's BeanProperty with classes

Hi Adarsh, I looked into your issue. The problem is that `var` generates Scala-style getters/setters and the annotation generates Java-style getters/setters. Right now Flink only supports one style in a POJO, I don't know why we have this restriction. I will work on a fix for that. Is it

Re: Unable to use Scala's BeanProperty with classes

Thanks Fabian, I need to sink data in Cassandra and direct sink with case class is not available (correct me if I am wrong) If we use Tuple then we are restricted to 22 fields What do you suggest here? On Wed, Feb 15, 2017 at 2:32 PM, Fabian Hueske wrote: > Hi Adarsh, >

Re: Unable to use Scala's BeanProperty with classes

Hi Adarsh, I think this is the same bug. I'm afraid you have to wait until the problem is fixed. The only workaround would be to use a different data type, for example a case class. Best, Fabian 2017-02-15 6:08 GMT+01:00 Adarsh Jain : > Any help will be highly

Re: Watermarks per key

Hi Jordan, it is not possible to generate watermarks per key. This feature has been requested a couple of times but I think there are no plans to implement that. As far as I understand, the management of watermarks would be quite expensive (maintaining several watermarks, purging watermarks of

Flink batch processing fault tolerance

Hi, all: I'm learning flink's doc and curious about the fault tolerance of batch process jobs. It seems that when one of task execution fails, the whole job will be restarted, is it true? If so, isn't it impractical to deploy large flink batch jobs? -- Liu, Renjie Software Engineer, MVAD