How can I confirm a savepoint is used for a new job?

Do we have any logs in JM/TM indicate the job is using a savepoint I passed in when I submit the job? Thanks

how does SQL mode work with PopularPlaces example?

Hi, I am following the PopularPlacesSQL example ( http://training.data-artisans.com/exercises/popularPlacesSql.html), but I am unable to understand why the following statement will pickup events with START flag only. "SELECT " + "toCoords(cell), wstart, wend, isStart, popCnt " + "FROM " +

Re: entrypoint for executing job in task manager

Thanks, let me clarify the requirement. Sorry that it wasn't clear in the original email. Here is our setup. these 3 dirs are added to classpath * flink/lib: core flink jars (like flink-dist_2.11, flink-shaded-hadoop2-uber) * spaaslib: many jars pulled in our internal platform * jobs: a single

Re: Strange behavior on filter, group and reduce DataSets

Hi, That was a bit too early. I found an issue with my approach. Will come back once I solved that. Best, Fabian 2018-03-21 23:45 GMT+01:00 Fabian Hueske : > Hi, > > I've opened a pull request [1] that should fix the problem. > It would be great if you could try change and

Re: Strange behavior on filter, group and reduce DataSets

Hi, I've opened a pull request [1] that should fix the problem. It would be great if you could try change and report back whether it fixes the problem. Thank you, Fabian [1] https://github.com/apache/flink/pull/5742 2018-03-21 9:49 GMT+01:00 simone : > Hi all, > >

InterruptedException when async function is cancelled

Hi all, When I cancel a job that has async functions, I see this sequence in the TaskManager logs: 2018-03-21 14:51:34,471 INFO org.apache.flink.runtime.taskmanager.Task - Attempting to cancel task AsyncFunctionName (1/1) (fcb7bbe7cd89f1167f8a656b0f2fdaf9). 2018-03-21

Out off memory when catching up

Hi. When our jobs are catching up they read with a factor 10-20 times normal rate but then we loose our task managers with OOM. We could increase the memory allocation but is there a way to figure out how high rate we can consume with the current memory and slot allocation and a way to limit

Re: ListCheckpointed function - what happens prior to restoreState() being called?

Hi Fabian, > On Mar 20, 2018, at 6:38 AM, Fabian Hueske wrote: > > Hi Ken, > > The documentation page describes that first the state is restored / > initialized and then the function's open() method is called [1]. Yes, thanks - my question was about

Re: Confluent Schema Registry DeserializationSchema

I added kafka tomy dependencies although i am not sure why this would be required... seems to work org.apache.kafka kafka_${kafka.scala.version} ${kafka.version} This is my full dependency

Re: Error running on Hadoop 2.7

Hi Piotrek, At this point we are simply trying to start a YARN session.  BTW, we are on Hortonworks HDP 2.6 which is on 2.7 Hadoop if anyone has experienced similar issues.  We actually pulled 2.6 binaries for the heck of it and ran into same issues.  I guess we are left with getting non-hadoop

Re: entrypoint for executing job in task manager

It would be great to understand a bit more what the exact requirements here are, and what setup you use. I am not a dependency injection expert, so let me know if what I am suggesting here is complete bogus. *(1) Fix set of libraries for Dependency Injection, or dedicated container images per

Re: Confluent Schema Registry DeserializationSchema

Hi, It looks like to me that kafka.utils.VerifiableProperties comes from org.apache.kafka:kafka package - please check and solve (if possible) dependency conflicts in your pom.xml regarding this package. Probably there is some version collision. Piotrek > On 21 Mar 2018, at 16:40, dim5b

Re: Error running on Hadoop 2.7

Hi Piotrek, Yes, this is a brand new Prod environment. 2.6 was in our lab. Thanks, -- Ashish On Wed, Mar 21, 2018 at 11:39 AM, Piotr Nowojski wrote: Hi, Have you replaced all of your old Flink binaries with freshly downloaded Hadoop 2.7 versions? Are you sure

Confluent Schema Registry DeserializationSchema

I trying to connect to schema registry and deserialize the project. I am building my project and on mvn build i get the error class file for kafka.utils.VerifiableProperties not found... import io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient; import

Re: Error running on Hadoop 2.7

Hi, Have you replaced all of your old Flink binaries with freshly downloaded Hadoop 2.7 versions? Are you sure that something hasn't mix in the process? Does some simple word count example works on the cluster after the upgrade? Piotrek > On 21 Mar

Error running on Hadoop 2.7

Hi All, We ran into a roadblock in our new Hadoop environment, migrating from 2.6 to 2.7. It was supposed to be an easy lift to get a YARN session but doesnt seem like :) We definitely are using 2.7 binaries but it looks like there is a call here to a private methos which screams runtime

Re: Migration to Flip6 Kubernetes

Hi Till, Thanks for the information. We are using the session cluster and is working quite good, but we would like to benefit from the new approach of per-job mode in order to have a better control over the jobs that are running on the cluster. I took a look to the YarnJobClusterEntrypoint and

Re: Kafka ProducerFencedException after checkpointing

Hi, But that’s exactly the case: producer’s transaction timeout starts when the external transaction starts - but FlinkKafkaProducer011 keeps an active Kafka transaction for the whole period between checkpoints. As I wrote in the previous message: > in case of failure, your timeout must also

Re: Apache Zookeeper vs Flink Zookeeper

Hi Alex, You can use vanilla Apache ZooKeeper. The class FlinkZooKeeperQuorumPeer is only used if you start ZooKeeper via the provided script bin/zookeeper.sh. FlinkZooKeeperQuorumPeer does not add any functionality except creating ZooKeeper's myid file. Best, Gary On Wed, Mar 21, 2018 at 12:02

Apache Zookeeper vs Flink Zookeeper

Hi, For standalone cluster configuration, is it possible to use vanilla Apache Zookeeper? I saw there's a wrapper around it in Flink - FlinkZooKeeperQuorumPeer. Is it mandatory to use it? Thank you, Alex

Re: Strange behavior on filter, group and reduce DataSets

Hi all, an update: following Stephan directives on how to diagnose the issue, making Person immutable, the problem does not occur. Simone. On 20/03/2018 20:20, Stephan Ewen wrote: To diagnose that, can you please check the following:   - Change the Person data type to be immutable (final

Re: Is Hadoop 3.0 integration planned?

That is definitely a good thing to have, would like to have a discussion about how to approach that after 1.5 is released. On Wed, Mar 21, 2018 at 5:39 AM, Jayant Ameta wrote: > > Jayant Ameta >

Re: Queryable State

Hi Vishal, As Fabian said, queryable state is just a feature that exposes the state kept within Flink, and it is not made to replace functionality that would otherwise be made by a sink. In the future the functionality will definitely evolve but for there are no discussions currently, for

Re: Let BucketingSink roll file on each checkpoint

Thank you! Fabian HDFS small file problem can be avoid with big checkpoint interval. Meanwhile, there is potential data lose problem in current BucketingSink. Say we consume data in kafka, when checkpoint is requested, kafka offset is update, but in-progress file in BucketingSink is remained. If