Savepoints and migrating value state data types

2017-09-20 Thread mrooding
Hi We've got a situation where we're merging several Kafka streams and for certain streams, we want to retain up to 6 days of history. We're trying to figure out how we can migrate savepoint data between application updates when the data type for a certain state buffer updates. Let's assume that

Re: Savepoints and migrating value state data types

2017-10-05 Thread mrooding
Gordon Thanks for the detailed response. I have verified your assumption and that is, unfortunately, the case. I also looked into creating a custom Kryo serializer but I got stuck on serializing arrays of complex types. It seems like this isn't trivial at all with Kryo. As an alternative, I've b

JobManager web interface redirect strategy when running in HA

2017-10-31 Thread mrooding
Hi We're running 3 job managers in high availability cluster mode backed by OpenStack/Openshift. We're currently exposing all 3 job managers using 3 different routes (flink-1.domain.tld, flink-2.domain.tld, flink-3.domain.tld). When accessing the route for a job manager which isn't the leader it a

Re: JobManager web interface redirect strategy when running in HA

2017-11-06 Thread mrooding
Chesnay, your solution is definitely the best approach. I was already wondering why the decision was made to only support the UI through the leading job manager only. Jürgen, I don't think that your solution will work in our setup. We're currently running 3 services, one for each job manager. We n

Re: Savepoints and migrating value state data types

2017-11-06 Thread mrooding
Hi Gordon I've been looking into creating a custom AvroSerializer without Kryo which would support Avro schemas and I'm starting to wonder if this is actually the most straightforward way to do it. If I extend a class from TypeSerializer I would also need to implement a TypeInformation class to