Hi Sergio, I wanted to close the loop on this email chain.
Unfortunately, we don't have the capacity to join the team evaluating Phase 2 right now. We currently lack a business requirement for Phase 2, and we're swamped with other projects. A quick glance at the FLIP, the direction looks good. I would have loved something "simpler" but given the requirements of Flink I do think this is a smart design. If something changes, I'll let you know and we'll collaborate again! Thanks, Ryan van Huuksloot Staff Engineer, Infrastructure | Streaming Platform [image: Shopify] <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> On Fri, Oct 9, 2026 at 5:37 PM Sergio Chong Loo <[email protected]> wrote: > Hi Ryan / Daniel / Gyula, > > I've finished the last round of asks, and PR #1043 is now out of draft and > ready for review: > https://github.com/apache/flink-kubernetes-operator/pull/1043. I rewrote > the description, so it should be a quicker read, with a suggested reading > order at the top. > > Highlights: > > 1. *Gate-clear timeout*. A new kubernetes.operator.bluegreen.gate.timeout > bounds the wait for the gate, and the transition aborts past it. Every > abort hands the gate back to the deployment that keeps running. > 2. *SQL / Table API support*. The gate reads each record's timestamp from > exactly one configured strategy. field-index reads a BIGINT, TIMESTAMP(p) > or TIMESTAMP_LTZ(p) column of RowData by its type, for SQL and Table API > jobs. The gate uses record time only, with no wall-clock fallback, and late > records are no longer dropped after the hand-over. > 3. *No-dup / no-loss e2e test*. The new > e2e-tests/test_bluegreen_advanced.sh runs a transition of a purpose-built > job and checks that every record is written exactly once, late records > included. It's wired into the CI matrix next to the existing Blue/Green > tests, which I fixed and enabled as well. All 32 jobs pass on my fork > across Flink 1.20, 2.0, 2.1 and 2.2, natively and standalone: > https://github.com/schongloo/flink-kubernetes-operator/actions/runs/37846378915. > The abort-after-cutover scenario isn't covered end to end yet; it's a TODO > in the script. > 4. *Gate chaining and parallelism*. The gate joins its neighbour's chain, > keeps the edge it replaces, and copies the neighbour's parallelism and max > parallelism. So it adds no vertex that could be rescaled on its own. > 5. *Docs*. There's a new Advanced Transition Mode section in the > Blue/Green Deployments page: how the hand-over works and what it > guarantees, teardown and aborts, requirements, the extraction strategies, > gate placement, SQL jobs and configuration. The related pages are updated > to match. > > On your exactly-once sinks point: the old deployment is now torn down only > once every one of its gate subtasks has handed over, and it's stopped with > a savepoint first, so sinks that commit on checkpoints commit what it wrote. > > Running the e2e tests also surfaced a few fixes, all in the PR. The main > one: only the standby sets the cutover point now, which closes a race that > could duplicate records. I also documented a Flink 2.x limitation: jobs > using the async state API can't take canonical savepoints, so they need the > native format for transitions. The operator now explains that failure and > rejects state backend changes that a native savepoint can't restore. The > known limitations are listed in the PR description. > > Could you continue your evaluation when you get a chance? Happy to walk > through any part of it. > > Thanks, > > On Aug 18, 2026, at 11:13 AM, Sergio Chong Loo <[email protected]> > wrote: > > Thanks for the feedback Ryan, I’ll re-focus on this soon and will keep you > posted. > > FYI: @Gyula > > - Sergio > > On Aug 10, 2026, at 10:12 AM, Ryan van Huuksloot via dev < > [email protected]> wrote: > > Hi Sergio, > > Thanks for driving this. We are still interested but are currently caught > up in other work, which explains the delay. > > Re: What we should include. > > The 4 points are good. > Additionally, your deferral is unclear. Are SQL/Table pipelines supported > if they use watermarks? That should be included. > I also think that having an e2e harness might need to be part of the > success criteria. If we have one test we may as well build it into the > harness. > > I agree with what is out of scope for Phase 2. > > The last thing I would point out is that the documentation will be really > important for this feature. Your callouts are great and I would do +100 to > what you have for documentation. > > Thanks > > Ryan van Huuksloot > Staff Engineer, Infrastructure | Streaming Platform > [image: Shopify] > <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> > > > On Mon, Jul 6, 2026 at 12:50 PM Sergio Chong Loo <[email protected]> > wrote: > > Hi Daniel / Ryan / Gyula, > > Overdue update, and apologies for the long gap since May. A no-code > watermark option jumped > the queue (a few teams were blocked without it), so I pushed that > through first. Quick > summary of what landed, then a proposal to lock the remaining scope so > we share a finish > line. I know it’s a bit lengthy but please bear with me. > > Landed since May: > - No-code watermark extraction: bluegreen.gate.watermark.field-path. > Instead of a custom > extractor-class, point the gate at a dot-notation field path: > bluegreen.gate.strategy: "WATERMARK" > bluegreen.gate.watermark.field-path: "eventTime" # or nested: > "metadata.timestamp" > It reads an epoch-ms long off the record (public field, class > hierarchy, or getXxx() > getter), cached with no per-record reflection, and works out of the > box with protobuf > classes. extractor-class stays for anything a field path cannot > express. Scope note, since > it is close to your WatermarkGenerator point but not the same: it > removes extractor code > for DataStream/POJO jobs, but it still reads an existing field and > does not cover the > SQL/Table case you raised (that stays open, see Defer). > - transitionMode: you were right that the unset case was a real gap (it > errored instead of > defaulting), so it now defaults to BASIC (existing basic deployments > upgrade cleanly) and > is immutable after the initial deployment (a change is rejected with a > validation error > and a Warning event). > - Rebased onto current operator main, and the gate-strategy layer is now > extensible > (watermark is one strategy). > > Proposed finish line: Phase 2 to be marked done when advanced Blue/Green > is correct > within a documented envelope, never hangs silently, and its > no-dup/no-loss claim is proven > by at least one real end-to-end test. Anything that widens the envelope > is a follow-up. > > Must-have: > 1. Gate-clear timeout. If a transition's gate never clears (watermark > stalls), the > controller currently waits indefinitely. Add a deadline so it fails > with a clear error > instead of hanging. > 2. One real no-dup/no-loss end-to-end test for the supported shape: > single source, > idempotent or transactional sink, happy-path cutover plus a failure > injected around the > cutover. This turns the headline claim from asserted to proven, and I > am happy to help > wire it into the e2e suite. Daniel, if your sandbox pipeline is > close, this is exactly it. > 3. Documented envelope and limitations: one meaningful watermark at the > gate point; sink > must be idempotent or transactional for end-to-end exactly-once; the > idleness caveat; > JVM-only auto-injection. > 4. Gate parallelism (your point). The agent already injects the gate at > the adjacent > operator's parallelism with a forward connection, so it adds no > shuffle and normally > chains into that vertex. The remaining work is to make that chaining > reliable (also match > the neighbor's max-parallelism and set the chaining strategy > explicitly) so the gate is > never a standalone vertex the autoscaler can rescale on its own, plus > a documented > pin/exclude fallback for the rare non-chainable case. That closes the > bottleneck-under-autoscaling concern. > > Defer (follow-ups): > - Full conformance matrix (every sink class by failure point by > idleness), beyond the one > core test. > - SQL/Table watermark-generator injection so table defs need no > timestamp field. This is > really SQL support, a separate surface from DataStream advanced BG; > proposing it as a > Phase 3 item. > - A broader shared e2e harness beyond the core test. > > Out of scope for Phase 2 (proposing we close these): > - Multiple sources with diverging watermarks, and mixed source types. > The gate assumes one > meaningful watermark; per-input readiness is a future gate strategy. > - Guaranteed no-loss when a gated source uses idleness. Documented > limitation; a future > per-source-readiness strategy could address it, at the cost of the > very stall the > gate-clear timeout is there to catch. > - Pyflink auto-injection. JVM-only this phase; the agent rewrites the > JVM job graph and > reads records as Java objects. > > Asks: > - Agree on the four must-haves as the finish line, and sign off on the > deferred and > out-of-scope lists. > - Daniel: how close is the sandbox pipeline to serving as the core test > (item 2 below)? And can > you confirm the exactly-once expectation to assert, that green's sink > transactions do not > commit until blue has taken and committed its final checkpoint? > > Thanks. This last stretch is short if we agree on the boundary. > > - Sergio > > Hi Daniel / Ryan, > > Again thanks a lot for the feedback. Here are some thoughts: > > The following are excellent points and I indeed think we can work on > > them immediately to incorporate them. I already have some ideas on > implementation but let me flesh them out more before I share them: > > - WatermarkGenerator > - Gate Parallelism > - TransitionMode default (trivial) > > Thanks for the Misc Testing proposal (especially with SQL), it will > > indeed facilitate development and make sure the outcome/output is precise. > Let me know if/when you choose to start this effort (or any other item for > that matter). > > > Exactly-once sinks. This needs explicit testing indeed. I haven't > > verified this one yet. I believe the gate operator should ensure green's > sink transactions don't begin until blue has completed its final checkpoint > and committed. This is a good candidate for a dedicated test case before > declaring phase 1 stable. > > > Non-idempotent sinks. I'm still not sure this is a problem meant to be > > solved by Blue/Green deployments. If we think closely about it, even within > a single Flink pipeline, strict global ordering is only guaranteed within a > single subtask's partition. The moment you have parallelism > 1, different > subtasks process different keys/partitions independently and emit records > at different rates. There's no global wall-clock ordering across subtasks. > Flink's watermark mechanism gives you event-time ordering, not > arrival-order consistency across the whole topology. For pipelines where > strict ordering to a non-idempotent sink is a hard requirement, the right > pattern is a savepoint based stop/restart rather than concurrent blue/green > execution. > > > On the other hand, assuming we pursue this, your proposal sounds > > straightforward, but using state would imply the new pipeline needs to > buffer an unbounded amount of data while waiting for the first deployment > to finish, which makes memory management and state sizing very difficult. > The gate watermark barrier already provides a temporal ordering guarantee > (green doesn't advance past blue's watermark), which covers the majority of > idempotency-sensitive sinks. Perhaps I'm looking at this from an overly > simplistic perspective, is there a concrete example you can share to > illustrate the scenario you have in mind? We should definitely keep this > topic open as we make progress on the other items. > > > I'll keep you posted on progress for the items at the top. > > ⁃ Sergio > > > On Jul 2, 2026, at 4:52 PM, Sergio Chong Loo <[email protected]> > > wrote: > > > Sorry for the late reply Vamshi, these are the right questions for a > > zero-loss/zero-dup rollout. > > > One thing up front, because it answers most of these: the gate works at > > a single point in > > the job, right after a source or right before a sink. There it reads > > one watermark > > (whatever Flink has combined at that point) and checks each record's > > event-time once > > against a single cutover value. So it assumes a simple, linear spot > > where one watermark > > cleanly governs the records passing through. In short: the pipeline has > > to give the gate > > one event-time per record to compare against one cutover, once. Most of > > your questions are > > really "what if it isn't that simple," and in this Advanced mode (Phase > > 2) the answer is: > > place the gate on a stretch where it stays simple. > > 1. Multiple sources / diverging watermarks. No special rule: the gate > > uses whatever > > watermark Flink has already combined at that point (for multiple > > inputs, the minimum across > > them), plus each record's own event-time. So it effectively waits for > > the slowest input; it > > doesn't track inputs individually. Jobs that need per-input logic > > should keep the gate on a > > single-watermark stretch. > > 2. Idleness / lag. A slow-but-active source just holds the watermark > > back, so the cutover > > waits. That is safe (no loss) but can stall, which is why we are adding > > a deadline so a > > permanent stall fails loudly rather than hanging. Idleness is the sharp > > edge, and a real > > caveat on no-loss: a source marked idle is dropped from the > > current/minimum watermark > > (by design, for liveness), so the gate can cut over ahead of it. > > Records it emits afterward carry > > an event-time before the cutover, and once the outgoing deployment is > > torn down they have > > nowhere to land, so they are dropped. This is the same > > completeness-versus-liveness > > tradeoff idleness always makes, turned into a hard boundary by the > > cutover. Stated plainly: > > no-loss holds only if no source feeding the gate is idle during the > > transition. If your > > sources use idleness, treat no-loss as best-effort across the cutover. > > (A future gate > > strategy could prob. wait for per-source readiness, including idle > > sources, at the cost of > > re-introducing that stall). > > 3. Mixed sources / event-time quality. Works for: event-time jobs with > > one solid watermark > > and a clear per-record event-time where the gate sits. Out of scope for > > Phase 2: weak or > > absent event-time, uneven event-time quality across inputs, or anything > > that needs several > > watermarks combined. > > 4. Sink guarantees. The gate makes a clean split (the old job emits up > > to the cutover, the > > new job after it), so normally each record goes out once. But the gate > > only coordinates the > > cutover; it can't take back a duplicate the sink already wrote. During > > the overlap, and > > especially if a job restarts and replays, true exactly-once still needs > > an idempotent or > > transactional sink. So I'd treat idempotent-or-transactional sinks as a > > requirement for the > > no-dup/no-loss claim. For Kafka, EOS transactions are the main > > backstop. The gate narrows > > the window; the sink closes it. > > 5. Conformance matrix. Not yet: we unit-test the controller and the > > gate's record-level > > logic, but there's no end-to-end no-dup/no-loss test grid, and that's > > the real gap (and my > > next step, happy to build it with you). The grid would cross: number of > > sources, watermark > > divergence, idleness, where a failure happens, and sink type, each > > checked for duplicates > > and loss. Honest status: the single-source and idleness rows are the > > near-term ones; the > > failover + sink-type rows (the ones that matter most for a hard > > zero-loss/zero-dup bar) are > > the bigger lift and tie into the cutover-recovery work, so I wouldn't > > call failover > > correctness proven yet. Send the scenarios that matter to you and I'll > > add them. > > > And none of the "out of scope" items are dead-ends. The watermark gate > > is just one strategy > > (bluegreen.gate.strategy=WATERMARK) on an extensible gate layer. The > > whole idea is that > > more complex cutover logic (multiple watermarks, per-input readiness, > > custom rules) can be > > added as a new gate implementation instead of complicating this one. So > > "out of scope for > > Phase 2" really means "a future strategy," and that's exactly where I'd > > welcome help. > > > This is the pressure-testing Phase 2 needs. Much appreciated. > > Hi Sergio, > Thanks for driving FLIP-504 forward. This is great and the top most need > for Flink B/G cutovers. I am evaluating Phase 2 against our current > production use cases, where zero-loss/zero-duplication during cutover > > is a > > hard requirement. I would appreciate a few clarifications on the > > intended > > implementation semantics. > > 1. For multi-source jobs, if watermarks diverge, what is the intended > cutover rule (min-watermark, per-input readiness, or other)? > 2. How should idleness/temporary lag on one source affect transition > readiness to avoid loss? > 3. For mixed source types/event-time quality, what behavior is > > supported > > vs out of scope in Phase 2? > 4. For “no-dup/no-loss” expectations, what sink guarantees are assumed > (idempotent/transactional required or recommended)? For Kafka sinks > specifically, should EOS transactional mode be treated as the primary > backstop? > 5. Is there a minimal conformance test matrix planned (divergence, > idleness, failover, duplicate/loss verification)? Any guidance here > > would > > really help teams roll out safely with clear correctness boundaries. > > Thanks, > Vamshi > > > > On Jun 21, 2026, at 2:57 PM, Jing-Jia Hung <[email protected]> > > wrote: > > > Hi Sergio, > > Late to the thread. The gate auto injection via the Java agent is cool. > > Most of the discussion so far has centered on data-plane correctness. > > The > > area I'd like to understand better is the control-plane side, > > specifically > > recovery when a transition doesn't complete. The FLIP notes the > > controller > > can be left in a bad state if a deployment fails mid-transition. From > reading the draft it looks like the teardown step waits on the gate > signaling CLEAR_TO_TEARDOWN. If the gate stalls (watermark never > > advances, > > or the job fails before it signals), how do you picture the transition > unwinding? Is there a deadline that forces a rollback, or is that part > > of > > the error handling still being worked out? > > Also, I wonder how this approach would work with Pyflink DataStream > > jobs. > > My understanding is that Java agent may not be able to inject the gate > > in > > that case. Curious to hear your thoughts. > > Thank you! > Jing > > > > On 2026/05/04 17:56:58 Sergio Chong Loo wrote: > Hi Daniel / Ryan, > > > Again thanks a lot for the feedback. Here are some thoughts: > > The > following are excellent points and I indeed think we can work on them > immediately to incorporate them. I already have some ideas on > implementation but let me flesh them out more before I share them: > - > WatermarkGenerator > - Gate Parallelism > - TransitionMode default > (trivial) > > Thanks for the Misc Testing proposal (especially with > > SQL), > > it will indeed facilitate development and make sure the outcome/output > > is > > precise. Let me know if/when you choose to start this effort (or any > > other > > item for that matter). > > Exactly-once sinks. This needs explicit > > testing > > indeed. I haven't verified this one yet. I believe the gate operator > > should > > ensure green's sink transactions don't begin until blue has completed > > its > > final checkpoint and committed. This is a good candidate for a dedicated > test case before declaring phase 1 stable. > > Non-idempotent sinks. I'm > still not sure this is a problem meant to be solved by Blue/Green > deployments. If we think closely about it, even within a single Flink > pipeline, strict global ordering is only guaranteed within a single > subtask's partition. The moment you have parallelism > 1, different > subtasks process different keys/partitions independently and emit > > records > > at different rates. There's no global wall-clock ordering across > > subtasks. > > Flink's watermark mechanism gives you event-time ordering, not > arrival-order consistency across the whole topology. For pipelines where > strict ordering to a non-idempotent sink is a hard requirement, the > > right > > pattern is a savepoint based stop/restart rather than concurrent > > blue/green > > execution. > > On the other hand, assuming we pursue this, your proposal > sounds straightforward, but using state would imply the new pipeline > > needs > > to buffer an unbounded amount of data while waiting for the first > deployment to finish, which makes memory management and state sizing > > very > > difficult. The gate watermark barrier already provides a temporal > > ordering > > guarantee (green doesn't advance past blue's watermark), which covers > > the > > majority of idempotency-sensitive sinks. Perhaps I'm looking at this > > from > > an overly simplistic perspective, is there a concrete example you can > > share > > to illustrate the scenario you have in mind? We should definitely keep > > this > > topic open as we make progress on the other items. > > I'll keep you > > posted > > on progress for the items at the top. > > ⁃ Sergio > > > > On Apr 20, > > 2026, > > at 8:40 AM, Sergio Chong Loo <[email protected]> wrote: > > > > Hi > > @Daniel / > > @Ryan, > > > > Thanks a lot for the input. Similarly we’re swamped in a > time crunch here but I’ll be taking a deep dive into your feedback > hopefully before EoW. > > > > Stay tuned! > > > > - Sergio > > > >> On > > Apr > > 15, 2026, at 3:57 PM, Daniel Rossos <[email protected]> wrote: > >> > > > Hey Sergio, > >> > >> I got around to running locally as well as doing a > deeper dive into the current implementation details (Sorry about the > delay). This all looks super awesome and huge thanks for taking the > > time to > > put this together. I have some comments / questions below. Some I think > will be answered as we test further and some are potential next steps / > features that might be nice for phase 2. > >> > >> Non-Idempotent sinks > > > I believe we will still have non-idempotent sinks issues by using > > these > > gate functions. Brief recap of when this was raised before, records in > > the > > “green” deployment could be produced before the “blue” deployment > > causing > > an out-of-order delivery in record processing order which could have > implications for downstream sinks. One potential solution I was thinking > about that fits into this PR was to have the gate-operator be stateful > > and > > have your “green” pipeline accumulate messages after watermark barrier > > and > > wait for the “blue” to communicate (via the configmap) that it is done > processing before passing its records on. This would introduce a minimal > processing delay (poll rate of the “green” on configmap), but would > > ensure > > that the ordering of the streams to down stream sinks remains > > consistent. > > Other complications arise here with regards to handling state, but want > > to > > get your opinion here. > >> > >> WatermarkGenerator > >> Inserting a > watermark generator instead of requiring watermark in data. On the SQL > > side > > of things I noticed it is required that a field be a timestamp field > > that > > can be converted into a watermark. I was wondering if an alternative > > would > > be to inject a watermark generator step (generate based off some other > value), that way existing table defs won’t need to be changed to > accommodate this new feature. > >> > >> Exactly-once-sinks > >> How does > this work with exactly_once downstream sinks compatibility wise? For > example we should test using exactly_once kafka sinks to see if there > > will > > be any conflicts there. > >> > >> Gate Parallelism > >> From my > understanding, the parallelism of the gate operator has to be same as > > the > > sink/source operator it is tied to. With autoscaling how do these stay > > in > > sync? Could this cause problems? Can this gate become a performance > bottleneck somehow? > >> > >> TransitionMode Default > >> > > `transitionMode` > > not being set to default `BASIC` means all phase 1 Blue-Green deployment > would be broken specs on upgrade. I think we will want to include that > default > >> > >> Misc Testing > >> This is something for the future > > (and > > something I might play around with as I test), is adding a case to the > e2e-tests that creates a Flink pipeline (sql and/or non-sql) and checks > output from sink to ensure there are no duplicates. I don’t have a > convenient pipeline on hand > >> > >> Going forward, I am going to try > > to > > get a good test pipeline created and test this on our sandbox > > environment > > to see if I can get a good e2e run on prod-like conditions. > >> > >> > Thanks again, > >> > >> Daniel > >> > >> On Wed, Apr 1, 2026 at 12:25 PM > Sergio Chong Loo <[email protected] <[email protected]>> wrote: > >>> > Absolutely no rush, take your time. > >>> > >>> I’m also still working > > on > > some details around error handling. I’ll reach out offline so you can > always work on the latest. > >>> > >>> Thank you both as well for the > feedback! > >>> > >>> - Sergio > >>> > >>> > >>>> On Apr 1, 2026, at > 8:16 AM, Daniel Rossos <[email protected] <[email protected]>> > wrote: > >>>> > >>>> Hi Sergio, > >>>> > >>>> Sorry about the delay on > > my > > end (just got back after a few weeks off). I have caught up with + agree > with all the feedback Ryan provided in this thread. > >>>> > >>>> The > > Gate > > auto-injection idea is really cool. I'm going to try and make some time > this week to test out your PR on my side and provide feedback and > > thoughts. > > Thanks again for driving this, > >>>> Daniel > >>>> > >>>> > > > On Thu, Mar 26, 2026 at 11:13 AM Ryan van Huuksloot via dev < > > [email protected] <[email protected]>> wrote: > >>>>> Awesome! > Thanks for the update, Sergio. I'm excited to see the plan - it is > > > > a cool idea so I'm glad it is working. > >>>>> > >>>>> Ryan van > > Huuksloot > > > Staff Engineer, Infrastructure | Streaming Platform > >>>>> [image: > > Shopify] > >>>>> < > https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> > > > On Thu, Mar 26, 2026 at 1:46 AM Sergio Chong Loo < > > [email protected] <[email protected]>> > >>>>> wrote: > >>>>> > >>>>> > > @Ryan / @Daniel, > >>>>> > > >>>>> > Good news! The > “GateInjectorPipelineExecutor” idea is successful!! > >>>>> > > >>>>> > > While the original approach of simply activating it with > >>>>> > > “execution.target” did not quite work, I was able to implement it via > > the > > *Instrumentation > >>>>> > API with a Java Agent* that injects it… the > > user > > doesn’t have to touch > >>>>> > their pipelines and I added 2 options, > > at > > least for now, to place/inject > >>>>> > the Gate after the source or > before the sink (complex DAG cases with > >>>>> > multiple sources or > > sinks > > for now are not supported). > >>>>> > > >>>>> > I’m documenting > > everything > > and prepping the Draft PR for your review, > >>>>> > probably a couple > > more > > days. > >>>>> > > >>>>> > Thanks, stay tuned. > >>>>> > > >>>>> > - > > Sergio > > On Mar 16, 2026, at 3:30 PM, Sergio Chong Loo > > <[email protected] <[email protected]>> wrote: > >>>>> > > >>>>> > > > Thanks > > for the ideas and the offer to help out Ryan! It’s invaluable to > > > > learn about how other users/teams scenarios. > >>>>> > > >>>>> > Indeed > > I > > have to pursue and evaluate the GateInjectorExecutor nonetheless > > > > for our internal development. Ideally it’d be great if the user can > > simply > > “invoke” the functionality, even give the user an option to > > specify “where" > >>>>> > the gating mechanism to be placed (e.g. right > after the source or before > >>>>> > sink), or for the most flexibility > they can incorporate and place the Gate > >>>>> > manually just like it > > is > > now. > >>>>> > > >>>>> > I’ll share the progress asap and we can all > > take > > it from there (this > >>>>> > should not exceed a couple weeks). I’ll > definitely need more of your > >>>>> > feedback to verify this with > > Flink > > SQL. > >>>>> > > >>>>> > Thanks again, > >>>>> > Sergio > >>>>> > > > > > On Mar 16, 2026, at 7:01 AM, Ryan van Huuksloot < > >>>>> > > > [email protected] <[email protected]>> wrote: > >>>>> > > > > > Hi Sergio, > >>>>> > > >>>>> > re: 1.1 > >>>>> > My thought is that a > > BlueGreen Mixin isn't Kubernetes specific and could > >>>>> > be reused > > by > > other deployment control planes. However, I do agree that > >>>>> > > attaching it to the sink has other implications so I am happy to pivot > > if > > > we can find an alternative solution. > >>>>> > > >>>>> > re: 2 > > I'm happy to leave it out of the Phase 2 implementation, but I > > think it > >>>>> > should be possible. For example we use Phase 1 with > cross cluster > >>>>> > migrations today. Phase 2 within a single > > cluster > > isn't particularly useful > >>>>> > for us. > >>>>> > > >>>>> > re: > GateInjectorExecutor > >>>>> > This sounds like a neat idea. I need to > > read > > more about how it would work > >>>>> > but from a high level, injecting > > an > > operator before your sinks sounds like > >>>>> > a good idea. Better > isolation, possible with SQL, no mixins, etc. > >>>>> > > >>>>> > I will > mention that part of the reason I want it before the sinks is > >>>>> > > because nine out of ten people building pipelines struggle to > > understand > > > where their state is and how Phase 2 would affect the correctness > > of their > >>>>> > state depending on where they put the gate. I > > understand > > that if you have a > >>>>> > remote lookup and want to save bandwidth, > > you > > could optimize your pipeline > >>>>> > by moving the gate before the > > remote > > call; however, that seems like an > >>>>> > optimization that can be > > made > > later. > >>>>> > > >>>>> > Thanks for driving this! Let me know how we > > can > > help. > >>>>> > > >>>>> > Ryan van Huuksloot > >>>>> > Staff Engineer, > Infrastructure | Streaming Platform > >>>>> > [image: Shopify] > >>>>> > > < > > https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> > > > On Mon, Mar 16, 2026 at 2:21 AM Sergio Chong > > Loo <[email protected] <[email protected]>> > >>>>> > wrote: > >>>>> > > > > Hi Ryan > >>>>> >> > >>>>> >> Thanks a lot for these details. For > > sure some of these observations > >>>>> >> popped up during our initial > discussions, and that’s why our initial goal > >>>>> >> was to introduce > this as simple as possible and gradually enhance it to > >>>>> >> cover > gaps. > >>>>> >> > >>>>> >> Allow me to address your concerns: > >>>>> > > > 1. I’m happy you stressed the point of “disruption to existing > > pipelines”. However, there’s a few points about attempting to > > build this > >>>>> >> functionality into the sinks (or sources) right > > off > > the bat (read further > >>>>> >> below for my alternative): > >>>>> >> > > 1. > > Kubernetes centric: as of now the Blue/Green Deployments > >>>>> >> > > support > > is a Kubernetes specific solution, adding a mixin directly > >>>>> >> > available to sinks would “leak” this support outside of K8s > >>>>> >> > > 2. A > > sink being aware of these deployment phases violates single > >>>>> >> > responsibility, but more importantly… > >>>>> >> 3. Flink currently has > many connectors, with the majority being > >>>>> >> maintained outside > > of > > the Flink code base, by separate teams, separate > >>>>> >> repos, > > separate > > release cycles. This would complicate things significantly > >>>>> >> > > as to > > try and add support for this for every potential flink connector > > > > project out there would be a cumbersome. Blue/Green Phase 2 then only > > would > > works with "gate-aware" sinks. > >>>>> >> 2. I’d leave the > > conversation about migrating jobs between K8s > >>>>> >> clusters > > outside > > of this scope, even Phase 1 is meant to only work in a > >>>>> >> single > cluster… > >>>>> >> 3. Watermarking, excellent point, it’s indeed a > requirement so I’ll > >>>>> >> make sure this is validated where > > applicable > > (by the concrete > >>>>> >> implementation) > >>>>> >> > >>>>> >> > > > > Having said what I said about point 1.1 above, I’m currently working > > on > > an approach which uses a “GateInjectorPipelineExecutor” so to > > speak; in > >>>>> >> other words a custom PipelineExecutor that would be > shipped with the K8s > >>>>> >> Operator, invoked by Flink Configuration > (via “execution.target:”). This > >>>>> >> custom piece would > > instantiate > > and inject the Gate at a fixed point in the > >>>>> >> StreamGraph right > before job submission. I still have to validate and > >>>>> >> ensure a > > few > > things are correctly taken care of (like Type Information, > >>>>> >> > > etc.) > > but the theory looks promising. > >>>>> >> > >>>>> >> For the most part > this works well with Flink SQL (same configuration), > >>>>> >> here’s > > my > > estimation: > >>>>> >> > >>>>> >> tEnv.executeSql("INSERT INTO my_sink > ...") > >>>>> >> └─> SQL planner → ExecNodeGraph → Transformation[] > > > > └─> StreamGraph > >>>>> >> └─> GateInjectorExecutor injects > > GateProcessFunction > >>>>> >> └─> StreamGraph' (mutated) → JobGraph > > > └─> Submit Job > >>>>> >> > >>>>> >> I’m aiming to share some > > updates along these lines in the next few weeks > >>>>> >> but hopefully > this falls inline with your objectives/thoughts overall. > >>>>> >> > > > > Sergio > >>>>> >> > >>>>> >> > >>>>> >> On Mar 6, 2026, at 3:36 PM, > > Ryan > > van Huuksloot via dev < > >>>>> >> [email protected] < > [email protected]>> wrote: > >>>>> >> > >>>>> >> Hi Sergio, > > > > Thanks for starting this conversation. > >>>>> >> > >>>>> >> A few > > thoughts > > regarding BlueGreen Phase 2: > >>>>> >> 1. The Gate Operator is > > interesting > > but I don't like that we would have to > >>>>> >> modify users' > > pipelines > > for them to use Phase 2. This gate function seems > >>>>> >> like it > > could > > be a Mixin that connectors would implement. If you want to > >>>>> >> > > use > > Phase 2, your sinks must implement this Mixin. I understand that a > > > > unique GateFunction has pros, but it works less well with FlinkSQL - > > and > > the trade-off doesn't seem worthwhile. > >>>>> >> 2. Regarding > > the ConfigMap. We should consider a solution that supports > >>>>> >> > migrating Flink jobs between Kubernetes clusters. Otherwise Phase 2 is > > > only > >>>>> >> useful for in cluster operations. > >>>>> >> 3. > > Watermarking is a requirement. Will the Flink Kubernetes Operator > > > > validate that the pipeline is using watermarks? > >>>>> >> > >>>>> >> > > What happens when idleness is configured? Watermarks will get ignored > > from > > these “slow” subtasks and advance, could records from > > the ignored subtasks > >>>>> >> eventually be lost? > >>>>> >> Yes they > would be lost, but that wo [message truncated...] > > > > > > >
