Do not change the CRD but you can roll back the operator itself I believe Gyula
On Mon, 11 Sep 2023 at 18:52, Evgeniy Lyutikov <eblyuti...@avito.ru> wrote: > Is it safe to rollback the operator version with replace to old CRDs? > ------------------------------ > *От:* Evgeniy Lyutikov <eblyuti...@avito.ru> > *Отправлено:* 11 сентября 2023 г. 23:50:26 > *Кому:* Gyula Fóra > > *Копия:* user@flink.apache.org > *Тема:* Re: Flink kubernets operator delete HA metadata after resuming > from suspend > > > Hi! > No, no one could restart jobmanager, > I monitored the pods in real time, they all deleted when suspended as > expected. > > > ------------------------------ > *От:* Gyula Fóra <gyula.f...@gmail.com> > *Отправлено:* 11 сентября 2023 г. 20:34:52 > *Кому:* Evgeniy Lyutikov > *Копия:* user@flink.apache.org > *Тема:* Re: Flink kubernets operator delete HA metadata after resuming > from suspend > > Hi! > > I could not reproduce your issue, last-state suspend/restore seems to work > as before. > However these 2 logs seem very suspicious: > > 2023-09-11 06:02:07,481 o.a.f.k.o.o.d.ApplicationObserver [INFO > ][rec-job/rec-job] Observing JobManager deployment. Previous status: MISSING > 2023-09-11 06:02:07,488 o.a.f.k.o.o.d.ApplicationObserver [INFO > ][rec-job/rec-job] JobManager is being deployed > > Looks like after suspending (and deleting the JobManager Deployment) > somebody restarted the JobManager manually. Is that possible? > > Cheers, > Gyula > > On Mon, Sep 11, 2023 at 2:59 PM Evgeniy Lyutikov <eblyuti...@avito.ru> > wrote: > >> Hi all! >> After updating the operator to version 1.6.0, suspended and resuming >> flink jobs stopped working. >> When job resumes, the high availability metadata is removed. >> >> Suspend job: >> 2023-09-11 06:01:41,548 o.a.f.k.o.l.AuditUtils [INFO >> ][rec-job/rec-job] >>> Event | Info | SPECCHANGED | UPGRADE >> change(s) detected (Diff: FlinkDeploymentSpec[job.state : running -> >> suspended]), starting reconciliation. >> 2023-09-11 06:01:41,548 o.a.f.k.o.r.d.AbstractJobReconciler [INFO >> ][rec-job/rec-job] Job is in running state, ready for upgrade with >> LAST_STATE >> 2023-09-11 06:01:41,558 o.a.f.k.o.l.AuditUtils [INFO >> ][rec-job/rec-job] >>> Event | Info | SUSPENDED | Suspending >> existing deployment. >> 2023-09-11 06:01:41,558 o.a.f.k.o.s.AbstractFlinkService [INFO >> ][rec-job/rec-job] Deleting cluster with Foreground propagation >> 2023-09-11 06:01:41,558 o.a.f.k.o.s.NativeFlinkService [INFO >> ][rec-job/rec-job] Deleting JobManager deployment while preserving HA >> metadata. >> 2023-09-11 06:01:41,598 o.a.f.k.o.s.AbstractFlinkService [INFO >> ][rec-job/rec-job] Waiting for cluster shutdown... >> 2023-09-11 06:01:45,667 o.a.f.k.o.s.AbstractFlinkService [INFO >> ][rec-job/rec-job] Waiting for cluster shutdown... (5s) >> 2023-09-11 06:01:50,730 o.a.f.k.o.s.AbstractFlinkService [INFO >> ][rec-job/rec-job] Waiting for cluster shutdown... (10s) >> 2023-09-11 06:01:55,837 o.a.f.k.o.s.AbstractFlinkService [INFO >> ][rec-job/rec-job] Waiting for cluster shutdown... (15s) >> 2023-09-11 06:02:00,885 o.a.f.k.o.s.AbstractFlinkService [INFO >> ][rec-job/rec-job] Waiting for cluster shutdown... (20s) >> 2023-09-11 06:02:01,895 o.a.f.k.o.s.AbstractFlinkService [INFO >> ][rec-job/rec-job] Cluster shutdown completed. >> 2023-09-11 06:02:01,973 o.a.f.k.o.l.AuditUtils [INFO >> ][rec-job/rec-job] >>> Status | Info | SUSPENDED | The resource >> (job) has been suspended >> 2023-09-11 06:02:01,981 o.a.f.k.o.r.d.AbstractFlinkResourceReconciler >> [INFO ][rec-job/rec-job] Resource fully reconciled, nothing to do... >> >> Resume: >> 2023-09-11 06:02:07,481 o.a.f.k.o.o.d.ApplicationObserver [INFO >> ][rec-job/rec-job] Observing JobManager deployment. Previous status: MISSING >> 2023-09-11 06:02:07,488 o.a.f.k.o.o.d.ApplicationObserver [INFO >> ][rec-job/rec-job] JobManager is being deployed >> 2023-09-11 06:02:07,563 o.a.f.k.o.l.AuditUtils [INFO >> ][rec-job/rec-job] >>> Status | Info | SUSPENDED | The resource >> (job) has been suspended >> 2023-09-11 06:02:07,576 o.a.f.k.o.l.AuditUtils [INFO >> ][rec-job/rec-job] >>> Event | Info | SPECCHANGED | UPGRADE >> change(s) detected (Diff: FlinkDeploymentSpec[job.state : suspended -> >> running]), starting reconciliation. >> 2023-09-11 06:02:07,649 o.a.f.k.o.l.AuditUtils [INFO >> ][rec-job/rec-job] >>> Status | Info | UPGRADING | The resource is >> being upgraded >> 2023-09-11 06:02:07,649 o.a.f.k.o.r.d.ApplicationReconciler [INFO >> ][rec-job/rec-job] Deleting deployment with terminated application before >> new deployment >> 2023-09-11 06:02:07,649 o.a.f.k.o.s.AbstractFlinkService [INFO >> ][rec-job/rec-job] Deleting cluster with Foreground propagation >> 2023-09-11 06:02:07,649 o.a.f.k.o.s.NativeFlinkService [INFO >> ][rec-job/rec-job] Deleting JobManager deployment and HA metadata. >> 2023-09-11 06:02:07,691 o.a.f.k.o.s.AbstractFlinkService [INFO >> ][rec-job/rec-job] Waiting for cluster shutdown... >> 2023-09-11 06:02:07,763 o.a.f.k.o.s.AbstractFlinkService [INFO >> ][rec-job/rec-job] Cluster shutdown completed. >> 2023-09-11 06:02:07,763 o.a.f.k.o.s.AbstractFlinkService [INFO >> ][rec-job/rec-job] Deleting Kubernetes HA metadata >> 2023-09-11 06:02:07,820 o.a.f.k.o.s.AbstractFlinkService [INFO >> ][rec-job/rec-job] Waiting for cluster shutdown... >> 2023-09-11 06:02:07,831 o.a.f.k.o.s.AbstractFlinkService [INFO >> ][rec-job/rec-job] Cluster shutdown completed. >> 2023-09-11 06:02:07,975 o.a.f.k.o.l.AuditUtils [INFO >> ][rec-job/rec-job] >>> Status | Info | UPGRADING | The resource is >> being upgraded >> 2023-09-11 06:02:07,987 o.a.f.k.o.l.AuditUtils [INFO >> ][rec-job/rec-job] >>> Event | Info | SUBMIT | Starting >> deployment >> 2023-09-11 06:02:07,987 o.a.f.k.o.s.AbstractFlinkService [INFO >> ][rec-job/rec-job] Deploying application cluster requiring last-state from >> HA metadata >> 2023-09-11 06:02:07,999 o.a.f.k.o.c.FlinkDeploymentController >> [ERROR][rec-job/rec-job] Flink recovery failed >> 2023-09-11 06:02:08,012 o.a.f.k.o.l.AuditUtils [INFO >> ][rec-job/rec-job] >>> Event | Warning | RESTOREFAILED | HA metadata not >> available to restore from last state. It is possible that the job has >> finished or terminally failed, or the configmaps have been deleted. Manual >> restore required. >> 2023-09-11 06:02:08,099 o.a.f.k.o.l.AuditUtils [INFO >> ][rec-job/rec-job] >>> Status | Error | UPGRADING | >> {"type":"org.apache.flink.kubernetes.operator.exception.RecoveryFailureException","message":"HA >> metadata not available to restore from last state. It is possible that the >> job has finished or terminally failed, or the configmaps have been deleted. >> Manual restore required.","additionalMetadata":{},"throwableList":[]} >> 2023-09-11 06:02:08,193 o.a.f.k.o.l.AuditUtils [INFO >> ][rec-job/rec-job] >>> Status | Info | UPGRADING | The resource is >> being upgraded >> 2023-09-11 06:02:08,218 o.a.f.k.o.l.AuditUtils [INFO >> ][rec-job/rec-job] >>> Event | Info | SUBMIT | Starting >> deployment >> 2023-09-11 06:02:08,218 o.a.f.k.o.s.AbstractFlinkService [INFO >> ][rec-job/rec-job] Deploying application cluster requiring last-state from >> HA metadata >> 2023-09-11 06:02:08,228 o.a.f.k.o.c.FlinkDeploymentController >> [ERROR][rec-job/rec-job] Flink recovery failed >> >> >> >> >> * ------------------------------ *“This message contains confidential >> information/commercial secret. If you are not the intended addressee of >> this message you may not copy, save, print or forward it to any third party >> and you are kindly requested to destroy this message and notify the sender >> thereof by email. >> Данное сообщение содержит конфиденциальную информацию/информацию, >> являющуюся коммерческой тайной. Если Вы не являетесь надлежащим адресатом >> данного сообщения, Вы не вправе копировать, сохранять, печатать или >> пересылать его каким либо иным лицам. Просьба уничтожить данное сообщение и >> уведомить об этом отправителя электронным письмом.” >> >