Thanks David & Teresa, We are thinking of similar kind of approach where we would have a dedicated Test env to test the Airflow upgrades. We have to preserve the metadata so we will be upgrading the Airflow version using Airflow cmd. Following steps would be done -> Setup Mysql Slave to Airflow's Mysql Metastore. -> Stop/Kill Airflow scheduler & server -> Detach Mysql Slave(It can be used for rollbacks) -> Upgrade Airflow Server & Scheduler Image -> Invoke airflow cli fro server image to upgrade DB -> Start Airflow server and scheduler. Restore metastore from slave If anything goes bad while upgrading Airflow cluster.
Thanks, Raman Gupta On 2019/06/12 17:39:11, David Klosowski <[email protected]> wrote: > Hi Raman, > > We do much the same that Teresa does. We just have a staging and > production instances and separated infrastructure environments for normal > testing and deployments. We do one of two things with upgrades > > 1. Build out a separate environment just for upgrade testing (which doesn't > take too long with Docker and cloud infrastructure) > 2. Block our CI pipeline while validation of core features is done in > staging > > We also. > > 1. Read the upgrade guide in the Airflow project > 2. Review the JIRA tickets between versions > 3. Assess any changes we may need to make to features like logging or our > components (subdags, retry handlers, alerting, airflow variables, airflow > configuration, UI etc) > 4. Do end-to-end tests of our core custom operators and sensors verifying > execution and retry cases > > All-in-all I don't think I'd feel comfortable automating this yet b/c I > think problems can exist in hidden places. > > Cheers, > David > > On Wed, Jun 12, 2019 at 5:57 AM Teresa Martyny < > [email protected]> wrote: > > > Hi Raman, > > We have a staging and preproduction instance. We block deploy of the docker > > image to production on the successful full run of our core pipeline dags in > > preproduction. > > > > If there's enough risk, we will disable the job that deploys to > > preproduction, duplicate it, and deploy a branch to preproduction. Then we > > can watch it run against the core pipeline and all external extracts for a > > few runs to feel even more confident. At which point we will let it through > > to master and our regular CI. > > > > Updating dependencies is the kind of scenario where we deploy a branch to > > preproduction or staging depending on what the dependency is since the risk > > is high. Another helpful thing is to wait a little and watch the issues in > > Jira. > > > > Hope that helps, > > Teresa > > > > > > On Wed, Jun 12, 2019, 1:14 AM [email protected] <[email protected]> > > wrote: > > > > > Hi All, > > > > > > Are there any recommended steps/process to upgrade Airflow in production. > > > We have a airflow setup in prod which is being actively used by different > > > users. So ideally we would want to have a seamless upgrade but it seems > > > Airflow upgrade are not backward compatible and might break few things. > > > So what are the best practices for Airflow Upgrade in Production. > > > > > > Thanks, > > > Raman Gupta > > > > > > > -- > > This email may contain material that is confidential and/or privileged for > > the sole use of the intended recipient. Any review, reliance, or > > distribution by others or forwarding without express permission is > > strictly > > prohibited. If you are not the intended recipient, please contact the > > sender and delete all copies. Also note that email is not an appropriate > > way to send protected health information to Omada Health employees. Please > > use your discretion when responding to this email. > > >
