wow OK I didn't know this was an option. I'll see what I can do and write back if we make progress. Thank you Daniel.
On 2021/06/10 01:01:23, Daniel Standish <[email protected]> wrote: > > Thanks Daniel. We would need to add this as an option to the K8s Helm > Chart and that is kinda outside our current scope. > > I may be misunderstanding what you're saying, but there's no need to modify > the helm chart. > > Generally speaking all airflow helm charts support using a metastore that > is deployed outside of kubernetes. > > Airflow just needs the creds. > > E.g. with the official helm chart, you can use an external database with > this configuration (i.e. in helm config.yaml): > > data: > metadataSecretName: metastore-uri > resultBackendSecretName: result-backend > > pgbouncer: > enabled: false > > postgresql: > enabled: false > > All that is required is you create k8s secrets with the conn URI. > > The secrets have to be structured like this: > > locals { > metastore_uri = "" > } > resource "kubernetes_secret" "metastore" { > metadata { > name = "metastore-uri" > namespace = var.environment_name > } > data = { > connection = "postgresql://${local.metastore_uri}" > } > type = "Opaque" > } > > resource "kubernetes_secret" "result_backend" { > metadata { > name = "result-backend" > namespace = var.environment_name > } > data = { > connection = "db+${kubernetes_secret.metastore.data.connection}" > } > type = "Opaque" > } > > > > > > > > > > On Wed, Jun 9, 2021 at 4:04 PM Lewis John McGibbney <[email protected]> > wrote: > > > Thanks Daniel. We would need to add this as an option to the K8s Helm > > Chart and that is kinda outside our current scope. > > Thanks for the suggestion. > > lewismc > > > > On 2021/06/09 05:46:00, Daniel Standish <[email protected]> wrote: > > > Perhaps it goes without saying but you might consider using cloud sql > > > option such as aws rds, which provides persistence even if you destroy > > and > > > rebuild your k8s cluster, and of course automated backups. > > > > > > On Tue, Jun 8, 2021, 10:41 PM Sumit Maheshwari <[email protected]> > > wrote: > > > > > > > If you are backing up data to safeguard against pod failures, then I > > > > believe that you can use a PV as data storage for PSql & it would > > survive > > > > any pod restarts. > > > > > > > > On Wed, Jun 9, 2021 at 5:01 AM Lewis John McGibbney < > > [email protected]> > > > > wrote: > > > > > > > >> Hi users@, > > > >> Does anyone have a recommended/best practice/preferred way for > > backing up > > > >> PostgreSQL when Airflow is deployed into K8s? > > > >> We were thinking of writing a maintenance DAG which would do this... > > > >> maybe even contributing it to > > > >> https://github.com/teamclairvoyant/airflow-maintenance-dags. > > > >> I'm thinking it would just authenticate into the K8s cluster, find the > > > >> postgresql pod and then perform a DB archival in s3 or something. > > > >> I also looked more into > > > >> https://www.postgresql.org/docs/9.1/continuous-archiving.html which > > > >> looks appealing. I'm interest to see what others are doing. > > > >> Any suggestions appreciated. > > > >> Thank you > > > >> lewismc > > > >> > > > > > > > > > >
