I think it's going to be an antipattern to write Python configuration in Airflow to configure a Kubernetes deployment since even a "simple" deployment would likely be more classes/objects than the DAG itself. I do like the idea of having a more featured operator than the PodOperator, but if I were to choose a second operator to implement, it would definitely be a Helm operator instead of a Kube API operator.
On Tue, Jul 3, 2018 at 12:47 PM, Daniel Imberman <daniel.imber...@gmail.com> wrote: > An example of creating a deployment using the k8s model architecture can be > found here: > https://github.com/kubernetes-client/python/blob/master/ > examples/deployment_examples.py > > def create_deployment_object(): > # Configureate Pod template container > container = client.V1Container( > name="nginx", > image="nginx:1.7.9", > ports=[client.V1ContainerPort(container_port=80)]) > # Create and configurate a spec section > template = client.V1PodTemplateSpec( > metadata=client.V1ObjectMeta(labels={"app": "nginx"}), > spec=client.V1PodSpec(containers=[container])) > # Create the specification of deployment > spec = client.ExtensionsV1beta1DeploymentSpec( > replicas=3, > template=template) > # Instantiate the deployment object > deployment = client.ExtensionsV1beta1Deployment( > api_version="extensions/v1beta1", > kind="Deployment", > metadata=client.V1ObjectMeta(name=DEPLOYMENT_NAME), > spec=spec) > > return deployment > > This would involve a more k8s knowledge from the user, but would have the > massive benefit that we would not have to maintain new features as the k8s > API updates (Would simply update version). A user would have to supply is a > deployment object and possibly a "success criteria" (i.e. an endpoint to > test). > > Conversely, we could make the API a bit easier by only requiring a spec and > an optional metadata, after which we would handle a lot of the boilerplate. > > On Tue, Jul 3, 2018 at 9:20 AM Daniel Imberman <daniel.imber...@gmail.com> > wrote: > > > Hi all, > > > > Enclosed is a proposal for a kubernetes deployment management operator. I > > think this would be a good addition to current k8s offerings s.t. users > can > > actually launch persistent applications from airflow DAGs. > > > > * What?* > > Add an operator that monitors a k8s deployment, declaring the task > > complete on proper deployment/accessibility of endpoint > > > > * Why?* > > Not all tasks are single pods, sometimes you would want to run one task > > that launches a service, and then a second task that smoke tests/stress > > tests/ > > gives state to an application deployment. This would give airflow extra > > functionality as a CI/CD tool in the k8s ecosystem. > > > > * Fix:* > > Create a modification (or extension) of the k8sPodOperator that can > > handle entire deployments (possibly using the k8s model API to ensure > > full flexibility of users). > > > > Thank you. > > >
