davlum commented on a change in pull request #6230: [AIRFLOW-5413] Allow K8S
worker pod to be configured from JSON/YAML file
URL: https://github.com/apache/airflow/pull/6230#discussion_r341365857
##########
File path: airflow/kubernetes/pod_generator.py
##########
@@ -62,76 +71,116 @@ class PodGenerator:
Contains Kubernetes Airflow Worker configuration logic
Represents a kubernetes pod and manages execution of a single pod.
+ Any configuration that is container specific gets applied to
+ the first container in the list of containers.
+
+ Parameters with a type of `kubernetes.client.models.*`/`k8s.*` can
+ often be replaced with their dictionary equivalent, e.i. the output of
+ `sanitize_for_serialization`.
:param image: The docker image
- :type image: str
+ :type image: Optional[str]
+ :param name: name in the metadata section (not the container name)
+ :type name: Optional[str]
+ :param namespace: pod namespace
+ :type namespace: Optional[str]
+ :param volume_mounts: list of kubernetes volumes mounts
+ :type volume_mounts: Optional[List[Union[k8s.V1VolumeMount, dict]]]
:param envs: A dict containing the environment variables
- :type envs: Dict[str, str]
- :param cmds: The command to be run on the pod
- :type cmds: List[str]
- :param secrets: Secrets to be launched to the pod
- :type secrets: List[airflow.kubernetes.models.secret.Secret]
+ :type envs: Optional[Dict[str, str]]
+ :param cmds: The command to be run on the first container
+ :type cmds: Optional[List[str]]
+ :param args: The arguments to be run on the pod
+ :type args: Optional[List[str]]
+ :param labels: labels for the pod metadata
+ :type labels: Optional[Dict[str, str]]
+ :param node_selectors: node selectors for the pod
+ :type node_selectors: Optional[Dict[str, str]]
+ :param ports: list of ports. Applies to the first container.
+ :type ports: Optional[List[Union[k8s.V1ContainerPort, dict]]]
+ :param volumes: Volumes to be attached to the first container
+ :type volumes: Optional[List[Union[k8s.V1Volume, dict]]]
:param image_pull_policy: Specify a policy to cache or always pull an image
:type image_pull_policy: str
+ :param restart_policy: The restart policy of the pod
+ :type restart_policy: str
:param image_pull_secrets: Any image pull secrets to be given to the pod.
If more than one secret is required, provide a comma separated list:
secret_a,secret_b
:type image_pull_secrets: str
+ :param init_containers: A list of init containers
+ :type init_containers: Optional[List[k8s.V1Container]]
+ :param service_account_name: Identity for processes that run in a Pod
+ :type service_account_name: Optional[str]
+ :param resources: Resource requirements for the first containers
+ :type resources: Optional[Union[k8s.V1ResourceRequirements, dict]]
+ :param annotations: annotations for the pod
+ :type annotations: Optional[Dict[str, str]]
:param affinity: A dict containing a group of affinity scheduling rules
- :type affinity: dict
+ :type affinity: Optional[dict]
:param hostnetwork: If True enable host networking on the pod
:type hostnetwork: bool
:param tolerations: A list of kubernetes tolerations
- :type tolerations: list
+ :type tolerations: Optional[list]
:param security_context: A dict containing the security context for the pod
- :type security_context: dict
+ :type security_context: Optional[Union[k8s.V1PodSecurityContext, dict]]
:param configmaps: Any configmap refs to envfrom.
- If more than one configmap is required, provide a comma separated list
- configmap_a,configmap_b
- :type configmaps: str
+ :type configmaps: Optional[List[str]]
:param dnspolicy: Specify a dnspolicy for the pod
- :type dnspolicy: str
- :param pod: The fully specified pod.
- :type pod: kubernetes.client.models.V1Pod
+ :type dnspolicy: Optional[str]
+ :param pod: The fully specified pod. Mutually exclusive with
`path_or_string`
+ :type pod: Optional[kubernetes.client.models.V1Pod]
+ :param pod_template_file_or_string: Path to JSON/YAML file. Mutually
exclusive with `pod`
+ :type pod_template_file_or_string: Optional[str]
+ :param extract_xcom: Whether to bring up a container for xcom
+ :type extract_xcom: bool
"""
def __init__( # pylint: disable=too-many-arguments,too-many-locals
self,
- image,
- name=None,
- namespace=None,
- volume_mounts=None,
- envs=None,
- cmds=None,
- args=None,
- labels=None,
- node_selectors=None,
- ports=None,
- volumes=None,
- image_pull_policy='IfNotPresent',
- restart_policy='Never',
- image_pull_secrets=None,
- init_containers=None,
- service_account_name=None,
- resources=None,
- annotations=None,
- affinity=None,
- hostnetwork=False,
- tolerations=None,
- security_context=None,
- configmaps=None,
- dnspolicy=None,
- pod=None,
- extract_xcom=False,
+ image: Optional[str] = None,
+ name: Optional[str] = None,
+ namespace: Optional[str] = None,
+ volume_mounts: Optional[List[Union[k8s.V1VolumeMount, dict]]] = None,
+ envs: Optional[Dict[str, str]] = None,
+ cmds: Optional[List[str]] = None,
+ args: Optional[List[str]] = None,
+ labels: Optional[Dict[str, str]] = None,
+ node_selectors: Optional[Dict[str, str]] = None,
+ ports: Optional[List[Union[k8s.V1ContainerPort, dict]]] = None,
+ volumes: Optional[List[Union[k8s.V1Volume, dict]]] = None,
+ image_pull_policy: str = 'IfNotPresent',
+ restart_policy: str = 'Never',
+ image_pull_secrets: Optional[str] = None,
+ init_containers: Optional[List[k8s.V1Container]] = None,
+ service_account_name: Optional[str] = None,
+ resources: Optional[Union[k8s.V1ResourceRequirements, dict]] = None,
+ annotations: Optional[Dict[str, str]] = None,
+ affinity: Optional[dict] = None,
+ hostnetwork: bool = False,
+ tolerations: Optional[list] = None,
+ security_context: Optional[Union[k8s.V1PodSecurityContext, dict]] =
None,
+ configmaps: Optional[List[str]] = None,
+ dnspolicy: Optional[str] = None,
+ pod: Optional[k8s.V1Pod] = None,
+ pod_template_file_or_string: Optional[str] = None,
Review comment:
The `extract_xcom` support works. You can see this in the [test
case](https://github.com/apache/airflow/blob/c337a63786675e3bb33f6d759e4f8f882b3d0353/tests/integration/kubernetes/test_kubernetes_pod_operator.py#L591).
What problems do you foresee having this as a parameter?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
