[GitHub] [airflow] ayman-albaz commented on issue #28372: Airflow KubernetesPodOperator task running despite no resources being available

2022-12-16 Thread GitBox 


ayman-albaz commented on issue #28372:
URL: https://github.com/apache/airflow/issues/28372#issuecomment-1355872345

   > You can limit the number of concurrent task run by airflow thanks to 
airflow pool
   > 
   > or concurrency settings on the dag
   
   Ya perhaps this is not necessarily a bug from airflow then but more of a 
feature request. What you suggested is a workaround to my problem, thanks for 
the suggestion.
   
   I think it would be pretty cool if Airflow scheduler was aware of resources 
+ auto scaling capability of a cluster, and then schedule accordingly (i.e. 
keep running jobs, and schedule the remainder that no resources can possibly be 
allocated for).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ayman-albaz commented on issue #28372: Airflow KubernetesPodOperator task running despite no resources being available

2022-12-14 Thread GitBox 


ayman-albaz commented on issue #28372:
URL: https://github.com/apache/airflow/issues/28372#issuecomment-1352559497

   Also some additional info
   ```
   
   kubectl -n airflow describe node
   Name:   kind-control-plane
   Roles:  master
   Labels: beta.kubernetes.io/arch=amd64
   beta.kubernetes.io/os=linux
   kubernetes.io/arch=amd64
   kubernetes.io/hostname=kind-control-plane
   kubernetes.io/os=linux
   node-role.kubernetes.io/master=
   Annotations:kubeadm.alpha.kubernetes.io/cri-socket: 
unix:///run/containerd/containerd.sock
   node.alpha.kubernetes.io/ttl: 0
   volumes.kubernetes.io/controller-managed-attach-detach: 
true
   CreationTimestamp:  Mon, 12 Dec 2022 21:54:55 -0500
   Taints: 
   Unschedulable:  false
   Lease:
 HolderIdentity:  kind-control-plane
 AcquireTime: 
 RenewTime:   Wed, 14 Dec 2022 22:50:52 -0500
   Conditions:
 Type Status  LastHeartbeatTime 
LastTransitionTimeReason   Message
  --  - 
----   ---
 MemoryPressure   False   Wed, 14 Dec 2022 22:48:10 -0500   Wed, 14 Dec 
2022 20:03:49 -0500   KubeletHasSufficientMemory   kubelet has sufficient 
memory available
 DiskPressure False   Wed, 14 Dec 2022 22:48:10 -0500   Wed, 14 Dec 
2022 20:03:49 -0500   KubeletHasNoDiskPressure kubelet has no disk pressure
 PIDPressure  False   Wed, 14 Dec 2022 22:48:10 -0500   Wed, 14 Dec 
2022 20:03:49 -0500   KubeletHasSufficientPID  kubelet has sufficient PID 
available
 ReadyTrueWed, 14 Dec 2022 22:48:10 -0500   Wed, 14 Dec 
2022 20:03:49 -0500   KubeletReady kubelet is posting ready 
status
   Addresses:
 Hostname:kind-control-plane
   Capacity:
 cpu:8
 ephemeral-storage:  382935608Ki
 hugepages-1Gi:  0
 hugepages-2Mi:  0
 memory: 16336528Ki
 pods:   110
   Allocatable:
 cpu:8
 ephemeral-storage:  382935608Ki
 hugepages-1Gi:  0
 hugepages-2Mi:  0
 memory: 16336528Ki
 pods:   110
   System Info:
 Kernel Version: 5.15.0-56-generic
 OS Image:   Ubuntu 20.10
 Operating System:   linux
 Architecture:   amd64
 Container Runtime Version:  containerd://1.4.0-106-gce4439a8
 Kubelet Version:v1.18.15
 Kube-Proxy Version: v1.18.15
   PodCIDR:  10.244.0.0/24
   PodCIDRs: 10.244.0.0/24
   ProviderID:   kind://docker/kind/kind-control-plane
   Non-terminated Pods:  (33 in total)
 Namespace   Name   
  CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
 -      
    --  ---  -  ---
 airflow airflow-postgresql-0   
  250m (3%) 0 (0%)  256Mi (1%)   0 (0%) 2d
 airflow airflow-scheduler-6dd68b485c-w8bhp 
  0 (0%)0 (0%)  0 (0%)   0 (0%) 2d
 airflow airflow-statsd-586dbdcc6b-h4mnr
  0 (0%)0 (0%)  0 (0%)   0 (0%) 2d
 airflow airflow-triggerer-95565b95d-phts7  
  0 (0%)0 (0%)  0 (0%)   0 (0%) 2d
 airflow airflow-webserver-599bb95bcd-7dtpk 
  0 (0%)0 (0%)  0 (0%)   0 (0%) 2d
 airflow my-task-dbeba55a80074c08bbdf023b3f0b885c   
  2001m (25%)   2 (25%) 512Mi (3%)   512Mi (3%) 7m39s
 airflow my-task-e2d57a9de7934aaabd9b4c481c0b8fde   
  2001m (25%)   2 (25%) 512Mi (3%)   512Mi (3%) 7m39s
 airflow my-task-f83ad2805d314be3a7307b7216a54e53   
  2001m (25%)   2 (25%) 512Mi (3%)   512Mi (3%) 7m35s
 airflow 
pipeline-my-task-0c6bba5cd622466fa6e234d3dbd9151b0 (0%)0 (0%)  
0 (0%)   0 (0%) 67s
 airflow 
pipeline-my-task-19efa573444842f0b259f72f016809940 (0%)0 (0%)  
0 (0%)   0 (0%) 59s
 airflow 
pipeline-my-task-22faa8c5d7ce4e439917be3e4cb9bbdf0 (0%)0 (0%)  
0 (0%)   0 (0%) 51s