dashton90 commented on PR #35839: URL: https://github.com/apache/airflow/pull/35839#issuecomment-1828524090
@ferruzzi Tried this example myself against the latest version of the providers package and it looks like there are a few things going on. 1. [Instances cannot be hibernated for a few minutes after they are started](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/troubleshoot-instance-hibernate.html#hibernate-troubleshooting-1). This is not a problem per se, but could cause flakiness in the example. I haven't found any documentation on checking if an instance is ready to hibernate. A hacky fix would be to sleep before running the hibernate operator. 2. Instance takes a long time to finish hibernating. This is anecdotal, but hibernate on a t3.micro is taking 10-15 minutes to go from `Stopping` to `Stopped`. I'm not sure if this could be solved by using an AMI with a larger root volume (the one we are using is 2GB vs 0.5GB of memory for the t3.micro so there should be sufficient space). Again, I can't find any docs on the "expected" amount of time it takes an instance to hibernate. We can add a longer duration for `wait_for_completion` in the operator and this should be fine. The downside is that it will now take the DAG 15-20 minutes to complete. If that's acceptable to the AWS team then we can go this route. 3. Instance can't be terminated while in `Stopping` state. I don't know if this is a botocore bug, or expected, but we can terminate a stopping instance through the console. [Botocore](https://github.com/boto/botocore/blob/81c0e57059de3ed7a56bf183e322a83f7a9e80ec/botocore/data/ec2/2016-11-15/waiters-2.json#L276-L281) explicitly disallows this. It could be a limitation of the EC2 API, but if not I can open an issue with botocore. This won't be a problem if we wait for hibernation to finish as outlined above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
