gudata commented on issue #55215:
URL: https://github.com/apache/airflow/issues/55215#issuecomment-3253594100
What I observed in Airflow 2.x is that the redis database become full.
Because I do know the internals of airflow I decided to wrote a small script
which set the TTL of all existing records to 30days.
I believe /but I am not sure/ that those are left overs from the celery
while it executes the tasks.
My solution was to run this script daily setting TTL on the keys which do
not have TTL.
I skip keys with prefix "unacked" and "_kombu.binding".
Now with the Airflow 3.x I am not sure if those prefixes are still valid.
I have purged now one redis instance and am waiting to see what keys will be
created. Then my plan is to return to the old solution by setting the TTL.
I thought I would see some keys if I ran some tasks and click around but
strange - I didn't saw any keys created with the new Airflow 3.0 . Maybe this
is not a problem now an Airflow is doing the cleanup - but then why my
production instance complain of full redis?!.
I don't know why Airflow do not set the TTL internally or include that to
the db clean task.
```bash
#!/bin/bash
# Usage message
usage() {
echo "Usage: $0 -h <host> -p <port>"
exit 1
}
# Default values
HOST=$1
PORT=$2
TTL_SECONDS=259200 # 3 days in seconds
# Parse command-line arguments
while getopts "h:p:" opt; do
case $opt in
h) HOST="$OPTARG" ;;
p) PORT="$OPTARG" ;;
*) usage ;;
esac
done
# Validate arguments
if [[ -z "$HOST" || -z "$PORT" ]]; then
usage
fi
# 54000 - 15days
# Inline Lua script
lua_script='
local keys = redis.call("KEYS", "*")
for _, key in ipairs(keys) do
if string.match(key, "^unacked") or string.match(key,
"^_kombu%.binding") then
-- Skip the key if it matches the pattern
else
if redis.call("TTL", key) == -1 then
redis.call("EXPIRE", key, 54000)
end
end
end
'
# Execute the Lua script
redis6-cli -h "$HOST" -p "$PORT" --eval <(echo "$lua_script")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
