GitHub user aipatr added a comment to the discussion: Trying to upgrade to 
Airflow 3

> > SQL: ALTER TABLE dag_code ALTER COLUMN dag_id SET NOT NULL]
> 
> I think that would be fixed by this PR #49563.
> 
> A temporary fix for now run `TRUNCATE dag_code` once first. Though was this 
> an empty DB or with existing data in it ?

We are one step forward.
Now this happens :
[2025-04-24T16:07:46.435+0200] {migration.py:618} INFO - Running upgrade 
9fc3fc5de720 -> eed27faa34e3, Remove pickled data from xcom table.
Traceback (most recent call last):
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/sqlalchemy/engine/base.py",
 line 1910, in _execute_context
    self.dialect.do_execute(
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/sqlalchemy/engine/default.py",
 line 736, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.InvalidTextRepresentation: invalid input syntax for type json
DETAIL:  Token "NaN" is invalid.
CONTEXT:  JSON data, line 1: ...{\"0\":null,\"1\":1.0}}", "magn_conf_margin": 
NaN...


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/miniconda3/envs/pytorch/bin/airflow", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/airflow/__main__.py",
 line 55, in main
    args.func(args)
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/airflow/cli/cli_config.py",
 line 48, in command
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/airflow/utils/cli.py",
 line 111, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/airflow/utils/providers_configuration_loader.py",
 line 55, in wrapped_function
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/airflow/cli/commands/db_command.py",
 line 197, in migratedb
    run_db_migrate_command(args, db.upgradedb, _REVISION_HEADS_MAP)
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/airflow/cli/commands/db_command.py",
 line 125, in run_db_migrate_command
    command(
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/airflow/utils/session.py",
 line 101, in wrapper
    return func(*args, session=session, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/airflow/utils/db.py",
 line 1142, in upgradedb
    command.upgrade(config, revision=to_revision or "heads")
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/alembic/command.py", 
line 408, in upgrade
    script.run_env()
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/alembic/script/base.py",
 line 586, in run_env
    util.load_python_file(self.dir, "env.py")
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/alembic/util/pyfiles.py",
 line 95, in load_python_file
    module = load_module_py(module_id, path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/alembic/util/pyfiles.py",
 line 113, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap_external>", line 999, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/airflow/migrations/env.py",
 line 138, in <module>
    run_migrations_online()
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/airflow/migrations/env.py",
 line 132, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/alembic/runtime/environment.py",
 line 946, in run_migrations
    self.get_context().run_migrations(**kw)
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/alembic/runtime/migration.py",
 line 623, in run_migrations
    step.migration_fn(**kw)
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/airflow/migrations/versions/0049_3_0_0_remove_pickled_data_from_xcom_table.py",
 line 101, in upgrade
    op.execute(
  File "<string>", line 8, in execute
  File "<string>", line 3, in execute
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/alembic/operations/ops.py",
 line 2552, in execute
    return operations.invoke(op)
           ^^^^^^^^^^^^^^^^^^^^^
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/alembic/operations/base.py",
 line 441, in invoke
    return fn(self, operation)
           ^^^^^^^^^^^^^^^^^^^
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/alembic/operations/toimpl.py",
 line 223, in execute_sql
    operations.migration_context.impl.execute(
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/alembic/ddl/impl.py",
 line 253, in execute
    self._exec(sql, execution_options)
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/alembic/ddl/impl.py",
 line 246, in _exec
    return conn.execute(construct, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/sqlalchemy/future/engine.py",
 line 286, in execute
    return self._execute_20(
           ^^^^^^^^^^^^^^^^^
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/sqlalchemy/engine/base.py",
 line 1710, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/sqlalchemy/sql/elements.py",
 line 334, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/sqlalchemy/engine/base.py",
 line 1577, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/sqlalchemy/engine/base.py",
 line 1953, in _execute_context
    self._handle_dbapi_exception(
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/sqlalchemy/engine/base.py",
 line 2134, in _handle_dbapi_exception
    util.raise_(
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/sqlalchemy/util/compat.py",
 line 211, in raise_
    raise exception
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/sqlalchemy/engine/base.py",
 line 1910, in _execute_context
    self.dialect.do_execute(
  File 
"/opt/miniconda3/envs/pytorch/lib/python3.12/site-packages/sqlalchemy/engine/default.py",
 line 736, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.DataError: (psycopg2.errors.InvalidTextRepresentation) invalid 
input syntax for type json
DETAIL:  Token "NaN" is invalid.
CONTEXT:  JSON data, line 1: ...{\"0\":null,\"1\":1.0}}", "magn_conf_margin": 
NaN...

[SQL: 
            ALTER TABLE xcom
            ALTER COLUMN value TYPE JSONB
            USING CASE
                WHEN value IS NOT NULL THEN CAST(CONVERT_FROM(value, 'UTF8') AS 
JSONB)
                ELSE NULL
            END
            ]
(Background on this error at: https://sqlalche.me/e/14/9h9h)

GitHub link: 
https://github.com/apache/airflow/discussions/49707#discussioncomment-12935993

----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]

Reply via email to