[
https://issues.apache.org/jira/browse/AIRFLOW-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620543#comment-16620543
]
k commented on AIRFLOW-2141:
I have run into the same issue. The problem is in
[cli.py#L331|[https://github.com/apache/incubator-airflow/blob/1.10.0/airflow/bin/cli.py#L331].]
{code:java}
try:
n = 0
for k, v in d.items():
if isinstance(v, dict):
Variable.set(k, v, serialize_json=True)
else:
Variable.set(k, v)
n += 1
except Exception:
pass
finally:
print("{} of {} variables successfully updated.".format(n, len(d)))
{code}
It only tries to serialize if it is a dict. You can temporarily fix it by
adjusting the line:
{code:java}
if isinstance(v, dict) or isinstance(v, list):
Variable.set(k, v, serialize_json=True){code}
But there are probably more values that need serialization. For example, when
you export the .json file it creates integers as integers and not as strings
and it also fails.
So perhaps something like this would work...
{code:java}
if not isinstance(v, string):
Variable.set(k, v, serialize_json=True){code}
> Cannot create airflow variables when there is a list of dictionary as a value
> -
>
> Key: AIRFLOW-2141
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2141
> Project: Apache Airflow
> Issue Type: Bug
> Components: aws
>Affects Versions: 1.8.0
>Reporter: Soundar
>Priority: Major
> Labels: beginner, newbie
> Attachments: airflow_cli.png, airflow_cli2_crop.png
>
>
> I'm trying to create Airflow variables using a json file. I am trying to
> import airflow variables using UI(webserver) when I upload the json file I
> get this error "Missing file or syntax error" and when I try to upload using
> airflow cli not all the variables gets uploaded properly. The catch is that I
> have a list of dictionary in my json file, say
> ex:
> {
> "demo_archivedir": "/home/ubuntu/folders/archive",
> "demo_filepattern": [
> { "id": "reference", "pattern": "Sample Data.xlsx" }
> ,
> { "id": "sale", "pattern": "Sales.xlsx" }
> ],
> "demo_sourcepath": "/home/ubuntu/folders/input",
> "demo_workdir": "/home/ubuntu/folders/working"
> }
> I've attached two images
> img1. Using airflow variables cli command I was able to create partial
> variables from my json file(airflow_cli.png)img2. After inserting logs in the
> "airflow/bin/cli.py" file, I got this error. (airflow_cli2_crop.png)
> The thing is I gave this value through the Admin UI one by one and it worked.
> Then I exported those same variable using "airflow variables" cli command and
> tried importing them, still it failed and the above mentioned error still
> occurs.
> Note:
> I am using Python 3.5 with Airflow version 1.8
> The stack trace is as follows
> .compute-1.amazonaws.com:22] out: 0 of 4 variables successfully updated.
> .compute-1.amazonaws.com:22] out: Traceback (most recent call last):
> .compute-1.amazonaws.com:22] out: File "/home/ubuntu/Env/bin/airflow", line
> 28, in
> .compute-1.amazonaws.com:22] out: args.func(args)
> .compute-1.amazonaws.com:22] out: File
> "/home/ubuntu/Env/lib/python3.5/site-packages/airflow/bin/cli.py", line 242,
> in variables
> .compute-1.amazonaws.com:22] out: import_helper(imp)
> .compute-1.amazonaws.com:22] out: File
> "/home/ubuntu/Env/lib/python3.5/site-packages/airflow/bin/cli.py", line 273,
> in import_helper
> .compute-1.amazonaws.com:22] out: Variable.set(k, v)
> .compute-1.amazonaws.com:22] out: File
> "/home/ubuntu/Env/lib/python3.5/site-packages/airflow/utils/db.py", line 53,
> in wrapper
> .compute-1.amazonaws.com:22] out: result = func(*args, **kwargs)
> .compute-1.amazonaws.com:22] out: File
> "/home/ubuntu/Env/lib/python3.5/site-packages/airflow/models.py", line 3615,
> in set
> .compute-1.amazonaws.com:22] out: session.add(Variable(key=key,
> val=stored_value))
> .compute-1.amazonaws.com:22] out: File "", line 4, in __init__
> .compute-1.amazonaws.com:22] out: File
> "/home/ubuntu/Env/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line
> 417, in _initialize_instance
> .compute-1.amazonaws.com:22] out: manager.dispatch.init_failure(self,
> args, kwargs)
> .compute-1.amazonaws.com:22] out: File
> "/home/ubuntu/Env/lib/python3.5/site-packages/sqlalchemy/util/langhelpers.py",
> line 66, in __exit__
> .compute-1.amazonaws.com:22] out: compat.reraise(exc_type, exc_value,
> exc_tb)
> .compute-1.amazonaws.com:22] out: File
> "/home/ubuntu/Env/lib/python3.5/site-packages/sqlalchemy/util/compat.py",
> line 187, in reraise
> .compute-1.amazonaws.com:22] out: raise value
> .compute-1.amazonaws.com:22] out: File
>
