[
https://issues.apache.org/jira/browse/AIRFLOW-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Soundar updated AIRFLOW-2141:
-
Description:
I'm trying to create Airflow variables using a json file. I am trying to import
airflow variables using UI(webserver) when I upload the json file I get this
error "Missing file or syntax error" and when I try to upload using airflow cli
not all the variables gets uploaded properly. The catch is that I have a list
of dictionary in my json file, say
ex:
{
"demo_archivedir": "/home/ubuntu/folders/archive",
"demo_filepattern": [
{ "id": "reference", "pattern": "Sample Data.xlsx" }
,
{ "id": "sale", "pattern": "Sales.xlsx" }
],
"demo_sourcepath": "/home/ubuntu/folders/input",
"demo_workdir": "/home/ubuntu/folders/working"
}
I've attached two images
img1. Using airflow variables cli command I was able to create partial
variables from my json file(airflow_cli.png)img2. After inserting logs in the
"airflow/bin/cli.py" file, I got this error. (airflow_cli2_crop.png)
The thing is I gave this value through the Admin UI one by one and it worked.
Then I exported those same variable using "airflow variables" cli command and
tried importing them, still it failed and the above mentioned error still
occurs.
Note:
I am using Python 3.5 with Airflow version 1.8
The stack trace is as follows
.compute-1.amazonaws.com:22] out: 0 of 4 variables successfully updated.
.compute-1.amazonaws.com:22] out: Traceback (most recent call last):
.compute-1.amazonaws.com:22] out: File "/home/ubuntu/Env/bin/airflow", line
28, in
.compute-1.amazonaws.com:22] out: args.func(args)
.compute-1.amazonaws.com:22] out: File
"/home/ubuntu/Env/lib/python3.5/site-packages/airflow/bin/cli.py", line 242, in
variables
.compute-1.amazonaws.com:22] out: import_helper(imp)
.compute-1.amazonaws.com:22] out: File
"/home/ubuntu/Env/lib/python3.5/site-packages/airflow/bin/cli.py", line 273, in
import_helper
.compute-1.amazonaws.com:22] out: Variable.set(k, v)
.compute-1.amazonaws.com:22] out: File
"/home/ubuntu/Env/lib/python3.5/site-packages/airflow/utils/db.py", line 53, in
wrapper
.compute-1.amazonaws.com:22] out: result = func(*args, **kwargs)
.compute-1.amazonaws.com:22] out: File
"/home/ubuntu/Env/lib/python3.5/site-packages/airflow/models.py", line 3615, in
set
.compute-1.amazonaws.com:22] out: session.add(Variable(key=key,
val=stored_value))
.compute-1.amazonaws.com:22] out: File "", line 4, in __init__
.compute-1.amazonaws.com:22] out: File
"/home/ubuntu/Env/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line
417, in _initialize_instance
.compute-1.amazonaws.com:22] out: manager.dispatch.init_failure(self, args,
kwargs)
.compute-1.amazonaws.com:22] out: File
"/home/ubuntu/Env/lib/python3.5/site-packages/sqlalchemy/util/langhelpers.py",
line 66, in __exit__
.compute-1.amazonaws.com:22] out: compat.reraise(exc_type, exc_value,
exc_tb)
.compute-1.amazonaws.com:22] out: File
"/home/ubuntu/Env/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line
187, in reraise
.compute-1.amazonaws.com:22] out: raise value
.compute-1.amazonaws.com:22] out: File
"/home/ubuntu/Env/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line
414, in _initialize_instance
.compute-1.amazonaws.com:22] out: return manager.original_init(*mixed[1:],
**kwargs)
.compute-1.amazonaws.com:22] out: File
"/home/ubuntu/Env/lib/python3.5/site-packages/sqlalchemy/ext/declarative/base.py",
line 700, in _declarative_constructor
.compute-1.amazonaws.com:22] out: setattr(self, k, kwargs[k])
compute-1.amazonaws.com:22] out: File "", line 1, in __set__
.compute-1.amazonaws.com:22] out: File
"/home/ubuntu/Env/lib/python3.5/site-packages/airflow/models.py", line 3550, in
set_val
.compute-1.amazonaws.com:22] out: self._val = FERNET.encrypt(bytes(value,
'utf-8')).decode()
.compute-1.amazonaws.com:22] out: TypeError: encoding without a string argument
.compute-1.amazonaws.com:22] out:
was:
I'm trying to create Airflow variables using a json file. I am trying to import
airflow variables using UI(webserver) when I upload the json file I get this
error "Missing file or syntax error" and when I try to upload using airflow cli
not all the variables gets uploaded properly. The catch is that I have a list
of dictionary in my json file, say
ex:
{
"demo_archivedir": "/home/ubuntu/folders/archive",
"demo_filepattern": [
{ "id": "reference", "pattern": "Sample Data.xlsx" }
,
{ "id": "sale", "pattern": "Sales.xlsx" }
],
"demo_sourcepath": "/home/ubuntu/folders/input",
"demo_workdir": "/home/ubuntu/folders/working"
}
I've attached two images
img1. Using airflow variables cli command I was able to create partial
variables from my json file(airflow_cli.png)img2. After inserting logs in the
"airflow/bin/cli.py" file, I got this error. (airflow_cli2_crop.png)
