Hi all, I'll push a fix today. I've created a Jira ticket: https://issues.apache.org/jira/browse/AIRFLOW-1731
Cheers, Fokko 2017-10-17 19:27 GMT+02:00 Driesprong, Fokko <[email protected]>: > Hi Andy, > > I see something weird. While debugging the code, the paths as documented > on the updating.md are not on the pythonpath: > ['/usr/local/bin', '/usr/local/lib/python36.zip', > '/usr/local/lib/python3.6', '/usr/local/lib/python3.6/lib-dynload', > '/usr/local/lib/python3.6/site-packages'] > > This is an installed instance of 1.9-alpha using the .tar.gz of Chris. I > would also expect more paths appended to the PYTHONPATH. So when I try to > run an example with the config provided by Andrew: > logging_config_class = airflow_logging_settings.LOGGING_CONFIG > > I end up with the same error as Andrew: > root@9e3cf03c0544:~# airflow test example_bash_operator runme_0 2017-01-01 > <20%2017%2001%2001> > Current path: > ['/usr/local/bin', '/usr/local/lib/python36.zip', > '/usr/local/lib/python3.6', '/usr/local/lib/python3.6/lib-dynload', > '/usr/local/lib/python3.6/site-packages'] > Traceback (most recent call last): > File "/usr/local/lib/python3.6/site-packages/airflow/logging_config.py", > line 40, in configure_logging > logging_config = import_string(logging_class_path) > File > "/usr/local/lib/python3.6/site-packages/airflow/utils/module_loading.py", > line 28, in import_string > module = import_module(module_path) > File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in > import_module > return _bootstrap._gcd_import(name[level:], package, level) > File "<frozen importlib._bootstrap>", line 978, in _gcd_import > File "<frozen importlib._bootstrap>", line 961, in _find_and_load > File "<frozen importlib._bootstrap>", line 948, in > _find_and_load_unlocked > ModuleNotFoundError: No module named 'airflow_logging_settings' > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/usr/local/bin/airflow", line 16, in <module> > from airflow import configuration > File "/usr/local/lib/python3.6/site-packages/airflow/__init__.py", line > 31, in <module> > from airflow import settings > File "/usr/local/lib/python3.6/site-packages/airflow/settings.py", line > 148, in <module> > configure_logging() > File "/usr/local/lib/python3.6/site-packages/airflow/logging_config.py", > line 52, in configure_logging > 'Unable to load custom logging from {}'.format(logging_class_path) > ImportError: Unable to load custom logging from airflow_logging_settings. > LOGGING_CONFIG > > > When explicitly passing the path when starting the test: > root@9e3cf03c0544:~# PYTHONPATH=/root/airflow/plugins/ airflow test > example_bash_operator runme_0 2017-01-01 <20%2017%2001%2001> > Current path: > ['/usr/local/bin', '/root/airflow/plugins', '/usr/local/lib/python36.zip', > '/usr/local/lib/python3.6', '/usr/local/lib/python3.6/lib-dynload', > '/usr/local/lib/python3.6/site-packages'] > [2017-10-17 17:21:19,793] {__init__.py:45} INFO - Using executor > SequentialExecutor > [2017-10-17 17:21:19,835] {models.py:186} INFO - Filling up the DagBag > from /root/airflow/dags > [2017-10-17 17:21:19,880] {dag.py:31} WARNING - test warn > [2017-10-17 17:21:19,880] {dag.py:32} INFO - test info > [2017-10-17 17:21:19,902] {models.py:1165} INFO - Dependencies all met for > <TaskInstance: example_bash_operator.runme_0 2017-01-01 00:00:00 [None]> > [2017-10-17 17:21:19,904] {models.py:1165} INFO - Dependencies all met for > <TaskInstance: example_bash_operator.runme_0 2017-01-01 00:00:00 [None]> > [2017-10-17 17:21:19,905] {models.py:1375} INFO - > ------------------------------------------------------------ > -------------------- > Starting attempt 1 of 1 > ------------------------------------------------------------ > -------------------- > > [2017-10-17 17:21:19,905] {models.py:1396} INFO - Executing > <Task(BashOperator): runme_0> on 2017-01-01 00:00:00 > [2017-10-17 17:21:19,916] {bash_operator.py:70} INFO - Tmp dir root > location: > /tmp > [2017-10-17 17:21:19,919] {bash_operator.py:80} INFO - Temporary script > location: /tmp/airflowtmpmco2abof//tmp/airflowtmpmco2abof/runme_0w53ja72x > [2017-10-17 17:21:19,919] {bash_operator.py:82} INFO - Running command: > echo "example_bash_operator__runme_0__20170101 <20%2017%2001%2001>" && > sleep 1 > [2017-10-17 17:21:19,924] {bash_operator.py:91} INFO - Output: > [2017-10-17 17:21:19,925] {bash_operator.py:95} INFO - > example_bash_operator__runme_0__20170101 <20%2017%2001%2001> > [2017-10-17 17:21:20,928] {bash_operator.py:99} INFO - Command exited with > return code 0 > > This will require a patch for 1.9 because I expected more paths being > loaded by default. I checked this before, and the paths mentioned in the > updating.md where present, anyone any idea? > > Cheers, Fokko > > > > 2017-10-17 13:04 GMT+02:00 Andrew Maguire <[email protected]>: > >> Hey, >> >> My airflow_logging_settings.py is attached. (is the same as the >> airflow_local_settings.py you mention - i think i saw both and >> 'airflow_local_settings.py' and an 'airflow_logging_settings.py' referenced >> in two separate places although i think it's pretty much the same thing, >> just slightly different filenames. I think one is mentioned in the >> updating.md file and the other is what the file is actually called in >> the github repo. Am guessing does not matter so long as i refrence the >> correct file in airflow.cfg). >> >> I had not renamed DEFAULT_LOGGING_CONFIG to LOGGING_CONFIG in my >> airflow_logging_settings.py. I did and uncommented the class line in my >> airflow.cfg and still got below error: >> >> andrew_maguire@airflow-server:~$ airflow list_dags >> Traceback (most recent call last): >> File "/usr/local/bin/airflow", line 16, in <module> >> from airflow import configuration >> File "/usr/local/lib/python2.7/dist-packages/airflow/__init__.py", line >> 31, in <module> >> from airflow import settings >> File "/usr/local/lib/python2.7/dist-packages/airflow/settings.py", line >> 148, in <module> >> configure_logging() >> File "/usr/local/lib/python2.7/dist-packages/airflow/logging_config.py", >> line 47, in configure_logging >> 'Unable to load custom logging from {}'.format(logging_class_path) >> ImportError: Unable to load custom logging from >> plugins.airflow_logging_settings.LOGGING_CONFIG >> >> So don't think my error even got that far. >> >> Happy to help debug or test or be of any use. >> >> Cheers, >> Andy >> >> >> On Tue, Oct 17, 2017 at 11:43 AM Driesprong, Fokko <[email protected]> >> wrote: >> >>> Hi Andy, >>> >>> Thanks for reaching out. We are debugging the new logging, and input from >>> the community is highly appreciated. >>> >>> If you are using Python 2, you'll need to put an empty __init__.py in all >>> the directory, so ~/airflow/plugins/__init__.py, this needs to be empty. >>> Could you share your airflow_local_settings.py? If there are any GCS >>> credentials, please remove them. Please check that you've renamed >>> the DEFAULT_LOGGING_CONFIG variable to LOGGING_CONFIG, this might not be >>> evident from the updating.md. >>> >>> Cheers, Fokko >>> >>> 2017-10-17 12:00 GMT+02:00 Andrew Maguire <[email protected]>: >>> >>> > Hi, >>> > >>> > I've updated to 1.9 but am having trouble setting the >>> logging_config_class >>> > class path in the airflow.cfg file. >>> > >>> > Currently i have below files in {AIRFLOW_HOME}/plugins >>> > >>> > [image: image.png] >>> > Where airflow_logging_settings.py is just a copy of this file >>> > <https://github.com/apache/incubator-airflow/blob/master/air >>> flow/config_templates/airflow_local_settings.py> >>> >>> > but with the GCS stuff uncommented and a line added for GCS_LOG_FOLDER >>> to >>> > be pulled from airflow.cfg just like BASE_LOG_FOLDER >>> > >>> > then in my {AIRFLOW_HOME}/airflow.cfg file i have this following line >>> to >>> > set up the log stuff: >>> > >>> > # The folder where airflow should store its log files >>> > # This path must be absolute >>> > base_log_folder = {AIRFLOW_HOME}/logs >>> > >>> > gcs_log_folder = gs://pmc-airflow/logs >>> > >>> > # Airflow can store logs remotely in AWS S3 or Google Cloud Storage. >>> Users >>> > # must supply an Airflow connection id that provides access to the >>> storage >>> > # location. >>> > remote_log_conn_id = my_gcp_connection >>> > encrypt_s3_logs = False >>> > >>> > # Logging level >>> > logging_level = INFO >>> > >>> > # Logging class >>> > # Specify the class that will specify the logging configuration >>> > # This class has to be on the python classpath >>> > logging_config_class = plugins.airflow_logging_settings.LOGGING_CONFIG >>> > >>> > # Log format >>> > log_format = [%%(asctime)s] {{%%(filename)s:%%(lineno)d}} >>> %%(levelname)s - %%(message)s >>> > simple_log_format = %%(asctime)s %%(levelname)s - %%(message)s >>> > >>> > >>> > However if i run "airflow list_dags" i now get: >>> > >>> > andrew_maguire@airflow-server:~/airflow$ airflow list_dags >>> > Traceback (most recent call last): >>> > File "/usr/local/bin/airflow", line 16, in <module> >>> > from airflow import configuration >>> > File "/usr/local/lib/python2.7/dist-packages/airflow/__init__.py", >>> line 31, in <module> >>> > from airflow import settings >>> > File "/usr/local/lib/python2.7/dist-packages/airflow/settings.py", >>> line 148, in <module> >>> > configure_logging() >>> > File "/usr/local/lib/python2.7/dist-packages/airflow/logging_config.py", >>> line 47, in configure_logging >>> > 'Unable to load custom logging from {}'.format(logging_class_path) >>> > ImportError: Unable to load custom logging from >>> plugins.airflow_logging_settings.LOGGING_CONFIG >>> > >>> > If i go back into my airflow.cfg and comment out the line: >>> > >>> > logging_config_class = plugins.airflow_logging_settings.LOGGING_CONFIG >>> > >>> > Things work again but i'm only doing local logging. >>> > >>> > So am sure i'm doing something wrong here in how i'm setting that line >>> in the airflow.cfg file. >>> > >>> > So what i did was. >>> > >>> > 1. create a folder {AIRFLOW_HOME}/plugins - i did this as updating.md >>> <https://github.com/apache/incubator-airflow/blob/master/UPDATING.md> >>> mentioned that "The logging configuration file that contains the >>> configuration needs te on the the PYTHONPATH, for example in ~/airflow/dags >>> or ~/airflow/plugins. These directories are loaded by default". >>> > >>> > 2. copy this file <https://github.com/apache/inc >>> ubator-airflow/blob/master/airflow/config_templates/airflow_ >>> local_settings.py> into {AIRFLOW_HOME}/plugins with the GCS changes i >>> mentioned. >>> > >>> > 3. create an __init__.py file in {AIRFLOW_HOME}/plugins - do i need to >>> put anyting in partiucular in here? >>> > >>> > 4. update {AIRFLOW_HOME}/airflow.cfg as above. >>> > >>> > Can someone help me figure out where i went wrong? >>> > >>> > I'm hesitant to change or add anytihng to the pythonpath as an not >>> 100% sure what i'm doing. So was hoping to just drop the logging config >>> file somewhere it would automatically be picked up. And i'm also not really >>> sure about python packages and class paths etc so kinda feeling my way >>> through it but not confident. >>> > >>> > Cheers >>> > >>> > Andy >>> > >>> > >>> > >>> >> >
