[
https://issues.apache.org/jira/browse/AIRFLOW-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Petrovsky updated AIRFLOW-1358:
-----------------------------------------
Issue Type: Bug (was: New Feature)
> SQL file renderer and on_failure_callback
> -----------------------------------------
>
> Key: AIRFLOW-1358
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1358
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
> Reporter: Alexander Petrovsky
>
> I have the follow dag:
> https://gist.github.com/juise/cc006584e17be72e78550aab53a1be9c
> When I broke my firs step (prepare), on_failure_callback function was called,
> but it's crashed too, due looks like in sql rendered something is broken, and
> he can't work with sql files (by example, by cleanup.sql file, see more info
> below). But, when I use flat sql instead of sql file, everything works
> perfectly.
> My cleanup.sql file:
> {code:java}
> $ cat cleanup.sql
> DROP TABLE IF EXISTS etl.{{ params.target_table }}_{{ ts_nodash }} CASCADE;
> {code:java}
> Log:
> {code:java}
> [2017-06-29 15:14:13,474] {models.py:168} INFO - Filling up the DagBag from
> /opt/airflow/dags/ga_full_stats_test/ga_full_stats.py
> [2017-06-29 15:14:13,852] {base_hook.py:67} INFO - Using connection to:
> v2.rtty.in
> [2017-06-29 15:14:13,858] {base_hook.py:67} INFO - Using connection to:
> vga.rtty.in
> [2017-06-29 15:14:13,893] {base_task_runner.py:112} INFO - Running: ['bash',
> '-c', u'airflow run ga_full_stats_test_v2 prepare 2017-06-06T10:27:00
> --job_id 123798 --raw -sd DAGS_FOLDER/ga_full_stats_test/ga_full_stats.py']
> [2017-06-29 15:14:14,391] {base_task_runner.py:95} INFO - Subtask:
> [2017-06-29 15:14:14,391] {__init__.py:57} INFO - Using executor
> CeleryExecutor
> [2017-06-29 15:14:14,924] {base_task_runner.py:95} INFO - Subtask:
> [2017-06-29 15:14:14,924] {models.py:168} INFO - Filling up the DagBag from
> /opt/airflow/dags/ga_full_stats_test/ga_full_stats.py
> [2017-06-29 15:14:15,088] {base_task_runner.py:95} INFO - Subtask:
> [2017-06-29 15:14:15,087] {base_hook.py:67} INFO - Using connection to:
> v2.rtty.in
> [2017-06-29 15:14:15,091] {base_task_runner.py:95} INFO - Subtask:
> [2017-06-29 15:14:15,091] {base_hook.py:67} INFO - Using connection to:
> vga.rtty.in
> [2017-06-29 15:14:15,109] {base_task_runner.py:95} INFO - Subtask:
> [2017-06-29 15:14:15,109] {models.py:1128} INFO - Dependencies all met for
> <TaskInstance: ga_full_stats_test_v2.prepare 2017-06-06 10:27:00 [queued]>
> [2017-06-29 15:14:15,112] {base_task_runner.py:95} INFO - Subtask:
> [2017-06-29 15:14:15,112] {models.py:1128} INFO - Dependencies all met for
> <TaskInstance: ga_full_stats_test_v2.prepare 2017-06-06 10:27:00 [queued]>
> [2017-06-29 15:14:15,112] {base_task_runner.py:95} INFO - Subtask:
> [2017-06-29 15:14:15,112] {models.py:1328} INFO -
> [2017-06-29 15:14:15,112] {base_task_runner.py:95} INFO - Subtask:
> --------------------------------------------------------------------------------
> [2017-06-29 15:14:15,112] {base_task_runner.py:95} INFO - Subtask: Starting
> attempt 1 of 1
> [2017-06-29 15:14:15,112] {base_task_runner.py:95} INFO - Subtask:
> --------------------------------------------------------------------------------
> [2017-06-29 15:14:15,112] {base_task_runner.py:95} INFO - Subtask:
> [2017-06-29 15:14:15,120] {base_task_runner.py:95} INFO - Subtask:
> [2017-06-29 15:14:15,120] {models.py:1352} INFO - Executing
> <Task(VerticaOperator): prepare> on 2017-06-06 10:27:00
> [2017-06-29 15:14:15,138] {base_task_runner.py:95} INFO - Subtask:
> [2017-06-29 15:14:15,137] {vertica_operator.py:45} INFO - Executing: CREATE
> TABLE etl.stats_full_daily_test_20170606T102700 1 LIKE
> public.stats_full_daily_test;
> [2017-06-29 15:14:15,141] {base_task_runner.py:95} INFO - Subtask:
> [2017-06-29 15:14:15,141] {base_hook.py:67} INFO - Using connection to:
> vga.rtty.in
> [2017-06-29 15:14:15,158] {base_task_runner.py:95} INFO - Subtask:
> [2017-06-29 15:14:15,158] {dbapi_hook.py:167} INFO - CREATE TABLE
> etl.stats_full_daily_test_20170606T102700 1 LIKE public.stats_full_daily_test;
> [2017-06-29 15:14:15,164] {base_task_runner.py:95} INFO - Subtask:
> [2017-06-29 15:14:15,164] {models.py:1427} ERROR - Severity: ERROR, Message:
> Syntax error at or near "1", Sqlstate: 42601, Position: 56, Routine:
> base_yyerror, File:
> /scratch_a/release/svrtar18593/vbuild/vertica/Parser/scan.l, Line: 1043, SQL:
> 'CREATE TABLE etl.stats_full_daily_test_20170606T102700 1 LIKE
> public.stats_full_daily_test;'
> [2017-06-29 15:14:15,164] {base_task_runner.py:95} INFO - Subtask: Traceback
> (most recent call last):
> [2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask: File
> "/usr/lib/python2.7/site-packages/airflow/models.py", line 1384, in run
> [2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask: result
> = task_copy.execute(context=context)
> [2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask: File
> "/usr/lib/python2.7/site-packages/airflow/contrib/operators/vertica_operator.py",
> line 47, in execute
> [2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask:
> hook.run(self.sql)
> [2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask: File
> "/usr/lib/python2.7/site-packages/airflow/hooks/dbapi_hook.py", line 171, in
> run
> [2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask:
> cur.execute(s)
> [2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask: File
> "/usr/lib/python2.7/site-packages/vertica_python/vertica/cursor.py", line
> 111, in execute
> [2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask: raise
> errors.QueryError.from_error_response(self._message, operation)
> [2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask:
> VerticaSyntaxError: Severity: ERROR, Message: Syntax error at or near "1",
> Sqlstate: 42601, Position: 56, Routine: base_yyerror, File:
> /scratch_a/release/svrtar18593/vbuild/vertica/Parser/scan.l, Line: 1043, SQL:
> 'CREATE TABLE etl.stats_full_daily_test_20170606T102700 1 LIKE
> public.stats_full_daily_test;'
> [2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask:
> [2017-06-29 15:14:15,164] {models.py:1451} INFO - Marking task as FAILED.
> [2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask:
> [2017-06-29 15:14:15,165] {vertica_operator.py:45} INFO - Executing:
> cleanup.sql
> [2017-06-29 15:14:15,169] {base_task_runner.py:95} INFO - Subtask:
> [2017-06-29 15:14:15,169] {base_hook.py:67} INFO - Using connection to:
> vga.rtty.in
> [2017-06-29 15:14:15,181] {base_task_runner.py:95} INFO - Subtask:
> [2017-06-29 15:14:15,181] {dbapi_hook.py:167} INFO - cleanup.sql
> [2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask:
> [2017-06-29 15:14:15,185] {models.py:1466} ERROR - Failed at executing
> callback
> [2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask:
> [2017-06-29 15:14:15,185] {models.py:1467} ERROR - Severity: ERROR, Message:
> Syntax error at or near "cleanup", Sqlstate: 42601, Position: 1, Routine:
> base_yyerror, File:
> /scratch_a/release/svrtar18593/vbuild/vertica/Parser/scan.l, Line: 1043, SQL:
> 'cleanup.sql'
> [2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask: Traceback
> (most recent call last):
> [2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask: File
> "/usr/lib/python2.7/site-packages/airflow/models.py", line 1464, in
> handle_failure
> [2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask:
> task.on_failure_callback(context)
> [2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask: File
> "/opt/airflow/dags/ga_full_stats_test/ga_full_stats.py", line 64, in
> cleanup_on_failure
> [2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask:
> cleanup.execute(context)
> [2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask: File
> "/usr/lib/python2.7/site-packages/airflow/contrib/operators/vertica_operator.py",
> line 47, in execute
> [2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask:
> hook.run(self.sql)
> [2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask: File
> "/usr/lib/python2.7/site-packages/airflow/hooks/dbapi_hook.py", line 171, in
> run
> [2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask:
> cur.execute(s)
> [2017-06-29 15:14:15,186] {base_task_runner.py:95} INFO - Subtask: File
> "/usr/lib/python2.7/site-packages/vertica_python/vertica/cursor.py", line
> 111, in execute
> [2017-06-29 15:14:15,186] {base_task_runner.py:95} INFO - Subtask: raise
> errors.QueryError.from_error_response(self._message, operation)
> [2017-06-29 15:14:15,186] {base_task_runner.py:95} INFO - Subtask:
> VerticaSyntaxError: Severity: ERROR, Message: Syntax error at or near
> "cleanup", Sqlstate: 42601, Position: 1, Routine: base_yyerror, File:
> /scratch_a/release/svrtar18593/vbuild/vertica/Parser/scan.l, Line: 1043, SQL:
> 'cleanup.sql'
> [2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask:
> [2017-06-29 15:14:15,191] {models.py:1472} ERROR - Severity: ERROR, Message:
> Syntax error at or near "1", Sqlstate: 42601, Position: 56, Routine:
> base_yyerror, File:
> /scratch_a/release/svrtar18593/vbuild/vertica/Parser/scan.l, Line: 1043, SQL:
> 'CREATE TABLE etl.stats_full_daily_test_20170606T102700 1 LIKE
> public.stats_full_daily_test;'
> [2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask: Traceback
> (most recent call last):
> [2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask: File
> "/usr/bin/airflow", line 28, in <module>
> [2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask:
> args.func(args)
> [2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask: File
> "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 422, in run
> [2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask:
> pool=args.pool,
> [2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask: File
> "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 53, in wrapper
> [2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask: result
> = func(*args, **kwargs)
> [2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask: File
> "/usr/lib/python2.7/site-packages/airflow/models.py", line 1384, in run
> [2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask: result
> = task_copy.execute(context=context)
> [2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask: File
> "/usr/lib/python2.7/site-packages/airflow/contrib/operators/vertica_operator.py",
> line 47, in execute
> [2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask:
> hook.run(self.sql)
> [2017-06-29 15:14:15,192] {base_task_runner.py:95} INFO - Subtask: File
> "/usr/lib/python2.7/site-packages/airflow/hooks/dbapi_hook.py", line 171, in
> run
> [2017-06-29 15:14:15,192] {base_task_runner.py:95} INFO - Subtask:
> cur.execute(s)
> [2017-06-29 15:14:15,192] {base_task_runner.py:95} INFO - Subtask: File
> "/usr/lib/python2.7/site-packages/vertica_python/vertica/cursor.py", line
> 111, in execute
> [2017-06-29 15:14:15,192] {base_task_runner.py:95} INFO - Subtask: raise
> errors.QueryError.from_error_response(self._message, operation)
> [2017-06-29 15:14:15,192] {base_task_runner.py:95} INFO - Subtask:
> vertica_python.errors.VerticaSyntaxError: Severity: ERROR, Message: Syntax
> error at or near "1", Sqlstate: 42601, Position: 56, Routine: base_yyerror,
> File: /scratch_a/release/svrtar18593/vbuild/vertica/Parser/scan.l, Line:
> 1043, SQL: 'CREATE TABLE etl.stats_full_daily_test_20170606T102700 1 LIKE
> public.stats_full_daily_test;'
> [2017-06-29 15:14:15,893] {jobs.py:2107} INFO - Task exited with return code 1
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
