date:20180218

[jira] [Commented] (AIRFLOW-83) Add MongoDB hook and operator

2018-02-18 Thread Marcin Szymanski (JIRA)


[ 
https://issues.apache.org/jira/browse/AIRFLOW-83?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16368727#comment-16368727
 ] 

Marcin Szymanski commented on AIRFLOW-83:
-

This is cool. I have one extra request. Can you add an additional parameter for 
a custom callable, and then, if set use the function to transform the result, 
somewhere around line 88 in mongo_to_s3.py? Comes in useful, when converting a 
nested mongo structure to something else, before loading to s3. Thanks 

> Add MongoDB hook and operator
> -
>
> Key: AIRFLOW-83
> URL: https://issues.apache.org/jira/browse/AIRFLOW-83
> Project: Apache Airflow
>  Issue Type: Wish
>  Components: contrib, hooks, operators
>Reporter: vishal srivastava
>Assignee: Ajay Yadava
>Priority: Minor
>
> A mongodb hook and operator will be really useful for people who use airflow 
> with the mongo database. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2122) SSHOperator throws an error

2018-02-18 Thread sam sen (JIRA)

sam sen created AIRFLOW-2122:


 Summary: SSHOperator throws an error
 Key: AIRFLOW-2122
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2122
 Project: Apache Airflow
  Issue Type: Bug
Reporter: sam sen


Here's my code:

 

 

{{dag = DAG('transfer_ftp_s3', 
default_args=default_args,schedule_interval=None) }}

{{task = SSHOperator(ssh_conn_id='ssh_node',}}
{{   task_id="check_ftp_for_new_files", }}
{{   command="echo 'hello world'", }}
{{   do_xcom_push=True, dag=dag,)}}

 

Here's the error
{code:java}
[2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask: Traceback 
(most recent call last):
[2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/bin/airflow", line 27, in 
[2018-02-19 06:48:02,692] {{base_task_runner.py:98}} INFO - Subtask: 
args.func(args)
[2018-02-19 06:48:02,693] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run
[2018-02-19 06:48:02,695] {{base_task_runner.py:98}} INFO - Subtask: 
pool=args.pool,
[2018-02-19 06:48:02,695] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in wrapper
[2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: result 
= func(*args, **kwargs)
[2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/lib/python2.7/site-packages/airflow/models.py", line 1496, in 
_run_raw_task
[2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: result 
= task_copy.execute(context=context)
[2018-02-19 06:48:02,697] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/lib/python2.7/site-packages/airflow/contrib/operators/ssh_operator.py", 
line 146, in execute
[2018-02-19 06:48:02,697] {{base_task_runner.py:98}} INFO - Subtask: raise 
AirflowException("SSH operator error: {0}".format(str(e)))
[2018-02-19 06:48:02,698] {{base_task_runner.py:98}} INFO - Subtask: 
airflow.exceptions.AirflowException: SSH operator error: 'bool' object has no 
attribute 'lower'
{code}
 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2122) SSHOperator throws an error

2018-02-18 Thread sam sen (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sam sen updated AIRFLOW-2122:
-
Description: 
Here's my code:

 

 

 
{code:java}
dag = DAG('transfer_ftp_s3', default_args=default_args,schedule_interval=None) 
}}
task = SSHOperator(ssh_conn_id='ssh_node', 
                                  task_id="check_ftp_for_new_files", 
                                  command="echo 'hello world'", 
                                  do_xcom_push=True, dag=dag,)
{code}
 

 

Here's the error
{code:java}
[2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask: Traceback 
(most recent call last):
[2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/bin/airflow", line 27, in 
[2018-02-19 06:48:02,692] {{base_task_runner.py:98}} INFO - Subtask: 
args.func(args)
[2018-02-19 06:48:02,693] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run
[2018-02-19 06:48:02,695] {{base_task_runner.py:98}} INFO - Subtask: 
pool=args.pool,
[2018-02-19 06:48:02,695] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in wrapper
[2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: result 
= func(*args, **kwargs)
[2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/lib/python2.7/site-packages/airflow/models.py", line 1496, in 
_run_raw_task
[2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: result 
= task_copy.execute(context=context)
[2018-02-19 06:48:02,697] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/lib/python2.7/site-packages/airflow/contrib/operators/ssh_operator.py", 
line 146, in execute
[2018-02-19 06:48:02,697] {{base_task_runner.py:98}} INFO - Subtask: raise 
AirflowException("SSH operator error: {0}".format(str(e)))
[2018-02-19 06:48:02,698] {{base_task_runner.py:98}} INFO - Subtask: 
airflow.exceptions.AirflowException: SSH operator error: 'bool' object has no 
attribute 'lower'
{code}
 

 

  was:
Here's my code:

 

 

{{dag = DAG('transfer_ftp_s3', 
default_args=default_args,schedule_interval=None) }}

{{task = SSHOperator(ssh_conn_id='ssh_node',}}
{{   task_id="check_ftp_for_new_files", }}
{{   command="echo 'hello world'", }}
{{   do_xcom_push=True, dag=dag,)}}

 

Here's the error
{code:java}
[2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask: Traceback 
(most recent call last):
[2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/bin/airflow", line 27, in 
[2018-02-19 06:48:02,692] {{base_task_runner.py:98}} INFO - Subtask: 
args.func(args)
[2018-02-19 06:48:02,693] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run
[2018-02-19 06:48:02,695] {{base_task_runner.py:98}} INFO - Subtask: 
pool=args.pool,
[2018-02-19 06:48:02,695] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in wrapper
[2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: result 
= func(*args, **kwargs)
[2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/lib/python2.7/site-packages/airflow/models.py", line 1496, in 
_run_raw_task
[2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: result 
= task_copy.execute(context=context)
[2018-02-19 06:48:02,697] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/lib/python2.7/site-packages/airflow/contrib/operators/ssh_operator.py", 
line 146, in execute
[2018-02-19 06:48:02,697] {{base_task_runner.py:98}} INFO - Subtask: raise 
AirflowException("SSH operator error: {0}".format(str(e)))
[2018-02-19 06:48:02,698] {{base_task_runner.py:98}} INFO - Subtask: 
airflow.exceptions.AirflowException: SSH operator error: 'bool' object has no 
attribute 'lower'
{code}
 

 


> SSHOperator throws an error
> ---
>
> Key: AIRFLOW-2122
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2122
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: sam sen
>Priority: Major
>
> Here's my code:
>  
>  
>  
> {code:java}
> dag = DAG('transfer_ftp_s3', 
> default_args=default_args,schedule_interval=None) }}
> task = SSHOperator(ssh_conn_id='ssh_node', 
>                                   task_id="check_ftp_for_new_files", 
>                                   command="echo 'hello world'", 
>                                   do_xcom_push=True, dag=dag,)
> {code}
>  
>  
> Here's the error
> {code:java}
> [2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask: 
> Traceback (most recent call last):
> [2018-02-19 06:48:02,691] {{

[jira] [Updated] (AIRFLOW-2122) SSHOperator throws an error

2018-02-18 Thread sam sen (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sam sen updated AIRFLOW-2122:
-
Description: 
Here's my code:

 

 

 
{code:java}
dag = DAG('transfer_ftp_s3', default_args=default_args,schedule_interval=None) 
}}
task = SSHOperator(ssh_conn_id='ssh_node', 
                   task_id="check_ftp_for_new_files", 
                   command="echo 'hello world'", 
                   do_xcom_push=True, dag=dag,)
{code}
 

 

Here's the error
{code:java}
[2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask: Traceback 
(most recent call last):
[2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/bin/airflow", line 27, in 
[2018-02-19 06:48:02,692] {{base_task_runner.py:98}} INFO - Subtask: 
args.func(args)
[2018-02-19 06:48:02,693] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run
[2018-02-19 06:48:02,695] {{base_task_runner.py:98}} INFO - Subtask: 
pool=args.pool,
[2018-02-19 06:48:02,695] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in wrapper
[2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: result 
= func(*args, **kwargs)
[2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/lib/python2.7/site-packages/airflow/models.py", line 1496, in 
_run_raw_task
[2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: result 
= task_copy.execute(context=context)
[2018-02-19 06:48:02,697] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/lib/python2.7/site-packages/airflow/contrib/operators/ssh_operator.py", 
line 146, in execute
[2018-02-19 06:48:02,697] {{base_task_runner.py:98}} INFO - Subtask: raise 
AirflowException("SSH operator error: {0}".format(str(e)))
[2018-02-19 06:48:02,698] {{base_task_runner.py:98}} INFO - Subtask: 
airflow.exceptions.AirflowException: SSH operator error: 'bool' object has no 
attribute 'lower'
{code}
 

 

  was:
Here's my code:

 

 

 
{code:java}
dag = DAG('transfer_ftp_s3', default_args=default_args,schedule_interval=None) 
}}
task = SSHOperator(ssh_conn_id='ssh_node', 
                                  task_id="check_ftp_for_new_files", 
                                  command="echo 'hello world'", 
                                  do_xcom_push=True, dag=dag,)
{code}
 

 

Here's the error
{code:java}
[2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask: Traceback 
(most recent call last):
[2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/bin/airflow", line 27, in 
[2018-02-19 06:48:02,692] {{base_task_runner.py:98}} INFO - Subtask: 
args.func(args)
[2018-02-19 06:48:02,693] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run
[2018-02-19 06:48:02,695] {{base_task_runner.py:98}} INFO - Subtask: 
pool=args.pool,
[2018-02-19 06:48:02,695] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in wrapper
[2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: result 
= func(*args, **kwargs)
[2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/lib/python2.7/site-packages/airflow/models.py", line 1496, in 
_run_raw_task
[2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: result 
= task_copy.execute(context=context)
[2018-02-19 06:48:02,697] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/lib/python2.7/site-packages/airflow/contrib/operators/ssh_operator.py", 
line 146, in execute
[2018-02-19 06:48:02,697] {{base_task_runner.py:98}} INFO - Subtask: raise 
AirflowException("SSH operator error: {0}".format(str(e)))
[2018-02-19 06:48:02,698] {{base_task_runner.py:98}} INFO - Subtask: 
airflow.exceptions.AirflowException: SSH operator error: 'bool' object has no 
attribute 'lower'
{code}
 

 


> SSHOperator throws an error
> ---
>
> Key: AIRFLOW-2122
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2122
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: sam sen
>Priority: Major
>
> Here's my code:
>  
>  
>  
> {code:java}
> dag = DAG('transfer_ftp_s3', 
> default_args=default_args,schedule_interval=None) }}
> task = SSHOperator(ssh_conn_id='ssh_node', 
>                    task_id="check_ftp_for_new_files", 
>                    command="echo 'hello world'", 
>                    do_xcom_push=True, dag=dag,)
> {code}
>  
>  
> Here's the error
> {code:java}
> [2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask: 
> Traceback (most recent call last):
> [2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask:   F

[jira] [Updated] (AIRFLOW-2122) SSHOperator throws an error

2018-02-18 Thread sam sen (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sam sen updated AIRFLOW-2122:
-
Description: 
Here's my code: 
{code:java}
dag = DAG('transfer_ftp_s3', default_args=default_args,schedule_interval=None) 
}}
task = SSHOperator(ssh_conn_id='ssh_node', 
                   task_id="check_ftp_for_new_files", 
                   command="echo 'hello world'", 
                   do_xcom_push=True, dag=dag,)
{code}
 

Here's the error
{code:java}
[2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask: Traceback 
(most recent call last):
[2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/bin/airflow", line 27, in 
[2018-02-19 06:48:02,692] {{base_task_runner.py:98}} INFO - Subtask: 
args.func(args)
[2018-02-19 06:48:02,693] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run
[2018-02-19 06:48:02,695] {{base_task_runner.py:98}} INFO - Subtask: 
pool=args.pool,
[2018-02-19 06:48:02,695] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in wrapper
[2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: result 
= func(*args, **kwargs)
[2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/lib/python2.7/site-packages/airflow/models.py", line 1496, in 
_run_raw_task
[2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: result 
= task_copy.execute(context=context)
[2018-02-19 06:48:02,697] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/lib/python2.7/site-packages/airflow/contrib/operators/ssh_operator.py", 
line 146, in execute
[2018-02-19 06:48:02,697] {{base_task_runner.py:98}} INFO - Subtask: raise 
AirflowException("SSH operator error: {0}".format(str(e)))
[2018-02-19 06:48:02,698] {{base_task_runner.py:98}} INFO - Subtask: 
airflow.exceptions.AirflowException: SSH operator error: 'bool' object has no 
attribute 'lower'
{code}
 

 

  was:
Here's my code:

 

 

 
{code:java}
dag = DAG('transfer_ftp_s3', default_args=default_args,schedule_interval=None) 
}}
task = SSHOperator(ssh_conn_id='ssh_node', 
                   task_id="check_ftp_for_new_files", 
                   command="echo 'hello world'", 
                   do_xcom_push=True, dag=dag,)
{code}
 

 

Here's the error
{code:java}
[2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask: Traceback 
(most recent call last):
[2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/bin/airflow", line 27, in 
[2018-02-19 06:48:02,692] {{base_task_runner.py:98}} INFO - Subtask: 
args.func(args)
[2018-02-19 06:48:02,693] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run
[2018-02-19 06:48:02,695] {{base_task_runner.py:98}} INFO - Subtask: 
pool=args.pool,
[2018-02-19 06:48:02,695] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in wrapper
[2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: result 
= func(*args, **kwargs)
[2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/lib/python2.7/site-packages/airflow/models.py", line 1496, in 
_run_raw_task
[2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: result 
= task_copy.execute(context=context)
[2018-02-19 06:48:02,697] {{base_task_runner.py:98}} INFO - Subtask:   File 
"/usr/lib/python2.7/site-packages/airflow/contrib/operators/ssh_operator.py", 
line 146, in execute
[2018-02-19 06:48:02,697] {{base_task_runner.py:98}} INFO - Subtask: raise 
AirflowException("SSH operator error: {0}".format(str(e)))
[2018-02-19 06:48:02,698] {{base_task_runner.py:98}} INFO - Subtask: 
airflow.exceptions.AirflowException: SSH operator error: 'bool' object has no 
attribute 'lower'
{code}
 

 


> SSHOperator throws an error
> ---
>
> Key: AIRFLOW-2122
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2122
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: sam sen
>Priority: Major
>
> Here's my code: 
> {code:java}
> dag = DAG('transfer_ftp_s3', 
> default_args=default_args,schedule_interval=None) }}
> task = SSHOperator(ssh_conn_id='ssh_node', 
>                    task_id="check_ftp_for_new_files", 
>                    command="echo 'hello world'", 
>                    do_xcom_push=True, dag=dag,)
> {code}
>  
> Here's the error
> {code:java}
> [2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask: 
> Traceback (most recent call last):
> [2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask:   File 
> "/usr/bin/airflow", line 27, in 
> [2018-02-19 06:48:02,692] {{b

[jira] [Commented] (AIRFLOW-83) Add MongoDB hook and operator

[jira] [Created] (AIRFLOW-2122) SSHOperator throws an error

[jira] [Updated] (AIRFLOW-2122) SSHOperator throws an error

[jira] [Updated] (AIRFLOW-2122) SSHOperator throws an error

[jira] [Updated] (AIRFLOW-2122) SSHOperator throws an error

5 matches

Site Navigation

Mail list logo

Footer information