[jira] [Commented] (AIRFLOW-83) Add MongoDB hook and operator
[ https://issues.apache.org/jira/browse/AIRFLOW-83?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16368727#comment-16368727 ] Marcin Szymanski commented on AIRFLOW-83: - This is cool. I have one extra request. Can you add an additional parameter for a custom callable, and then, if set use the function to transform the result, somewhere around line 88 in mongo_to_s3.py? Comes in useful, when converting a nested mongo structure to something else, before loading to s3. Thanks > Add MongoDB hook and operator > - > > Key: AIRFLOW-83 > URL: https://issues.apache.org/jira/browse/AIRFLOW-83 > Project: Apache Airflow > Issue Type: Wish > Components: contrib, hooks, operators >Reporter: vishal srivastava >Assignee: Ajay Yadava >Priority: Minor > > A mongodb hook and operator will be really useful for people who use airflow > with the mongo database. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2122) SSHOperator throws an error
sam sen created AIRFLOW-2122: Summary: SSHOperator throws an error Key: AIRFLOW-2122 URL: https://issues.apache.org/jira/browse/AIRFLOW-2122 Project: Apache Airflow Issue Type: Bug Reporter: sam sen Here's my code: {{dag = DAG('transfer_ftp_s3', default_args=default_args,schedule_interval=None) }} {{task = SSHOperator(ssh_conn_id='ssh_node',}} {{ task_id="check_ftp_for_new_files", }} {{ command="echo 'hello world'", }} {{ do_xcom_push=True, dag=dag,)}} Here's the error {code:java} [2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask: Traceback (most recent call last): [2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/bin/airflow", line 27, in [2018-02-19 06:48:02,692] {{base_task_runner.py:98}} INFO - Subtask: args.func(args) [2018-02-19 06:48:02,693] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run [2018-02-19 06:48:02,695] {{base_task_runner.py:98}} INFO - Subtask: pool=args.pool, [2018-02-19 06:48:02,695] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in wrapper [2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: result = func(*args, **kwargs) [2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1496, in _run_raw_task [2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: result = task_copy.execute(context=context) [2018-02-19 06:48:02,697] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/contrib/operators/ssh_operator.py", line 146, in execute [2018-02-19 06:48:02,697] {{base_task_runner.py:98}} INFO - Subtask: raise AirflowException("SSH operator error: {0}".format(str(e))) [2018-02-19 06:48:02,698] {{base_task_runner.py:98}} INFO - Subtask: airflow.exceptions.AirflowException: SSH operator error: 'bool' object has no attribute 'lower' {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2122) SSHOperator throws an error
[ https://issues.apache.org/jira/browse/AIRFLOW-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam sen updated AIRFLOW-2122: - Description: Here's my code: {code:java} dag = DAG('transfer_ftp_s3', default_args=default_args,schedule_interval=None) }} task = SSHOperator(ssh_conn_id='ssh_node', task_id="check_ftp_for_new_files", command="echo 'hello world'", do_xcom_push=True, dag=dag,) {code} Here's the error {code:java} [2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask: Traceback (most recent call last): [2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/bin/airflow", line 27, in [2018-02-19 06:48:02,692] {{base_task_runner.py:98}} INFO - Subtask: args.func(args) [2018-02-19 06:48:02,693] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run [2018-02-19 06:48:02,695] {{base_task_runner.py:98}} INFO - Subtask: pool=args.pool, [2018-02-19 06:48:02,695] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in wrapper [2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: result = func(*args, **kwargs) [2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1496, in _run_raw_task [2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: result = task_copy.execute(context=context) [2018-02-19 06:48:02,697] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/contrib/operators/ssh_operator.py", line 146, in execute [2018-02-19 06:48:02,697] {{base_task_runner.py:98}} INFO - Subtask: raise AirflowException("SSH operator error: {0}".format(str(e))) [2018-02-19 06:48:02,698] {{base_task_runner.py:98}} INFO - Subtask: airflow.exceptions.AirflowException: SSH operator error: 'bool' object has no attribute 'lower' {code} was: Here's my code: {{dag = DAG('transfer_ftp_s3', default_args=default_args,schedule_interval=None) }} {{task = SSHOperator(ssh_conn_id='ssh_node',}} {{ task_id="check_ftp_for_new_files", }} {{ command="echo 'hello world'", }} {{ do_xcom_push=True, dag=dag,)}} Here's the error {code:java} [2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask: Traceback (most recent call last): [2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/bin/airflow", line 27, in [2018-02-19 06:48:02,692] {{base_task_runner.py:98}} INFO - Subtask: args.func(args) [2018-02-19 06:48:02,693] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run [2018-02-19 06:48:02,695] {{base_task_runner.py:98}} INFO - Subtask: pool=args.pool, [2018-02-19 06:48:02,695] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in wrapper [2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: result = func(*args, **kwargs) [2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1496, in _run_raw_task [2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: result = task_copy.execute(context=context) [2018-02-19 06:48:02,697] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/contrib/operators/ssh_operator.py", line 146, in execute [2018-02-19 06:48:02,697] {{base_task_runner.py:98}} INFO - Subtask: raise AirflowException("SSH operator error: {0}".format(str(e))) [2018-02-19 06:48:02,698] {{base_task_runner.py:98}} INFO - Subtask: airflow.exceptions.AirflowException: SSH operator error: 'bool' object has no attribute 'lower' {code} > SSHOperator throws an error > --- > > Key: AIRFLOW-2122 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2122 > Project: Apache Airflow > Issue Type: Bug >Reporter: sam sen >Priority: Major > > Here's my code: > > > > {code:java} > dag = DAG('transfer_ftp_s3', > default_args=default_args,schedule_interval=None) }} > task = SSHOperator(ssh_conn_id='ssh_node', > task_id="check_ftp_for_new_files", > command="echo 'hello world'", > do_xcom_push=True, dag=dag,) > {code} > > > Here's the error > {code:java} > [2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask: > Traceback (most recent call last): > [2018-02-19 06:48:02,691] {{
[jira] [Updated] (AIRFLOW-2122) SSHOperator throws an error
[ https://issues.apache.org/jira/browse/AIRFLOW-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam sen updated AIRFLOW-2122: - Description: Here's my code: {code:java} dag = DAG('transfer_ftp_s3', default_args=default_args,schedule_interval=None) }} task = SSHOperator(ssh_conn_id='ssh_node', task_id="check_ftp_for_new_files", command="echo 'hello world'", do_xcom_push=True, dag=dag,) {code} Here's the error {code:java} [2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask: Traceback (most recent call last): [2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/bin/airflow", line 27, in [2018-02-19 06:48:02,692] {{base_task_runner.py:98}} INFO - Subtask: args.func(args) [2018-02-19 06:48:02,693] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run [2018-02-19 06:48:02,695] {{base_task_runner.py:98}} INFO - Subtask: pool=args.pool, [2018-02-19 06:48:02,695] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in wrapper [2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: result = func(*args, **kwargs) [2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1496, in _run_raw_task [2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: result = task_copy.execute(context=context) [2018-02-19 06:48:02,697] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/contrib/operators/ssh_operator.py", line 146, in execute [2018-02-19 06:48:02,697] {{base_task_runner.py:98}} INFO - Subtask: raise AirflowException("SSH operator error: {0}".format(str(e))) [2018-02-19 06:48:02,698] {{base_task_runner.py:98}} INFO - Subtask: airflow.exceptions.AirflowException: SSH operator error: 'bool' object has no attribute 'lower' {code} was: Here's my code: {code:java} dag = DAG('transfer_ftp_s3', default_args=default_args,schedule_interval=None) }} task = SSHOperator(ssh_conn_id='ssh_node', task_id="check_ftp_for_new_files", command="echo 'hello world'", do_xcom_push=True, dag=dag,) {code} Here's the error {code:java} [2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask: Traceback (most recent call last): [2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/bin/airflow", line 27, in [2018-02-19 06:48:02,692] {{base_task_runner.py:98}} INFO - Subtask: args.func(args) [2018-02-19 06:48:02,693] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run [2018-02-19 06:48:02,695] {{base_task_runner.py:98}} INFO - Subtask: pool=args.pool, [2018-02-19 06:48:02,695] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in wrapper [2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: result = func(*args, **kwargs) [2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1496, in _run_raw_task [2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: result = task_copy.execute(context=context) [2018-02-19 06:48:02,697] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/contrib/operators/ssh_operator.py", line 146, in execute [2018-02-19 06:48:02,697] {{base_task_runner.py:98}} INFO - Subtask: raise AirflowException("SSH operator error: {0}".format(str(e))) [2018-02-19 06:48:02,698] {{base_task_runner.py:98}} INFO - Subtask: airflow.exceptions.AirflowException: SSH operator error: 'bool' object has no attribute 'lower' {code} > SSHOperator throws an error > --- > > Key: AIRFLOW-2122 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2122 > Project: Apache Airflow > Issue Type: Bug >Reporter: sam sen >Priority: Major > > Here's my code: > > > > {code:java} > dag = DAG('transfer_ftp_s3', > default_args=default_args,schedule_interval=None) }} > task = SSHOperator(ssh_conn_id='ssh_node', > task_id="check_ftp_for_new_files", > command="echo 'hello world'", > do_xcom_push=True, dag=dag,) > {code} > > > Here's the error > {code:java} > [2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask: > Traceback (most recent call last): > [2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask: F
[jira] [Updated] (AIRFLOW-2122) SSHOperator throws an error
[ https://issues.apache.org/jira/browse/AIRFLOW-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam sen updated AIRFLOW-2122: - Description: Here's my code: {code:java} dag = DAG('transfer_ftp_s3', default_args=default_args,schedule_interval=None) }} task = SSHOperator(ssh_conn_id='ssh_node', task_id="check_ftp_for_new_files", command="echo 'hello world'", do_xcom_push=True, dag=dag,) {code} Here's the error {code:java} [2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask: Traceback (most recent call last): [2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/bin/airflow", line 27, in [2018-02-19 06:48:02,692] {{base_task_runner.py:98}} INFO - Subtask: args.func(args) [2018-02-19 06:48:02,693] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run [2018-02-19 06:48:02,695] {{base_task_runner.py:98}} INFO - Subtask: pool=args.pool, [2018-02-19 06:48:02,695] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in wrapper [2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: result = func(*args, **kwargs) [2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1496, in _run_raw_task [2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: result = task_copy.execute(context=context) [2018-02-19 06:48:02,697] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/contrib/operators/ssh_operator.py", line 146, in execute [2018-02-19 06:48:02,697] {{base_task_runner.py:98}} INFO - Subtask: raise AirflowException("SSH operator error: {0}".format(str(e))) [2018-02-19 06:48:02,698] {{base_task_runner.py:98}} INFO - Subtask: airflow.exceptions.AirflowException: SSH operator error: 'bool' object has no attribute 'lower' {code} was: Here's my code: {code:java} dag = DAG('transfer_ftp_s3', default_args=default_args,schedule_interval=None) }} task = SSHOperator(ssh_conn_id='ssh_node', task_id="check_ftp_for_new_files", command="echo 'hello world'", do_xcom_push=True, dag=dag,) {code} Here's the error {code:java} [2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask: Traceback (most recent call last): [2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/bin/airflow", line 27, in [2018-02-19 06:48:02,692] {{base_task_runner.py:98}} INFO - Subtask: args.func(args) [2018-02-19 06:48:02,693] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run [2018-02-19 06:48:02,695] {{base_task_runner.py:98}} INFO - Subtask: pool=args.pool, [2018-02-19 06:48:02,695] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in wrapper [2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: result = func(*args, **kwargs) [2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1496, in _run_raw_task [2018-02-19 06:48:02,696] {{base_task_runner.py:98}} INFO - Subtask: result = task_copy.execute(context=context) [2018-02-19 06:48:02,697] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/contrib/operators/ssh_operator.py", line 146, in execute [2018-02-19 06:48:02,697] {{base_task_runner.py:98}} INFO - Subtask: raise AirflowException("SSH operator error: {0}".format(str(e))) [2018-02-19 06:48:02,698] {{base_task_runner.py:98}} INFO - Subtask: airflow.exceptions.AirflowException: SSH operator error: 'bool' object has no attribute 'lower' {code} > SSHOperator throws an error > --- > > Key: AIRFLOW-2122 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2122 > Project: Apache Airflow > Issue Type: Bug >Reporter: sam sen >Priority: Major > > Here's my code: > {code:java} > dag = DAG('transfer_ftp_s3', > default_args=default_args,schedule_interval=None) }} > task = SSHOperator(ssh_conn_id='ssh_node', > task_id="check_ftp_for_new_files", > command="echo 'hello world'", > do_xcom_push=True, dag=dag,) > {code} > > Here's the error > {code:java} > [2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask: > Traceback (most recent call last): > [2018-02-19 06:48:02,691] {{base_task_runner.py:98}} INFO - Subtask: File > "/usr/bin/airflow", line 27, in > [2018-02-19 06:48:02,692] {{b