[GitHub] [airflow] potiuk edited a comment on pull request #11276: Switched to Run Checks for Building Images.

2020-10-05 Thread GitBox
potiuk edited a comment on pull request #11276: URL: https://github.com/apache/airflow/pull/11276#issuecomment-704068686 FY: This is even nicer. I could not see it reliably working in my fork, but I see that it nicely works in Airflow. The "Status of the Image Build" checks are also

[GitHub] [airflow] potiuk commented on pull request #11276: Switched to Run Checks for Building Images.

2020-10-05 Thread GitBox
potiuk commented on pull request #11276: URL: https://github.com/apache/airflow/pull/11276#issuecomment-704068686 FY: This is even nicer. I could not see it reliably working in my fork, but I see that it nicely works. The "Status of the Image Build" checks are also showing up now directly

[jira] [Commented] (AIRFLOW-3607) Decreasing scheduler delay between tasks

2020-10-05 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17208513#comment-17208513 ] ASF GitHub Bot commented on AIRFLOW-3607: - yuqian90 commented on pull request #

[GitHub] [airflow] yuqian90 commented on pull request #11010: [AIRFLOW-3607] Only query DB once per DAG run for TriggerRuleDep

2020-10-05 Thread GitBox
yuqian90 commented on pull request #11010: URL: https://github.com/apache/airflow/pull/11010#issuecomment-704065719 > @yuqian90 you will need this as well: #7503 Thanks for pointing out. I'm not using the features fixed in #7503 myself so it didn't affect me. However, you are right i

[GitHub] [airflow] TobKed commented on a change in pull request #11287: Improve handling of job_id in BigQuery operators

2020-10-05 Thread GitBox
TobKed commented on a change in pull request #11287: URL: https://github.com/apache/airflow/pull/11287#discussion_r500037926 ## File path: airflow/providers/google/cloud/hooks/bigquery.py ## @@ -1443,6 +1445,12 @@ def get_job( job = client.get_job(job_id=job_id, projec

[GitHub] [airflow] potiuk opened a new pull request #11292: Replaces depreated set-env with env file

2020-10-05 Thread GitBox
potiuk opened a new pull request #11292: URL: https://github.com/apache/airflow/pull/11292 Github Actions deprecated the set-env action due to moderate security vulnerability they found. https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-comma

[GitHub] [airflow] kishvanchee edited a comment on pull request #11277: Move dagrun_operator.py to dagrun.py (#11178)

2020-10-05 Thread GitBox
kishvanchee edited a comment on pull request #11277: URL: https://github.com/apache/airflow/pull/11277#issuecomment-703783997 I just noticed. In tests it is named as `test_trigger_dagrun_operator_conf` and `test_trigger_dagrun_operator_templated_conf`. Should those be changed to `test_trig

[GitHub] [airflow] jacobmhoffman commented on a change in pull request #11285: [AIRFLOW-11041] Create AirflowMacroPluginRemovedRule to ease upgrade to Airflow 2.0

2020-10-05 Thread GitBox
jacobmhoffman commented on a change in pull request #11285: URL: https://github.com/apache/airflow/pull/11285#discussion_r500019774 ## File path: tests/upgrade/rules/test_airflow_macro_plugin_removed.py ## @@ -0,0 +1,73 @@ +# Licensed to the Apache Software Foundation (ASF) und

[GitHub] [airflow] boring-cyborg[bot] commented on issue #11291: Airflow 1.10.12 : webserver gunicorn issue - Arguments to config option are malformed: python:airflow.www.gunicorn_config

2020-10-05 Thread GitBox
boring-cyborg[bot] commented on issue #11291: URL: https://github.com/apache/airflow/issues/11291#issuecomment-704020784 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automate

[GitHub] [airflow] rajeshkatkarnice opened a new issue #11291: Airflow 1.10.12 : webserver gunicorn issue - Arguments to config option are malformed: python:airflow.www.gunicorn_config

2020-10-05 Thread GitBox
rajeshkatkarnice opened a new issue #11291: URL: https://github.com/apache/airflow/issues/11291 **Apache Airflow version**: **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): **Environment**: - **Cloud provider or hardware c

[GitHub] [airflow] nullhack edited a comment on pull request #9464: Fix DockerOperator xcom

2020-10-05 Thread GitBox
nullhack edited a comment on pull request #9464: URL: https://github.com/apache/airflow/pull/9464#issuecomment-703541914 Thank you, yes you're right. They're failing because the test assume a binary string `b'container log'` One of the modifications of this PR is changing this to a d

[GitHub] [airflow] nullhack edited a comment on pull request #9464: Fix DockerOperator xcom

2020-10-05 Thread GitBox
nullhack edited a comment on pull request #9464: URL: https://github.com/apache/airflow/pull/9464#issuecomment-703541914 Thank you, yes you're right. They're failing because the test assume a binary string `b'container log'` One of the modifications of this PR is changing this to a d

[GitHub] [airflow] boring-cyborg[bot] commented on issue #11290: Logout issues with Gihub Enterprise auth enabled

2020-10-05 Thread GitBox
boring-cyborg[bot] commented on issue #11290: URL: https://github.com/apache/airflow/issues/11290#issuecomment-704006597 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automate

[GitHub] [airflow] srthok opened a new issue #11290: Logout issues with Gihub Enterprise auth enabled

2020-10-05 Thread GitBox
srthok opened a new issue #11290: URL: https://github.com/apache/airflow/issues/11290 **Apache Airflow version**: 1.10.9 **Environment**: - **Cloud provider or hardware configuration**: AWS - **OS** (e.g. from /etc/os-release): Ubuntu 18.04.4 LTS - **Kernel** (e

[GitHub] [airflow] atalimekhian opened a new issue #11289: Payment d

2020-10-05 Thread GitBox
atalimekhian opened a new issue #11289: URL: https://github.com/apache/airflow/issues/11289 **Apache Airflow version**: **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): **Environment**: - **Cloud provider or hardware confi

[GitHub] [airflow] boring-cyborg[bot] commented on issue #11289: Payment d

2020-10-05 Thread GitBox
boring-cyborg[bot] commented on issue #11289: URL: https://github.com/apache/airflow/issues/11289#issuecomment-703999010 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automate

[GitHub] [airflow] github-actions[bot] commented on pull request #11090: Add rules to ensure users are using pod_template_file for KubernetesE…

2020-10-05 Thread GitBox
github-actions[bot] commented on pull request #11090: URL: https://github.com/apache/airflow/pull/11090#issuecomment-703998825 [The Build Workflow run](https://github.com/apache/airflow/actions/runs/290596969) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$

[GitHub] [airflow] KIRY4 edited a comment on pull request #8745: Install python-ldap library for FAB LDAP authorization with Python3

2020-10-05 Thread GitBox
KIRY4 edited a comment on pull request #8745: URL: https://github.com/apache/airflow/pull/8745#issuecomment-703981497 Hello! I'm using stable helm chart (image: 1.10.12-python3.6) and when I tried to install python-ldap using extraPipPackage I got following: unable to execute 'g

[GitHub] [airflow] KIRY4 commented on pull request #8745: Install python-ldap library for FAB LDAP authorization with Python3

2020-10-05 Thread GitBox
KIRY4 commented on pull request #8745: URL: https://github.com/apache/airflow/pull/8745#issuecomment-703981497 Hello! I'm using stable helm chart (image: 1.10.12-python3.6) and when I tried to install python-ldap using extraPipPackage I got following: -- unable to execute 'gcc':

[GitHub] [airflow] dyi1 commented on issue #8160: SimpleHttpOperator aborts connection after 5 minutes

2020-10-05 Thread GitBox
dyi1 commented on issue #8160: URL: https://github.com/apache/airflow/issues/8160#issuecomment-703936612 @anirudhbagri Could you give me details as to what the root cause of your issue was and how you fixed it? I'm currently experiencing the same issue and I think it would help a lot.

[airflow] branch master updated (c51016b -> 4847460)

2020-10-05 Thread dimberman
This is an automated email from the ASF dual-hosted git repository. dimberman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from c51016b Add LocalToAzureDataLakeStorageOperator (#10814) add 4847460 Add CeleryKubernetesExecutor to helm

[airflow] branch master updated (c51016b -> 4847460)

2020-10-05 Thread dimberman
This is an automated email from the ASF dual-hosted git repository. dimberman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from c51016b Add LocalToAzureDataLakeStorageOperator (#10814) add 4847460 Add CeleryKubernetesExecutor to helm

[airflow] branch master updated (c51016b -> 4847460)

2020-10-05 Thread dimberman
This is an automated email from the ASF dual-hosted git repository. dimberman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from c51016b Add LocalToAzureDataLakeStorageOperator (#10814) add 4847460 Add CeleryKubernetesExecutor to helm

[GitHub] [airflow] dimberman merged pull request #11288: Add CeleryKubernetesExecutor to helm chart

2020-10-05 Thread GitBox
dimberman merged pull request #11288: URL: https://github.com/apache/airflow/pull/11288 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [airflow] dimberman opened a new pull request #11288: Add CeleryKubernetesExecutor to helm chart

2020-10-05 Thread GitBox
dimberman opened a new pull request #11288: URL: https://github.com/apache/airflow/pull/11288 Users of the CeleryKubernetesExecutor will require both Celery and Kubernetes features to launch tasks. This PR will also serve as the basis for integration tests for this executor

[GitHub] [airflow] ashb commented on a change in pull request #10956: Officially support running more than one scheduler concurrently.

2020-10-05 Thread GitBox
ashb commented on a change in pull request #10956: URL: https://github.com/apache/airflow/pull/10956#discussion_r499883416 ## File path: airflow/models/pool.py ## @@ -91,7 +96,15 @@ def slots_stats(session: Session = None) -> Dict[str, PoolStats]: pools: Dict[str, P

[GitHub] [airflow] ashb commented on a change in pull request #10956: Officially support running more than one scheduler concurrently.

2020-10-05 Thread GitBox
ashb commented on a change in pull request #10956: URL: https://github.com/apache/airflow/pull/10956#discussion_r499882907 ## File path: airflow/models/dag.py ## @@ -1573,34 +1680,28 @@ def create_dagrun(self, @classmethod @provide_session -def bulk_sync_to_db(c

[GitHub] [airflow] ashb commented on a change in pull request #10956: Officially support running more than one scheduler concurrently.

2020-10-05 Thread GitBox
ashb commented on a change in pull request #10956: URL: https://github.com/apache/airflow/pull/10956#discussion_r499878674 ## File path: airflow/utils/sqlalchemy.py ## @@ -141,3 +142,107 @@ def skip_locked(session: Session) -> Dict[str, Any]: return {'skip_locked': Tru

[GitHub] [airflow] nikste commented on issue #7907: End-to-end DAG testing

2020-10-05 Thread GitBox
nikste commented on issue #7907: URL: https://github.com/apache/airflow/issues/7907#issuecomment-703882074 Sorry, this somehow drowned in other messages. So, I have a DAG and some specified input, and i expect a certain output. I want to run all operators end to end. Currently I'm

[airflow] branch master updated (93475e9 -> c51016b)

2020-10-05 Thread turbaszek
This is an automated email from the ASF dual-hosted git repository. turbaszek pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from 93475e9 Single/Multi-Namespace mode for helm chart (#11034) add c51016b Add LocalToAzureDataLakeStorageOpe

[airflow] branch master updated (93475e9 -> c51016b)

2020-10-05 Thread turbaszek
This is an automated email from the ASF dual-hosted git repository. turbaszek pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from 93475e9 Single/Multi-Namespace mode for helm chart (#11034) add c51016b Add LocalToAzureDataLakeStorageOpe

[GitHub] [airflow] turbaszek merged pull request #10814: Add LocalToAzureDataLakeStorageOperator

2020-10-05 Thread GitBox
turbaszek merged pull request #10814: URL: https://github.com/apache/airflow/pull/10814 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [airflow] turbaszek commented on pull request #9464: Fix DockerOperator xcom

2020-10-05 Thread GitBox
turbaszek commented on pull request #9464: URL: https://github.com/apache/airflow/pull/9464#issuecomment-703861860 I would be in favor of 2. + deprecation warning. I think we should return strings. @feluelle @mik-laj any opinions? -

[GitHub] [airflow] turbaszek commented on a change in pull request #10956: Officially support running more than one scheduler concurrently.

2020-10-05 Thread GitBox
turbaszek commented on a change in pull request #10956: URL: https://github.com/apache/airflow/pull/10956#discussion_r499841037 ## File path: airflow/jobs/scheduler_job.py ## @@ -1714,53 +1321,158 @@ def _run_scheduler_loop(self) -> None: ) bre

[GitHub] [airflow] turbaszek commented on a change in pull request #10956: Officially support running more than one scheduler concurrently.

2020-10-05 Thread GitBox
turbaszek commented on a change in pull request #10956: URL: https://github.com/apache/airflow/pull/10956#discussion_r499840548 ## File path: airflow/jobs/scheduler_job.py ## @@ -556,12 +556,15 @@ def execute_callbacks( :param session: DB session. """

[GitHub] [airflow] ashb commented on a change in pull request #10956: Officially support running more than one scheduler concurrently.

2020-10-05 Thread GitBox
ashb commented on a change in pull request #10956: URL: https://github.com/apache/airflow/pull/10956#discussion_r499834162 ## File path: airflow/models/dag.py ## @@ -1939,6 +2099,37 @@ def deactivate_deleted_dags(cls, alive_dag_filelocs: List[str], session=None):

[GitHub] [airflow] ashb commented on a change in pull request #10956: Officially support running more than one scheduler concurrently.

2020-10-05 Thread GitBox
ashb commented on a change in pull request #10956: URL: https://github.com/apache/airflow/pull/10956#discussion_r499833094 ## File path: airflow/models/dagrun.py ## @@ -494,12 +542,13 @@ def verify_integrity(self, session: Session = None): session.add(ti)

[GitHub] [airflow] ashb commented on a change in pull request #10956: Officially support running more than one scheduler concurrently.

2020-10-05 Thread GitBox
ashb commented on a change in pull request #10956: URL: https://github.com/apache/airflow/pull/10956#discussion_r499833303 ## File path: airflow/models/dagrun.py ## @@ -313,14 +360,16 @@ def update_state(self, session: Session = None) -> List[TI]: :rtype ready_tis: li

[GitHub] [airflow] ashb commented on a change in pull request #10956: Officially support running more than one scheduler concurrently.

2020-10-05 Thread GitBox
ashb commented on a change in pull request #10956: URL: https://github.com/apache/airflow/pull/10956#discussion_r499832547 ## File path: airflow/jobs/scheduler_job.py ## @@ -1705,62 +1305,216 @@ def _run_scheduler_loop(self) -> None: loop_duration = loop_end_time -

[GitHub] [airflow] ashb commented on pull request #10956: Officially support running more than one scheduler concurrently.

2020-10-05 Thread GitBox
ashb commented on pull request #10956: URL: https://github.com/apache/airflow/pull/10956#issuecomment-703848887 The "The job has been cancelled by another workflow." message is slightly confusing -- I'll look at updating that. In this case it's cancelled this PR, because the Build Image wo

[GitHub] [airflow] github-actions[bot] commented on pull request #10956: Officially support running more than one scheduler concurrently.

2020-10-05 Thread GitBox
github-actions[bot] commented on pull request #10956: URL: https://github.com/apache/airflow/pull/10956#issuecomment-703845760 [The Build Workflow run](https://github.com/apache/airflow/actions/runs/289987522) is cancelling this PR. The job has been cancelled by another workflow. ---

[GitHub] [airflow] ashb commented on a change in pull request #10956: Officially support running more than one scheduler concurrently.

2020-10-05 Thread GitBox
ashb commented on a change in pull request #10956: URL: https://github.com/apache/airflow/pull/10956#discussion_r499826337 ## File path: airflow/jobs/scheduler_job.py ## @@ -1714,53 +1321,158 @@ def _run_scheduler_loop(self) -> None: ) break

[GitHub] [airflow] ashb commented on a change in pull request #10956: Officially support running more than one scheduler concurrently.

2020-10-05 Thread GitBox
ashb commented on a change in pull request #10956: URL: https://github.com/apache/airflow/pull/10956#discussion_r499826539 ## File path: airflow/jobs/scheduler_job.py ## @@ -1714,53 +1321,158 @@ def _run_scheduler_loop(self) -> None: ) break

[GitHub] [airflow] XD-DENG commented on pull request #11196: Use priority queue algorithm to handle priority in executors

2020-10-05 Thread GitBox
XD-DENG commented on pull request #11196: URL: https://github.com/apache/airflow/pull/11196#issuecomment-703836194 Will further proceed once we have more clarity on the Scheduler HA feature (changing some common files). This

[GitHub] [airflow] github-actions[bot] commented on pull request #10956: Officially support running more than one scheduler concurrently.

2020-10-05 Thread GitBox
github-actions[bot] commented on pull request #10956: URL: https://github.com/apache/airflow/pull/10956#issuecomment-703834310 [The Build Workflow run](https://github.com/apache/airflow/actions/runs/289987522) is cancelling this PR. The job has been cancelled by another workflow. ---

[GitHub] [airflow] github-actions[bot] commented on pull request #10838: Fix autocommit property for snowflake connection

2020-10-05 Thread GitBox
github-actions[bot] commented on pull request #10838: URL: https://github.com/apache/airflow/pull/10838#issuecomment-703833344 [The Build Workflow run](https://github.com/apache/airflow/actions/runs/290013570) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$

[GitHub] [airflow] ivica-k commented on a change in pull request #11227: includes the STS token if STS credentials are used

2020-10-05 Thread GitBox
ivica-k commented on a change in pull request #11227: URL: https://github.com/apache/airflow/pull/11227#discussion_r499802779 ## File path: airflow/providers/amazon/aws/transfers/redshift_to_s3.py ## @@ -104,29 +107,50 @@ def __init__( # pylint: disable=too-many-arguments

[GitHub] [airflow] ashb commented on a change in pull request #11189: WIP: Prepend `DAG:` to dag permissions

2020-10-05 Thread GitBox
ashb commented on a change in pull request #11189: URL: https://github.com/apache/airflow/pull/11189#discussion_r499800190 ## File path: airflow/www/security.py ## @@ -520,18 +560,23 @@ def merge_pv(perm, view_menu): def update_admin_perm_view(self): """

[GitHub] [airflow] drsantos20 commented on a change in pull request #11283: Refactor

2020-10-05 Thread GitBox
drsantos20 commented on a change in pull request #11283: URL: https://github.com/apache/airflow/pull/11283#discussion_r499791981 ## File path: setup.py ## @@ -65,12 +64,6 @@ class CleanCommand(Command): description = "Tidy up the project root" user_options = [] # typ

[GitHub] [airflow] rootcss commented on issue #11161: Improved health checks

2020-10-05 Thread GitBox
rootcss commented on issue #11161: URL: https://github.com/apache/airflow/issues/11161#issuecomment-703810947 @mik-laj Shall I pick this one up? This is an automated message from the Apache Git Service. To respond to the mess

[GitHub] [airflow] ashb commented on a change in pull request #11158: WIP: Migrate existing permissions to resource-based permissions

2020-10-05 Thread GitBox
ashb commented on a change in pull request #11158: URL: https://github.com/apache/airflow/pull/11158#discussion_r499784748 ## File path: airflow/cli/commands/role_command.py ## @@ -41,3 +39,259 @@ def roles_create(args): appbuilder = cached_app().appbuilder # pylint: disa

[GitHub] [airflow] turbaszek commented on a change in pull request #11227: includes the STS token if STS credentials are used

2020-10-05 Thread GitBox
turbaszek commented on a change in pull request #11227: URL: https://github.com/apache/airflow/pull/11227#discussion_r499783147 ## File path: airflow/providers/amazon/aws/transfers/redshift_to_s3.py ## @@ -104,29 +107,50 @@ def __init__( # pylint: disable=too-many-arguments

[GitHub] [airflow] jhtimmins commented on a change in pull request #11189: WIP: Prepend `DAG:` to dag permissions

2020-10-05 Thread GitBox
jhtimmins commented on a change in pull request #11189: URL: https://github.com/apache/airflow/pull/11189#discussion_r499777685 ## File path: airflow/cli/cli_parser.py ## @@ -1228,6 +1228,12 @@ class GroupCommand(NamedTuple): func=lazy_load_command('airflow.cli.comman

[GitHub] [airflow] jhtimmins commented on a change in pull request #11189: WIP: Prepend `DAG:` to dag permissions

2020-10-05 Thread GitBox
jhtimmins commented on a change in pull request #11189: URL: https://github.com/apache/airflow/pull/11189#discussion_r499776966 ## File path: airflow/www/security.py ## @@ -520,18 +560,23 @@ def merge_pv(perm, view_menu): def update_admin_perm_view(self): """

[GitHub] [airflow] jhtimmins commented on a change in pull request #11189: WIP: Prepend `DAG:` to dag permissions

2020-10-05 Thread GitBox
jhtimmins commented on a change in pull request #11189: URL: https://github.com/apache/airflow/pull/11189#discussion_r499776424 ## File path: airflow/www/security.py ## @@ -320,33 +317,75 @@ def get_accessible_dags(self, user_actions, user, session=None): for perm

[GitHub] [airflow] ivica-k commented on pull request #11227: includes the STS token if STS credentials are used

2020-10-05 Thread GitBox
ivica-k commented on pull request #11227: URL: https://github.com/apache/airflow/pull/11227#issuecomment-703793262 Does CI need to be triggered again because of cancelled checks? If so, how? :) This is an automated message f

[GitHub] [airflow] turbaszek commented on issue #11282: BigQuery hook in providers creates non-unique job_ids

2020-10-05 Thread GitBox
turbaszek commented on issue #11282: URL: https://github.com/apache/airflow/issues/11282#issuecomment-703792725 @nathadfield #11287 I think this should do, WDYT? This is an automated message from the Apache Git Service. To re

[GitHub] [airflow] ashb commented on a change in pull request #11075: WIP: Replace existing perms w new perms

2020-10-05 Thread GitBox
ashb commented on a change in pull request #11075: URL: https://github.com/apache/airflow/pull/11075#discussion_r499775503 ## File path: airflow/www/views.py ## @@ -393,6 +393,7 @@ def health(self): @expose('/home') @has_access +@permission_name() Review commen

[GitHub] [airflow] github-actions[bot] commented on pull request #11277: Move dagrun_operator.py to dagrun.py (#11178)

2020-10-05 Thread GitBox
github-actions[bot] commented on pull request #11277: URL: https://github.com/apache/airflow/pull/11277#issuecomment-703792547 [The Build Workflow run](https://github.com/apache/airflow/actions/runs/289866879) is cancelling this PR. The job has been cancelled by another workflow. ---

[GitHub] [airflow] nathadfield commented on issue #11282: BigQuery hook in providers creates non-unique job_ids

2020-10-05 Thread GitBox
nathadfield commented on issue #11282: URL: https://github.com/apache/airflow/issues/11282#issuecomment-703789894 Totally agree! This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [airflow] turbaszek commented on issue #11282: BigQuery hook in providers creates non-unique job_ids

2020-10-05 Thread GitBox
turbaszek commented on issue #11282: URL: https://github.com/apache/airflow/issues/11282#issuecomment-703789283 @nathadfield I can push a simple fix and then we can improve it... better done than perfect 😄 This is an automa

[GitHub] [airflow] nathadfield commented on issue #11282: BigQuery hook in providers creates non-unique job_ids

2020-10-05 Thread GitBox
nathadfield commented on issue #11282: URL: https://github.com/apache/airflow/issues/11282#issuecomment-703788869 I understand. Its not critical for us right now as we'll just continue to use the current operators. Perhaps this is something for the next round of backports then? ---

[GitHub] [airflow] turbaszek commented on issue #11282: BigQuery hook in providers creates non-unique job_ids

2020-10-05 Thread GitBox
turbaszek commented on issue #11282: URL: https://github.com/apache/airflow/issues/11282#issuecomment-703787636 > @turbaszek Extending the timestamp - perhaps to millisecond - would probably fix it but I feel it would be better overall if there was a consistent methodology for creating the

[GitHub] [airflow] nathadfield commented on issue #11282: BigQuery hook in providers creates non-unique job_ids

2020-10-05 Thread GitBox
nathadfield commented on issue #11282: URL: https://github.com/apache/airflow/issues/11282#issuecomment-703786851 @turbaszek Extending the timestamp - perhaps to millisecond - would probably fix it but I feel it would be better overall if there was a consistent methodology for creating the

[GitHub] [airflow] kishvanchee commented on pull request #11277: Move dagrun_operator.py to dagrun.py (#11178)

2020-10-05 Thread GitBox
kishvanchee commented on pull request #11277: URL: https://github.com/apache/airflow/pull/11277#issuecomment-703783997 I just noticed. In tests it is named as `test_trigger_dagrun_operator_conf` and `test_trigger_dagrun_operator_templated_conf`. Should those be changed to `test_trigger_dag

[GitHub] [airflow] turbaszek edited a comment on issue #11282: BigQuery hook in providers creates non-unique job_ids

2020-10-05 Thread GitBox
turbaszek edited a comment on issue #11282: URL: https://github.com/apache/airflow/issues/11282#issuecomment-703779936 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [airflow] nathadfield commented on issue #11282: BigQuery hook in providers creates non-unique job_ids

2020-10-05 Thread GitBox
nathadfield commented on issue #11282: URL: https://github.com/apache/airflow/issues/11282#issuecomment-703783010 Not sure how you want to categorise it. Yes, all the old operators use a random id. This came about because I'm trying to use the new operators in existing production D

[airflow] branch master updated (a33a919 -> 93475e9)

2020-10-05 Thread dimberman
This is an automated email from the ASF dual-hosted git repository. dimberman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from a33a919 Switched to Run Checks for Building Images. (#11276) add 93475e9 Single/Multi-Namespace mode for h

[airflow] branch master updated (a33a919 -> 93475e9)

2020-10-05 Thread dimberman
This is an automated email from the ASF dual-hosted git repository. dimberman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from a33a919 Switched to Run Checks for Building Images. (#11276) add 93475e9 Single/Multi-Namespace mode for h

[GitHub] [airflow] dimberman merged pull request #11034: Single/Multi-Namespace mode for helm chart

2020-10-05 Thread GitBox
dimberman merged pull request #11034: URL: https://github.com/apache/airflow/pull/11034 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [airflow] turbaszek commented on issue #11282: BigQuery hook in providers creates non-unique job_ids

2020-10-05 Thread GitBox
turbaszek commented on issue #11282: URL: https://github.com/apache/airflow/issues/11282#issuecomment-703779936 Is this a bug or request feature? I think previously all operators used a random job_id, right? This is an autom

[GitHub] [airflow] turbaszek opened a new pull request #11287: Improve idempotency of BigQuery operators

2020-10-05 Thread GitBox
turbaszek opened a new pull request #11287: URL: https://github.com/apache/airflow/pull/11287 Previously the job_id was always automatically generated but in refactor this was lost. Closes: #11282 Closes: #11280 --- **^ Add meaningful description above**

[GitHub] [airflow] nathadfield commented on issue #11280: BigQueryInsertJobOperator will fail if the DagID contains the '.' character

2020-10-05 Thread GitBox
nathadfield commented on issue #11280: URL: https://github.com/apache/airflow/issues/11280#issuecomment-703772922 Well, perhaps this is something that I shouldn't have done early on, but we've specified a DAG version as part of the DagID using a '.'. Not had any problems with this until n

[GitHub] [airflow] saisiddhant12 commented on issue #11280: BigQueryInsertJobOperator will fail if the DagID contains the '.' character

2020-10-05 Thread GitBox
saisiddhant12 commented on issue #11280: URL: https://github.com/apache/airflow/issues/11280#issuecomment-703766096 > Btw. I'm not sure if using `.` in dag_id is the best idea. The dot is special for subdags if I'm not mistaken yes, it creates a sub-dag if we use . between parent & c

[GitHub] [airflow] turbaszek commented on issue #11280: BigQueryInsertJobOperator will fail if the DagID contains the '.' character

2020-10-05 Thread GitBox
turbaszek commented on issue #11280: URL: https://github.com/apache/airflow/issues/11280#issuecomment-703764813 Btw. I'm not sure if using `.` in dag_id is the best idea. The dot is special for subdags if I'm not mistaken T

[GitHub] [airflow] nathadfield commented on issue #11282: BigQuery hook in providers creates non-unique job_ids

2020-10-05 Thread GitBox
nathadfield commented on issue #11282: URL: https://github.com/apache/airflow/issues/11282#issuecomment-703763320 @potiuk We're more than happy to give it a try but we've just got to find the time. This is an automated messa

[GitHub] [airflow] turbaszek commented on issue #11280: BigQueryInsertJobOperator will fail if the DagID contains the '.' character

2020-10-05 Thread GitBox
turbaszek commented on issue #11280: URL: https://github.com/apache/airflow/issues/11280#issuecomment-703762921 Or should we build the string and the use the following code with added `.`? https://github.com/apache/airflow/blob/a33a91951b731f1a5496be08e47d503527382968/airflow/providers/g

[GitHub] [airflow] turbaszek edited a comment on issue #11280: BigQueryInsertJobOperator will fail if the DagID contains the '.' character

2020-10-05 Thread GitBox
turbaszek edited a comment on issue #11280: URL: https://github.com/apache/airflow/issues/11280#issuecomment-703762921 Or should we build the string and then use the following code with added `.`? https://github.com/apache/airflow/blob/a33a91951b731f1a5496be08e47d503527382968/airflow/pro

[GitHub] [airflow] turbaszek commented on a change in pull request #11285: [AIRFLOW-11041] Create AirflowMacroPluginRemovedRule to ease upgrade to Airflow 2.0

2020-10-05 Thread GitBox
turbaszek commented on a change in pull request #11285: URL: https://github.com/apache/airflow/pull/11285#discussion_r499744010 ## File path: tests/upgrade/rules/test_airflow_macro_plugin_removed.py ## @@ -0,0 +1,73 @@ +# Licensed to the Apache Software Foundation (ASF) under o

[GitHub] [airflow] eladkal edited a comment on pull request #11209: Change prefix of Aws dynamodb hook file

2020-10-05 Thread GitBox
eladkal edited a comment on pull request #11209: URL: https://github.com/apache/airflow/pull/11209#issuecomment-703760379 I'm not sure what is the issue with the black static test. It doesnt happen on my local fork. This is

[GitHub] [airflow] potiuk commented on issue #11282: BigQuery hook in providers creates non-unique job_ids

2020-10-05 Thread GitBox
potiuk commented on issue #11282: URL: https://github.com/apache/airflow/issues/11282#issuecomment-703761047 And @nathadfield -> maybe you or someone from your team would like to fix it ? I am sure we can help with having out the details and review. ---

[GitHub] [airflow] potiuk commented on issue #11282: BigQuery hook in providers creates non-unique job_ids

2020-10-05 Thread GitBox
potiuk commented on issue #11282: URL: https://github.com/apache/airflow/issues/11282#issuecomment-703760483 cc: @turbaszek - WDYT ? This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [airflow] eladkal commented on pull request #11209: Change prefix of Aws dynamodb hook file

2020-10-05 Thread GitBox
eladkal commented on pull request #11209: URL: https://github.com/apache/airflow/pull/11209#issuecomment-703760379 I'm not sure how to fix the black static test. It doesnt happen on my local fork. This is an automated messag

[GitHub] [airflow] potiuk commented on issue #11280: BigQueryInsertJobOperator will fail if the DagID contains the '.' character

2020-10-05 Thread GitBox
potiuk commented on issue #11280: URL: https://github.com/apache/airflow/issues/11280#issuecomment-703759481 I think it we already do similar thing elsewhere in Airflow - we are replacing . with __dot__ I believe. It is in "views.py" - and it is mostly to prevent some subdag matching. I do

[GitHub] [airflow] turbaszek commented on a change in pull request #11285: [AIRFLOW-11041] Create AirflowMacroPluginRemovedRule to ease upgrade to Airflow 2.0

2020-10-05 Thread GitBox
turbaszek commented on a change in pull request #11285: URL: https://github.com/apache/airflow/pull/11285#discussion_r499741937 ## File path: airflow/upgrade/rules/airflow_macro_plugin_removed.py ## @@ -0,0 +1,55 @@ +# Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [airflow] turbaszek commented on a change in pull request #11285: [AIRFLOW-11041] Create AirflowMacroPluginRemovedRule to ease upgrade to Airflow 2.0

2020-10-05 Thread GitBox
turbaszek commented on a change in pull request #11285: URL: https://github.com/apache/airflow/pull/11285#discussion_r499741673 ## File path: airflow/upgrade/rules/airflow_macro_plugin_removed.py ## @@ -0,0 +1,55 @@ +# Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [airflow] saisiddhant12 commented on issue #11280: BigQueryInsertJobOperator will fail if the DagID contains the '.' character

2020-10-05 Thread GitBox
saisiddhant12 commented on issue #11280: URL: https://github.com/apache/airflow/issues/11280#issuecomment-703759064 @nathadfield @potiuk I would like to give it a shot, I can think introducing macro like ts_no_dash with a hyphen separation of each version should solve the problem, or

[GitHub] [airflow] prakshalj0512 opened a new issue #11286: Configs under `config` in values.yaml aren't applying to worker pods

2020-10-05 Thread GitBox
prakshalj0512 opened a new issue #11286: URL: https://github.com/apache/airflow/issues/11286 **Apache Airflow version**: **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): ```Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", G

[GitHub] [airflow] turbaszek commented on a change in pull request #11285: [AIRFLOW-11041] Create AirflowMacroPluginRemovedRule to ease upgrade to Airflow 2.0

2020-10-05 Thread GitBox
turbaszek commented on a change in pull request #11285: URL: https://github.com/apache/airflow/pull/11285#discussion_r499740657 ## File path: airflow/upgrade/rules/airflow_macro_plugin_removed.py ## @@ -0,0 +1,55 @@ +# Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [airflow] turbaszek commented on a change in pull request #11285: [AIRFLOW-11041] Create AirflowMacroPluginRemovedRule to ease upgrade to Airflow 2.0

2020-10-05 Thread GitBox
turbaszek commented on a change in pull request #11285: URL: https://github.com/apache/airflow/pull/11285#discussion_r499739588 ## File path: airflow/upgrade/rules/airflow_macro_plugin_removed.py ## @@ -0,0 +1,55 @@ +# Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [airflow] potiuk commented on pull request #10956: Officially support running more than one scheduler concurrently.

2020-10-05 Thread GitBox
potiuk commented on pull request #10956: URL: https://github.com/apache/airflow/pull/10956#issuecomment-703756198 Unfortunately, there is no "rerun failed job up to x times" feature in GitHub and this part is a bit outside of our control. --

[GitHub] [airflow] potiuk commented on pull request #10956: Officially support running more than one scheduler concurrently.

2020-10-05 Thread GitBox
potiuk commented on pull request #10956: URL: https://github.com/apache/airflow/pull/10956#issuecomment-703755603 And "re-running" is not happening at all - if the build image fails (like in this case transiently - on pushing to the Github registry) then it is treated as any other failure

[GitHub] [airflow] potiuk commented on pull request #10956: Officially support running more than one scheduler concurrently.

2020-10-05 Thread GitBox
potiuk commented on pull request #10956: URL: https://github.com/apache/airflow/pull/10956#issuecomment-703754865 You need to restart the workflow or commit --amend and push --force. This is an automated message from the Apac

[GitHub] [airflow] potiuk commented on pull request #10956: Officially support running more than one scheduler concurrently.

2020-10-05 Thread GitBox
potiuk commented on pull request #10956: URL: https://github.com/apache/airflow/pull/10956#issuecomment-703754413 @ashb "Same "unknown blob" if you follow the link and look at the failed job. - I got it also today several times with my personal account so it seems like a transient instabil

[GitHub] [airflow] turbaszek commented on pull request #10838: Fix autocommit property for snowflake connection

2020-10-05 Thread GitBox
turbaszek commented on pull request #10838: URL: https://github.com/apache/airflow/pull/10838#issuecomment-703753991 This may help: ``` git fetch --all git rebase apache/master git cherrypick 8a8bee4 git log ``` If you see in log your commit then do `git push --force-wit

[GitHub] [airflow] turbaszek edited a comment on pull request #10838: Fix autocommit property for snowflake connection

2020-10-05 Thread GitBox
turbaszek edited a comment on pull request #10838: URL: https://github.com/apache/airflow/pull/10838#issuecomment-703753991 This may help: ``` git fetch --all git reset --hard apache/master git cherrypick 8a8bee4 git log ``` If you see in log your commit then do `git pus

[GitHub] [airflow] OmairK edited a comment on pull request #9597: [WIP] Add read-only endpoints for task instances

2020-10-05 Thread GitBox
OmairK edited a comment on pull request #9597: URL: https://github.com/apache/airflow/pull/9597#issuecomment-703751945 > @OmairK @mik-laj Any updates here? Waiting for @mik-laj's review on the approach I followed.

[GitHub] [airflow] OmairK commented on pull request #9597: [WIP] Add read-only endpoints for task instances

2020-10-05 Thread GitBox
OmairK commented on pull request #9597: URL: https://github.com/apache/airflow/pull/9597#issuecomment-703751945 > @OmairK @mik-laj Any updates here? Waiting for @mik-laj's review on the approach I followed, This is an

[GitHub] [airflow] potiuk commented on pull request #11276: Switched to Run Checks for Building Images.

2020-10-05 Thread GitBox
potiuk commented on pull request #11276: URL: https://github.com/apache/airflow/pull/11276#issuecomment-703748893 > I think we should merge this one and adjust with the time being. I'm sorry but I don't have enough capacity to fully understand the problem now Yep. This is something I

[airflow] branch master updated (baa980f -> a33a919)

2020-10-05 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from baa980f When sending tasks to celery from a sub-process, reset signal handlers (#11278) add a33a919 Switched

  1   2   3   4   >