Re: [PR] Add a public interface for custom weight_rule implementation [airflow]

2023-12-02 Thread via GitHub
potiuk commented on PR #35210: URL: https://github.com/apache/airflow/pull/35210#issuecomment-1837140121 Hey @hussein-awala @jscheffl -> I've proposed https://github.com/apache/airflow/pull/36022 (subject to feedback and discussion - describing the way I understand the current model of DAG

Re: [PR] Add a public interface for custom weight_rule implementation [airflow]

2023-11-28 Thread via GitHub
potiuk commented on PR #35210: URL: https://github.com/apache/airflow/pull/35210#issuecomment-1830836923 BTW. I realized that this is something we should describe in our Security Model https://airflow.apache.org/docs/apache-airflow/stable/security/security_model.html. I will draft PR

Re: [PR] Add a public interface for custom weight_rule implementation [airflow]

2023-11-28 Thread via GitHub
potiuk commented on PR #35210: URL: https://github.com/apache/airflow/pull/35210#issuecomment-1830836013 >UPDATE: Okay I was wrong. Just tested with an example DAG and if you use the module name . then it is possible to inject code from the DAG directly. yep. it is > If the

Re: [PR] Add a public interface for custom weight_rule implementation [airflow]

2023-11-28 Thread via GitHub
hussein-awala commented on PR #35210: URL: https://github.com/apache/airflow/pull/35210#issuecomment-1830756014 > import_string I will take a look tomorrow at the history of this security issue and if it's the case for the new feature. > Are you sure it can inject "any" code

Re: [PR] Add a public interface for custom weight_rule implementation [airflow]

2023-11-28 Thread via GitHub
jscheffl commented on PR #35210: URL: https://github.com/apache/airflow/pull/35210#issuecomment-1830653754 > I believe this change breaks this assumption. This `priority_strategy_class = import_string(strategy_name)` will be executed in schecduler, and the strategy can be defined in DAG -

Re: [PR] Add a public interface for custom weight_rule implementation [airflow]

2023-11-28 Thread via GitHub
potiuk commented on PR #35210: URL: https://github.com/apache/airflow/pull/35210#issuecomment-1830553450 cc: @hussein-awala -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Add a public interface for custom weight_rule implementation [airflow]

2023-11-28 Thread via GitHub
potiuk commented on PR #35210: URL: https://github.com/apache/airflow/pull/35210#issuecomment-1830551874 Late to the game - but I think (maybe I am wrong) we have potential security issue here (but it's easy to fix). For quite some time wa have been very careful to not allow to execute the

Re: [PR] Add a public interface for custom weight_rule implementation [airflow]

2023-11-28 Thread via GitHub
hussein-awala merged PR #35210: URL: https://github.com/apache/airflow/pull/35210 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Add a public interface for custom weight_rule implementation [airflow]

2023-11-28 Thread via GitHub
hussein-awala commented on PR #35210: URL: https://github.com/apache/airflow/pull/35210#issuecomment-1830342908 I merge to test it and get some feebacks before 2.8.0 RC. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Add a public interface for custom weight_rule implementation [airflow]

2023-11-28 Thread via GitHub
hussein-awala commented on PR #35210: URL: https://github.com/apache/airflow/pull/35210#issuecomment-1830342380 I merge to test it and get some feebacks before 2.8.0 RC. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Add a public interface for custom weight_rule implementation [airflow]

2023-11-12 Thread via GitHub
eladkal commented on PR #35210: URL: https://github.com/apache/airflow/pull/35210#issuecomment-1807219135 This looks really good! I will find time this week to test it as well -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Add a public interface for custom weight_rule implementation [airflow]

2023-10-29 Thread via GitHub
hussein-awala commented on PR #35210: URL: https://github.com/apache/airflow/pull/35210#issuecomment-1784249764 I had to make some changes to provide a task instance instead of a task to get_weight method. I tested it in Breeze (CeleryExecutor + Postgres) with this dag: ```python

Re: [PR] Add a public interface for custom weight_rule implementation [airflow]

2023-10-28 Thread via GitHub
hussein-awala commented on PR #35210: URL: https://github.com/apache/airflow/pull/35210#issuecomment-1783932883 > I am just not sure whether the pickling of DAGs after parsing is sufficient such that a custom weight rule gets to Scheduler. Have you tested with separated DAG parser and

[PR] Add a public interface for custom weight_rule implementation [airflow]

2023-10-26 Thread via GitHub
hussein-awala opened a new pull request, #35210: URL: https://github.com/apache/airflow/pull/35210 This PR is a proposition to add a new public interface for weight rules (weight strategy) to allow the users to create custom classes to calculate the task priority weight. -- This is an