Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

MassyB commented on PR #38982: URL: https://github.com/apache/airflow/pull/38982#issuecomment-2402792332 @mobuchowski do you think this is only useful for Airflow ? I would like to have the same configuration for `dbt-ol` as well. The signature of the function would be different (an empt

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

mobuchowski commented on PR #38982: URL: https://github.com/apache/airflow/pull/38982#issuecomment-2242711641 Thanks @anandhimurali for implementing this feature 🚀 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

boring-cyborg[bot] commented on PR #38982: URL: https://github.com/apache/airflow/pull/38982#issuecomment-2242711124 Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional contributions.

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

mobuchowski merged PR #38982: URL: https://github.com/apache/airflow/pull/38982 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airf

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

JDarDagran commented on code in PR #38982: URL: https://github.com/apache/airflow/pull/38982#discussion_r1686210312 ## tests/providers/openlineage/utils/custom_facet_fixture.py: ## @@ -0,0 +1,87 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contribu

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

anandhimurali commented on code in PR #38982: URL: https://github.com/apache/airflow/pull/38982#discussion_r1684583435 ## tests/system/providers/google/cloud/datapipelines/resources/__init__.py: ## Review Comment: This was added by the pre-commit hook. I tried again now by

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

kacpermuda commented on code in PR #38982: URL: https://github.com/apache/airflow/pull/38982#discussion_r1684148107 ## tests/system/providers/google/cloud/datapipelines/resources/__init__.py: ## Review Comment: This file looks like some leftover from another branch -- T

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

anandhimurali commented on code in PR #38982: URL: https://github.com/apache/airflow/pull/38982#discussion_r1683261420 ## tests/providers/openlineage/utils/test_utils.py: ## @@ -482,3 +484,151 @@ def test_get_task_groups_details_nested(): def test_get_task_groups_details_no_t

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

kacpermuda commented on code in PR #38982: URL: https://github.com/apache/airflow/pull/38982#discussion_r1683042949 ## tests/providers/openlineage/utils/test_utils.py: ## @@ -482,3 +484,151 @@ def test_get_task_groups_details_nested(): def test_get_task_groups_details_no_task

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

anandhimurali commented on PR #38982: URL: https://github.com/apache/airflow/pull/38982#issuecomment-2234939275 >@anandhimurali If you need any help to make the CI pass let me know, If you run the pre-commits locally then all of the static checks will pass. When it comes to 2.7 compatibilit

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

anandhimurali commented on code in PR #38982: URL: https://github.com/apache/airflow/pull/38982#discussion_r1667195011 ## airflow/providers/openlineage/utils/utils.py: ## @@ -78,11 +78,38 @@ def get_job_name(task: TaskInstance) -> str: def get_custom_facets(task_instance: T

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

kacpermuda commented on PR #38982: URL: https://github.com/apache/airflow/pull/38982#issuecomment-2210548250 @anandhimurali If you need any help to make the CI pass let me know, If you run the pre-commits locally then all of the static checks will pass. When it comes to 2.7 compatibility te

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

kacpermuda commented on code in PR #38982: URL: https://github.com/apache/airflow/pull/38982#discussion_r1666431372 ## airflow/providers/openlineage/utils/utils.py: ## @@ -78,11 +78,38 @@ def get_job_name(task: TaskInstance) -> str: def get_custom_facets(task_instance: Task

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

anandhimurali commented on code in PR #38982: URL: https://github.com/apache/airflow/pull/38982#discussion_r1666420076 ## airflow/providers/openlineage/utils/utils.py: ## @@ -78,11 +78,38 @@ def get_job_name(task: TaskInstance) -> str: def get_custom_facets(task_instance: T

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

anandhimurali commented on code in PR #38982: URL: https://github.com/apache/airflow/pull/38982#discussion_r1666331170 ## airflow/providers/openlineage/utils/utils.py: ## @@ -78,11 +78,38 @@ def get_job_name(task: TaskInstance) -> str: def get_custom_facets(task_instance: T

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

anandhimurali commented on PR #38982: URL: https://github.com/apache/airflow/pull/38982#issuecomment-2210305119 > FYI. Sometimes --clean-build flag helps. Thanks @potiuk, `--clean-build` was helpful to overcome those documentation build errors. -- This is an automated message from

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

anandhimurali commented on code in PR #38982: URL: https://github.com/apache/airflow/pull/38982#discussion_r1666331170 ## airflow/providers/openlineage/utils/utils.py: ## @@ -78,11 +78,38 @@ def get_job_name(task: TaskInstance) -> str: def get_custom_facets(task_instance: T

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

anandhimurali commented on code in PR #38982: URL: https://github.com/apache/airflow/pull/38982#discussion_r1666320760 ## airflow/providers/openlineage/utils/utils.py: ## @@ -78,11 +78,38 @@ def get_job_name(task: TaskInstance) -> str: def get_custom_facets(task_instance: T

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

anandhimurali commented on code in PR #38982: URL: https://github.com/apache/airflow/pull/38982#discussion_r1666303994 ## airflow/providers/openlineage/utils/utils.py: ## @@ -78,11 +78,38 @@ def get_job_name(task: TaskInstance) -> str: def get_custom_facets(task_instance: T

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

anandhimurali commented on code in PR #38982: URL: https://github.com/apache/airflow/pull/38982#discussion_r1666300729 ## docs/apache-airflow-providers-openlineage/guides/developer.rst: ## @@ -446,15 +446,96 @@ Conversion from Airflow Table entity to OpenLineage Dataset is made

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

anandhimurali commented on code in PR #38982: URL: https://github.com/apache/airflow/pull/38982#discussion_r1666300729 ## docs/apache-airflow-providers-openlineage/guides/developer.rst: ## @@ -446,15 +446,96 @@ Conversion from Airflow Table entity to OpenLineage Dataset is made

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

anandhimurali commented on code in PR #38982: URL: https://github.com/apache/airflow/pull/38982#discussion_r1666299656 ## airflow/providers/openlineage/provider.yaml: ## @@ -104,6 +104,13 @@ config: example: full.path.to.ExtractorClass;full.path.to.AnotherExtractorClass

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

kacpermuda commented on code in PR #38982: URL: https://github.com/apache/airflow/pull/38982#discussion_r1665780045 ## airflow/providers/openlineage/provider.yaml: ## @@ -104,6 +104,13 @@ config: example: full.path.to.ExtractorClass;full.path.to.AnotherExtractorClass

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

potiuk commented on PR #38982: URL: https://github.com/apache/airflow/pull/38982#issuecomment-2208558429 > Getting some documentation error on `breeze build-docs openlineage`. Trying to debug and fix this. > > https://private-user-images.githubusercontent.com/68486238/345697063-3a27cd

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

anandhimurali commented on PR #38982: URL: https://github.com/apache/airflow/pull/38982#issuecomment-2208102010 Getting some documentation error on `breeze build-docs openlineage`. Trying to debug and fix this. https://github.com/apache/airflow/assets/68486238/3a27cddb-334e-441f-a63e-

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

anandhimurali commented on PR #38982: URL: https://github.com/apache/airflow/pull/38982#issuecomment-2208101080 >I think it would be good to rename this functionality to custom_run_facets, we might need to inject other facets too, specifically Job.facets. Addressed in 3a11b7625c896252

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

ahidalgob commented on PR #38982: URL: https://github.com/apache/airflow/pull/38982#issuecomment-2205728582 I think it would be good to rename this functionality to `custom_run_facets`, I think we might need to inject other facets too, specifically `Job.facets`. -- This is an automated m

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

mobuchowski commented on PR #38982: URL: https://github.com/apache/airflow/pull/38982#issuecomment-2200253493 @anandhimurali any interest in getting this PR into shape and approved? I think this is good, really useful feature. -- This is an automated message from the Apache Git Service. T

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

kacpermuda commented on code in PR #38982: URL: https://github.com/apache/airflow/pull/38982#discussion_r1607990291 ## airflow/providers/openlineage/utils/utils.py: ## @@ -61,11 +61,31 @@ def get_job_name(task: TaskInstance) -> str: def get_custom_facets(task_instance: Task

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

anandhimurali commented on PR #38982: URL: https://github.com/apache/airflow/pull/38982#issuecomment-2114094133 > I have two points I think it's worth addressing: > 1. Whether we pass all the necessary info. I think given task_instance we can call get_template_context and get all the data

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

anandhimurali commented on code in PR #38982: URL: https://github.com/apache/airflow/pull/38982#discussion_r1602650978 ## airflow/providers/openlineage/utils/utils.py: ## @@ -61,11 +61,20 @@ def get_job_name(task: TaskInstance) -> str: def get_custom_facets(task_instance: T

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

anandhimurali commented on code in PR #38982: URL: https://github.com/apache/airflow/pull/38982#discussion_r1602602899 ## airflow/providers/openlineage/conf.py: ## @@ -65,6 +65,19 @@ def custom_extractors() -> set[str]: return set(extractor.strip() for extractor in option.s

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

anandhimurali commented on code in PR #38982: URL: https://github.com/apache/airflow/pull/38982#discussion_r1602602899 ## airflow/providers/openlineage/conf.py: ## @@ -65,6 +65,19 @@ def custom_extractors() -> set[str]: return set(extractor.strip() for extractor in option.s

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

anandhimurali commented on code in PR #38982: URL: https://github.com/apache/airflow/pull/38982#discussion_r1602593146 ## airflow/providers/openlineage/conf.py: ## @@ -65,6 +65,19 @@ def custom_extractors() -> set[str]: return set(extractor.strip() for extractor in option.s

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

anandhimurali commented on code in PR #38982: URL: https://github.com/apache/airflow/pull/38982#discussion_r1602589624 ## airflow/providers/openlineage/utils/utils.py: ## @@ -61,11 +61,20 @@ def get_job_name(task: TaskInstance) -> str: def get_custom_facets(task_instance: T

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

anandhimurali commented on code in PR #38982: URL: https://github.com/apache/airflow/pull/38982#discussion_r1602585312 ## airflow/providers/openlineage/utils/utils.py: ## @@ -61,11 +61,20 @@ def get_job_name(task: TaskInstance) -> str: def get_custom_facets(task_instance: T

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

anandhimurali commented on code in PR #38982: URL: https://github.com/apache/airflow/pull/38982#discussion_r1602580045 ## airflow/providers/openlineage/utils/utils.py: ## @@ -61,11 +61,20 @@ def get_job_name(task: TaskInstance) -> str: def get_custom_facets(task_instance: T

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

anandhimurali commented on PR #38982: URL: https://github.com/apache/airflow/pull/38982#issuecomment-2110598999 >Hey @anandhimurali @XD-DENG, is there anything i could help you with here? I really like the idea and would like to see it happen, let me know 😄 Hi @kacpermuda, Thanks for

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

kacpermuda commented on PR #38982: URL: https://github.com/apache/airflow/pull/38982#issuecomment-2110380257 Hey @anandhimurali @XD-DENG, is there anything i could help you with here? I really like the idea and would like to see it happen, let me know 😄 -- This is an automated message fr

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

mobuchowski commented on code in PR #38982: URL: https://github.com/apache/airflow/pull/38982#discussion_r1592624751 ## airflow/providers/openlineage/conf.py: ## @@ -65,6 +65,19 @@ def custom_extractors() -> set[str]: return set(extractor.strip() for extractor in option.spl

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

kacpermuda commented on code in PR #38982: URL: https://github.com/apache/airflow/pull/38982#discussion_r1579417082 ## airflow/providers/openlineage/utils/utils.py: ## @@ -61,11 +61,20 @@ def get_job_name(task: TaskInstance) -> str: def get_custom_facets(task_instance: Task

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

mobuchowski commented on PR #38982: URL: https://github.com/apache/airflow/pull/38982#issuecomment-2076821285 Hey @anandhimurali @XD-DENG I think ability to add custom facets to arbitrary operators is a great idea. I have two points I think it's worth addressing: 1. Whether we

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

eladkal commented on PR #38982: URL: https://github.com/apache/airflow/pull/38982#issuecomment-2065139485 Ping @mobuchowski for review :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

XD-DENG commented on PR #38982: URL: https://github.com/apache/airflow/pull/38982#issuecomment-2064794678 Hi @eladkal , @anandhimurali is my colleague from the same team and I'm really excited about her 1st PR to Airflow! Do you have an idea who among the committers may be the best to

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

eladkal commented on code in PR #38982: URL: https://github.com/apache/airflow/pull/38982#discussion_r1564573821 ## airflow/providers/openlineage/provider.yaml: ## @@ -101,6 +101,15 @@ config: example: full.path.to.ExtractorClass;full.path.to.AnotherExtractorClass

[PR] Ability to add custom facet in OpenLineage events [airflow]

anandhimurali opened a new pull request, #38982: URL: https://github.com/apache/airflow/pull/38982 **Motivation** Currently, users need to write custom extractor inorder to add custom facets in the lineage events. Managing the entire extractor will be hard while users might just want to

Re: [PR] Ability to add custom facet in OpenLineage events [airflow]

boring-cyborg[bot] commented on PR #38982: URL: https://github.com/apache/airflow/pull/38982#issuecomment-2052695209 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors'