Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2024-01-14 Thread via GitHub
potiuk commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1891082583 > I understood that. The high level pattern is adding developer churn to the process of review, regardless of the specific details. I believe it adds more to the process of reviewe

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2024-01-14 Thread via GitHub
vsoch commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1891074974 I understood that. The high level pattern is adding developer churn to the process of review, regardless of the specific details. -- This is an automated message from the Apache Git Ser

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2024-01-14 Thread via GitHub
potiuk commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1891065228 @vsoch :) -> yep I know what you talk about, but I think in this case there is reverse cause <> effect. What we are discussing here (ned to resolve comments to merge PR) is just

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2024-01-13 Thread via GitHub
vsoch commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1890824882 Going to agree with you @bolkedebruin. I had similar issues about 5 years ago, and while I think the intention is best, when it results in a PR being open on the order of years and not a

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2024-01-13 Thread via GitHub
potiuk commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1890484550 I simply thinnk that individual "little" time on reviewing the comments and resolving them is very little price to pay for "project" positive effect of making sure we have not forgotten

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2024-01-13 Thread via GitHub
bolkedebruin commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1890482790 I'm of the opposite opinion here @potiuk . I understand why you would like to have it, I do not support it though as I think it a administrative waste tbh. Trust the authors and co

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2024-01-13 Thread via GitHub
potiuk commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1890392275 > Side note @potiuk: this is where I do not like "unresolved conversations". They are hiding in outdated changes :/ @bolkedebruin - you just gave the most important reason why enab

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2024-01-12 Thread via GitHub
potiuk commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1889563795 :tada: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscr

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2024-01-12 Thread via GitHub
hamedhsn commented on code in PR #22253: URL: https://github.com/apache/airflow/pull/22253#discussion_r1450609588 ## docs/apache-airflow-providers-google/operators/cloud/kubernetes_engine.rst: ## @@ -117,7 +117,7 @@ There are two operators available in order to run a pod on a G

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2024-01-12 Thread via GitHub
hamedhsn commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1889539241 @bolkedebruin Thanks for the review, glad it finally merged 😌 🎉 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2024-01-12 Thread via GitHub
hamedhsn commented on code in PR #22253: URL: https://github.com/apache/airflow/pull/22253#discussion_r1450609990 ## airflow/providers/cncf/kubernetes/operators/pod.py: ## @@ -95,7 +95,7 @@ class KubernetesPodOperator(BaseOperator): .. seealso:: For more informat

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2024-01-12 Thread via GitHub
hamedhsn commented on code in PR #22253: URL: https://github.com/apache/airflow/pull/22253#discussion_r1450609588 ## docs/apache-airflow-providers-google/operators/cloud/kubernetes_engine.rst: ## @@ -117,7 +117,7 @@ There are two operators available in order to run a pod on a G

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2024-01-12 Thread via GitHub
bolkedebruin merged PR #22253: URL: https://github.com/apache/airflow/pull/22253 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@air

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2024-01-12 Thread via GitHub
bolkedebruin commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1889199986 Side note @potiuk: this is where I do not like "unresolved conversations". They are hiding in outdated changes :/ -- This is an automated message from the Apache Git Service. To

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2024-01-12 Thread via GitHub
bolkedebruin commented on code in PR #22253: URL: https://github.com/apache/airflow/pull/22253#discussion_r1444501416 ## docs/apache-airflow-providers-google/operators/cloud/kubernetes_engine.rst: ## @@ -117,7 +117,7 @@ There are two operators available in order to run a pod on

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2024-01-12 Thread via GitHub
bolkedebruin commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1889195609 Jihaa! Thank you for your patience and persistence @hamedhsn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2024-01-12 Thread via GitHub
hamedhsn commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1888911881 @bolkedebruin sounds like all tests are green now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2024-01-11 Thread via GitHub
hamedhsn commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1887703064 @bolkedebruin can you approve the build pls? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2024-01-10 Thread via GitHub
bolkedebruin commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1884432760 Almost there @hamedhsn ! ``` airflow/providers/cncf/kubernetes/operators/custom_object_launcher.py:106: error: Incompatible default for argument "driver" (default has t

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2024-01-09 Thread via GitHub
dstandish commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1884156435 well, docs are green, just have a few other issues to resolve, it seems 👏 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2024-01-08 Thread via GitHub
bolkedebruin commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1880862580 @hamedhsn documentation generation is always a bit tough especially in the beginning and I speak from experience. I have added some comments. Make sure to check the error message c

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2024-01-08 Thread via GitHub
hamedhsn commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1880820881 > > If I could run that locally it would have been much faster to test. > > You can. > > See the output of the failed build It contains detailed instructions right after t

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2024-01-05 Thread via GitHub
potiuk commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1879072512 > If I could run that locally it would have been much faster to test. You can. See the output of the failed build It contains detailed instructions right after the error m

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2024-01-05 Thread via GitHub
hamedhsn commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1879067690 @bolkedebruin build fails while rendering doc which I am not sure how to fix. If I could run that locally it would have been much faster to test. -- This is an automated mes

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2024-01-05 Thread via GitHub
hamedhsn commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1878784704 @bolkedebruin I am not sure why it's failing. it complains about a JSON sample code in docs I added but I am not sure how to solve that. also, how can I run this locally? -- Thi

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2024-01-04 Thread via GitHub
bolkedebruin commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1876936019 @hamedhsn great that you have added the docs. If you can address the build issues, I'll do a final check and (hopefully) I can merge. -- This is an automated message from the Apa

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2024-01-03 Thread via GitHub
hamedhsn commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1875219366 @bolkedebruin I added a related doc for the operator to [operators.rst](https://github.com/apache/airflow/blob/main/docs/apache-airflow-providers-cncf-kubernetes/operators.rst) let m

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2024-01-02 Thread via GitHub
hamedhsn commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1874375370 > @bolkedebruin do you know why the build is failing? sounds like it resolved itself -- This is an automated message from the Apache Git Service. To respond to the message, pl

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2024-01-02 Thread via GitHub
hamedhsn commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1874249131 @bolkedebruin do you know why the build is failing? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2023-12-10 Thread via GitHub
bolkedebruin commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1848914425 No worries @hamedhsn , get better first! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2023-12-08 Thread via GitHub
hamedhsn commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1847371472 @bolkedebruin Sorry was sick for the last few days, I will finish the documentation soon. will ping you when it is ready -- This is an automated message from the Apache Git Servi

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2023-12-08 Thread via GitHub
bolkedebruin commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1846898571 ping @hamedhsn , would love to have this in after all the work you did. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2023-11-23 Thread via GitHub
bolkedebruin commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1824404326 > @bolkedebruin I will fix your above comment. What is the best place to put some usage examples, can you point me to the location? I'd like to see documentation here as a se

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2023-11-23 Thread via GitHub
hamedhsn commented on code in PR #22253: URL: https://github.com/apache/airflow/pull/22253#discussion_r1403317000 ## airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py: ## @@ -17,173 +17,271 @@ # under the License. from __future__ import annotations -import date

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2023-11-23 Thread via GitHub
hamedhsn commented on code in PR #22253: URL: https://github.com/apache/airflow/pull/22253#discussion_r1403314801 ## airflow/providers/cncf/kubernetes/operators/custom_object_launcher.py: ## @@ -0,0 +1,366 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or mo

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2023-11-23 Thread via GitHub
hamedhsn commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1824328879 @bolkedebruin I will fix your above comment. What is the best place to put some usage examples, can you point me to the location? -- This is an automated message from the Apache Git

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2023-11-23 Thread via GitHub
bolkedebruin commented on code in PR #22253: URL: https://github.com/apache/airflow/pull/22253#discussion_r1403221124 ## airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py: ## @@ -17,173 +17,271 @@ # under the License. from __future__ import annotations -import

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2023-11-23 Thread via GitHub
hamedhsn commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1824067974 @bolkedebruin build is green now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2023-11-22 Thread via GitHub
potiuk commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1822378364 I think you need to rebase and solve conflicts @hamedhsn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2023-11-20 Thread via GitHub
hamedhsn commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1818764539 > Are you on this @hamedhsn ? yeah sorry been busy last week, will sort it out this week. -- This is an automated message from the Apache Git Service. To respond to the messag

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2023-11-16 Thread via GitHub
bolkedebruin commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1815200659 Are you on this @hamedhsn ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2023-11-13 Thread via GitHub
hamedhsn commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1808085466 great! I will add the description :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2023-11-13 Thread via GitHub
bolkedebruin commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1807888797 ping @hamedhsn ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2023-11-06 Thread via GitHub
bolkedebruin commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1794576775 Oh and given the fact that this is outside the comfort area of both the kubernetes provider and spark provider, can you please add some usage documentation? Otherwise, I think it w

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2023-11-06 Thread via GitHub
bolkedebruin commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1794569307 I think this looks good. Can you ensure that tests pass and with a recent operator? Then I'll merge this. -- This is an automated message from the Apache Git Service. To respond

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2023-10-12 Thread via GitHub
hamedhsn commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1759455224 @hussein-awala when do you have time to review this? if you could give me a date I will make it ready to review -- This is an automated message from the Apache Git Service. To resp

Re: [PR] Add SparkKubernetesOperator crd implementation [airflow]

2023-10-11 Thread via GitHub
github-actions[bot] commented on PR #22253: URL: https://github.com/apache/airflow/pull/22253#issuecomment-1758720195 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for you