Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-04-01 Thread via GitHub
HyukjinKwon closed pull request #45232: [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile URL: https://github.com/apache/spark/pull/45232 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-04-01 Thread via GitHub
HyukjinKwon commented on PR #45232: URL: https://github.com/apache/spark/pull/45232#issuecomment-2030829152 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-04-01 Thread via GitHub
wbo4958 commented on PR #45232: URL: https://github.com/apache/spark/pull/45232#issuecomment-2030792790 Hi @HyukjinKwon, Could you help merge it? Thx -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-04-01 Thread via GitHub
wbo4958 commented on PR #45232: URL: https://github.com/apache/spark/pull/45232#issuecomment-2030753074 > Please make sure that the follow work does not get lost. Sure, I will get it done. -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-04-01 Thread via GitHub
wbo4958 commented on PR #45232: URL: https://github.com/apache/spark/pull/45232#issuecomment-2029532177 > In contrast to the regular spark API this implementation doesn't manage the lifecycle of of the resource request. Can you create a follow up Jira that removes the resource request from

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-31 Thread via GitHub
wbo4958 commented on code in PR #45232: URL: https://github.com/apache/spark/pull/45232#discussion_r1545872296 ## connector/connect/common/src/main/protobuf/spark/connect/base.proto: ## @@ -375,6 +375,9 @@ message ExecutePlanResponse { // Response type informing if the stre

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-26 Thread via GitHub
grundprinzip commented on code in PR #45232: URL: https://github.com/apache/spark/pull/45232#discussion_r1540499756 ## connector/connect/common/src/main/protobuf/spark/connect/base.proto: ## @@ -375,6 +375,9 @@ message ExecutePlanResponse { // Response type informing if the

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-20 Thread via GitHub
wbo4958 commented on PR #45232: URL: https://github.com/apache/spark/pull/45232#issuecomment-2009170881 Hi @grundprinzip, @HyukjinKwon, @zhengruifeng, This PR has been there for a while, could you help review/merge it? Thx -- This is an automated message from the Apache Git Service. To re

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-13 Thread via GitHub
wbo4958 commented on PR #45232: URL: https://github.com/apache/spark/pull/45232#issuecomment-1996052736 Hi @grundprinzip, I would be grateful if you could kindly take another look at this PR, Thx. -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-11 Thread via GitHub
wbo4958 commented on PR #45232: URL: https://github.com/apache/spark/pull/45232#issuecomment-1989639850 Hi @grundprinzip, Could you help review it again? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-11 Thread via GitHub
wbo4958 commented on code in PR #45232: URL: https://github.com/apache/spark/pull/45232#discussion_r1519241310 ## connector/connect/common/src/main/protobuf/spark/connect/relations.proto: ## @@ -892,6 +893,9 @@ message MapPartitions { // (Optional) Whether to use barrier mo

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-11 Thread via GitHub
wbo4958 commented on code in PR #45232: URL: https://github.com/apache/spark/pull/45232#discussion_r1519241310 ## connector/connect/common/src/main/protobuf/spark/connect/relations.proto: ## @@ -892,6 +893,9 @@ message MapPartitions { // (Optional) Whether to use barrier mo

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-11 Thread via GitHub
wbo4958 commented on code in PR #45232: URL: https://github.com/apache/spark/pull/45232#discussion_r1519235937 ## connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectBuildResourceProfileHandler.scala: ## @@ -0,0 +1,75 @@ +/* + * Licensed to t

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-11 Thread via GitHub
wbo4958 commented on code in PR #45232: URL: https://github.com/apache/spark/pull/45232#discussion_r1519232917 ## connector/connect/common/src/main/protobuf/spark/connect/base.proto: ## @@ -1011,5 +1039,7 @@ service SparkConnectService { // FetchErrorDetails retrieves the m

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-11 Thread via GitHub
wbo4958 commented on code in PR #45232: URL: https://github.com/apache/spark/pull/45232#discussion_r1519232536 ## connector/connect/common/src/main/protobuf/spark/connect/relations.proto: ## @@ -892,6 +893,9 @@ message MapPartitions { // (Optional) Whether to use barrier mo

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-08 Thread via GitHub
grundprinzip commented on code in PR #45232: URL: https://github.com/apache/spark/pull/45232#discussion_r1518491571 ## connector/connect/common/src/main/protobuf/spark/connect/relations.proto: ## @@ -892,6 +893,9 @@ message MapPartitions { // (Optional) Whether to use barri

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-08 Thread via GitHub
tgravescs commented on PR #45232: URL: https://github.com/apache/spark/pull/45232#issuecomment-1985925035 changes look fine to me -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-07 Thread via GitHub
wbo4958 commented on code in PR #45232: URL: https://github.com/apache/spark/pull/45232#discussion_r1517085186 ## python/pyspark/resource/tests/test_connect_resources.py: ## @@ -0,0 +1,46 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-07 Thread via GitHub
wbo4958 commented on code in PR #45232: URL: https://github.com/apache/spark/pull/45232#discussion_r1517085041 ## python/pyspark/resource/tests/test_connect_resources.py: ## @@ -0,0 +1,46 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-07 Thread via GitHub
wbo4958 commented on code in PR #45232: URL: https://github.com/apache/spark/pull/45232#discussion_r1517084819 ## python/pyspark/resource/profile.py: ## @@ -114,14 +122,23 @@ def id(self) -> int: int A unique id of this :class:`ResourceProfile` """

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-07 Thread via GitHub
wbo4958 commented on code in PR #45232: URL: https://github.com/apache/spark/pull/45232#discussion_r1517084721 ## python/pyspark/resource/profile.py: ## @@ -114,14 +122,23 @@ def id(self) -> int: int A unique id of this :class:`ResourceProfile` """

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-07 Thread via GitHub
tgravescs commented on code in PR #45232: URL: https://github.com/apache/spark/pull/45232#discussion_r1516375405 ## python/pyspark/resource/tests/test_connect_resources.py: ## @@ -0,0 +1,46 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributo

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-06 Thread via GitHub
wbo4958 commented on PR #45232: URL: https://github.com/apache/spark/pull/45232#issuecomment-1982322729 > > Does this PR introduce any user-facing change? > > Yes, Users can pass ResourceProfile to mapInPandas/mapInArrow through the connect pysprark client. > > I think you are addi

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-06 Thread via GitHub
wbo4958 commented on code in PR #45232: URL: https://github.com/apache/spark/pull/45232#discussion_r1515490084 ## dev/sparktestsupport/modules.py: ## @@ -554,6 +554,7 @@ def __hash__(self): "pyspark.resource.profile", # unittests "pyspark.resource.test

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-06 Thread via GitHub
wbo4958 commented on code in PR #45232: URL: https://github.com/apache/spark/pull/45232#discussion_r1515485716 ## python/pyspark/sql/connect/resource/profile.py: ## @@ -0,0 +1,69 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license a

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-06 Thread via GitHub
zhengruifeng commented on code in PR #45232: URL: https://github.com/apache/spark/pull/45232#discussion_r1515455169 ## python/pyspark/sql/connect/resource/profile.py: ## @@ -0,0 +1,69 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor lice

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-06 Thread via GitHub
zhengruifeng commented on code in PR #45232: URL: https://github.com/apache/spark/pull/45232#discussion_r1515448887 ## dev/sparktestsupport/modules.py: ## @@ -554,6 +554,7 @@ def __hash__(self): "pyspark.resource.profile", # unittests "pyspark.resource

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-06 Thread via GitHub
tgravescs commented on PR #45232: URL: https://github.com/apache/spark/pull/45232#issuecomment-1980935581 >Does this PR introduce any user-facing change? > Yes, Users can pass ResourceProfile to mapInPandas/mapInArrow through the connect pysprark client. I think you are adding the

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-06 Thread via GitHub
HyukjinKwon commented on code in PR #45232: URL: https://github.com/apache/spark/pull/45232#discussion_r1514004559 ## python/pyspark/resource/profile.py: ## @@ -114,14 +122,26 @@ def id(self) -> int: int A unique id of this :class:`ResourceProfile`

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-06 Thread via GitHub
wbo4958 commented on PR #45232: URL: https://github.com/apache/spark/pull/45232#issuecomment-1980292725 Hi @HyukjinKwon, Could you help review again, thx very much. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-06 Thread via GitHub
wbo4958 commented on code in PR #45232: URL: https://github.com/apache/spark/pull/45232#discussion_r1513993480 ## python/pyspark/sql/connect/resource/profile.py: ## @@ -0,0 +1,69 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license a

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-06 Thread via GitHub
wbo4958 commented on code in PR #45232: URL: https://github.com/apache/spark/pull/45232#discussion_r1513991688 ## python/pyspark/resource/profile.py: ## @@ -114,14 +122,26 @@ def id(self) -> int: int A unique id of this :class:`ResourceProfile` """

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-05 Thread via GitHub
wbo4958 commented on code in PR #45232: URL: https://github.com/apache/spark/pull/45232#discussion_r1513987011 ## python/pyspark/resource/profile.py: ## @@ -114,14 +122,26 @@ def id(self) -> int: int A unique id of this :class:`ResourceProfile` """

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-04 Thread via GitHub
HyukjinKwon commented on PR #45232: URL: https://github.com/apache/spark/pull/45232#issuecomment-1976100589 Looks fine in general -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-04 Thread via GitHub
HyukjinKwon commented on code in PR #45232: URL: https://github.com/apache/spark/pull/45232#discussion_r1510840968 ## python/pyspark/resource/profile.py: ## @@ -114,14 +122,26 @@ def id(self) -> int: int A unique id of this :class:`ResourceProfile`

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-04 Thread via GitHub
HyukjinKwon commented on code in PR #45232: URL: https://github.com/apache/spark/pull/45232#discussion_r1510824227 ## python/pyspark/sql/connect/resource/profile.py: ## @@ -0,0 +1,69 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor licen

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-04 Thread via GitHub
HyukjinKwon commented on code in PR #45232: URL: https://github.com/apache/spark/pull/45232#discussion_r1510818206 ## python/pyspark/resource/profile.py: ## @@ -114,14 +122,26 @@ def id(self) -> int: int A unique id of this :class:`ResourceProfile`

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-03-04 Thread via GitHub
HyukjinKwon commented on code in PR #45232: URL: https://github.com/apache/spark/pull/45232#discussion_r1510814720 ## connector/connect/common/src/main/protobuf/spark/connect/base.proto: ## @@ -967,6 +967,34 @@ message FetchErrorDetailsResponse { } } +message BuildResource

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-02-28 Thread via GitHub
wbo4958 commented on PR #45232: URL: https://github.com/apache/spark/pull/45232#issuecomment-1970509831 Hi @tgravescs, This PR changed ResourceProfile a little bit to support connect, Could you help review it? Thx very much. -- This is an automated message from the Apache Git Service. To

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-02-25 Thread via GitHub
wbo4958 commented on PR #45232: URL: https://github.com/apache/spark/pull/45232#issuecomment-1963270400 Hi @tgravescs @WeichenXu123 @zhengruifeng @Ngone51, Could you also please help review it. Thx. -- This is an automated message from the Apache Git Service. To respond to the message, pl

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-02-25 Thread via GitHub
wbo4958 commented on code in PR #45232: URL: https://github.com/apache/spark/pull/45232#discussion_r1502014261 ## python/pyspark/resource/profile.py: ## @@ -99,6 +99,11 @@ def __init__( _exec_req: Optional[Dict[str, ExecutorResourceRequest]] = None, Review Comment:

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-02-25 Thread via GitHub
wbo4958 commented on PR #45232: URL: https://github.com/apache/spark/pull/45232#issuecomment-1963263080 ## With dynamic allocation enabled. ``` bash start-connect-server.sh --master spark://192.168.0.106:7077 \ --jars jars/spark-connect_2.13-4.0.0-SNAPSHOT.jar \ --co

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-02-25 Thread via GitHub
wbo4958 commented on PR #45232: URL: https://github.com/apache/spark/pull/45232#issuecomment-1963261901 # Manual tests The manual tests were conducted on a spark Standalone cluster with only 1 worker which has 6 cpu cores. ## With dynamic allocation disabled. ``` bas

Re: [PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-02-25 Thread via GitHub
HyukjinKwon commented on code in PR #45232: URL: https://github.com/apache/spark/pull/45232#discussion_r1501943441 ## python/pyspark/resource/profile.py: ## @@ -99,6 +99,11 @@ def __init__( _exec_req: Optional[Dict[str, ExecutorResourceRequest]] = None, Review Comment:

[PR] [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile [spark]

2024-02-23 Thread via GitHub
wbo4958 opened a new pull request, #45232: URL: https://github.com/apache/spark/pull/45232 ### What changes were proposed in this pull request? Support stage-level scheduling for PySpark connect DataFrame APIs (mapInPandas and mapInArrow). ### Why are the changes needed?