[GitHub] [spark] WeichenXu123 commented on a diff in pull request #39299: [WIP][SPARK-41593][PYTHON][ML] Adding logging from executors

WeichenXu123 commented on code in PR #39299: URL: https://github.com/apache/spark/pull/39299#discussion_r1082428873 ## python/pyspark/ml/torch/log_communication.py: ## @@ -0,0 +1,201 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor licen

[GitHub] [spark] WeichenXu123 commented on a diff in pull request #39299: [WIP][SPARK-41593][PYTHON][ML] Adding logging from executors

WeichenXu123 commented on code in PR #39299: URL: https://github.com/apache/spark/pull/39299#discussion_r1082420774 ## python/pyspark/ml/torch/log_communication.py: ## @@ -0,0 +1,201 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor licen

[GitHub] [spark] WeichenXu123 commented on a diff in pull request #39299: [WIP][SPARK-41593][PYTHON][ML] Adding logging from executors

WeichenXu123 commented on code in PR #39299: URL: https://github.com/apache/spark/pull/39299#discussion_r1082420394 ## python/pyspark/ml/torch/log_communication.py: ## @@ -0,0 +1,201 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor licen

[GitHub] [spark] WeichenXu123 commented on a diff in pull request #39299: [WIP][SPARK-41593][PYTHON][ML] Adding logging from executors

WeichenXu123 commented on code in PR #39299: URL: https://github.com/apache/spark/pull/39299#discussion_r1082416985 ## python/pyspark/ml/torch/log_communication.py: ## @@ -0,0 +1,201 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor licen

[GitHub] [spark] WeichenXu123 commented on a diff in pull request #39299: [WIP][SPARK-41593][PYTHON][ML] Adding logging from executors

WeichenXu123 commented on code in PR #39299: URL: https://github.com/apache/spark/pull/39299#discussion_r1082414835 ## python/pyspark/ml/torch/log_communication.py: ## @@ -0,0 +1,201 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor licen

[GitHub] [spark] WeichenXu123 commented on a diff in pull request #39299: [WIP][SPARK-41593][PYTHON][ML] Adding logging from executors

WeichenXu123 commented on code in PR #39299: URL: https://github.com/apache/spark/pull/39299#discussion_r1080811703 ## python/pyspark/ml/torch/log_communication.py: ## @@ -0,0 +1,201 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor licen

[GitHub] [spark] WeichenXu123 commented on a diff in pull request #39299: [WIP][SPARK-41593][PYTHON][ML] Adding logging from executors

WeichenXu123 commented on code in PR #39299: URL: https://github.com/apache/spark/pull/39299#discussion_r1080811334 ## python/pyspark/ml/torch/log_communication.py: ## @@ -0,0 +1,201 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor licen

[GitHub] [spark] WeichenXu123 commented on a diff in pull request #39299: [WIP][SPARK-41593][PYTHON][ML] Adding logging from executors

WeichenXu123 commented on code in PR #39299: URL: https://github.com/apache/spark/pull/39299#discussion_r1080808104 ## python/pyspark/ml/torch/log_communication.py: ## @@ -0,0 +1,201 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor licen

[GitHub] [spark] WeichenXu123 commented on a diff in pull request #39299: [WIP][SPARK-41593][PYTHON][ML] Adding logging from executors

WeichenXu123 commented on code in PR #39299: URL: https://github.com/apache/spark/pull/39299#discussion_r1080786715 ## python/pyspark/ml/torch/distributor.py: ## @@ -501,26 +537,42 @@ def _run_distributed_training( ) -> Optional[Any]: if not framework_wrapper_fn:

[GitHub] [spark] WeichenXu123 commented on a diff in pull request #39299: [WIP][SPARK-41593][PYTHON][ML] Adding logging from executors

WeichenXu123 commented on code in PR #39299: URL: https://github.com/apache/spark/pull/39299#discussion_r1080782505 ## python/pyspark/ml/torch/log_communication.py: ## @@ -0,0 +1,201 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor licen

[GitHub] [spark] WeichenXu123 commented on a diff in pull request #39299: [WIP][SPARK-41593][PYTHON][ML] Adding logging from executors

WeichenXu123 commented on code in PR #39299: URL: https://github.com/apache/spark/pull/39299#discussion_r1080781810 ## python/pyspark/ml/torch/log_communication.py: ## @@ -0,0 +1,201 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor licen

[GitHub] [spark] WeichenXu123 commented on a diff in pull request #39299: [WIP][SPARK-41593][PYTHON][ML] Adding logging from executors

WeichenXu123 commented on code in PR #39299: URL: https://github.com/apache/spark/pull/39299#discussion_r1080781706 ## python/pyspark/ml/torch/log_communication.py: ## @@ -0,0 +1,201 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor licen

[GitHub] [spark] WeichenXu123 commented on a diff in pull request #39299: [WIP][SPARK-41593][PYTHON][ML] Adding logging from executors

WeichenXu123 commented on code in PR #39299: URL: https://github.com/apache/spark/pull/39299#discussion_r1080780895 ## python/pyspark/ml/torch/log_communication.py: ## @@ -0,0 +1,201 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor licen

[GitHub] [spark] WeichenXu123 commented on a diff in pull request #39299: [WIP][SPARK-41593][PYTHON][ML] Adding logging from executors

WeichenXu123 commented on code in PR #39299: URL: https://github.com/apache/spark/pull/39299#discussion_r1080780209 ## python/pyspark/ml/torch/log_communication.py: ## @@ -0,0 +1,201 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor licen

[GitHub] [spark] WeichenXu123 commented on a diff in pull request #39299: [WIP][SPARK-41593][PYTHON][ML] Adding logging from executors

WeichenXu123 commented on code in PR #39299: URL: https://github.com/apache/spark/pull/39299#discussion_r1080780209 ## python/pyspark/ml/torch/log_communication.py: ## @@ -0,0 +1,201 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor licen

[GitHub] [spark] WeichenXu123 commented on a diff in pull request #39299: [WIP][SPARK-41593][PYTHON][ML] Adding logging from executors

WeichenXu123 commented on code in PR #39299: URL: https://github.com/apache/spark/pull/39299#discussion_r1080779250 ## python/pyspark/ml/torch/log_communication.py: ## @@ -0,0 +1,201 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor licen

[GitHub] [spark] WeichenXu123 commented on a diff in pull request #39299: [WIP][SPARK-41593][PYTHON][ML] Adding logging from executors

WeichenXu123 commented on code in PR #39299: URL: https://github.com/apache/spark/pull/39299#discussion_r1080778092 ## python/pyspark/ml/torch/distributor.py: ## @@ -72,6 +77,19 @@ def get_conf_boolean(sc: SparkContext, key: str, default_value: str) -> bool: ) +def get