[jira] [Updated] (SPARK-44264) DeepSpeed Distrobutor

[ https://issues.apache.org/jira/browse/SPARK-44264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rithwik Ediga Lakhamsani updated SPARK-44264: - Attachment: Trying to Run Deepspeed Funcs.html > DeepSpeed Distrobutor

[jira] [Resolved] (SPARK-42103) Add Instrumentation

[ https://issues.apache.org/jira/browse/SPARK-42103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rithwik Ediga Lakhamsani resolved SPARK-42103. -- Resolution: Not A Problem > Add Instrumentation > ---

[jira] [Resolved] (SPARK-41590) Implement Baseline API Code

[ https://issues.apache.org/jira/browse/SPARK-41590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rithwik Ediga Lakhamsani resolved SPARK-41590. -- Resolution: Fixed > Implement Baseline API Code >

[jira] [Updated] (SPARK-41916) Address General Fixes

[ https://issues.apache.org/jira/browse/SPARK-41916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rithwik Ediga Lakhamsani updated SPARK-41916: - Description: We want the distributor to have the ability to run

[jira] [Updated] (SPARK-41916) Address General Fizes

[ https://issues.apache.org/jira/browse/SPARK-41916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rithwik Ediga Lakhamsani updated SPARK-41916: - Summary: Address General Fizes (was: Address

[jira] [Resolved] (SPARK-41776) Implement support for PyTorch Lightning

[ https://issues.apache.org/jira/browse/SPARK-41776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rithwik Ediga Lakhamsani resolved SPARK-41776. -- Resolution: Fixed Not needed, since we are now using

[jira] [Updated] (SPARK-41916) Address `spark.task.resource.gpu.amount > 1`

[ https://issues.apache.org/jira/browse/SPARK-41916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rithwik Ediga Lakhamsani updated SPARK-41916: - Description: We want the distributor to have the ability to run multiple

[jira] [Updated] (SPARK-41916) Address General Fixes

[ https://issues.apache.org/jira/browse/SPARK-41916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rithwik Ediga Lakhamsani updated SPARK-41916: - Summary: Address General Fixes (was: Address General Fizes) > Address

[jira] [Updated] (SPARK-41916) Address `spark.task.resource.gpu.amount > 1`

[ https://issues.apache.org/jira/browse/SPARK-41916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rithwik Ediga Lakhamsani updated SPARK-41916: - Description: We want the distributor to have the ability to run multiple

[jira] [Updated] (SPARK-41776) Implement support for PyTorch Lightning

[ https://issues.apache.org/jira/browse/SPARK-41776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rithwik Ediga Lakhamsani updated SPARK-41776: - Description: This requires us to just call train() on each spark task

[jira] [Updated] (SPARK-41776) Implement support for PyTorch Lightning

[ https://issues.apache.org/jira/browse/SPARK-41776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rithwik Ediga Lakhamsani updated SPARK-41776: - Description: This requires us to just call train() on each spark task

[jira] [Resolved] (SPARK-41915) Change API so that the user doesn't have to explicitly set pytorch-lightning

[ https://issues.apache.org/jira/browse/SPARK-41915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rithwik Ediga Lakhamsani resolved SPARK-41915. -- Resolution: Fixed This is already resolved within

[jira] [Created] (SPARK-42103) Add Instrumentation

Rithwik Ediga Lakhamsani created SPARK-42103: Summary: Add Instrumentation Key: SPARK-42103 URL: https://issues.apache.org/jira/browse/SPARK-42103 Project: Spark Issue Type:

[jira] [Updated] (SPARK-41775) Implement training functions as input

[ https://issues.apache.org/jira/browse/SPARK-41775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rithwik Ediga Lakhamsani updated SPARK-41775: - Description: Sidenote: make formatting updates described in

[jira] [Created] (SPARK-41916) Address `spark.task.resource.gpu.amount > 1`

Rithwik Ediga Lakhamsani created SPARK-41916: Summary: Address `spark.task.resource.gpu.amount > 1` Key: SPARK-41916 URL: https://issues.apache.org/jira/browse/SPARK-41916 Project: Spark

[jira] [Created] (SPARK-41915) Change API so that the user doesn't have to explicitly set pytorch-lightning

Rithwik Ediga Lakhamsani created SPARK-41915: Summary: Change API so that the user doesn't have to explicitly set pytorch-lightning Key: SPARK-41915 URL:

[jira] [Updated] (SPARK-41775) Implement training functions as input

[ https://issues.apache.org/jira/browse/SPARK-41775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rithwik Ediga Lakhamsani updated SPARK-41775: - Description: Currently, `Distributor().run(...)` takes only files as

[jira] [Updated] (SPARK-41775) Implement training functions as input

[ https://issues.apache.org/jira/browse/SPARK-41775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rithwik Ediga Lakhamsani updated SPARK-41775: - Description: Currently, `Distributor().run(...)` takes only files as

[jira] [Updated] (SPARK-41775) Implement training functions as input

[ https://issues.apache.org/jira/browse/SPARK-41775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rithwik Ediga Lakhamsani updated SPARK-41775: - Description: Currently, `Distributor().run(...)` takes only files as

[jira] [Updated] (SPARK-41775) Implement training functions as input

[ https://issues.apache.org/jira/browse/SPARK-41775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rithwik Ediga Lakhamsani updated SPARK-41775: - Description: Currently, `Distributor().run(...)` takes only files as

[jira] [Created] (SPARK-41777) Add Integration Tests

Rithwik Ediga Lakhamsani created SPARK-41777: Summary: Add Integration Tests Key: SPARK-41777 URL: https://issues.apache.org/jira/browse/SPARK-41777 Project: Spark Issue Type:

[jira] [Created] (SPARK-41776) Implement support for PyTorch Lightning

Rithwik Ediga Lakhamsani created SPARK-41776: Summary: Implement support for PyTorch Lightning Key: SPARK-41776 URL: https://issues.apache.org/jira/browse/SPARK-41776 Project: Spark

[jira] [Updated] (SPARK-41775) Implement training functions as input

[ https://issues.apache.org/jira/browse/SPARK-41775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rithwik Ediga Lakhamsani updated SPARK-41775: - Component/s: ML > Implement training functions as input >

[jira] [Created] (SPARK-41775) Implement training functions as input

Rithwik Ediga Lakhamsani created SPARK-41775: Summary: Implement training functions as input Key: SPARK-41775 URL: https://issues.apache.org/jira/browse/SPARK-41775 Project: Spark

[jira] [Updated] (SPARK-41589) PyTorch Distributor

[ https://issues.apache.org/jira/browse/SPARK-41589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rithwik Ediga Lakhamsani updated SPARK-41589: - Description: This is a project to make it easier for PySpark users to

[jira] [Commented] (SPARK-41589) PyTorch Distributor

[ https://issues.apache.org/jira/browse/SPARK-41589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17649526#comment-17649526 ] Rithwik Ediga Lakhamsani commented on SPARK-41589: -- [~xkrogen] I created a new copy,

[jira] [Updated] (SPARK-41589) PyTorch Distributor

[ https://issues.apache.org/jira/browse/SPARK-41589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rithwik Ediga Lakhamsani updated SPARK-41589: - Description: This is a project to make it easier for PySpark users to

[jira] [Commented] (SPARK-41589) PyTorch Distributor

[ https://issues.apache.org/jira/browse/SPARK-41589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17649516#comment-17649516 ] Rithwik Ediga Lakhamsani commented on SPARK-41589: -- Sorry, I need update it with a new

[jira] [Comment Edited] (SPARK-41589) PyTorch Distributor

[ https://issues.apache.org/jira/browse/SPARK-41589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17649511#comment-17649511 ] Rithwik Ediga Lakhamsani edited comment on SPARK-41589 at 12/20/22 12:27 AM:

[jira] [Updated] (SPARK-41589) PyTorch Distributor

[ https://issues.apache.org/jira/browse/SPARK-41589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rithwik Ediga Lakhamsani updated SPARK-41589: - Description: This is a project to make it easier for PySpark users to

[jira] [Commented] (SPARK-41589) PyTorch Distributor

[ https://issues.apache.org/jira/browse/SPARK-41589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17649511#comment-17649511 ] Rithwik Ediga Lakhamsani commented on SPARK-41589: -- Oh sorry, let me fix that!  >

[jira] [Created] (SPARK-41592) Implement functionality for training a PyTorch file on the executors

Rithwik Ediga Lakhamsani created SPARK-41592: Summary: Implement functionality for training a PyTorch file on the executors Key: SPARK-41592 URL: https://issues.apache.org/jira/browse/SPARK-41592

[jira] [Created] (SPARK-41593) Implement logging from the executor nodes

Rithwik Ediga Lakhamsani created SPARK-41593: Summary: Implement logging from the executor nodes Key: SPARK-41593 URL: https://issues.apache.org/jira/browse/SPARK-41593 Project: Spark

[jira] [Created] (SPARK-41591) Implement functionality for training a PyTorch file locally

Rithwik Ediga Lakhamsani created SPARK-41591: Summary: Implement functionality for training a PyTorch file locally Key: SPARK-41591 URL: https://issues.apache.org/jira/browse/SPARK-41591

[jira] [Commented] (SPARK-41589) PyTorch Distributor

[ https://issues.apache.org/jira/browse/SPARK-41589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17649509#comment-17649509 ] Rithwik Ediga Lakhamsani commented on SPARK-41589: -- I am working on this. > PyTorch

[jira] [Updated] (SPARK-41589) PyTorch Distributor

[ https://issues.apache.org/jira/browse/SPARK-41589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rithwik Ediga Lakhamsani updated SPARK-41589: - Description: This is a project to make it easier for PySpark users to

[jira] [Created] (SPARK-41590) Implement Baseline API Code

Rithwik Ediga Lakhamsani created SPARK-41590: Summary: Implement Baseline API Code Key: SPARK-41590 URL: https://issues.apache.org/jira/browse/SPARK-41590 Project: Spark Issue

[jira] [Created] (SPARK-41589) PyTorch Distributor

Rithwik Ediga Lakhamsani created SPARK-41589: Summary: PyTorch Distributor Key: SPARK-41589 URL: https://issues.apache.org/jira/browse/SPARK-41589 Project: Spark Issue Type: