min-guk commented on PR #2126:
URL: https://github.com/apache/systemds/pull/2126#issuecomment-2380632190
Here are my implementation intentions, current implementation, issues, and
questions.
### 1. Original Implementation Intent
I aimed to implement the **Federated Roll Function** as follows:
1. **Coordinator Role**:
- Shift the `begindims` and `enddims` of the `fedworker` matrices stored
in `fedmap` by a specified `shift` value.
2. **Matrix Splitting Due to Shift**:
- If the `shift` causes a specific `fedworker` matrix to be split into
two, dividing the entire matrix between the end and the beginning:
- **Existing Element**: Modify `fedrange` to point to the end of the
entire matrix, keeping `feddata` the same (data from a non-split `fedworker`).
- **Add New Element**: Add a new entry with `fedrange` pointing to the
beginning of the entire matrix, keeping `feddata` the same (data from a
non-split `fedworker`).
3. **Handling Split `fedworkers`**:
- Send an `EXEC_INST(rightindex)` `fedrequest` only to the `fedworkers`
that are being split, to divide and store their `feddata`.
### 2. Current Implementation Method
Due to uncertainties in implementing step **(3)** as intended, I had to
deviate and implement it as follows:
3. **Sending `EXEC_UDP(SplitRow)` `fedrequest` to All `fedworkers`**:
- **Non-split `fedworkers`**: Set the existing matrix as `output`.
- **Split `fedworkers`**: Perform two `fedrequests`:
1. Slice the existing matrix to become the initial part of the entire
matrix and set it as `output`.
2. Slice the existing matrix to become the final part of the entire
matrix and set it as `output`.
4. **Collecting Results**:
- Retrieve the results using `MatrixObject out =
ec.getMatrixObject(output);`.
### 3. Problems and Questions
#### **Problem 1: Data Interference Issues in Split `fedworkers`**
- **Situation**:
- Split `fedworkers` perform two `fedrequests` through a `fedmap` that
points to the same `data` but with different `ranges`.
- The results are saved using
`ec.setMatrixOutput(String.valueOf(_outputID), resBlock)`, resulting in only
one of the execution results being stored.
- **Question**:
- Is there a way to access the same `feddata` during `fedrequest`
execution and save it as different `feddata`?
#### **Problem 2: Avoiding `fedrequest` Communication with Non-split
`fedworkers`**
- **Situation**:
- Currently, results are aggregated using `MatrixObject out =
ec.getMatrixObject(output);`, which requires communicating with non-split
`fedworkers` to set their existing matrices as `output`.
- **Question**:
- Is there a method to set the `input` as `output` within the coordinator
without communicating with specific `fedworkers`?
#### **Problem 3: Using `EXEC_UDP` Instead of `EXEC_INST`**
- **Situation**:
- The federated `rightindex` slices the entire matrix rather than a
specific `fedworker`'s matrix.
- Therefore, the current approach of splitting a specific `fedworker`'s
matrix into two different `feddata` entries does not seem compatible.
- **Questions**:
- Can federated `rightindex` be applied to the current situation?
- Are there potential issues or significant reductions in code reliability
when using `EXEC_UDP`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
