Baunsgaard commented on PR #1665: URL: https://github.com/apache/systemds/pull/1665#issuecomment-1225824243
> There is a minor problem I don't know how to best fix. I need to select a path at the sites for the workers to write their partition, I am currently choosing to create a (most likely) unique filename and write into the LOCAL_TEMP_DIR (defined by the configuration). This usually works, but the federated python testcases have it set to `/tmp/systemds` which is apparently not available for our git runners. @Baunsgaard maybe we could change the configuration for the tests? > > The problems mostly impact our testcases, but I also don't really have a favorite from a user side. Other choices for where to write and downsides: > > * current working directory (cwd): the cwd of our testcases is the root, therefore we would clutter it full of partitions > > * the scratch space: will be deleted when the worker is killed and in our case, because we just start the new workers from the same JVM, also when the testcase finishes. Therefore, we can't check the results. > > * A new configuration directory: pretty clean solution IMO, but would only be used by workers and I don't want to add it, except if we agree it is necessary > > > The other aspects are finished and this PR is ready for review. hmm, good question what the best option is. I find the last option with a new configuration tempting. It could be made such that the federated workers use a configured path to make a directory, furthermore we can use their process ID to make a unique sub directory, and maintain a static counter inside to guarantee incrementing folder/file names, while appending the user specified file name in the end. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
