Baunsgaard commented on PR #2154: URL: https://github.com/apache/systemds/pull/2154#issuecomment-2565554800
LGTM, I have now merged it. While merging I played around with parallelizing the Python API and found out that it spawns a thread per connection. There is indeed an overhead in this connection, but it is not the main problem. To make the transfer more efficient we could: 1. Reduce the number of calls by fusing many operations into single calls to java. 2. Reduce the current serialization bottleneck by slicing up the array into many smaller byte arrays when sending over. I see around 20% utilization of my CPU when transferring 10k by 10k integer matrices, so there is room for improvement. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
