Re: [PR] [Bugfix][Disco] Handle NDArray larger than OS buffer for pipe [tvm]

2024-05-15 Thread via GitHub
Lunderberg commented on PR #16992: URL: https://github.com/apache/tvm/pull/16992#issuecomment-2112491349 > Ah I see, the Stream interface was actually created to be in analogy of FileStream interface(which do not have partial writes). Thank you, and that history makes a lot of sense.

Re: [PR] [Bugfix][Disco] Handle NDArray larger than OS buffer for pipe [tvm]

2024-05-14 Thread via GitHub
tqchen commented on PR #16992: URL: https://github.com/apache/tvm/pull/16992#issuecomment-2110865776 The socket interface was mainly meant to accomodate for the non-blocking case where full read is not always possible -- This is an automated message from the Apache Git Service. To respond

Re: [PR] [Bugfix][Disco] Handle NDArray larger than OS buffer for pipe [tvm]

2024-05-14 Thread via GitHub
tqchen commented on PR #16992: URL: https://github.com/apache/tvm/pull/16992#issuecomment-2110864271 Ah I see, the Stream interface was actually created to be in analogy of FileStream interface(which do not have partial writes). After seeing your fix, i realized that this PR actually did th

Re: [PR] [Bugfix][Disco] Handle NDArray larger than OS buffer for pipe [tvm]

2024-05-14 Thread via GitHub
Lunderberg commented on PR #16992: URL: https://github.com/apache/tvm/pull/16992#issuecomment-2110618499 @tqchen The changes for the long-term fix are implemented in https://github.com/dmlc/dmlc-core/pull/686, with TVM compatibility changes implemented in https://github.com/apache/tvm/pull/

Re: [PR] [Bugfix][Disco] Handle NDArray larger than OS buffer for pipe [tvm]

2024-05-14 Thread via GitHub
Lunderberg commented on PR #16992: URL: https://github.com/apache/tvm/pull/16992#issuecomment-2110474112 I agree. This is a stop-gap measure, as the longer-term fix will require updating the `dmlc::Stream` API to return `size_t` from `Write`, similar to `Read`. Without that API change, th

Re: [PR] [Bugfix][Disco] Handle NDArray larger than OS buffer for pipe [tvm]

2024-05-14 Thread via GitHub
tqchen commented on PR #16992: URL: https://github.com/apache/tvm/pull/16992#issuecomment-2110470795 Thanks @Lunderberg ! In this case, i think it is better to introduce a ReadAll and WriteAll function, in pairing with https://github.com/apache/tvm/blob/main/src/support/socket.h#L492,

Re: [PR] [Bugfix][Disco] Handle NDArray larger than OS buffer for pipe [tvm]

2024-05-14 Thread via GitHub
Lunderberg merged PR #16992: URL: https://github.com/apache/tvm/pull/16992 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@tvm.apach

[PR] [Bugfix][Disco] Handle NDArray larger than OS buffer for pipe [tvm]

2024-05-13 Thread via GitHub
Lunderberg opened a new pull request, #16992: URL: https://github.com/apache/tvm/pull/16992 Prior to this commit, using `disco.Session` methods to transfer `NDArray` instances to workers could raise an exception if the `NDArray` is larger than the buffer allocated by the OS for the controll