I could easily be confusing the C# API with the Win32 API, but I recall seeing some StackOverflow posts which indicated there was no way to do what I wanted to do. This was some time ago and my memory ain't so great. My hack-y solution was to spin up a thread for each file/pipe I needed to read from and send all the data down a thread safe queue, which could then implement a 'select()' like mechanism, which is what I really wanted.
... but I no longer have to care about that. At least until I do again. ~John On Wed, Mar 3, 2021 at 2:02 PM Kenton Varda <ken...@cloudflare.com> wrote: > Hmm when I say "async I/O" I mean the same thing as "non-blocking I/O". > > I haven't worked in C#, but have done some stuff with the Win32 C API, > where they call it "overlapped" I/O. > > -Kenton > > On Wed, Mar 3, 2021 at 3:54 PM John Demme <teqdr...@gmail.com> wrote: > >> I think in Windows, async IO is different from non-blocking IO. I haven't >> used Windows async IO, other than briefly trying to use C# async IO, >> getting very confused, and (perhaps prematurely) that C# async is intended >> to be async everywhere (without some really nasty hacks) and you can only >> use async IO in async code. >> >> ~John >> >> On Wed, Mar 3, 2021 at 1:42 PM Kenton Varda <ken...@cloudflare.com> >> wrote: >> >>> On Wed, Mar 3, 2021 at 2:11 PM John Demme <teqdr...@gmail.com> wrote: >>> >>>> Some operating systems (*cough* Windows *cough*) don't support >>>> non-blocking file (or inter-process pipe) I/O without some unreliable >>>> hacks. AFAICT. >>>> >>> >>> Hate to say it, but my understanding is that Windows has much more >>> robust async filesystem support than Linux does. The comment that Pepijn >>> found is about Linux, not Windows. >>> >>> Linux filesystem implementations are not async (at least for metadata / >>> directory walking), therefore the only way to access them in an async way >>> is to use threads -- either in the kernel, or in userspace. The kernel >>> isn't very excited about spawning kernel threads transparently -- it would >>> rather than you create userspace threads and make blocking calls. >>> >>> That said, this is changing a bit with io_uring, where kernel threads >>> running asynchronously from userspace threads are becoming more of a thing. >>> Still, on the kernel side, there are threads, even if it looks like "async" >>> I/O from the userspace side. >>> >>> Disclaimer: I'm not a kernel developer, this is my second-hand >>> understanding. Also, I know almost nothing about Windows, except that async >>> file I/O has featured much more prominently in the Win32 API going back >>> decades, so I'm guessing it's also baked deeper into the kernel. >>> >>> -Kenton >>> >>> >>>> >>>> ~John >>>> >>>> On Wed, Mar 3, 2021 at 11:02 AM pepijn de vos <pepijnde...@gmail.com> >>>> wrote: >>>> >>>>> Oh I just went hunting for the asynch bits and in the header it >>>>> actually say there is no such thing as async file IO >>>>> >>>>> // >>>>> ======================================================================================= >>>>> // The filesystem API >>>>> // >>>>> // This API is strictly synchronous because, unfortunately, there's >>>>> no such thing as asynchronous >>>>> // filesystem access in practice. The filesystem drivers on Linux are >>>>> written to assume they can >>>>> // block. The AIO API is only actually asynchronous for >>>>> reading/writing the raw file blocks, but if >>>>> // the filesystem needs to be involved (to allocate blocks, update >>>>> metadata, etc.) that will block. >>>>> // It's best to imagine that the filesystem is just another tier of >>>>> memory that happens to be >>>>> // slower than RAM (which is slower than L3 cache, which is slower >>>>> than L2, which is slower than >>>>> // L1). You can't do asynchronous RAM access so why asynchronous >>>>> filesystem? The only way to >>>>> // parallelize these is using threads. >>>>> // >>>>> // All KJ filesystem objects are thread-safe, and so all methods are >>>>> marked "const" (even write >>>>> // methods). Of course, if you concurrently write the same bytes of a >>>>> file from multiple threads, >>>>> // it's unspecified which write will "win". >>>>> >>>>> On Wed, Mar 3, 2021 at 7:57 PM pepijn de vos <pepijnde...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hey Kenton, >>>>>> >>>>>> Thanks for the answer, and sorry my frustration in trying to find out >>>>>> how to do async file IO in cap'n proto came across rude. >>>>>> >>>>>> What puzzles me is your suggestion that I don't need to use KJ. >>>>>> What puzzles me even more is that the KJ file I just obtained seems >>>>>> to not support async operations at all. >>>>>> Are you suggesting it's actually fine to just do blocking IO in the >>>>>> RPC eventloop? >>>>>> Is there some KJ thing to do async file IO, or a way to use other >>>>>> libraries for async file IO with the KJ eventloop? >>>>>> >>>>>> Thanks to all the people working on cap'n proto, it seems pretty neat >>>>>> :) >>>>>> >>>>>> Cheers, >>>>>> Pepijn >>>>>> >>>>>> On Wed, Mar 3, 2021 at 4:42 PM Kenton Varda <ken...@cloudflare.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Pepijn, >>>>>>> >>>>>>> The full comment you refer to says: >>>>>>> >>>>>>> // DO NOT CALL THIS except at the top level of your program, e.g. in >>>>>>> main(). Anywhere else, you >>>>>>> // should instead have your caller pass in a Filesystem object, or a >>>>>>> specific Directory object, >>>>>>> // or whatever it is that your code needs. This ensures that your >>>>>>> code supports dependency >>>>>>> // injection, which makes it more reusable and testable. >>>>>>> >>>>>>> As the comment says, the idea is that you construct your Filesystem >>>>>>> object at the top level of your program, then you pass it (or specific >>>>>>> File >>>>>>> or Directory objects) down to whatever parts of your program need it. >>>>>>> That >>>>>>> ensures that those parts of your program can easily be unit-tested >>>>>>> against >>>>>>> a fake in-memory filesystem, by swapping out the objects that you pass >>>>>>> down. This is trying to help you make your code more flexible, but you >>>>>>> can >>>>>>> ignore it if you want. >>>>>>> >>>>>>> You also don't have to use the KJ filesystem API. In fact, nothing >>>>>>> else in KJ or Cap'n Proto cares what you use to open files. KJ is a >>>>>>> toolkit, meaning it provides a bunch of tools, but doesn't force you to >>>>>>> use >>>>>>> them if you don't want to. >>>>>>> >>>>>>> KJ documentation can be found here: >>>>>>> https://github.com/capnproto/capnproto/tree/master/kjdoc >>>>>>> >>>>>>> Please note that Cap'n Proto and KJ are open source software >>>>>>> provided for free. I'd like them to be useful but I don't get any actual >>>>>>> benefit from you using them, and I'm not interested in helping people >>>>>>> who >>>>>>> are rude. So, if you have further questions, please try to tone it down >>>>>>> a >>>>>>> bit. Thanks. >>>>>>> >>>>>>> -Kenton >>>>>>> >>>>>>> On Wed, Mar 3, 2021 at 5:54 AM pepij...@gmail.com < >>>>>>> pepijnde...@gmail.com> wrote: >>>>>>> >>>>>>>> Hey, >>>>>>>> >>>>>>>> I'm just getting started and pretty confused. >>>>>>>> The RPC server is all async and stuff, so no blocking IO in there. >>>>>>>> Then there are some vague mentions that you should use KJ for most >>>>>>>> things. >>>>>>>> But KJ documentation is nowhere to be found... >>>>>>>> >>>>>>>> All I want to do is just open a file and write to it. >>>>>>>> After resorting to reading the source code, I found >>>>>>>> >>>>>>>> https://github.com/capnproto/capnproto/blob/master/c%2B%2B/src/kj/filesystem.h >>>>>>>> So I managed to create a path so far. >>>>>>>> Then I started to look for how to open a file. >>>>>>>> I found some methods on Directory. >>>>>>>> So then I started to look how to make a directory. >>>>>>>> Then I found a Filesystem, which is abstract. >>>>>>>> Then I found newDiskFilesystem which tells me >>>>>>>> DO NOT CALL THIS except in main() >>>>>>>> Well **** how am I supposed to open a file inside my server handler >>>>>>>> then?? >>>>>>>> >>>>>>>> Pepijn >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Cap'n Proto" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to capnproto+unsubscr...@googlegroups.com. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/capnproto/cd5ce66c-b12e-4612-b383-32462f047f69n%40googlegroups.com >>>>>>>> <https://groups.google.com/d/msgid/capnproto/cd5ce66c-b12e-4612-b383-32462f047f69n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Cap'n Proto" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to capnproto+unsubscr...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/capnproto/CANPfQguo2bq4G0tWPeAgzphdSpU4isMkxRUzNmxjCjyvvaxO0w%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/capnproto/CANPfQguo2bq4G0tWPeAgzphdSpU4isMkxRUzNmxjCjyvvaxO0w%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/CAOjmg%3DvY10X8N7SjXUj6%3DFJRE0PGw%2Bi%3DmDfkCdHYpJtb1iuNWQ%40mail.gmail.com.
