On Tue, 16 Aug 2016 11:17:01 -0300 Gustavo Sverzut Barbieri <[email protected]> said:
> On Tue, Aug 16, 2016 at 4:28 AM, Carsten Haitzler <[email protected]> > wrote: > > On Tue, 16 Aug 2016 01:43:43 -0300 Gustavo Sverzut Barbieri > > <[email protected]> said: > > > >> On Mon, Aug 15, 2016 at 11:37 PM, Carsten Haitzler <[email protected]> > >> wrote: > >> > On Mon, 15 Aug 2016 22:35:58 -0300 Gustavo Sverzut Barbieri > >> > <[email protected]> said: > >> > > >> >> On Mon, Aug 15, 2016 at 8:13 PM, Carsten Haitzler <[email protected]> > >> >> wrote: > >> >> > On Mon, 15 Aug 2016 12:07:16 -0300 Gustavo Sverzut Barbieri > >> >> > <[email protected]> said: > >> [...] > >> >> It is the same, but you do not need to replicate this in every class > >> >> like done in Ecore_Exe, Ecore_Con, Ecore_Con_URL... :-) > >> >> > >> >> I was thinking just like you, but after talking to Tasn a bit I got > >> >> what he meant with a "thin wrapper around syscalls" and at the end it > >> >> does make sense, more sense actually. > >> > > >> > if it is just a thin wrapper then what value does it provide? > >> > >> Uniform access to the calls. > >> > >> Like in Linux, you do get read(2),write(2),close(2) and file > >> descriptors to work on almost every basic resource. But when you go to > >> higher level resources, like when doing HTTP over libcurl, then you > >> cannot call "read(2)" directly... > >> > >> With the API I'm proposing you get that simplicity of Unix FD's back. > >> It's almost the same call and behavior. > >> > >> Then you can write a simple code that monitors a source, see when > >> there is data to read, read some data, wait until the destination can > >> hold more data, then write it... in a loop. This is the Efl.Io.Copier. > >> > >> Check: > >> https://git.enlightenment.org/core/efl.git/log/?h=devs/barbieri/efl-io-interfaces > >> > >> You will see I already provide Stdin, Stdout, Stderr and File. Those > >> are "useless" since you could do with pure POSIX calls. But when I add > >> the objects implemented on complex libraries such as cURL, then that > >> code will "just work". > > > > unix fd's are NOT - simple. not if you want to be non-blocking. you have to > > handle write failures and figure out what was and was not written from your > > buffer, handle select() on the when it is available again and write then - > > and for all you know you may be able to just write a single byte and then > > ave to try again and so on. > > > > unix read/write and fd's push the logic of this up the stack into the app. > > the alternative is to do blocking i/o and that is just not viable for > > something that multiplexes all its i/o through an event loop. > > > > what i have read of this so far means pushing the "kernel buffer is full, > > write failed now or partly failed" back off into the app. and that is not > > even close to replacing ecore_con - it fundamentally misses the "i'll > > buffer that for you, don't worry about it" nature of it that takes the > > kernel's limited buffering and extends it to "infinite" that saves a lot of > > pain and agony. > > Raster, check the code and see it's all in there. When I mean "simple" > is that its nature is pretty simple, concepts are well understood... > not that it's simple or easy to use. > > Since we must deal with that *at least* for the POSIX, then we'll do > it on our side at least once. > > As this POSIX is very simple, it's implementable everywhere with > minimum effort, thus works well for this level of API. > > I completely agree that for end users, having infinite buffers and > even line-buffers for most tasks is *must have*, that's why I'm > focusing on having this in an uniform way. > > You seem to just not get the layers and split roles... maybe because > in previous ecore all elements replicated that? It's the only > difference, the logic is there, but moved outside so we don't > replicate it all. i am only seeing a low level atm in your examples. the code samples look to be not much above a simple read/write/fd wrapper that i've seen. > Benefits everyone: > > - efl developers reader/writer providers become simpler, no need to > reimplement all on their own, no need to inherit a big bulk of a base > class > > - efl users know that they all look and feel the same, they can use > the efl.io.copier on them all and the signals, buffering and behavior > are the same. No need to understand Ecore_Exe, Ecore_Con, > Ecore_Con_URL events. none of these api's should be even using these low level api's because at the efl.io level so far that i've seen it's either blocking i/o OR it's non-blocking and then writes can fail witn 0 or partial byte writes and thus the api user has to deal with that failure. this just shouldn't have to be done anywhere. in any efl api. :( > I know you're worried about "common cases", so am I. At the end of > this project I'll have to migrate all Ecore_Con + Ecore_Con_URL users > to the new API and I'll be the first one to come with helpers for > these common cases... then if the common case is to read all the > information to memory, like download a JSON to memory and use it... > this should be very easy and not require dozen lines more than the > legacy code :-) i'm not seeing that atm in your design. :) i'm seeing something REALLY low level with all the pain and problems of the low level api;s >( > > >> >> >> Efl.Io.Copier does and keeps a "read_chunk" segment that is used as > >> >> >> memory for the given slice. > >> >> >> > >> >> >> This is why the Eina_Slice and Eina_Rw_Slice plays well in this > >> >> >> scenario. For example you can get a slice of the given binbuf in > >> >> >> order to handle to other functions that will write/read to/from it. > >> >> >> It doesn't require any a new binbuf to be created or COW logic. > >> >> > > >> >> > it requires a new eina slice struct to be allocated that points to the > >> >> > data which is EXACTLY the below binbuf api i mention. > >> >> > >> >> eina slice is a pair of 2 values, will always be. There is no opaque > >> >> or need for pointer, or allocate. The eina_slice.h API is mostly about > >> >> passing struct value, not reference/pointer. > >> >> > >> >> with binbuf indeed you're right, given its complexity you end with an > >> >> allocated opaque memory handle, magic validation, etc. > >> > > >> > and that's what a slice is - it's an allocated opaque handle over a blob > >> > of memory... is that not just binbuf? > >> > >> it's not opaque handle. It's a public structure, you allocate it on > >> stack... same cost as doing "const void *x, size_t xlen". But the pair > >> is carried, in sync, easy to use, easy to understand. > > > > it'll need to be allocated if you ever have buffering... and the data it > > points to will have to be managed. > > I'm officially giving up on this until you check the code. I'm talking > one thing, you're talking another. :-/ i looked at your samples. specifically efl_io_reader_fd.c and writer_fd.c - i see a writer that handles EINTR but will fail on a write and you keep having to call it to write with your slice (track if it wrote all of it or just some) etc. exactly as i described. the user of the interface has to do all the work that a user of read/write in raw would have to. i dislike this interface as a result. :( the only solution with your current impl is then to either mak,e it block so all writes are a full success (all written) OR some i/o error has happened (fatal), OR to implement extra buffering at the efl io writer level.. and this buffering would require the same memcpy's you are wanting to avoid that ecore_con has been doing. what i see is an interface that is NOT friendly to usage ... and if it were made friendly it would not be performance friendly. > >> >> This can be done transparently with the current API proposal. > >> > > >> > but in your current one - writes will fail because you don't allocate or > >> > expand an existing buffer - right? once full.. then what? > >> > >> It's just like read(2)/write(2) that you know very well. If you want > >> to copy using them, you need an intermediate buffer. > >> > >> Efl.Io.Copier is that code and holds that buffer. You can limit it or not. > >> > >> If unlimited, reads() up to a maximum chunk size and keeps expanding > >> the buffer. Once write() returns positive value, that amount is > >> removed from the buffer, that can shrink. > >> > >> If limited, it will stop monitoring read (partially implemented), thus > >> will not call read(2), thus will not reach the kernel and eventually > >> its internal buffer will be full and the writer process will be > >> informed. > > > > so you are FORCING an api that HAS to memcpy() at the time a slice is > > passed in before the func returns. that means either it always has to > > memcpy somewhere (or has to once writes() start failing when a kernel > > buffer is full) OR it requires a blocking api... > > I'm not forcing anything neither requires a blocking API. as above. it either has to copy, or block OR fail writes given your current design. failing is unfriendly. blocking it mainloop unfriendly. copying is performance unfriendly. > The current read() and write() methods behave like POSIX, that is, you > handle them a buffer. WHAT is this buffer is outside of scope. This > means if you can negotiate a DMA, then you can simply get the DMA > handle from the writer object and use it with the reader and there is > no memcpy() (This is a great idea... something to look as an extra > method for the Efl.Io.Writer class and use in Efl.Io.Copier, like when > using mmap() files). Likewise readers could offer DMA. In these cases > the copier class can avoid the internal buffer altogether. > > NOTE: I'm not doing the DMA now since this is a base for Efl.Net stuff > which have higher priority, but as you can see it's doable and I can > get to them later. > > > > > what i see here is that you are designing either: > > > > 1. a blocking api (unacceptable from any main loop construct) > > There are 2 properties/events: Reader.can_read, Writer.can_write. If > you write when can_write is false, then you can either block or get an > EAGAIN/EWOULDBLOCK. If you read when can_read is false, that's the > same. That's the equivalent of select()/poll(). > > But if you read when "can_read" or write when "can_write", it's able > to provide/take at least one byte without blocking/failing. > > Efl.Io.Copier is an Efl.Loop.User and internally will keep an > efl_loop_job(efl_loop_user_loop_get(self)). Thus it will process > chunks in the main loop... being friendly to the rest. > > Note that this is all hidden from the user. Shall we want to use a > thread, we can move all this operation to threads. > > > > or > > 2. an api where writes can fail when buffers are full and that requires the > > caller handle buffering and write failures themselves (which makes the api a > > pain to use and no better than raw read/write with a raw fd) > > this is the case for the Readers/Writers. The pain is taken care by > Efl.Io.Copier. Just different roles, before it used to be bundled > inside all other classes. there shouldn't even BE an api layer where writes can fail. i dont see a value in such an api. it's technically impossible to have reads NOT fail unless you want to block, but writes are different ad ALL DATA to write is known. read - that is not the case. > > > or > > 3. an api that requires a memcpy of data on write ALWAYS once kernel buffers > > fill up and no ability to zero copy (which goes against the whole original > > idea of you wanting to make it efficient). > > See the DMA part above. > > > > -- > Gustavo Sverzut Barbieri > -------------------------------------- > Mobile: +55 (16) 99354-9890 > > ------------------------------------------------------------------------------ > _______________________________________________ > enlightenment-devel mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/enlightenment-devel > -- ------------- Codito, ergo sum - "I code, therefore I am" -------------- The Rasterman (Carsten Haitzler) [email protected] ------------------------------------------------------------------------------ _______________________________________________ enlightenment-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
