Jonas Drewsen wrote: >On 18/05/11 10.09, Johannes Pfau wrote: >> jdrewsen wrote: >>> Please see comments below. >>> >>> Den 17-05-2011 16:42, Andrei Alexandrescu skrev: >>>> Thanks for the response! A few more answers and comments within >>>> (everything deleted counts as "sounds great"). >>>> >>>> On 5/17/11 3:50 AM, Jonas Drewsen wrote: >>>>>> 14. Isn't the max redirect configurable via a parameter? >>>>> >>>>> Yes. It is called Http.followLocation from libcurls followLocation >>>>> option. I will rename it to maxRedirections for clearity. >>>> >>>> You mention this about many higher-level functions: "Maximum >>>> redirections are set to 10." Then I'm thinking, if it's an >>>> important parameter, make it a defaulted parameter for the >>>> function. >>> >>> I guess I should just remove that comment because other defaults has >>> been selected as well e.g. timeouts. >>> >>>>>> 16. Documentation should point to descriptions of the HTTP >>>>>> methods wrapped (e.g. "post", "del" etc). >>>>> >>>>> Do you mean point to the relevant RFC? >>>> >>>> Yah, or a good tutorial. (I didn't know DEL existed!) >>> >>> Well DEL is actually called DELETE in the HTTP RFC. But delete is a >>> reserved word i D so I used del() instead. delete() wouldn't work in >>> following code in think: >>> >>> with(auto http = ...) { >>> delete(...); >>> } >>> >>>>>> 22. byLine/byChunk should not expose a string or generally data >>>>>> that can be shared safely by the client. That's because it would >>>>>> need to create a new string with each iteration, which is very >>>>>> inefficient. Instead, they should expose char[]/ubyte[] just like >>>>>> File.byLine does, and reuse the buffer with each call. >>>>>> Essentially byLine/byChunk should be near-zero-overhead means to >>>>>> transfer and inspect arbitrarily large streams. >>>>> >>>>> byLine/byChunk is currenly only available for the async methods >>>>> which is implemented using message passing. This means that for >>>>> passing the message a copy has to be made because a message is >>>>> by-value or immutable data. Is there another way for me to pass >>>>> the message without doing a copy... some kind of move semantic? >>>> >>>> A library element is planned but not on the short list yet. You can >>>> work around that by internally doing a cast. As long as you observe >>>> the commitment that buffers are not accessed simultaneously by more >>>> than one thread you're well within defined behavior. >>> >>> ok nice to know. >>> >>>> One more suggestion - you may want to also provide classic boring >>>> synchronous read/write methods that take a user-provided buffer. >>>> I'm sure people could use such e.g. when they want to stream data >>>> inside their own threads, or even in the main thread if they don't >>>> mind blocking. >>> >>> This would mean hooking into libcurl and selecting on its sockets. >>> Totally doable. >> I'm not sure if it'd be that easy. Using select to implement a >> blocking api is possible, but select is only used to wait until data >> is ready, reading and writing the data is still done by curl. And as >> curl does not allow you to supply the data buffer, having >> user-provided buffers is afaik impossible. > >_Doable_ - not easy :) > >Select will wait for data to be ready and ask curl to handle the data >chunk. Curl in turn calls back to a registered callback handler with >the data read. That handler fills the buffer provided by the user. If >not enough data has been receive an new select is performed until the >requested amount of data is read. Then the blocking method can return.
I think it's not necessary to do multiple calls if the first didn't return enough data. Most read calls are only guaranteed to return a maximum of data, less is always possible. It's a bigger problem if the user supplied buffer is smaller than the curl buffer. And you always end up copying data. So it might be better to just return a reference to the buffer allocated by curl, or just forget about this api, curl just doesn't work well this way. >>> But I would rather stop here feature wise and wrap >>> this thing up. I would like to focus my efforts on a candidate for >>> std.net.event/reactor/proactor module. >> >> Have you already started working on std.net.event? Do you plan to >> base that on libev / libevent? > >I'll finish the curl wrapper before starting on it. > >I have had a look at libev/event before and they are very nice libs. >But I think I'll have to implement it from scratch for two reasons: > >1, AFAIK we cannot include code in std phobos that is not boost >licensed or a license as liberal. libev for example requires you to >distribute the license with your software. That could indeed be a problem. I think libev/libevent even have the same license. Probably it's really best to implement it from scratch, but it will be quite some work. I'd still have a look at the libev documentation though, it contains some details about OS quirks. And try to avoid Linux AIO ;-) >2, It introduces a dependency to an external project in phobos. >Currently no dependencies are present. This makes phobos very easy to >install and use out of the box. > >The optimal solution to these problems from my point of view would be >to get that "much discussed" package tool in place soon (CPAN,apt >alike). > >Heck, now I think about it maybe I should do that before any std.net >stuff. Let's see how the wind blows - many interesting projects :) > >> I've been trying to get familiar with libev lately, so I have >> bindings for livev 3.9 / 4.04 (combined, version can be selected >> with version statements): >> http://dl.dropbox.com/u/24218791/d/src/libev.html (comments in the c >> header have been turned into doc comments. The binding is partially >> based on Leandro Lucarella bindings: >> http://git.llucax.com.ar/?p=software/ev.d.git) >> > >Very nice. I've also stumbled upon some other async D libs out there >that could be a good starting point. > >It seems that Leandro has both bindings and a wrapper in place. And a >wrapper is really nice to have to make it more D'ish. Yes, I have a higher level wrapper for libev as well, but the new 'shared' stuff made it a lot more complicated to get that right. >/Jonas > > -- Johannes Pfau
signature.asc
Description: PGP signature