Re: A study on immutability usage
On Monday, 1 October 2012 at 12:28:33 UTC, bearophile wrote: I am back. This study regards how "final" is used in Java programs, and generally how immutability is used and is useful: http://whiley.org/2012/09/30/profiling-field-initialisation-in-java/ ... Reminds me of C# 'readonly' http://msdn.microsoft.com/en-us/library/acdd6hb7(v=vs.71).aspx /Jonas
Re: When will we have std.log?
This is great news. Really looking forward to a std.log module. /Jonas
C++ to catch up?
It seems like the C++ committee is speeding up development adding lots of the goodies from D like Ranges, static if, template contraints etc. Will D still have a case when C++ gets this done? I wonder if Andrei is part of the C++ Ranges Study Group? The Future of C++: http://channel9.msdn.com/Events/Build/2012/2-005 /Jonas
Re: DConf 2013 to be recorded
That is great! I will not be able to attend the conference but I did go for the lousy tshirt pledge. I live in Europe and wanted to set the shipping address to one of my friends in the US. How do you do that? /Jonas
Re: A new web newsreader
On Saturday, 10 December 2011 at 07:07:11 UTC, Vladimir Panteleev wrote: On Saturday, �� December 2011 at 00:10:26 UTC, Jesse Phillips wrote: You do realize that > is a formatting feature, and like it is visible to others. As a language news group having a way to segregate code would be useful even if only to be fixed. Reddit uses for leading spaces? Using ">" for quotes is a widely-implemented standard (see e.g. RFC 3676) which applies directly to our situation. How do you feel about creating a setting that would make regions formatted for code? Possibly default off. Somebody might want this enough to send a patch. /Jonas
Re: Package manager - interacting with the compiler
On Saturday, 10 December 2011 at 08:55:57 UTC, Jacob Carlborg wrote: I think I've come so far in my development of a package manager that it's time to think how it should interact with the compiler. Currently I see two use cases: 1. When the package manager installs (and builds) a package 2. When a user (developer) builds a project and want's to use installed packages In the best of worlds the user wouldn't have to do anything and it just works. The package manger needs to somehow pass import paths to the compiler and libraries to link with. I'm not entirely sure what the best method to do this would be. But I'm thinking that if the compiler could accept compiler flags passed via environment variables use case 1 would be easy to implement. For use case 2 it would be a bit more problematic. In this use case the user would need to somehow tell the package manager that I want to use these packages, something like: // project.obspec orb "foo" orb "bar" $ orb use project.obspec or for single packages $ orb use foobar $ dmd project.d If environment variables are used in this case, then the package manager would need a shell script wrapper, the same way as DVM does it, to be able to set environment variables for the parent (the shell). The reason for this is that a child process (the package manager) can't set environment variables for the parent process (the shell). This complicates the implementation and installation of the package manager and requires different implementations for Posix and Windows. Another idea would be to manipulate the dmd.conf/sc.ini file but that seems to be quite complicated and messy. On the other hand, this wouldn't require any changes to the compiler. Any other ideas? https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D https://github.com/jacob-carlborg/orbit For use case 1 the package manager could just as well call dmd directly with the correct flags ie. no need for using environment variables. Use case 2 does not belong to a package manager in my opinion. It is the job of a build tool to configure packages for a project. What would be nice to have support for using packages without a build tool. Maybe something like what pkg-config provides: dmd -ofhello `orb -lib foo` hello.d where "org -lib foo" returns the flags to use the foo package. /Jonas
Re: Package manager - interacting with the compiler
On Sunday, 11 December 2011 at 21:22:37 UTC, Jacob Carlborg wrote: On 2011-12-10 22:17, jdrewsen wrote: On Saturday, 10 December 2011 at 08:55:57 UTC, Jacob Carlborg wrote: I think I've come so far in my development of a package manager that it's time to think how it should interact with the compiler. Currently I see two use cases: 1. When the package manager installs (and builds) a package 2. When a user (developer) builds a project and want's to use installed packages In the best of worlds the user wouldn't have to do anything and it just works. The package manger needs to somehow pass import paths to the compiler and libraries to link with. I'm not entirely sure what the best method to do this would be. But I'm thinking that if the compiler could accept compiler flags passed via environment variables use case 1 would be easy to implement. For use case 2 it would be a bit more problematic. In this use case the user would need to somehow tell the package manager that I want to use these packages, something like: // project.obspec orb "foo" orb "bar" $ orb use project.obspec or for single packages $ orb use foobar $ dmd project.d If environment variables are used in this case, then the package manager would need a shell script wrapper, the same way as DVM does it, to be able to set environment variables for the parent (the shell). The reason for this is that a child process (the package manager) can't set environment variables for the parent process (the shell). This complicates the implementation and installation of the package manager and requires different implementations for Posix and Windows. Another idea would be to manipulate the dmd.conf/sc.ini file but that seems to be quite complicated and messy. On the other hand, this wouldn't require any changes to the compiler. Any other ideas? https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D https://github.com/jacob-carlborg/orbit For use case 1 the package manager could just as well call dmd directly with the correct flags ie. no need for using environment variables. I was thinking that the package manager just invokes a build tool like make, rdmd, dsss, shell script and so on. Use case 2 does not belong to a package manager in my opinion. It is the job of a build tool to configure packages for a project. What would be nice to have support for using packages without a build tool. Maybe something like what pkg-config provides: dmd -ofhello `orb -lib foo` hello.d where "org -lib foo" returns the flags to use the foo package. /Jonas I would say that the preferred way is to use a build tool then there is no problem. The build tool just asks the package manager which import paths to use for the given packages and pass the information to the compiler. But I don't want my package manager to depend on a built tool, I want it to be usable on its own. And for that I think the pkg-config method is the way to go. Setting environment vars brings unneeded state into you development session. Another option would be to just wrap dmd in a e.g. orbdmd command and handle it there. Btw: have you considered renaming from orb to something that makes sense to newbies e.g. dpack? -Jonas
Re: Second Round CURL Wrapper Review
On Monday, 12 December 2011 at 10:26:52 UTC, Somedude wrote:
Le 05/12/2011 22:36, Jonas Drewsen a écrit :
Anyway - I _have_ said that I will add a data availability poll
functionality so I guess your initial concern is also covered.
/Jonas
Just an idea: would it be possible/useful to use the
signals/slots
mechanism for this kind of synch ?
This would be most useful if there were some kind of main loop.
This is needed because the request is done in another thread and
the response should be handled in the main thread. Without an
main loop in the main thread or a destinct call to e.g. a poll()
method the signal/slot s would not work.
As a side note I would very much like D to enter a main loop with
a task scheduler as part of the runtime. That way co-routines and
event handling would be much nicer to use out of the box. For
example:
// The main function is running a fiber in the main loop.
// the main loop terminates when no more fibers are active
void main(string[] args) {
fiber( { auto content= download("google.com"); write(content);
} );
fiber( { sleep(2.0); writeln("Hello in 2 secs"); } );
writeln("first line written");
}
The download() yields until data is downloaded and the sleep()
yield until the time has passed. You wouldn't notice this when
not using fibers because you would only have the main() fiber
which terminates the main loop (and the program) when it is done.
Optimally the runtime should support replacing the standard
mainloop scheduler with your own.
/Jonas
Re: Second Round CURL Wrapper Review
On Monday, 12 December 2011 at 00:53:14 UTC, dsimcha wrote: Here's my review. Remember, review ends on December 16. Overall, this library has massively improved due to the rounds of review it's been put through. I only found a few minor nitpicks. However, a recurring pattern is minor grammar mistakes in the documentation. Please proofread all documentation again. Docs: "The high level API is build" -> "The high level API is built" "LibCurl is licensed under a MIT/X derivate license" -> "LibCurl is licensed under an MIT/X derivative license" AutoConnect: "Connection type used when the url should be used to auto detect protocol." -> "auto detect THE protocol" ok Why is there a link to curl_easy_set_opt in the byLineAsync and byChunkAsync docs? will fix In onSend: "The length of the void[] specifies the maximum number of bytes that can be send." -> "can be SENT" ok What is the use case for exposing struct Curl? I prefer if this were unexposed because we'll obviously be unable to provide a replacement if/when the backend to this library is rewritten in pure D. Initially it was not exposed but there were a couple of requests to expose it. Thats why. Actually, that leads to another question: Should this module really be named etc.curl/std.curl/std.net.curl, or should the fact that it currently uses Curl as a backend be relegated to an implementation detail? I think it should have curl in the name. I do not believe that a native D network library should have the same API as this module. It is limited by the design of libcurl and should be improved by a native D net library. Code: pragma(lib) basically doesn't work on Linux because the object format doesn't support it. Don't rely on it. ok Should the protocol detection be case-insensitive, i.e. "ftp://"; == "FTP://"? yes Thanks for the feedback /Jonas
Re: Second Round CURL Wrapper Review
On Monday, 12 December 2011 at 01:05:25 UTC, Jonathan M Davis
wrote:
On Sunday, December 11, 2011 19:55:28 dsimcha wrote:
On 12/11/2011 7:53 PM, dsimcha wrote:
> Should the protocol detection be case-insensitive, i.e.
> "ftp://"; ==
> "FTP://"?
Oh, one more thing: Factor the protocol detection out into a
function.
You have the same expression cut and pasted everywhere:
if(url.startsWith("ftp://";) || url.startsWith("ftps://") ...
And startsWith will take multiple arguments, so the code
definitely shouldn't have
x.startsWith(y) || x.startsWith(z) || ...
It's more verbose and less efficient that way.
- Jonathan M Davis
I hadn't noticed that multiple arguments is accepted. Thanks.
/Jonas
Re: Second Round CURL Wrapper Review
On Monday, 12 December 2011 at 14:42:17 UTC, Andrei Alexandrescu
wrote:
On 12/12/11 6:25 AM, jdrewsen wrote:
On Monday, 12 December 2011 at 01:05:25 UTC, Jonathan M Davis
wrote:
On Sunday, December 11, 2011 19:55:28 dsimcha wrote:
On 12/11/2011 7:53 PM, dsimcha wrote:
> Should the protocol detection be case-insensitive, i.e. >
> "ftp://"; ==
> "FTP://"?
Oh, one more thing: Factor the protocol detection out into a
function.
You have the same expression cut and pasted everywhere:
if(url.startsWith("ftp://";) || url.startsWith("ftps://") ...
And startsWith will take multiple arguments, so the code
definitely
shouldn't have
x.startsWith(y) || x.startsWith(z) || ...
It's more verbose and less efficient that way.
- Jonathan M Davis
I hadn't noticed that multiple arguments is accepted. Thanks.
/Jonas
You can also pass a predicate for case insensitive comparison.
Andrei
Perfect
Re: Second Round CURL Wrapper Review
On Tuesday, 13 December 2011 at 00:47:26 UTC, David Nadlinger wrote: On 12/12/11 1:18 PM, jdrewsen wrote: On Monday, 12 December 2011 at 10:26:52 UTC, Somedude wrote: Just an idea: would it be possible/useful to use the signals/slots mechanism for this kind of synch ? This would be most useful if there were some kind of main loop. This is needed because the request is done in another thread and the response should be handled in the main thread. Without an main loop in the main thread or a destinct call to e.g. a poll() method the signal/slot s would not work. I don't know if you already have a solution in the works, but maybe the future interface I did for Thrift is similar to what you are looking for: http://klickverbot.at/code/gsoc/thrift/docs/thrift.util.future.html David I don't have any plans for this in the curl stuff. But it is something like your future implementation and a builtin mainloop I think would make async programming much easier in D. /Jonas
Re: Second Round CURL Wrapper Review
On Thursday, 15 December 2011 at 07:46:56 UTC, Jonathan M Davis wrote: On Friday, December 02, 2011 23:26:10 dsimcha wrote: I volunteered ages ago to manage the review for the second round of Jonas Drewsen's CURL wrapper. After the first round it was decided that, after a large number of minor issues were fixed, a second round would be necessary. Significant open issues: 1. Should libcurl be bundled with DMD on Windows? I'd argue that it should be a separate download but that it should definitely be provided. 2. etc.curl, std.curl, or std.net.curl? (We had a vote a while back but it was buried deep in a thread and a lot of people may have missed it: http://www.easypolls.net/poll.html?p=4ebd3219011eb0e4518d35ab ) If we're going to follow a policy that wrappers around 3rd party libraries go in etc, then it should be etc.curl. Otherwise, I think that it should be std.net.curl. I'd argue that AutoConnection should be AutoConnect, because it's shorter and just as informative. AutoConnect sounds like a command to connection automatically which might be confusing since that is not what it does. Therefore I went with AutoConnection which I still believe is better. I'd argue that the acronyms should be in all caps if camelcasing would require that the first letter of the acronym be capitalized and all lower case if camelcasing would require that the first letter of the acronym be lower case. We're not completely consistent in how we using acronyms in names in Phobos, but I believe that we primarily follow that rule when putting them in symbol names. So, for instance, it would be HTTP, FTP, and SMTP rather than Http, Ftp, and Smtp. I like Http better as personal taste. But if HTTP is preferred then I'll do that. Actually I think I will make a pull request to extend the dlang.org/dstyle.html doc with and example that we can point to when someone asks about styling. I've spend too much time restyling because there is no such style doc already and people are complaining about style in reviews anyway. I'd rename T to C or Char in the template parameters, since it's a character type rather than a generic type. Not a huge deal, but I think that it makes the code quicker to understand. I've named it T because it can also be ubyte. In that case C for Char is confusing. Why are some parameters const(char)[] and others string? Why not make them all const(char)[] or even actually templatize them on character type (one per parameter which is a string)? The parameters that are const(char)[] will accept both string and char[] which is what I want because libcurl make an internal copy of it anyway. If I typed it as string then you would have to idup a char[] for the parameter without any reason. The parameters that are string is internally passed to functions (in other std modules) that only accept strings. I could accept const(char)[] and idup it in order to have consistant parameter types but that gives an unnecessary overhead. Shorten the URL in your examples. Its long length increases the risk of making the examples too wide. It's a made up URL, so there's no need for it to be that long. Something like foo.bar.com is probably more than enough. Actually the long urls point to a google appspot app that fits the example. I've done it this way so that people can just copy/paste the example and have something working. Not too many people have a test HTTP POST server setup for a quick test AFAIK. Why does byLine use '\x0a' instead of '\n'? '\n' is clear, whereas most people will have to look up what '\x0a' is, so I really think that it should be '\n'. I looked at how it was done elsewhere in phobos and did like that: http://dlang.org/phobos/std_stdio.html#byLine But I agree with you that \n is better and will change it. Also, if byLine is going to return an internal range type instead of having an externally defined one, its documentation needs to explain what Char is used for. As it stands, it looks like a pointless template parameter. To do that, the return section should probably say something like "A range of Char[]..." All of this goes for byLineAsync as well. Ok. In general, the functions should do a better job of documenting their parameters. Many of them don't document their parameters at all. For instance, while it's fairly easy to guess what keepTerminator and terminator are used for, there is no explanation about them whatsoever in either byLine or byLineAsync's documentation. Again I looked at phobos to see what was current the style: http://dlang.org/phobos/std_stdio.html#byLine I'll include some more detail though. If any of the range return types of these functions have non-standard functions on them, they need to be properly documented. e.g. wait(Duration) is mentioned in passing in byLineAsync's documentation, but no explanation for its purpose is given. The documentation needs to explain everything the progra
Re: Second Round CURL Wrapper Review
On Thursday, 15 December 2011 at 08:51:29 UTC, Jonathan M Davis
wrote:
Line# 235 is identical to line# 239. Shouldn't line# 235 be
creating an Http object, not an Ftp object? That mistake
definitely makes it look like download hasn't been properly
tested.
It has been tested as you can see by the unittest just below the
function. It just did not fail because libcurl requires the same
setup for this download situation for both ftp and http. Will fix
it of course.
You should create template similar to
template isCurlConn(Conn)
{
auto isCurlConn = is(Conn : Http) || is(Conn : Ftp) ||
is(Conn : AutoConnection);
}
It would reduce code duplication.
ok
The functions which have a template parameter T = char really
need to have that parameter properly explained in the
documentation. From the documentation, I would have thought
that it was any character type, but it's char or ubyte. There
is no hint whatsoever in the documentation that ubyte would
work. All of the parameters need to be made clear. That goes
for all functions.
ok
Is there a reason that the functions which have a template
argument T which can be either a char or a ubyte can't work
with immutable char? Glancing at _basicHttp, I don't see any
reason why T couldn't be immutable char. Yes, it would require
casting to immutable(char)[], but you're already casting to
char[], and the data being returned appears to be unique such
that it could be safely cast to immutable. That being the case,
I'd encourage you to not only make it work with immutable char
but to make immutable char the default instead of char.
Is is indeed unique and can be cast to immutable. I'll add that
option.
Why do you think immutable char as the default is better than
char? I know that the return type in that case would be string
and not char[] - but why is that better?
Once you've fixed the exception types like I requested in my
first review, you can and should use
enforceEx!CurlException(cond, msg) instead of enforce(cond, new
CurlException(msg)).
ok
it may not matter, since we're dealing with strings here, but
I'd argue that !empty should be used rather than length > 0
(e.g. line# 475). In stuff other than strings it definitely can
be more efficient, and even with strings, it may be, since I
think that checking whether the length is 0 (as empty does) is
slightly more efficient than checking whether it's greater than
0.
ok
Please make sure that opening braces on on their own line.
That's the way that the rest of Phobos does it and it's one of
the few formatting rules that we've been trying to use
consistently throughout Phobos. For the most part, you get it
right, but not everywhere - nested functions in particular seem
to have braces on the same line for some reason.
When I look through the code it seems ok with regards to braces
on own line. Maybe by nested functions you mean delegates? The
delegates I use do indeed have braces on same line which is ok
afaik.
It's not a big deal, but you should use auto more. For
instance, lines #626, #638, and #640 don't use auto when they
could.
I'm a bit in doubt. On one hand it is great to make everything
auto in order to make it easy to change type and to remove
redundancy. One the other hand it is very convenient to be able
to see the type when you read the code. I know that a clever
editor would be able to figure out the type for me. But I also
read code in normal editors for diffs etc. or on github.
Anyway - I've changed as much as possible to auto now.
I have no idea why you keep putting parens around uses of the
in operator, but it's not necessary and makes the code harder
to read IMHO. It's certainly not required that you change that,
but I'd appreciate it if you did.
Bad habit. Will remove.
I see that regexes are used in the module. Please make sure
that they still work correctly with the new std.regex. They
probably do, but it's not 100% backwards compatible.
ok
byLine's template constraint needs to verify that the types of
Terminator and Char are valid. As it is, I could try and pass
it something like an Http struct if I wanted to. In general,
_all_ templates need to verify that their arguments are of the
appropriate type for _all_ of their arguments.
ok
It's odd that popFront (line #780) is not @safe when empty and
front are. Does findSplit or findSplitAfter prevent it?
exactly
With a name like Char on byLine, I'd expect it to take any char
type, but not only do you not verify the type (as previously
mentioned), but you instantiate _basicHttp with it, which works
with char and ubyte, not any char. If Char is going to take
char and ubyte specifically, then Char is a bad name for it.
I'll verify the type. For byLine it will only accept char types.
I'd suggest changing line# 823 to
auto result = get(url, isFtpUrl(url) ? FTP() : Http());
Not possible. get() is a template function where the template
parameter is connection t
Re: Second Round CURL Wrapper Review
On Saturday, 17 December 2011 at 20:38:46 UTC, Jakob Ovrum wrote: On Saturday, 17 December 2011 at 11:56:04 UTC, jdrewsen wrote: AutoConnect sounds like a command to connection automatically which might be confusing since that is not what it does. Therefore I went with AutoConnection which I still believe is better. How about something completely different then, like URLInfer? Also, some instances in the docs of "url" in lowercase should probably be changed to uppercase. Will change case. Don't know if the URLInfer name is better though. /Jonas
Re: Second Round CURL Wrapper Review
On Sunday, 18 December 2011 at 01:30:10 UTC, Jonathan M Davis wrote: On Saturday, December 17, 2011 09:21:42 Justin C Calvarese wrote: On Sat, Dec 17, 2011 at 5:56 AM, jdrewsen wrote: > On Thursday, 15 December 2011 at 07:46:56 UTC, Jonathan M > Davis wrote: >>> ... > > I'd argue that the acronyms should be in all caps if > camelcasing would > >> require that the first letter of the acronym be capitalized >> and all >> lower case if camelcasing would require that the first >> letter of the >> acronym be lower case. We're not completely consistent in >> how we >> using acronyms in names in Phobos, but I believe that we >> primarily >> follow that rule when putting them in symbol names. So, for >> instance, >> it would be HTTP, FTP, and SMTP rather than Http, Ftp, and >> Smtp. > > I like Http better as personal taste. But if HTTP is > preferred then I'll > do that. Actually I think I will make a pull request to > extend the > dlang.org/dstyle.html doc with and example that we can point > to when > someone asks about styling. I've spend too much time > restyling because > there is no such style doc already and people are > complaining about > style > in reviews anyway. I absolutely agree that the "D Style" page should be revised and extended to reflect the current standards that new Phobos modules are expected to adhere to. I found an open pull request that indicates a past effort in this regard: https://github.com/D-Programming-Language/d-programming-language.org/pull/16 (I thought that another document was also floating around that was a proposed new "Style Guide for Phobos", but I don't recall who authored it and I don't remember where it was located.) Yeah. I need to get back to that. Portions of it were completely agreed upon and others need further discussion, and I keep forgetting to get those discussed among the Phobos devs and get those questions settled. - Jonathan m Davis Nice to see that something is in progress. Please also add a note about acronyms in names then. /Jonas
Re: d future or plans for d3
On Sunday, 18 December 2011 at 17:19:33 UTC, Andrei Alexandrescu wrote: On 12/18/11 6:19 AM, Alex Rønne Petersen wrote: On 18-12-2011 12:45, Somedude wrote: Le 18/12/2011 12:13, Ruslan Mullakhmetov a écrit : I do not want to make a flame over D vs C++11. Walter and I are both interested in making the GC precise. Wouldn't support for precise GC collection mean larger executable because type information is needed at runtime for precise collection? If so, could extra info be used to improve runtime reflection as well? /Jonas
Re: Incubated modules for Phobos
On Sunday, 18 December 2011 at 14:18:34 UTC, Piotr Szturmaj wrote: Hi all, Normal Phobos submission procedure is usually like that: 1. write entire module from scratch by oneself 2. submit for voting 3. rewrite wrong parts, if there are none then add it to Phobos 4. otherwise goto 2 It is hard for one person to write entire module in such way it satisfies everyone, especially for very new or complex additions, such as database handling or cryptography. Here, I propose that we add experimental "exp" hierarchy to Phobos for such projects. I know etc hierarchy may me used for that but IMHO separate "exp" would be more appropriate. In this namespace "beta" modules will slowly evolve into official std modules. Some (obvious) advantages are: * developers may receive feedback very early in the process, saving them from mass rewrites when in the opinion of community they made some wrong coding decission. * developers may receive coding help, i.e. many of them may collaborate on one big module. * users may test experimental modules early. This is important, because usability issues may be catched earlier, not after submission when API usually becomes frozen and it is too late. "Exp" code may be shipped with each release just like "etc" code. Users using experimental code should be aware of breaking changes that may be introduced with each release or even with each commit. Thoughts? I think that people should just their WIP + links to github/docs on the "review queue" wiki page linked from the dlang.org left menu. /Jonas
Re: Second Round CURL Wrapper Review
On Sunday, 18 December 2011 at 01:27:45 UTC, Jonathan M Davis wrote: On Saturday, December 17, 2011 23:10:00 jdrewsen wrote: On Thursday, 15 December 2011 at 08:51:29 UTC, Jonathan M Davis wrote: > Line# 235 is identical to line# 239. Shouldn't line# 235 be > creating an Http object, not an Ftp object? That mistake > definitely makes it look like download hasn't been properly > tested. It has been tested as you can see by the unittest just below the function. It just did not fail because libcurl requires the same setup for this download situation for both ftp and http. Will fix it of course. Well, when there's code like that that is clearly wrong, it makes it look like it's either not being tested by the unit tests or that the unit tests haven't been run. I guess that this is just a case where the unit tests are unable to catch the problem. > Is there a reason that the functions which have a template > argument T which can be either a char or a ubyte can't work > with immutable char? Glancing at _basicHttp, I don't see any > reason why T couldn't be immutable char. Yes, it would > require > casting to immutable(char)[], but you're already casting to > char[], and the data being returned appears to be unique such > that it could be safely cast to immutable. That being the > case, > I'd encourage you to not only make it work with immutable > char > but to make immutable char the default instead of char. Is is indeed unique and can be cast to immutable. I'll add that option. Why do you think immutable char as the default is better than char? I know that the return type in that case would be string and not char[] - but why is that better? strings can't be sliced with impunity, because their elements are immutable. With char[] and const(char)[], you have to worry about the elements changing, so you're often forced to duplicate the string rather than simply slice it. As such, string is far preferrable to char[] or const(char)[] in the general case. It's even better when there's a choice between them, but if you have to choose, you should almost always choose string (there are exceptions though - e.g. buffers like in std.stdio.ByLine where it reuses its char[] buffer rather than allocating a new one). > Please make sure that opening braces on on their own line. > That's the way that the rest of Phobos does it and it's one > of > the few formatting rules that we've been trying to use > consistently throughout Phobos. For the most part, you get it > right, but not everywhere - nested functions in particular > seem > to have braces on the same line for some reason. When I look through the code it seems ok with regards to braces on own line. Maybe by nested functions you mean delegates? The delegates I use do indeed have braces on same line which is ok afaik. In general, just make sure that braces are on their own line unless you're dealing with a one line lambda or something similar. You do have nested functions or delegates which don't do that. > It's not a big deal, but you should use auto more. For > instance, lines #626, #638, and #640 don't use auto when they > could. I'm a bit in doubt. On one hand it is great to make everything auto in order to make it easy to change type and to remove redundancy. One the other hand it is very convenient to be able to see the type when you read the code. I know that a clever editor would be able to figure out the type for me. But I also read code in normal editors for diffs etc. or on github. Anyway - I've changed as much as possible to auto now. Just in general, it's best practice to use auto unless you can't. There _are_ exceptions to that rule, but that's almost always the way that it should be. > I'd suggest changing line# 823 to > > auto result = get(url, isFtpUrl(url) ? FTP() : Http()); Not possible. get() is a template function where the template parameter is connection type ie. one of FTP or HTTP. isFtpUrl(url) is evaluated at runtime and not compile time. Yeah. I missed that. Stuff like that makes me wish that we had some kind of static ternary operator, but it would probably complicate the language too much. In some cases though, you might be able to use alias to fix the problem (static if the alias and then use the alias in the function call so that the entire function call isn't duplicated). A static ternary operator wouldn't work in this case since isFtpUrl(url) cannot be evaluated at compile time. /Jonas
Re: [Deimos] Request repositories for mediainfo, systemd, libev
On Friday, 23 December 2011 at 11:47:46 UTC, Johannes Pfau wrote: I have the following libraries that I'd like to contribute to deimos: libmediainfo: MediaInfo supplies technical and tag information about your video or audio files. systemd: systemd is a system and service manager for Linux, compatible with SysV and LSB init scripts. libev: A full-featured and high-performance event loop that is loosely modelled after libevent, but without its limitations and bugs. Could someone please create these repositories? libev.. yay! /Jonas
Re: Reference counted containers prototype
On Monday, 26 December 2011 at 17:25:12 UTC, Andrei Alexandrescu wrote: Hello, I've been playing with a new approach to reference counting, in particular for the containers library. A small prototype is at http://pastebin.com/WnSQY1Jw. The prototype features a simple doubly-linked list implementation DListImpl. That is not supposed to be manipulated directly (or it might, in case the user wants a simple garbage collected implementation - this is a point in the discussion). DListImpl has only a couple of primitives implemented. The only interesting points that it tries to make are: (a) the presence of the dispose() primitive, which deallocates all memory and brings the object back to its .init state (b) the presence of the dup() primitive, which creates a full-blown duplicate of the object. The interesting part of the sample is RefImpl, which has a couple of quite interesting details: (a) All interaction with the held object is done via opDispatch. In fact opDispatch can be engineered to statically enforce no reference to the held object escapes. (b) A const/immutable call against a non-existing is silently converted into a call against a default-initialized object. (c) If a call works and returns the same type for a non-const and a const object, then the const version is preferred. This is to reduce the number of calls to ensureUnique(). Destroy. Andrei Maybe the fact that the container is duplicated on opDispatch call time and not on assignment time could be confusing to at least developers coming from c++. It does benefit from lazy duplication by doing it this way which might justify it. Otherwise it seems good. /Jonas
Re: CURL Wrapper: Congratulations Next up: std.serialize
On Tuesday, 27 December 2011 at 02:01:51 UTC, dsimcha wrote: By a vote of 14-0, Jonas Drewsen's CURL wrapper (std.net.curl) has been accepted into Phobos. Thanks to Jonas for his hard work and his persistence through the multiple rounds of review that it took to get this module up to Phobos's high and increasing quality standard. Keep the good work coming. Next in line, if it's ready, is Jacob Carlborg's std.serialize. Jacob, please post here when you've got something ready to go. I'll go fix the last issues mentioned in the final review thread and the do a pull request. Thanks to all who provided valuable feedback.
Re: CURL Wrapper: Congratulations Next up: std.serialize
On Wednesday, 28 December 2011 at 16:01:50 UTC, Jacob Carlborg wrote: On 2011-12-27 03:01, dsimcha wrote: By a vote of 14-0, Jonas Drewsen's CURL wrapper (std.net.curl) has been accepted into Phobos. Thanks to Jonas for his hard work and his persistence through the multiple rounds of review that it took to get this module up to Phobos's high and increasing quality standard. Keep the good work coming. Next in line, if it's ready, is Jacob Carlborg's std.serialize. Jacob, please post here when you've got something ready to go. Project page (two tutorials): http://dsource.org/projects/orange Repository: https://github.com/jacob-carlborg/orange Documentation: http://dl.dropbox.com/u/18386187/orange_docs/orange.serialization.Serializer.html (Don't forget the "Package" tab) Unit tests are available in the "tests" directory. These unit tests are not like regular unit tests that test individual functions. These unit tests are on a higher level. The most important part to review is the "serialization" package. The rest is mostly utility modules. After I quick look: I think that all other modules/packages under the orange package should either be integrated into other existing phobos modules or as new phobos modules since they are really not serialization specific. Furthermore I think that all suppport for tango in the code should be stripped before going into phobos. /Jonas
Re: CURL Wrapper: Congratulations Next up: std.serialize
On Wednesday, 28 December 2011 at 22:21:09 UTC, Jacob Carlborg wrote: On 2011-12-28 22:19, jdrewsen wrote: On Wednesday, 28 December 2011 at 16:01:50 UTC, Jacob Carlborg wrote: On 2011-12-27 03:01, dsimcha wrote: By a vote of 14-0, Jonas Drewsen's CURL wrapper (std.net.curl) has been accepted into Phobos. Thanks to Jonas for his hard work and his persistence through the multiple rounds of review that it took to get this module up to Phobos's high and increasing quality standard. Keep the good work coming. Next in line, if it's ready, is Jacob Carlborg's std.serialize. Jacob, please post here when you've got something ready to go. Project page (two tutorials): http://dsource.org/projects/orange Repository: https://github.com/jacob-carlborg/orange Documentation: http://dl.dropbox.com/u/18386187/orange_docs/orange.serialization.Serializer.html (Don't forget the "Package" tab) Unit tests are available in the "tests" directory. These unit tests are not like regular unit tests that test individual functions. These unit tests are on a higher level. The most important part to review is the "serialization" package. The rest is mostly utility modules. After I quick look: I think that all other modules/packages under the orange package should either be integrated into other existing phobos modules or as new phobos modules since they are really not serialization specific. I agree with that. Furthermore I think that all suppport for tango in the code should be stripped before going into phobos. /Jonas That is obvious. I don't know if this have been clear or not but I've requested a pre-review. A review that should be as a regular review but before I do a Phobos integration. If it gets accepted into Phobos I'll do the necessary modifications (remove all Tango related code and so on) to integrate it into Phobos and then we can have a second review. I don't want to waste time on removing the Tango support if I don't know for sure that it will be accepted. Ok that wasn't clear to me. I've updated the review queue: http://prowiki.org/wiki4d/wiki.cgi?ReviewQueue /Jonas
Re: Second Round CURL Wrapper Review
On Friday, 16 December 2011 at 09:42:50 UTC, Jonathan M Davis
wrote:
Please make sure that you remove trailing whitespace from the
file. A lot of the lines have trailing whitespace. Also, make
sure that you don't have any tabs in the file. There are a few
places where you used tabs.
ok
Line# 916 claims that the code there won't work and that it
needs to be fixed, so please fix it before it goes into Phobos
(assuming that it's voted in).
ok
On line# 1051, just do
auto arr = new Char[](bufferSize);
There's no reason to create it and then assign to its length
property.
ok
The braces in the block follow line# 1131 need to be fixed. The
indentation is wrong on some of them, and not all of them are
on their own line.
ok
You should see if the cast(void[])'s can be removed from
byLineAsync. They might still be necessary, but null was given
a type with the last release, so it might work without the cast
now. The same goes with byChunkAsync.
Still doesn't work.
Same with line# 1215 as 1051. There's no point in wasting a
line of code by declaring an array and the assigning to its
length variable. The code is probably slightly less efficient
that way too. Just allocate a new array of the correct length.
ok
On line#1297, the braces need to be fixed. Same with line# 1308
and line# 1334. Just check over your braces in general to make
sure that they're consistent and are on their own line unless
you're dealing with something like a one line lambda.
ok
And as I mentioned before, all of these enforces which take a
new CurlException, should be changed to use enforceEx after
you've fixed CurlException's constructor.
ok
The documentation on Protocol's isValid is wrong. It claims
that isValid is true if the instance is stoppend and invalid.
Wouldn't that mean that the object _wasn't_ valid?
ok
There are quite a few places where you're concatenating several
strings and format would make the code much cleaner (e.g.
Protocol's netInterface function). Unless you're avoiding
format in an attempt to allow functions to be pure (since
unfortunately, format can't currently be pure), you really
should be using format.
ok
I'd argue that it's better to use empty than check whether a
string is equal to "". e.g. domain.empty instead of domain !=
"" on line# 1538.
ok
There bodies of Http's constructor, static opCall, and dup
functions are all very similar - especially the constructor and
static opCall. You should look at refactoring those so that
they don't have to duplicate so much code. Maybe it can't be
reasonably done easily, but it would certainly be better if
that code duplication could be reduced.
ok
Change setTimeCondition to take a SysTime. DateTime is not
intended for timestamps. It's intended for calendar-based time,
not the system time. Also, SysTime makes it easier to convert
to time_t. In general, the fact that you're looking to deal
with a time_t is a sign that you should be using SysTime. This
is especially true, since DateTime is going to give the wrong
time_t in general, since it has no time zone. As it stands, the
timestamp is assumed to be in UTC. That's just begging for
bugs. Users won't expect that. Definitely make it a SysTime.
And once that's done, line# 2016 would disappear, and line#
2017 would become
p.curl.set(CurlOption.timevalue, timestamp.toUnixTime());
ok
On line# 2247, why do you use format with no format string?
There should be no ~ or to!string in a call to format. It's
just creating unnecessary memory allocations. It should be
something more like
format("%s%s(%s.", code, reason, majorVersion, minorVersion);
Though I find that '(' to be rather odd, since it has no
matching closing paren, so one may need to be added.
Regardless, that line shouldn't be concatenating strings or use
conv.to. format takes care of all of that and should do it with
fewer memory allocations.
ok - ')' was missing
Pointers really should have their * next to the type not
floating in space like on line #2274. As it is, the code looks
like a multiplication. Unlike in C/C++, the * is clearly
associated with the type. e.g.
int* a, b;
creates an int* and an int in C, but it creates int* and int*
in D. Separating the * from the type has a tendancy to make the
code harder to understand.
If that's the D style I'll do that.
Ftp's consturctor and static opCall have a code duplication
problem similar to that of Http.
ok
Make encoding take a string. Any time that you are _always_
going to idup a parameter which is a character array, make it a
string, _not_ const. That way, if it's a string, no idup is
necessary, and if it's char[], then it can be iduped when it's
passed in, and iduping only occurs when it's actually necessary.
ok
In Smtp's constructor, yo ushould probably create a variable
for the toLowered url. e.g.
?
auto lowered = url.toLower();
That way, you avoid having to lower it twice if the else branch
of the if-
Re: Second Round CURL Wrapper Review
On Saturday, 17 December 2011 at 20:01:04 UTC, Jakob Ovrum wrote: The docs for onReceiveHeader, http://freeze.steamwinter.com/D/web/phobos/etc_curl.html#onReceiveHeader explicitly says that the const string parameters are not valid after the function returns. This is all well and good; but a minor improvement would be to change the signature to: @property void onReceiveHeader(void delegate(in char[] key, in char[] value) callback); to add `scope` to the parameters. This documents the fact that the constant strings shouldn't be escaped as-is in both code and documentation. I suspect many other instances of `const(char)[]` could be changed to `in char[]` to better document how they use the parameters (I also personally think it looks better as it is more concise). I agree. Actually I thought that 'in' was just an alias of 'const' but now I see that it is actually an alias of 'const scope'.
Re: Second Round CURL Wrapper Review
On Monday, 19 December 2011 at 18:14:29 UTC, Somedude wrote: Le 19/12/2011 19:05, Somedude a écrit : Le 03/12/2011 05:26, dsimcha a écrit : I volunteered ages ago to manage the review for the second round of Jonas Drewsen's CURL wrapper. After the first round it was decided that, after a large number of minor issues were fixed, a second round would be necessary. Significant open issues: 1. Should libcurl be bundled with DMD on Windows? 2. etc.curl, std.curl, or std.net.curl? (We had a vote a while back but it was buried deep in a thread and a lot of people may have missed it: http://www.easypolls.net/poll.html?p=4ebd3219011eb0e4518d35ab ) Code: https://github.com/jcd/phobos/blob/curl-wrapper/etc/curl.d Docs: http://freeze.steamwinter.com/D/web/phobos/etc_curl.html For those of you on Windows, a libcurl binary built by DMC is available at http://gool.googlecode.com/files/libcurl_7.21.7.zip. Review starts now and ends on December 16, followed by one week of voting. A bit late, but following Andrei's important remark that some operations are blocking, I also noticed that curl-multi API is not covered. http://curl.haxx.se/libcurl/c/libcurl-multi.html Is there a reason why ? Would it break the current API to add it later ? It looks like curl_multi_perform's behavior is more interesting than curl_easy_perform. * From the tutorial: http://curl.haxx.se/libcurl/c/libcurl-tutorial.html "curl_multi_perform(3) is asynchronous. It will only execute as little as possible and then return back control to your program. It is designed to never block. If it returns CURLM_CALL_MULTI_PERFORM you better call it again soon, as that is a signal that it still has local data to send or remote data to receive. " I am aware of the multi_perform API. The multi API actually uses easy handles and therefore it would be natural to extend the wrapper with multi support later on. It will allow for read() and write() methods on the HTTP/FTP structs as well. For now I want to limit the scope and stick to the easy handles. /Jonas
Re: Second Round CURL Wrapper Review
On Saturday, 17 December 2011 at 22:25:07 UTC, Andrei
Alexandrescu wrote:
On 12/2/11 10:26 PM, dsimcha wrote:
I volunteered ages ago to manage the review for the second
round of
Jonas Drewsen's CURL wrapper. After the first round it was
decided that,
after a large number of minor issues were fixed, a second
round would be
necessary.
Significant open issues:
1. Should libcurl be bundled with DMD on Windows?
Yes.
2. etc.curl, std.curl, or std.net.curl? (We had a vote a while
back but
it was buried deep in a thread and a lot of people may have
missed it:
http://www.easypolls.net/poll.html?p=4ebd3219011eb0e4518d35ab )
std.net.curl
ok
Code:
https://github.com/jcd/phobos/blob/curl-wrapper/etc/curl.d
Docs: http://freeze.steamwinter.com/D/web/phobos/etc_curl.html
I have a few comments. They are not critical for library's
admission (apologies I didn't make the deadline) and could be
integrated before the commit.
1. Tables must be centered (this is a note to self)
2. Replace d-p-l.org in examples with dlang.org
ok
3. "Compared to using libcurl directly this module provides a
simpler API for performing common tasks." -> "Compared to using
libcurl directly this module allows simpler client code for
common uses, requires no unsafe operations, and integrates
better with the rest of the language."
ok
4. From the high level ops it seems there's no way to issue a
PUT/POST and then direct the result to a file, or read the
result with a foreach.
Use byLine/Chunk with a connection initialized with post/put data:
auto http = HTTP();
http.postData = "hello world";
foreach (line; byLine("dlang.org/there", KeepTerminator.no, '\n',
http))
writeln(line);
Same deal for directing to file using download()
5. Also there seems to be no way to issue a PUT/POST from a
range.
Correct. It will be a later extension.
6. s/Examples:/Example:/
ok
7. s/HTTP http = HTTP("www.d-p-l.org");/auto http =
HTTP("www.d-p-l.org");/
ok
8. "receives a header or a data buffer" -> "receives a header
and a data buffer, respectively"
ok
9. "In this simple example, the headers are writting to stdout
and the data is ignored." How can we make it to stop the
request? A sentence about that would be great.
ok
10. "Finally the HTTP request is started by calling perform()."
-> "Finally the HTTP request is effected by calling perform(),
which is synchronous."
ok
11. Use InferredProtocol instead of AutoConnect(ion). The type
is exactly what it claims. If "inferred" is too long,
"AutoProtocol" would be fine too. But it's not the connection
that is automated. The word must not be a verb, i.e.
InferProtocol/DeduceProtocol would not be good.
ok
12. The example for InferredProtocol nee AutoConnection does
not show any interesting example, e.g. ftp.
improved
13. I think download() and upload() are good convenience
functions but are very limited. Such operations can last
arbitrarily long and it's impossible to stop them. Therefore,
they will foster poor program design (e.g. program hangs until
you kill it etc). We should keep these for convenience in
scripts, and we already have byXxx() for downloading, but we
don't have anything for uploading. We should offer output
ranges for uploading sync/async. Maybe in the next minor
release.
Using the same method as for post it is possible to set timeouts.
auto http = HTTP();
http.dataTimeout = dur!"minutes"(2);
foreach (line; byLine("dlang.org/there", KeepTerminator.no, '\n',
http))
writeln(line);
Regarding uploading ranges this will be handled by the same
extension as
for POST/PUT output ranges.
14. Parameter doc for put(), post(), del() etc. is messed up.
ok
15. I don't think "connection" is the right term. For example
we have
T[] get(Conn = AutoConnection, T = char)(const(char)[] url,
Conn conn = Conn());
The connection is established only during the call to get(), so
conn is not really a connection - more like a sort of a bundle
of connection parameters. But then it does clean up the
connection once established, so I'm unsure...
But you can provide an existing connection to use has that is
already connected to the server. Hence the name connection.
16. For someone who'd not versed in HTTP options, the example
string content =
options("d-programming-language.appspot.com/testUrl2",
"something");
isn't terribly informative.
ok - improved
17. In byLine: "A range of lines is returned when the request
is complete." This suggests the last byte has been read as the
client gets to the first, which is not the case. "Data will be
read on demand as the foreach loop makes progress." Similar for
byChunk.
This will change when multi API is supported as well.
18. In byXxxAsync wait(Duration) should be cross-referenced to
its definition.
ok
=
Overall: I think this is a valuable addition to Phobos, but I
have the feeling we don't have a good scenario for interrupting
connections. F
Re: Happy New Year in 2012....
On Tuesday, 3 January 2012 at 02:45:20 UTC, Walter Bright wrote: On 12/31/2011 9:34 PM, Andrei Alexandrescu wrote: to the entire D community! New Zealand Tanzania Japan Russia France U.S. D is taking over zee vorld!!! Mua-hah-ha-ha! Add Denmark to the list. Happy new year!
Andrei's castle located
Map of C++ http://herbsutter.com/2012/01/10/map-of-c/ /Jonas
ref const array error
Recently the encoding.safeDecode stopped working for some of my
existing code. This example outlines the issue:
import std.encoding;
void main(string[] args) {
auto e = EncodingScheme.create("utf-8");
auto a = new byte[100];
e.safeDecode(a);
}
Results in:
Error: function std.encoding.EncodingScheme.safeDecode (ref
const(ubyte)[] s) const is not callable using argument types
(byte[])
Isn't this an error in the compiler?
/Jonas
Re: ref const array error
On Wednesday, 18 January 2012 at 19:43:52 UTC, Timon Gehr wrote:
On 01/18/2012 08:31 PM, jdrewsen wrote:
Recently the encoding.safeDecode stopped working for some of
my existing
code. This example outlines the issue:
import std.encoding;
void main(string[] args) {
auto e = EncodingScheme.create("utf-8");
auto a = new byte[100];
e.safeDecode(a);
}
Results in:
Error: function std.encoding.EncodingScheme.safeDecode (ref
const(ubyte)[] s) const is not callable using argument types
(byte[])
Isn't this an error in the compiler?
/Jonas
No, this is a bugfix. The operation is unsound:
immutable(ubyte)[] foo(ref const(ubyte)[] s){
auto r = new immutable(ubyte)[1];
s = r;
return r;
}
void main() {
ubyte[] x;
immutable(ubyte)[] y = foo(x);
static assert(is(typeof(y[0])==immutable));
auto oldy0 = y[0];
x[0]=oldy0+1;
assert(oldy0 == y[0]); // fail
}
The functionality is not going away; You will be able to use
inout for the same purpose once my enhancement request gets
implemented: http://d.puremagic.com/issues/show_bug.cgi?id=7105
Wouldn't a nicer solution be to let the compiler ensure that
an immutable array cannot escape through a ref const array
parameter?
/Jonas
Re: ref const array error
On Wednesday, 18 January 2012 at 20:13:04 UTC, Timon Gehr wrote:
On 01/18/2012 08:59 PM, jdrewsen wrote:
On Wednesday, 18 January 2012 at 19:43:52 UTC, Timon Gehr
wrote:
On 01/18/2012 08:31 PM, jdrewsen wrote:
Recently the encoding.safeDecode stopped working for some of
my existing
code. This example outlines the issue:
import std.encoding;
void main(string[] args) {
auto e = EncodingScheme.create("utf-8");
auto a = new byte[100];
e.safeDecode(a);
}
Results in:
Error: function std.encoding.EncodingScheme.safeDecode (ref
const(ubyte)[] s) const is not callable using argument types
(byte[])
Isn't this an error in the compiler?
/Jonas
No, this is a bugfix. The operation is unsound:
immutable(ubyte)[] foo(ref const(ubyte)[] s){
auto r = new immutable(ubyte)[1];
s = r;
return r;
}
void main() {
ubyte[] x;
immutable(ubyte)[] y = foo(x);
static assert(is(typeof(y[0])==immutable));
auto oldy0 = y[0];
x[0]=oldy0+1;
assert(oldy0 == y[0]); // fail
}
The functionality is not going away; You will be able to use
inout for
the same purpose once my enhancement request gets implemented:
http://d.puremagic.com/issues/show_bug.cgi?id=7105
Wouldn't a nicer solution be to let the compiler ensure that
an immutable array cannot escape through a ref const array
parameter?
/Jonas
That would not suffice.
ubyte[] foo(ref const(ubyte)[] s){
auto r = new ubyte[1];
s = r;
return r;
}
void main() {
immutable(ubyte)[] x;
ubyte[] y = foo(x);
static assert(is(typeof(x[0])==immutable));
auto oldx0 = x[0];
y[0]=oldx0+1;
assert(oldx0 == x[0]); // fail
}
In the example foo is actually using the ref s parameter as an
out parameter. The compiler could catch that you're doing this
and show an error.
This would force you to let foo look like:
ubyte[] foo(out const(ubyte)[] s);
Wouldn't that fix it?
Re: ref const array error
On Wednesday, 18 January 2012 at 23:09:56 UTC, Timon Gehr wrote:
On 01/18/2012 10:12 PM, jdrewsen wrote:
On Wednesday, 18 January 2012 at 20:13:04 UTC, Timon Gehr
wrote:
On 01/18/2012 08:59 PM, jdrewsen wrote:
On Wednesday, 18 January 2012 at 19:43:52 UTC, Timon Gehr
wrote:
On 01/18/2012 08:31 PM, jdrewsen wrote:
Recently the encoding.safeDecode stopped working for some
of my
existing
code. This example outlines the issue:
import std.encoding;
void main(string[] args) {
auto e = EncodingScheme.create("utf-8");
auto a = new byte[100];
e.safeDecode(a);
}
Results in:
Error: function std.encoding.EncodingScheme.safeDecode (ref
const(ubyte)[] s) const is not callable using argument
types (byte[])
Isn't this an error in the compiler?
/Jonas
No, this is a bugfix. The operation is unsound:
immutable(ubyte)[] foo(ref const(ubyte)[] s){
auto r = new immutable(ubyte)[1];
s = r;
return r;
}
void main() {
ubyte[] x;
immutable(ubyte)[] y = foo(x);
static assert(is(typeof(y[0])==immutable));
auto oldy0 = y[0];
x[0]=oldy0+1;
assert(oldy0 == y[0]); // fail
}
The functionality is not going away; You will be able to
use inout for
the same purpose once my enhancement request gets
implemented:
http://d.puremagic.com/issues/show_bug.cgi?id=7105
Wouldn't a nicer solution be to let the compiler ensure that
an immutable array cannot escape through a ref const array
parameter?
/Jonas
That would not suffice.
ubyte[] foo(ref const(ubyte)[] s){
auto r = new ubyte[1];
s = r;
return r;
}
void main() {
immutable(ubyte)[] x;
ubyte[] y = foo(x);
static assert(is(typeof(x[0])==immutable));
auto oldx0 = x[0];
y[0]=oldx0+1;
assert(oldx0 == x[0]); // fail
}
In the example foo is actually using the ref s parameter as an
out
parameter. The compiler could catch that you're doing this and
show an
error.
This would force you to let foo look like:
ubyte[] foo(out const(ubyte)[] s);
Wouldn't that fix it?
If it is ref or out is irrelevant for the example, so how would
this fix anything? The compiler could, in principle, treat
const similarly to inout (just without the context sensitivity
and parameter matching etc) for 'ref' parameters and do all the
type checking at the call site. However, that would then
restrict what the callee can do and introduce a strange special
case. inout is the way to go.
But in the example you're using s as an out parameter and that
should trigger the error I got originally of course. But if ref
parameters were disallowed to be used as out parameters the
compiler would catch the error in your example wouldn't it?
Re: ref const array error
On Thursday, 19 January 2012 at 17:00:59 UTC, Timon Gehr wrote:
On 01/19/2012 03:47 PM, jdrewsen wrote:
On Wednesday, 18 January 2012 at 23:09:56 UTC, Timon Gehr
wrote:
On 01/18/2012 10:12 PM, jdrewsen wrote:
On Wednesday, 18 January 2012 at 20:13:04 UTC, Timon Gehr
wrote:
On 01/18/2012 08:59 PM, jdrewsen wrote:
On Wednesday, 18 January 2012 at 19:43:52 UTC, Timon Gehr
wrote:
On 01/18/2012 08:31 PM, jdrewsen wrote:
Recently the encoding.safeDecode stopped working for
some of my
existing
code. This example outlines the issue:
import std.encoding;
void main(string[] args) {
auto e = EncodingScheme.create("utf-8");
auto a = new byte[100];
e.safeDecode(a);
}
Results in:
Error: function std.encoding.EncodingScheme.safeDecode
(ref
const(ubyte)[] s) const is not callable using argument
types
(byte[])
Isn't this an error in the compiler?
/Jonas
No, this is a bugfix. The operation is unsound:
immutable(ubyte)[] foo(ref const(ubyte)[] s){
auto r = new immutable(ubyte)[1];
s = r;
return r;
}
void main() {
ubyte[] x;
immutable(ubyte)[] y = foo(x);
static assert(is(typeof(y[0])==immutable));
auto oldy0 = y[0];
x[0]=oldy0+1;
assert(oldy0 == y[0]); // fail
}
The functionality is not going away; You will be able to
use inout
for
the same purpose once my enhancement request gets
implemented:
http://d.puremagic.com/issues/show_bug.cgi?id=7105
Wouldn't a nicer solution be to let the compiler ensure
that
an immutable array cannot escape through a ref const array
parameter?
/Jonas
That would not suffice.
ubyte[] foo(ref const(ubyte)[] s){
auto r = new ubyte[1];
s = r;
return r;
}
void main() {
immutable(ubyte)[] x;
ubyte[] y = foo(x);
static assert(is(typeof(x[0])==immutable));
auto oldx0 = x[0];
y[0]=oldx0+1;
assert(oldx0 == x[0]); // fail
}
In the example foo is actually using the ref s parameter as
an out
parameter. The compiler could catch that you're doing this
and show an
error.
This would force you to let foo look like:
ubyte[] foo(out const(ubyte)[] s);
Wouldn't that fix it?
If it is ref or out is irrelevant for the example, so how
would this
fix anything? The compiler could, in principle, treat const
similarly
to inout (just without the context sensitivity and parameter
matching
etc) for 'ref' parameters and do all the type checking at the
call
site. However, that would then restrict what the callee can
do and
introduce a strange special case. inout is the way to go.
But in the example you're using s as an out parameter and that
should
trigger the error I got originally of course. But if ref
parameters were
disallowed to be used as out parameters the compiler would
catch the
error in your example wouldn't it?
The compiler would catch something in my example, but not the
error. The error happens at the call site.
Actually when I think about it using semantic for out/ref that I
suggested does not solve the problem at all. I guess the inout
thing is the way to go.
A pitty that safeDecode only supports the ref const(ubyte)[]
version and not ref ubyte[]. I wonder how big of an issue this is
for the rest of phobos functions accepting ref parameters.
/Jonas
Re: Message-Passing
On Friday, 20 January 2012 at 13:10:37 UTC, Manu wrote: On 20 January 2012 00:36, Sean Kelly wrote: Thanks :-) If you have ideas on how it could be improved, please let me know. On Jan 19, 2012, at 12:58 PM, Nathan M. Swan wrote: > I want to applaud Sean Kelly and everyone who worked on > std.concurrency for a great API, and wish that I could easily write Cocoa applications with it. > > I'm writing a screen recording program in Objective-C, and > to make sure each frame has an equal length, I have two threads: one that takes the screenshot at certain intervals, the other that assembles it into a quicktime movie. After writing an implementation of a thread-safe queue, where the picture-taking thread adds and the assembling thread removes, I realized the huge similarity between what I'd done and std.concurrency. > > The std.concurrency module would have made hours of effort > in ObjC take five minutes in D. Well done! I had some troubles with std.concurrency which I thought it might be nice to address. Perhaps the most major problem I had was related to the concept of thread ownership. If a spawned threads parent thread dies, it also receives a signal to kill its self, but it seems impossible to reassign ownership. In my case I had threads A, B and C... Thread A is the main thread, which may spawn many temporary worker threads B, which may last 1-2 seconds. During the life of B, it may spawn C, which may persist for hours. B promptly dies, and any C's that were spawned receive the kill signal, which I ignore. Thread A, the main thread may exit, and I would really like all C's to receive that notification, but they were are all orphaned when their B died. The problem is, when I spawn a C, it should be created as a child of A somehow, rather than a child of the transient B... Some mechanism to solve this sort of problem would be useful. Another usability problem I had was that, intuitively, the simpler function with intuitive usage pattern receiveOnly() should be named receive(). And receive() with the complex var-arg list of delegates should be named something more complex. receiveTimeout() has no receiveTimeoutOnly(), which is the function I almost always want to use... and receiveTimeout() didn't actually work for me anyway (it wouldn't receive a duration as per the documentation, I had to pass an int) I wonder if there is a solution to the 'shared' problem. Basically every single line of code that uses the std.concurrenty api is riddled with casts to/from shared... really ugly. I think the problem here is that I'm not SHARING values between threads, I'm actually passing ownership of something to another thread. So I wonder if some improvement can be made in this area. I was going to complain about the documentation, but I just checked, and it seems to have had work since I was reading it. Looks much better now! :) It still needs to document that the ops in "void receive(T...)(T ops)" can return false if they did not handle the message. This is useful when you want to receive a message from a specific spawned subthread (tid). /Jonas
Bug tracking and assigned to
So I've been thinking about helping out with fixing some bugs in dmd. I made search in the bug tracker for bugs with NEW state but quickly noticed that a lot of them are actually already being worked on or have a pending pull request. It would be great if the state was changed to ASSIGNED. It would also make the stats a bit nicer to look at I guess. I don't have the solution to this problem... just a rant :) /Jonas
Re: Bug tracking and assigned to
On Friday, 20 January 2012 at 22:42:59 UTC, Nick Sabalausky wrote: "jdrewsen" wrote in message news:ybillabaoehmexgjo...@dfeed.kimsufi.thecybershadow.net... So I've been thinking about helping out with fixing some bugs in dmd. I made search in the bug tracker for bugs with NEW state but quickly noticed that a lot of them are actually already being worked on or have a pending pull request. It would be great if the state was changed to ASSIGNED. It would also make the stats a bit nicer to look at I guess. I don't have the solution to this problem... just a rant :) If you're still looking for DMD bugs to work on, I've got a few I can [selfishly] suggest :) Yes still looking. But I'm just starting to dig into the dmd source so I guess it shouldn't be the tough bugs to start with.
Re: Review of Jose Armando Garcia Sancio's std.log
A first quick observation:
I vote for a debug severity level. Then make that default to the
template parameter for log:
template log(Severity severity = Severity.debug)
That would make it nice for good old print debugging.
log("This is a dbg message");
/Jonas
Re: Review of Jose Armando Garcia Sancio's std.log
On Tuesday, 14 February 2012 at 02:28:11 UTC, Jose Armando Garcia
wrote:
On Mon, Feb 13, 2012 at 6:44 PM, jdrewsen
wrote:
A first quick observation:
I vote for a debug severity level. Then make that default to
the template
parameter for log:
template log(Severity severity = Severity.debug)
That would make it nice for good old print debugging.
log("This is a dbg message");
I like the idea of having a default. Not sure about adding
debug. What
are you trying to do with default that log!info and vlog(#)
doesn't
let you do?
As Sean mentioned the vlog function may be the one I want. Maybe
it is okey not to have a debug severity but then a default on the
vlog level parameter would be nice. That would make quick debug
prints a tad simpler
vlog("This is a dbg message");
I know this is a small thing... but to prevent death by a 1000
cuts.
/Jonas
Re: Review of Jose Armando Garcia Sancio's std.log
On Monday, 13 February 2012 at 15:50:05 UTC, David Nadlinger wrote: There are several modules in the review queue right now, and to get things going, I have volunteered to manage the review of Jose's std.log proposal. Barring any objections, the review period starts now and ends in three weeks, on March 6th, followed by a week of voting. --- Code: https://github.com/jsancio/phobos/commit/d114420e0791c704f6899d81a0293cbd3cc8e6f5 Docs: http://jsancio.github.com/phobos/phobos/std_log.html Known remaining issues: - Proof-reading of the docs is required. - Not yet fully tested on Windows. Depends on: https://github.com/D-Programming-Language/druntime/pull/141 (will be part of 2.058) --- Earlier drafts of this library were discussed last year, just search the NG and ML archives for "std.log". I think getting this right is vitally important so that we can avoid an abundance of partly incompatible logging libraries like in Java. Thus, I'd warmly encourage everyone to actively try out the module or compare it with any logging solution you might already be using in your project. Please post all feedback in this thread, and remember: Although comprehensive reviews are obviously appreciated, short comments are very welcome as well! David Updated wiki: http://prowiki.org/wiki4d/wiki.cgi?ReviewQueue /Jonas
Re: Review of Jose Armando Garcia Sancio's std.log
On Tuesday, 14 February 2012 at 17:48:06 UTC, so wrote:
On Tuesday, 14 February 2012 at 16:21:42 UTC, Jose Armando
Garcia wrote:
On Tue, Feb 14, 2012 at 8:42 AM, jdrewsen
wrote:
On Tuesday, 14 February 2012 at 02:28:11 UTC, Jose Armando
Garcia wrote:
On Mon, Feb 13, 2012 at 6:44 PM, jdrewsen
wrote:
A first quick observation:
I vote for a debug severity level. Then make that default
to the template
parameter for log:
template log(Severity severity = Severity.debug)
That would make it nice for good old print debugging.
log("This is a dbg message");
I like the idea of having a default. Not sure about adding
debug. What
are you trying to do with default that log!info and vlog(#)
doesn't
let you do?
As Sean mentioned the vlog function may be the one I want.
Maybe it is okey
not to have a debug severity but then a default on the vlog
level parameter
would be nice. That would make quick debug prints a tad
simpler
If we do set a default what should it be? It is not clear to
me what
value we should pick so if you have any suggestions let me
know.
IMO a default severity level is not a good idea, not explicit
to begin with.
As i suggested on another reply, getting rid of the
instantiations solve it.
We lose nothing and gain a common keyword. I used to have
severity levels for my logging library in c++. As soon as i got
the C++0x options i thrashed them all.
Now instead of:
txt(error) << "this is: " << it;
i just got:
error("this is: ", it);
Win win for every aspect of it. And got rid of the keyword
"txt".
I also like your proposal better. In the end I just want a simple
to type function call syntax or it will be tempting to fallback
to writeln(...).
/Jonas
Re: Review of Jose Armando Garcia Sancio's std.log
On Tuesday, 14 February 2012 at 16:12:57 UTC, Jose Armando Garcia
wrote:
On Mon, Feb 13, 2012 at 1:50 PM, David Nadlinger
wrote:
There are several modules in the review queue right now, and
to get things
going, I have volunteered to manage the review of Jose's
std.log proposal.
Barring any objections, the review period starts now and ends
in three
weeks, on March 6th, followed by a week of voting.
---
Code:
https://github.com/jsancio/phobos/commit/d114420e0791c704f6899d81a0293cbd3cc8e6f5
Docs: http://jsancio.github.com/phobos/phobos/std log.html
Updated the document an implementation to reflect every suggest
for I
replied to with the exception of providing a format parameter
for
thread name. Should have that ready later today. Let me know if
I
missed anything. API changes:
1. Dropped log!info("message"), etc. Use info("message), etc.
2. opCall now alias to format. Ie. info("Format %s message",
Severity.info).
this means that to concatenate strings you need to
info.write("Hello ", "world");
In the introduction text the references to Configuration etc.
should be made into links.
/Jonas
Re: Review of Jose Armando Garcia Sancio's std.log
On Tuesday, 14 February 2012 at 18:51:20 UTC, Jose Armando Garcia
wrote:
On Tue, Feb 14, 2012 at 4:44 PM, jdrewsen
wrote:
On Tuesday, 14 February 2012 at 16:12:57 UTC, Jose Armando
Garcia wrote:
On Mon, Feb 13, 2012 at 1:50 PM, David Nadlinger
wrote:
There are several modules in the review queue right now, and
to get
things
going, I have volunteered to manage the review of Jose's
std.log
proposal.
Barring any objections, the review period starts now and
ends in three
weeks, on March 6th, followed by a week of voting.
---
Code:
https://github.com/jsancio/phobos/commit/d114420e0791c704f6899d81a0293cbd3cc8e6f5
Docs: http://jsancio.github.com/phobos/phobos/std log.html
Updated the document an implementation to reflect every
suggest for I
replied to with the exception of providing a format parameter
for
thread name. Should have that ready later today. Let me know
if I
missed anything. API changes:
1. Dropped log!info("message"), etc. Use info("message), etc.
2. opCall now alias to format. Ie. info("Format %s message",
Severity.info).
this means that to concatenate strings you need to
info.write("Hello ", "world");
In the introduction text the references to Configuration etc.
should be made
into links.
I would love to do that. How do you do that with ddoc?
For example: $(LREF Configuration)
The LREF macro is defined in
https://github.com/D-Programming-Language/d-programming-language.org/blob/master/std.ddoc#L293
along with other useful macros.
/Jonas
receiveOnly behavior
import std.concurrency;
void main(string[] args) {
receiveOnly!bool();
}
This simple program results in segmentation fault. I know i does not
spawn a thread to receive anything from but is this the expected behavior?
/Jonas
Curl wrapper
Hi, I've been working on a wrapper for the etc.c.curl module. It is now pretty stable and I would very much like some feedback on the API. http://freeze.steamwinter.com/D/web/phobos/etc_curl.html BTW I use template mixins which doesn't seem to get included in the generated docs. Is there any way I can make this work? /Jonas
Re: Curl wrapper
Please see comments below.
Den 17-05-2011 16:42, Andrei Alexandrescu skrev:
Thanks for the response! A few more answers and comments within
(everything deleted counts as "sounds great").
On 5/17/11 3:50 AM, Jonas Drewsen wrote:
14. Isn't the max redirect configurable via a parameter?
Yes. It is called Http.followLocation from libcurls followLocation
option. I will rename it to maxRedirections for clearity.
You mention this about many higher-level functions: "Maximum
redirections are set to 10." Then I'm thinking, if it's an important
parameter, make it a defaulted parameter for the function.
I guess I should just remove that comment because other defaults has
been selected as well e.g. timeouts.
16. Documentation should point to descriptions of the HTTP methods
wrapped (e.g. "post", "del" etc).
Do you mean point to the relevant RFC?
Yah, or a good tutorial. (I didn't know DEL existed!)
Well DEL is actually called DELETE in the HTTP RFC. But delete is a
reserved word i D so I used del() instead. delete() wouldn't work in
following code in think:
with(auto http = ...) {
delete(...);
}
22. byLine/byChunk should not expose a string or generally data that can
be shared safely by the client. That's because it would need to create a
new string with each iteration, which is very inefficient. Instead, they
should expose char[]/ubyte[] just like File.byLine does, and reuse the
buffer with each call. Essentially byLine/byChunk should be
near-zero-overhead means to transfer and inspect arbitrarily large
streams.
byLine/byChunk is currenly only available for the async methods which is
implemented using message passing. This means that for passing the
message a copy has to be made because a message is by-value or immutable
data. Is there another way for me to pass the message without doing a
copy... some kind of move semantic?
A library element is planned but not on the short list yet. You can work
around that by internally doing a cast. As long as you observe the
commitment that buffers are not accessed simultaneously by more than one
thread you're well within defined behavior.
ok nice to know.
One more suggestion - you may want to also provide classic boring
synchronous read/write methods that take a user-provided buffer. I'm
sure people could use such e.g. when they want to stream data inside
their own threads, or even in the main thread if they don't mind blocking.
This would mean hooking into libcurl and selecting on its sockets.
Totally doable. But I would rather stop here feature wise and wrap this
thing up. I would like to focus my efforts on a candidate for
std.net.event/reactor/proactor module.
Anyone who wants to implement this read/write in the curl wrapper are
ofcourse very welcome.
24. The shutdown mechanism should be handled properly. Shutting down
libcurl would have all pending transfers instantly throw a special
exception. Without a shutdown API, applications won't be able to
implement e.g. a Quit button.
If you're only using this API this should be handled. But I see that it
is not documented.
Yah, in the future we need to put an onShutdown protocol inside core,
similar to atexit() in C. Each library that may block would plant a
hook. That way an application will be able to exit quickly and
gracefully even if it has blocked threads.
That would be neat.
/Jonas
Re: Curl wrapper
Den 17-05-2011 16:57, Robert Jacques skrev:
On Tue, 17 May 2011 07:24:16 -0400, Jonas Drewsen
wrote:
On 17/05/11 08.03, Robert Jacques wrote:
On Mon, 16 May 2011 17:07:57 -0400, jdrewsen
wrote:
Hi,
I've been working on a wrapper for the etc.c.curl module. It is now
pretty stable and I would very much like some feedback on the API.
http://freeze.steamwinter.com/D/web/phobos/etc_curl.html
BTW I use template mixins which doesn't seem to get included in the
generated docs. Is there any way I can make this work?
/Jonas
[snip]
I don't know what ditto does. I'll try to look at the ddoc documentation.
"If a documentation comment for a declaration consists only of the
identifier ditto then the documentation comment for the previous
declaration at the same declaration scope is applied to this declaration
as well."
[snip]
http.onReceiveHeader has issues with it's 'See' link. It's example is
too wide. You don't need really need the parameter documentation,
particularly if you improve the example, i.e.:
string[string] headerInformation;
with(auto http = new Http("http://www.google.com";)) {
onReceiveHeader = (string key, string value) {
if ( value.length <= 10 )
headerInformation[key.idup] = value.idup;
};
onReceive = (ubyte[] data) { };
perform;
}
I see your point. The question is how we would like phobos docs to be:
1, consistent with all parameters documented for each method and each
method explained and exemplified
2, minimal with examples and explanations where absolutely needed.
or somewhere in between. The current style if the curl wrapper is
closer to 1 than 2.
Anything official about this?
Official? I don't know. But looking over std.algorithm, Param: is only
used 3 times, for each of the find variants, in order to clarify what
haystack and needle are and in one case make the
limitations/requirements on haystack/needle explicit. Looking over the
rest of phobos, I see a mix of styles, depending on the author, and some
things (like datetime) seem to do both.
[snip]
CurlTimeCond is not calld Http.TimeCondition because it comes from a
etc.c.curl. I could alias it in Http ofcourse.
I think where it's defined is less of an issue than where the
documentation is.
[snip]
HttpResult.text should have huge warning flags with regard to use and/or
have its design re-thought. Is there some reason you're not at least
caching the result and throwing errors on invalid uses of content, etc.
Calling text multiple times should be okay. And calling content after
text should be either logically valid, or throw.
It is mostly based on storage concerns. Initially the data is stored
as ubyte[] and can be returned by content(). When calling text() to
get a string it will be decoded and returned. I now have a coule of
options:
1, to store the string in the result for future calls to text(). I can
do that and keep the original data for future calls to content(). This
will double the amount of memory needed.
2, generated the string on each call to text(). This way no copy of
the data is kept in HttpResult.
3, generated and store the string and then throw away the original
data. Again no copy is kept in HttpResult.
I've chosen the 2. version since I guessed that once you have called
text() it is more likely that you'll call text again and not content().
Hmm... I take it that content and text are going to be large enough, and
result long lived enough to warrant not having the array cached inside
result. In that case you might want to rename text to toString or
decode. 'text' feels like it should be cheaply reusable, while
'toString'/'decode' doesn't. (this is a verb/noun thing) Also,
'toString' tends to compose better with the rest of D while 'decode' is
similar to the std.utf functions. Also, if memory re-use is a concern, a
'toStringInPlace' or 'decodeInPlace' with the old behavior might be
appropriate, although I might recommend setting content to the utf-8
string and updating the encodingSchemeName as appropriate.
I think I'll go for the toString() version.
/Jonas
Re: Curl wrapper
Den 18-05-2011 14:52, Johannes Pfau skrev:
Jonas Drewsen wrote:
On 18/05/11 10.09, Johannes Pfau wrote:
jdrewsen wrote:
Please see comments below.
Den 17-05-2011 16:42, Andrei Alexandrescu skrev:
Thanks for the response! A few more answers and comments within
(everything deleted counts as "sounds great").
On 5/17/11 3:50 AM, Jonas Drewsen wrote:
14. Isn't the max redirect configurable via a parameter?
Yes. It is called Http.followLocation from libcurls followLocation
option. I will rename it to maxRedirections for clearity.
You mention this about many higher-level functions: "Maximum
redirections are set to 10." Then I'm thinking, if it's an
important parameter, make it a defaulted parameter for the
function.
I guess I should just remove that comment because other defaults has
been selected as well e.g. timeouts.
16. Documentation should point to descriptions of the HTTP
methods wrapped (e.g. "post", "del" etc).
Do you mean point to the relevant RFC?
Yah, or a good tutorial. (I didn't know DEL existed!)
Well DEL is actually called DELETE in the HTTP RFC. But delete is a
reserved word i D so I used del() instead. delete() wouldn't work in
following code in think:
with(auto http = ...) {
delete(...);
}
22. byLine/byChunk should not expose a string or generally data
that can be shared safely by the client. That's because it would
need to create a new string with each iteration, which is very
inefficient. Instead, they should expose char[]/ubyte[] just like
File.byLine does, and reuse the buffer with each call.
Essentially byLine/byChunk should be near-zero-overhead means to
transfer and inspect arbitrarily large streams.
byLine/byChunk is currenly only available for the async methods
which is implemented using message passing. This means that for
passing the message a copy has to be made because a message is
by-value or immutable data. Is there another way for me to pass
the message without doing a copy... some kind of move semantic?
A library element is planned but not on the short list yet. You can
work around that by internally doing a cast. As long as you observe
the commitment that buffers are not accessed simultaneously by more
than one thread you're well within defined behavior.
ok nice to know.
One more suggestion - you may want to also provide classic boring
synchronous read/write methods that take a user-provided buffer.
I'm sure people could use such e.g. when they want to stream data
inside their own threads, or even in the main thread if they don't
mind blocking.
This would mean hooking into libcurl and selecting on its sockets.
Totally doable.
I'm not sure if it'd be that easy. Using select to implement a
blocking api is possible, but select is only used to wait until data
is ready, reading and writing the data is still done by curl. And as
curl does not allow you to supply the data buffer, having
user-provided buffers is afaik impossible.
_Doable_ - not easy :)
Select will wait for data to be ready and ask curl to handle the data
chunk. Curl in turn calls back to a registered callback handler with
the data read. That handler fills the buffer provided by the user. If
not enough data has been receive an new select is performed until the
requested amount of data is read. Then the blocking method can return.
I think it's not necessary to do multiple calls if the first didn't
return enough data. Most read calls are only guaranteed to return a
maximum of data, less is always possible. It's a bigger problem if
the user supplied buffer is smaller than the curl buffer. And you
always end up copying data. So it might be better
to just return a reference to the buffer allocated by curl, or just
forget about this api, curl just doesn't work well this way.
Yeah. That's part of the reason why I'm not interested in implementing it.
But I would rather stop here feature wise and wrap
this thing up. I would like to focus my efforts on a candidate for
std.net.event/reactor/proactor module.
Have you already started working on std.net.event? Do you plan to
base that on libev / libevent?
I'll finish the curl wrapper before starting on it.
I have had a look at libev/event before and they are very nice libs.
But I think I'll have to implement it from scratch for two reasons:
1, AFAIK we cannot include code in std phobos that is not boost
licensed or a license as liberal. libev for example requires you to
distribute the license with your software.
That could indeed be a problem. I think libev/libevent even have the
same license. Probably it's really best to implement it from scratch,
but it will be quite some work. I'd still have a look at the libev
documentation though, it contains some details about OS quirks. And try
to avoid Linux AIO ;-)
2, It introduces a dependency to an external project in phobos.
Curr
Re: Curl wrapper
Den 18-05-2011 16:53, Andrei Alexandrescu skrev: On 5/18/11 6:07 AM, Jonas Drewsen wrote: Select will wait for data to be ready and ask curl to handle the data chunk. Curl in turn calls back to a registered callback handler with the data read. That handler fills the buffer provided by the user. If not enough data has been receive an new select is performed until the requested amount of data is read. Then the blocking method can return. Perhaps this would be too complicated. In any case the core functionality must be paid top attention. And the core functionality is streaming. Currently there are two proposed ways to stream data from an HTTP address: (a) by using the onReceive callback, and (b) by using byLine/byChunk. If either of these perform slower than the best-of-the-breed streaming using libcurl, we have failed. The onReceive method is not particularly appealing because the client and libcurl block each other: the client is blocked while libcurl is waiting for data, and the client blocks libcurl while inside the callback. (Please correct me if I'm wrong.) To make byLine/byChunk fast, the basic setup should include a hidden thread that does the download in separation from the client's thread. There should be K buffers allocated (K = 2 to e.g. 10), and a simple protocol for passing the buffers back and forth between the client thread and the hidden thread. That way, in the quiescent state, there is no memory allocation and either both client and libcurl are busy doing work, or one is much slower than the other, which waits. The same mechanism should be used in byChunkAsync or byFileAsync. If byChunk is using a hidden thread to download into buffers, then how does it differ from the byChunkAsync that you mention? The current curl wrapper actually does the hidden thread trick (based on a hint you gave me a while ago). It does not reuse buffers because I thought that all data had to be immutable or by value to go through the message passing system. I'll fix this since it is a good place to do some type casting to allow passing the buffers for reuse. I think that we have to consider the context of the streaming before we can tell the best solution. I do not have any number to back the following up, but this is how I see it: If data that is read is going to be processed (e.g. compressed) in some way it is most likely a benefit to spawn a thread to handle the data buffering. If no processing is done (e.g. a simple copy from net to disk) I believe keeping things in the same thread and simply select on sockets (disk or net) is fastest. This way no message passing and context switching is taking place and does cause any overhead. libcurl can give you access to the file descriptors for this exact purpose but it does have some drawbacks: you are not in control of the buffers used by libcurl. This means that reading from one curl connection and sending on another you would have to copy the data. libcurl does in fact provide even simpler methods where you can provide your own buffers for read/writes. Unfortunately this is only supported for HTTP and a lot of the convenience features such as redirections are lost. The more you want to control to get the last drop of performance, the more you have to manually handle yourself. In my opinion I think that providing the performance of the standard libcurl API in the D wrapper is the way to go (as done in the current curl wrapper). Generic and efficient streaming across protocols is best done in std.net where buffers can be handled entirely in D. I know this is not a small task which is why I started out with wrapping libcurl. Thanks Jonas
Re: libcurl
Den 18-05-2011 19:59, Walter Bright skrev: On 5/18/2011 10:50 AM, Andrei Alexandrescu wrote: I sat down to write an announcement to the libcurl mailing list that we have support for it starting with 2.053. To my surprise, when I tried to provide a link to the documentation, there wasn't any available. Furthermore, when I tried to generate documentation it came messed up. Obviously ddoc had never been run against the module. What should we do? Fix this or wait until Jonas' D native bindings come up? Fix it now. If there's anything I need to do then let me hear. /Jonas
Re: Curl wrapper
Den 19-05-2011 00:54, Andrei Alexandrescu skrev: On 5/18/11 5:29 PM, jdrewsen wrote: Den 18-05-2011 16:53, Andrei Alexandrescu skrev: On 5/18/11 6:07 AM, Jonas Drewsen wrote: Select will wait for data to be ready and ask curl to handle the data chunk. Curl in turn calls back to a registered callback handler with the data read. That handler fills the buffer provided by the user. If not enough data has been receive an new select is performed until the requested amount of data is read. Then the blocking method can return. Perhaps this would be too complicated. In any case the core functionality must be paid top attention. And the core functionality is streaming. Currently there are two proposed ways to stream data from an HTTP address: (a) by using the onReceive callback, and (b) by using byLine/byChunk. If either of these perform slower than the best-of-the-breed streaming using libcurl, we have failed. The onReceive method is not particularly appealing because the client and libcurl block each other: the client is blocked while libcurl is waiting for data, and the client blocks libcurl while inside the callback. (Please correct me if I'm wrong.) To make byLine/byChunk fast, the basic setup should include a hidden thread that does the download in separation from the client's thread. There should be K buffers allocated (K = 2 to e.g. 10), and a simple protocol for passing the buffers back and forth between the client thread and the hidden thread. That way, in the quiescent state, there is no memory allocation and either both client and libcurl are busy doing work, or one is much slower than the other, which waits. The same mechanism should be used in byChunkAsync or byFileAsync. If byChunk is using a hidden thread to download into buffers, then how does it differ from the byChunkAsync that you mention? Sorry, byChunkAsync and byLineAsync (which I wrongly denoted as byFileAsync) would be methods File. The current curl wrapper actually does the hidden thread trick (based on a hint you gave me a while ago). It does not reuse buffers because I thought that all data had to be immutable or by value to go through the message passing system. I'll fix this since it is a good place to do some type casting to allow passing the buffers for reuse. Great, thanks. Don't forget there's great motivation to do so. I think that we have to consider the context of the streaming before we can tell the best solution. I do not have any number to back the following up, but this is how I see it: If data that is read is going to be processed (e.g. compressed) in some way it is most likely a benefit to spawn a thread to handle the data buffering. If no processing is done (e.g. a simple copy from net to disk) I believe keeping things in the same thread and simply select on sockets (disk or net) is fastest. Not at all. If operating with the network and operating with the disk block each other, you're guaranteed to be slower than the slowest of them. Consider that disk speed is V1 MB/s and network speed is V2 MB/s, and that they're independent of each other. If you do one thing at a time, you need to take 1/V1 + 1/V2 seconds to transfer one MB. The speed of the process is therefore 1/(1/V1 + 1/V2) = V1 * V2 / (V1 + V2). > If the two devices have comparable speeds, you're halving the speed. As soon as you do buffering with two threads you can easily reach close to the minimum of the two speeds, which is the theoretical best. It see your point. By buffering data asynchronously the reads and writes don't block each other and this increases performance. The thing is that the OS already does buffering for us. So while we're writing data to disk the OS is buffering incoming data from the network asynchronously. In my opinion I think that providing the performance of the standard libcurl API in the D wrapper is the way to go (as done in the current curl wrapper). Generic and efficient streaming across protocols is best done in std.net where buffers can be handled entirely in D. I know this is not a small task which is why I started out with wrapping libcurl. Sounds reasonable, although if you take care of recycling the buffers in the implementation you have, your wrapper may as well be the best of breed. Andrei
Re: libcurl
Den 19-05-2011 00:46, Andrei Alexandrescu skrev: On 5/18/11 5:34 PM, jdrewsen wrote: Den 18-05-2011 19:59, Walter Bright skrev: On 5/18/2011 10:50 AM, Andrei Alexandrescu wrote: I sat down to write an announcement to the libcurl mailing list that we have support for it starting with 2.053. To my surprise, when I tried to provide a link to the documentation, there wasn't any available. Furthermore, when I tried to generate documentation it came messed up. Obviously ddoc had never been run against the module. What should we do? Fix this or wait until Jonas' D native bindings come up? Fix it now. If there's anything I need to do then let me hear. /Jonas Would be great if you found the time to make a pull request with fixed documentation for etc/c/curl. Generate the html document however you wish and I'll take care of integrating it with the general template etc. Thanks, Andrei It seems to be on digitalmars.com now so I guess you did it yourself :) /Jonas
Re: libcurl
Den 19-05-2011 22:50, Andrei Alexandrescu skrev: On 5/19/11 3:28 PM, jdrewsen wrote: [snip] Speaking of which, what's the status on recycling them buffers? :o) I'm slowly going through all the suggestions I received in the "Curl wrapper" thread. I still haven't fixed the recycling buffers though. /Jonas
Re: Curl wrapper
Den 23-05-2011 21:02, Masahiro Nakagawa skrev: > On Wed, 18 May 2011 20:07:01 +0900, Jonas Drewsen > wrote: > >> On 18/05/11 10.09, Johannes Pfau wrote: >>> jdrewsen wrote: >>>> >>>> But I would rather stop here feature wise and wrap >>>> this thing up. I would like to focus my efforts on a candidate for >>>> std.net.event/reactor/proactor module. >>> >>> Have you already started working on std.net.event? Do you plan to base >>> that on libev / libevent? >> >> I'll finish the curl wrapper before starting on it. >> >> I have had a look at libev/event before and they are very nice libs. >> But I think I'll have to implement it from scratch for two reasons: >> >> 1, AFAIK we cannot include code in std phobos that is not boost >> licensed or a license as liberal. libev for example requires you to >> distribute the license with your software. >> >> 2, It introduces a dependency to an external project in phobos. >> Currently no dependencies are present. This makes phobos very easy to >> install and use out of the box. >> >> The optimal solution to these problems from my point of view would be >> to get that "much discussed" package tool in place soon (CPAN,apt alike). >> >> Heck, now I think about it maybe I should do that before any std.net >> stuff. Let's see how the wind blows - many interesting projects :) > > Good! > > Last year, I tried to implement std.event. In Windows, using IOCP with > Overlapped-IO for proactor. > Others, IO multiplexing with worker-thread for proactor emulation. > But Johannes already started implementing event module using libev, > so I deleted this task from my tasks. > If you have any code for inspiration or that could be used as a base I would very much like to see/use it. Especially if it is under Boost license. I would like to create reactor initially and proceed to build the proactor on top of that (not on top for windows and IOCP of course). Reactor and proactor designs each have their forces and therefore I do not think targeting a proactor only design is the way to go. /Jonas
Re: Handy API examples for HTTP requests
Den 28-05-2011 00:43, bearophile skrev: I've seen this with some examples of handy API for HTTP requests, useful for the Phobos devs that want to add a Phobos module to perform such things: http://pydanny.blogspot.com/2011/05/python-http-requests-for-humans.html http://pypi.python.org/pypi/requests Bye, bearophile That API is not far from the one I'm working on for etc.curl. I'm currently implementing the suggestions from the last round of reviews and it will probably be ready for the official review queue after that. /Jonas
Re: Curl wrapper
Den 30-05-2011 01:16, Jimmy Cao skrev: Is there any reason why this code (using SMTP + Curl): https://gist.github.com/998214 segfaults on Windows but not on Fedora 15? I found an easy fix, but it looks like a bug somewhere that I can't figure out. etc.curl is currently a moving target. The last version I pushed to github is outdated. Currently the focus is on getting http and ftp working and the other protocol hasn't been tested. Out of curiosity I would very much like to known what your easy fix was? /Jonas
std.concurrency extension
Hi,
Currently each thread spawned has an associated Tid when using
std.concurrency. When calling receive() this is always the Tid (and
associated messagebox) that you receive from.
This becomes a problem when you spawn several worker threads that send
messages to the main thread since each call to receive must know how to
handle all kind on messages.
For example: the current version of the curl wrapper I'm working on
receives HTTP data from worker threads using receive. The user of this
wrapper can also create his own worker threads that will send data to
the main thread. This means that the curl wrapper must be able to handle
the users worker thread messages.
The GO language has solved this problem with channels. My suggestion is
to allow a thread to create additional Tids that can act just like
channels. Then by adding a receive() call that accepts a Tid as first
argument we can use them as channels.
e.g.
-
void run() {
auto channel1 = receiveOnly!Tid();
channel1.send("Hello world);
}
Tid channel1 = Tid();
auto tid1 = spawn(&run);
tid.send(channel1);
// receive only from the specified work on the channel
writeln(channel1.receiveOnly!string());
-
This would decouple handling of different worker threads if needed.
This is something I need for the curl wrapper I think. I can create
patch for this if it is something that will be accepted upstream?
/Jonas
Re: std.concurrency extension
Den 08-06-2011 16:47, Sean Kelly skrev: On Jun 7, 2011, at 3:07 PM, jdrewsen wrote: Hi, Currently each thread spawned has an associated Tid when using std.concurrency. When calling receive() this is always the Tid (and associated messagebox) that you receive from. This becomes a problem when you spawn several worker threads that send messages to the main thread since each call to receive must know how to handle all kind on messages. For example: the current version of the curl wrapper I'm working on receives HTTP data from worker threads using receive. The user of this wrapper can also create his own worker threads that will send data to the main thread. This means that the curl wrapper must be able to handle the users worker thread messages. The GO language has solved this problem with channels. My suggestion is to allow a thread to create additional Tids that can act just like channels. Then by adding a receive() call that accepts a Tid as first argument we can use them as channels. A Tid is a thread ID, thus "tid". The current design allows you to design any message format you want. In Erlang (on which this design is based), it's common practice to have the sender's Tid included in the message. As for channels... I hope that they can be built on top of the existing messaging system. They might not be as optimal as a from-scratch approach though, since a bit of extra dispatching would need to be done in the wrapper. I took a look at the std.concurrency code and noticed that one of my assumptions was wrong: A receive call does not necessarily handle messages in the messagebox in the order they arrive. If a message cannot be handled by the receive call then the next message available in the messagebox is tried and so on. My assumption was that it would throw. Now that I've checked in TDPL I can see that it is actually documented there. The library API documentation on the web could use some fleshing out though. Sorry about the noise. /Jonas
Re: Historical moment: D2 is now D
Den 10-06-2011 19:06, Andrei Alexandrescu skrev: I think we all agree that it is appropriate to characterize D2 to mean "The D Programming Language". From here on, we have changed the website http://d-programming-language.org to reflect that reality. D simply refers to what was formerly known as D2, and D1 stays D1. Also, today Walter will change the "D" links from digitalmars.com to point to http://d-programming-language.org, which is now the official site of the D programming language. Expect (and please contribute) many improvements of that site going forward. This is great news! I guess the documentation for etc.curl is an error? /Jonas
Re: Flag proposal
Den 11-06-2011 03:26, David Nadlinger skrev:
On 6/11/11 3:21 AM, bearophile wrote:
Andrei:
This module won't compile in today's D, but not for a matter of
principles; it's just a random limitation of the language. (It does work
if you import from within a class or struct.) You can insert most
declarations in a scope, so the ones you can't insert are just awkward
exceptions, unless there's a good reason to actively disable them. Any
code should work if you just wrap another scope around it.
Walter and I call the above a "turtle feature", in allusion to the
"turtles all the way down" meme. Walter wants to fix this, and similar
limitations that act "unturtly". Imports inside a scope won't be visible
outside that scope, and importing the same module from several different
functions will come at no cost in compilation time.
You are allowed to import modules inside functions in Python too
(there is a just a minor restriction), but I have never asked for this
feature in D because this feature has a cost too. Putting all imports
at the top of the module is more tidy, it allows the person that reads
the code to find all the used imports very quickly. If they are spread
in the module they become less easy to find, you need to use your
editor/IDE to search for them.
Bye,
bearophile
At least allowing imports in unittests would be nice though – I
frequently find myself writing »version (unittest) {}« blocks before the
actual unit tests just to import some modules not needed during regular
builds (yeah, I suppose I'm somewhat pedantic about that).
+1
Re: htod converts C long to int?
Den 12-06-2011 20:05, David Nadlinger skrev: While having a look at etc.curl which has a »converted by htod« comment in its header, I noticed that values having type »long« in the original C headers are declared as »int« there – shouldn't this rather be c_long from core.stdc.config to avoid problems on non-LLP64 (i.e. non-Windows) 64 bit systems? Does htod generally behave like this? I just ran htod without any special tricks so I guess this is the default behavior. I will fix this asap. thx. /Jonas
Re: What remains to be done for D2?
Den 14-06-2011 15:17, Andrei Alexandrescu skrev: On 6/14/11 4:07 AM, Don wrote: Petr Janda wrote: Hi, Can someone please explain what needs to be done (other than fixing the plethora of bugs) to call D2 final? And if someone can provide an approximate estimate of when? There are a couple of areas of major missing functionality: * CTFE should support pointers (upcoming release 2.054) and classes (2.055). * alias this, inout, safe aren't fully implemented, and will probably require language changes/clarifications. * Built-in struct functions (opEquals, toString, etc) need more thought. * I suspect that we'll see changes to the modifiers for function parameters (in, out, inout, ref, auto ref) * Work last kinks out of qualified constructors and destructors * Low-level threading support with shared; change language to match TDPL (that means e.g. shared/synchronized methods is at class level, not method level). * Minor touches such as user-defined operator '$', @disable etc. Work that is not quintessential for using D but very important: * Finish typechecking for SafeD * Make SafeD == CompileTimeD We should put this list somewhere. At the current rate of progress I estimate we are about six months from having the language implemented (but still with bugs). But there will still be bugs which can only be fixed by making small changes to the language spec. Phobos is quite a bit further away from being "final". * Define allocator abstraction * Overhaul std.container to use it; add classic containers to std.container (doubly-linked list, hash, deque) * Define streaming abstraction (already in my head) * Add high-level networking (Jonas, where art thou?) Finishing off some major changes to the curl wrapper. I'm posting a RFC on the updated code in a sec. * Redo xml (Tomek) * Many other changes and additions Andrei
Curl wrapper round two
Hi, I've finally got through all the very constructive comments from the last review of the curl wrapper and performed the needed changes. Here is the github branch: https://github.com/jcd/phobos/tree/curl-wrapper And the generated docs: http://freeze.steamwinter.com/D/web/phobos/etc_curl.html I do have some problems getting ddoc to show the documentation of mixins. So in order to view the doc for byLine/byChunk methods you have to look at the source. Anyway...this is what I've been up to: New features: * Full support for async/sync by line/chunk * FTP support extended from only allowing download of a file sync into full async/sync by line/chunk support * Allow providing parameters such as credentials/timeouts when using the convenience statis methods. Changes caused by last review: * rethink byLine/... to not return string in order to prevent allocations. they should return char[]/ubyte[] * 80 chars * Http.Result not HttpResult * gramma for http.postData * len -> length * perform http request -> perform a http ... * authMethod to property * curltimecond alias into module * followlocation -> maxredirs * http not class anymore but struct * timecondition use std.datetime * timeouts use core.duration * Spelling "callbacks is not supported" * refer to HTTP RFC describing the methods * login/password example * chuncked -> chunked * max redirs; use uint.max and not -1 * isRunning returining short * 4 chars tabs in examples. * no space in examples. * Send/recv use special structs in order not to mess with other communications Comments are welcome. /Jonas
Re: Curl wrapper round two
Den 18-06-2011 22:52, Jimmy Cao skrev: Would an SMTP protocol struct be beneficial? My immediate goal is to provide HTTP support and basic FTP support through libcurl. I believe these are the most important protocols to get in place in order to improve the adoption of D. I have currently no plans of adding more protocols to the curl wrapper. Patches are welcome :) I would rather do some work on native async net support since I believe that would give better performance. /Jonas This looks great, thanks for you work. On Sat, Jun 18, 2011 at 3:36 PM, jdrewsen mailto:jdrew...@nospam.com>> wrote: Hi, I've finally got through all the very constructive comments from the last review of the curl wrapper and performed the needed changes. Here is the github branch: https://github.com/jcd/phobos/__tree/curl-wrapper <https://github.com/jcd/phobos/tree/curl-wrapper> And the generated docs: http://freeze.steamwinter.com/__D/web/phobos/etc_curl.html <http://freeze.steamwinter.com/D/web/phobos/etc_curl.html> I do have some problems getting ddoc to show the documentation of mixins. So in order to view the doc for byLine/byChunk methods you have to look at the source. Anyway...this is what I've been up to: New features: * Full support for async/sync by line/chunk * FTP support extended from only allowing download of a file sync into full async/sync by line/chunk support * Allow providing parameters such as credentials/timeouts when using the convenience statis methods. Changes caused by last review: * rethink byLine/... to not return string in order to prevent allocations. they should return char[]/ubyte[] * 80 chars * Http.Result not HttpResult * gramma for http.postData * len -> length * perform http request -> perform a http ... * authMethod to property * curltimecond alias into module * followlocation -> maxredirs * http not class anymore but struct * timecondition use std.datetime * timeouts use core.duration * Spelling "callbacks is not supported" * refer to HTTP RFC describing the methods * login/password example * chuncked -> chunked * max redirs; use uint.max and not -1 * isRunning returining short * 4 chars tabs in examples. * no space in examples. * Send/recv use special structs in order not to mess with other communications Comments are welcome. /Jonas
Re: Curl wrapper round two
Den 19-06-2011 11:12, Johannes Pfau skrev: Jimmy Cao wrote: Also, why the bool dummy argument in the Curl struct constructor? I guess that's because structs can't have default constructors. Is there a better solution to this problem? That is why yes. I could just make a static "create()" function instead, but I went for this model. I would very much like to know a better solution.
Re: Curl wrapper round two
Den 19-06-2011 11:08, Johannes Pfau skrev:
jdrewsen wrote:
Hi,
I've finally got through all the very constructive comments from
the
last review of the curl wrapper and performed the needed changes.
Here is the github branch:
https://github.com/jcd/phobos/tree/curl-wrapper
And the generated docs:
http://freeze.steamwinter.com/D/web/phobos/etc_curl.html
I do have some problems getting ddoc to show the documentation of
mixins. So in order to view the doc for byLine/byChunk methods you
have to look at the source.
That's bad because lots of useful stuff hides in the protocol mixin.
The url property for example is essential for keep-alive requests, but
it doesn't show up in the documentation :-(
I agree. And also in the ByLineAsync etc. mixins. I would very much like
to get a hint on how to do it if anyone knows.
Also, a keep alive example would be great:
auto client = Http("http://api.vevo.com/mobile/v2/authentication.json";);
client.addHeader("User-Agent", "Android API Connector");
client.addHeader("Connection", "Keep-Alive");
client.method = Http.Method.post;
client.onReceive = (ubyte[] data) { write(cast(char[])data); return
data.length; };
client.postData = "p=android&v=1.05";
client.perform();
//2nd request
client.url = "http://api.vevo.com/mobile/v1/featured/carousel.json";;
client.method = Http.Method.get;
client.perform();
Maybe something like this. (+points if the code uses existing
websites)
I'll include that. And I need a "header(key, value)" parameter on
Http.(Async)Result as well. That way your example could be written:
auto r = Http.post("http://api.vevo.com/mobile/v2/authentication.json";,
"p=android&v=1.05")
.header("User-Agent", "Android API Connector")
.header("Connection", "Keep-Alive"));
write(r.bytes);
BTW: The curl verbose output is great. I guess it won't
be activated in phobos by default, but is it possible to activate it
manually? If so, this very useful feature should be documented.
Yes - verbose should be made in a property by itself.
Thank you for the comments!
/Jonas
Re: Curl wrapper round two
Den 19-06-2011 05:16, Jimmy Cao skrev: On Sat, Jun 18, 2011 at 4:16 PM, jdrewsen mailto:jdrew...@nospam.com>> wrote: Den 18-06-2011 22:52, Jimmy Cao skrev: Would an SMTP protocol struct be beneficial? My immediate goal is to provide HTTP support and basic FTP support through libcurl. I believe these are the most important protocols to get in place in order to improve the adoption of D. I have currently no plans of adding more protocols to the curl wrapper. Patches are welcome :) Here's a crude implementation for supporting the SMTP protocol: https://gist.github.com/1033711 What do you think? Very nice. A couple of things I believe would help: 1, Get rid of MailMessageData and use curl.onSend() and a delegate that keeps a reference to the message. That way you don't have to use the lower level Curl.set(infile/readfunction) calls as well. 2, It would be nice if the static sendMail(...) function worked like the Http/Ftp counterparts. They return a Result object that you can change before performing the actual task. That way you can easily set timeouts etc. If there shouldn't be support for async smtp then this is probably not important though. My biggest concern is whether SMTP protocol support would really be necessary within this module. Personally it is not very high on my list. Initially I would like to have the curl wrapper accepted with only Http and basic Ftp just to reach a milestone. After that other protocols could be added. But thats just how I imagine the process. /Jonas
Re: Curl wrapper round two
Den 19-06-2011 14:52, Jacob Carlborg skrev: On 2011-06-19 11:44, jdrewsen wrote: Den 19-06-2011 11:12, Johannes Pfau skrev: Jimmy Cao wrote: Also, why the bool dummy argument in the Curl struct constructor? I guess that's because structs can't have default constructors. Is there a better solution to this problem? That is why yes. I could just make a static "create()" function instead, but I went for this model. I would very much like to know a better solution. A static opCall function allow this syntax: auto s = Struct(); Thanks
Re: Curl wrapper round two
Den 19-06-2011 13:26, Max Klyga skrev: On 2011-06-19 12:44:57 +0300, jdrewsen said: Den 19-06-2011 11:12, Johannes Pfau skrev: Jimmy Cao wrote: Also, why the bool dummy argument in the Curl struct constructor? I guess that's because structs can't have default constructors. Is there a better solution to this problem? That is why yes. I could just make a static "create()" function instead, but I went for this model. I would very much like to know a better solution. Declare static opCall and then to instatiate: auto foo = Bar(); But the user must remember to initialize variables properly. Thanks. I guess that I can initialize variables in the opCall and that way the user should not have to remember to initialize variables? /Jonas
Re: Curl wrapper round two
Den 19-06-2011 14:03, David Nadlinger skrev: Does curl_global_init really have to be called for every thread? If not, invoke it in a shared static constructor. I should indeed only be called once for all threads. Thanks. David On 6/18/11 10:36 PM, jdrewsen wrote: Hi, I've finally got through all the very constructive comments from the last review of the curl wrapper and performed the needed changes. Here is the github branch: https://github.com/jcd/phobos/tree/curl-wrapper And the generated docs: http://freeze.steamwinter.com/D/web/phobos/etc_curl.html I do have some problems getting ddoc to show the documentation of mixins. So in order to view the doc for byLine/byChunk methods you have to look at the source. Anyway...this is what I've been up to: New features: * Full support for async/sync by line/chunk * FTP support extended from only allowing download of a file sync into full async/sync by line/chunk support * Allow providing parameters such as credentials/timeouts when using the convenience statis methods. Changes caused by last review: * rethink byLine/... to not return string in order to prevent allocations. they should return char[]/ubyte[] * 80 chars * Http.Result not HttpResult * gramma for http.postData * len -> length * perform http request -> perform a http ... * authMethod to property * curltimecond alias into module * followlocation -> maxredirs * http not class anymore but struct * timecondition use std.datetime * timeouts use core.duration * Spelling "callbacks is not supported" * refer to HTTP RFC describing the methods * login/password example * chuncked -> chunked * max redirs; use uint.max and not -1 * isRunning returining short * 4 chars tabs in examples. * no space in examples. * Send/recv use special structs in order not to mess with other communications Comments are welcome. /Jonas
Re: Curl wrapper round two
Den 19-06-2011 14:00, Jacob Carlborg skrev: On 2011-06-18 22:36, jdrewsen wrote: Hi, I've finally got through all the very constructive comments from the last review of the curl wrapper and performed the needed changes. Here is the github branch: https://github.com/jcd/phobos/tree/curl-wrapper And the generated docs: http://freeze.steamwinter.com/D/web/phobos/etc_curl.html I do have some problems getting ddoc to show the documentation of mixins. So in order to view the doc for byLine/byChunk methods you have to look at the source. Anyway...this is what I've been up to: New features: * Full support for async/sync by line/chunk * FTP support extended from only allowing download of a file sync into full async/sync by line/chunk support * Allow providing parameters such as credentials/timeouts when using the convenience statis methods. Changes caused by last review: * rethink byLine/... to not return string in order to prevent allocations. they should return char[]/ubyte[] * 80 chars * Http.Result not HttpResult * gramma for http.postData * len -> length * perform http request -> perform a http ... * authMethod to property * curltimecond alias into module * followlocation -> maxredirs * http not class anymore but struct * timecondition use std.datetime * timeouts use core.duration * Spelling "callbacks is not supported" * refer to HTTP RFC describing the methods * login/password example * chuncked -> chunked * max redirs; use uint.max and not -1 * isRunning returining short * 4 chars tabs in examples. * no space in examples. * Send/recv use special structs in order not to mess with other communications Comments are welcome. /Jonas Is the wrapper really supposed to be in the etc package? I thought that was just for the bindings? I don't know where best place to put it is. In some way I think modules that introduces dependencies (libcurl in this case) is best handled by the hopefully upcoming dget/build2/??? functionality and thereby keeping phobos free of licensing issues. Then there could be some official wrappers that are "blessed" meaning that the phobos community ensures that the quality is as good as phobos itself. I think the curl wrapper would fit in there nicely. If this can be agreed upon then the module should probably be moved out of the etc package and become a root module. The etc.c.curl should probably be moved out of phobos at the same time. /Jonas
Re: Curl wrapper round two
Den 19-06-2011 17:43, Johannes Pfau skrev:
jdrewsen wrote:
Den 19-06-2011 11:08, Johannes Pfau skrev:
jdrewsen wrote:
Hi,
I've finally got through all the very constructive comments from
the
last review of the curl wrapper and performed the needed changes.
Here is the github branch:
https://github.com/jcd/phobos/tree/curl-wrapper
And the generated docs:
http://freeze.steamwinter.com/D/web/phobos/etc_curl.html
I do have some problems getting ddoc to show the documentation of
mixins. So in order to view the doc for byLine/byChunk methods you
have to look at the source.
That's bad because lots of useful stuff hides in the protocol mixin.
The url property for example is essential for keep-alive requests,
but it doesn't show up in the documentation :-(
I agree. And also in the ByLineAsync etc. mixins. I would very much
like to get a hint on how to do it if anyone knows.
Also, a keep alive example would be great:
auto client =
Http("http://api.vevo.com/mobile/v2/authentication.json";);
client.addHeader("User-Agent", "Android API Connector");
client.addHeader("Connection", "Keep-Alive"); client.method =
Http.Method.post; client.onReceive = (ubyte[] data)
{ write(cast(char[])data); return data.length; };
client.postData = "p=android&v=1.05";
client.perform();
//2nd request
client.url = "http://api.vevo.com/mobile/v1/featured/carousel.json";;
client.method = Http.Method.get;
client.perform();
Maybe something like this. (+points if the code uses existing
websites)
I'll include that.
Feel free to use this example, but please note that the vevo api is not
public, so it could break any time.
And I need a "header(key, value)" parameter on
Http.(Async)Result as well. That way your example could be written:
auto r = Http.post("http://api.vevo.com/mobile/v2/authentication.json";,
"p=android&v=1.05")
.header("User-Agent", "Android API Connector")
.header("Connection", "Keep-Alive"));
write(r.bytes);
Looks great, but I guess it won't help for keep-alive sessions? Or is
it possible to reuse the Curl client with the static methods?
Not in the linked version. But is it a nice idea that I'm working on now.
I also found another small problem related to keep-alive:
How can I change a header that's been added with addHeader? I have to
reuse the client to use keep-alive, however calling addHeader with the
same key again just appends the header, but it doesn't replace it. An
easy solution is to expose a clearHeader function, but this means if the
user wants to change 1 header all other headers must be set
again. It seems the curl api is too limited to do something more
advanced though. Having a D associative array for the headers might be
possible, but then it has to be converted for curl in every request.
I'm actually storing a curl_slist to keep the headers. This makes it
possible to manipulate the headers as you request. I'll fix it so that
you can change and remove individual headers.
BTW: The curl verbose output is great. I guess it won't
be activated in phobos by default, but is it possible to activate it
manually? If so, this very useful feature should be documented.
Yes - verbose should be made in a property by itself.
Thank you for the comments!
/Jonas
You're welcome!
Re: Curl wrapper round two
Den 19-06-2011 19:42, David Nadlinger skrev:
On 6/19/11 7:34 PM, jdrewsen wrote:
I guess that I can initialize variables in the opCall and that way the
user should not have to remember to initialize variables?
Yes, you can just do:
---
struct Foo {
Foo static opCall() {
Foo result;
result. = ;
return result;
}
…
}
---
The thing is that the static opCall is not invoked if the user writes
»Foo foo;« instead of »auto foo = Foo();«, so you need to carefully
document that users need to use the opCall syntax.
This shouldn't be a problem really since the struct is private.
/Jonas
Re: Curl wrapper round two
Den 19-06-2011 17:57, Jimmy Cao skrev: On Sun, Jun 19, 2011 at 5:29 AM, jdrewsen mailto:jdrew...@nospam.com>> wrote: Very nice. A couple of things I believe would help: 1, Get rid of MailMessageData and use curl.onSend() and a delegate that keeps a reference to the message. That way you don't have to use the lower level Curl.set(infile/readfunction) calls as well. Ah, that makes it much better. 2, It would be nice if the static sendMail(...) function worked like the Http/Ftp counterparts. They return a Result object that you can change before performing the actual task. That way you can easily set timeouts etc. If there shouldn't be support for async smtp then this is probably not important though. There should be support for async SMTP. The problem is this: SMTP.sendMailAsync(...).connectTimeout(dur!"seconds"(60)).localPort(25).? byLine, byChunk, etc don't make much sense there. I think it would be better to get rid of the static sendMail function, and write a performAsync method. I'm not sure though. Something like this: https://gist.github.com/1034433 Maybe i doesn't make sense to provide the async interface at all? Users needing to do it async could just as well create a delegate and call spawn themselves. /Jonas
Re: Curl wrapper round two
Den 18-06-2011 22:36, jdrewsen skrev: Hi, I've finally got through all the very constructive comments from the last review of the curl wrapper and performed the needed changes. Here is the github branch: https://github.com/jcd/phobos/tree/curl-wrapper And the generated docs: http://freeze.steamwinter.com/D/web/phobos/etc_curl.html I've made the changes as suggested from your comments and pushed to the github branch above. Changes: * Change and delete individual headers when using static convenience methods * Make keep-alive work when using static convenience methods * Add as extra modifiable parameters on follow requests (keep-alive): headers, method, url, postData * Add verbose property to Protocol * No dummy bool in constructors Comments are welcome /Jonas
Re: Curl wrapper round two
Den 21-06-2011 02:52, Jose Armando Garcia skrev: On Mon, Jun 20, 2011 at 8:33 PM, jdrewsen wrote: Den 18-06-2011 22:36, jdrewsen skrev: Hi, I've finally got through all the very constructive comments from the last review of the curl wrapper and performed the needed changes. Here is the github branch: https://github.com/jcd/phobos/tree/curl-wrapper And the generated docs: http://freeze.steamwinter.com/D/web/phobos/etc_curl.html I've made the changes as suggested from your comments and pushed to the github branch above. Changes: * Change and delete individual headers when using static convenience methods * Make keep-alive work when using static convenience methods * Add as extra modifiable parameters on follow requests (keep-alive): headers, method, url, postData * Add verbose property to Protocol * No dummy bool in constructors Comments are welcome /Jonas Hi Jonas, Was reading your implementation but I had to context switch. Only go to line 145 :(. I see that you are refcounting by sharing a uint* but what about all the other private fields? What happens if you pass the Curl object around functions and those values are modified? Thanks, -Jose The refcounting is actually just needed to keep the "handle" alive under construction of the Curl object using "Curl()". I'm using "Curl()" by defining opCall on Curl in order not to have a struct constructor with a dummy parameter (structs cannot have a default constructor defined). After that the Curl instance will always be assigned to a member variable of Http/Ftp classes. Instances of Http/Ftp are not to be copied because they are used for RAII. I hope this makes sense. Maybe I should look for another solution for this since it might be too difficult to figure out for the uninitiated. /Jonas
Re: Curl wrapper round two
Den 21-06-2011 12:49, Johannes Pfau skrev: jdrewsen wrote: Den 18-06-2011 22:36, jdrewsen skrev: Hi, I've finally got through all the very constructive comments from the last review of the curl wrapper and performed the needed changes. Here is the github branch: https://github.com/jcd/phobos/tree/curl-wrapper And the generated docs: http://freeze.steamwinter.com/D/web/phobos/etc_curl.html I've made the changes as suggested from your comments and pushed to the github branch above. Changes: * Change and delete individual headers when using static convenience methods * Make keep-alive work when using static convenience methods * Add as extra modifiable parameters on follow requests (keep-alive): headers, method, url, postData * Add verbose property to Protocol * No dummy bool in constructors Comments are welcome /Jonas Great, thanks for including those changes! Another small feature request: Could curl_easy_escape and curl_easy_unescape be made available in the wrapper? I don't like the fact that Curl requires a client instance for these functions, but they're quite useful nevertheless. I'll add this yes. It pretty weird that there is deprecated curl_escape() which does not need a curl instance. That function refers to the newer curl_easy_escape(). They must have had a good reason to do so i guess. BTW: http://d-programming-language.org/ has newer css stylesheets for documentation. I think with this new style the documentation looks much better. So it'd be great if you could rebuild the documentation with the new std.ddoc and stylesheets from https://github.com/D-Programming-Language/d-programming-language.org Done /Jonas
Re: Curl wrapper round two
Den 21-06-2011 12:55, Johannes Pfau skrev: jdrewsen wrote: Den 18-06-2011 22:36, jdrewsen skrev: Hi, I've finally got through all the very constructive comments from the last review of the curl wrapper and performed the needed changes. Here is the github branch: https://github.com/jcd/phobos/tree/curl-wrapper And the generated docs: http://freeze.steamwinter.com/D/web/phobos/etc_curl.html I've made the changes as suggested from your comments and pushed to the github branch above. Changes: * Change and delete individual headers when using static convenience methods * Make keep-alive work when using static convenience methods * Add as extra modifiable parameters on follow requests (keep-alive): headers, method, url, postData * Add verbose property to Protocol * No dummy bool in constructors Comments are welcome /Jonas Oh, and btw are you reading the phobos-dev mailing list? Seems like there are no code reviews planned right now, so if you think etc.curl was ready for review, you could propose it now? It don't know anything about a phobos-dev mailing list. The only reviews I've seen so far is on this D newsgroup. How do I get on that mailing list? I'll think It'll soon be ready for an official code review though. /Jonas
Re: State of curl wrapper?
Den 22-06-2011 05:05, Jonathan M Davis skrev: Is the curl wrapper at about the point that we could look at doing a formal review of it followed by voting for inclusion in Phobos, or does it need more bake time? It's fine either way. I'd just like to know where it sits. If it's ready for review, then we can start a formal review process for it. If not, then Andrei has a pull request which would create a std.benchmark (though a good chunk of it is from std.datetime) which some of expressed the desire to have formally reviewed rather than just reviewed on github. So, if the curl wrapper isn't ready for review yet, then we can review that. So, is the curl wrapper about ready for formal review, or does it need more bake time first? - Jonathan M Davis I've got a couple of small features that I would like to add before it is ready for review. But I think it'll be ready within a week or so. If std.benchmark is ready then let's do that one first. /Jonas
Re: Curl wrapper round two
Den 22-06-2011 10:25, Johannes Pfau skrev: jdrewsen wrote: Den 21-06-2011 12:55, Johannes Pfau skrev: jdrewsen wrote: Den 18-06-2011 22:36, jdrewsen skrev: Hi, I've finally got through all the very constructive comments from the last review of the curl wrapper and performed the needed changes. Here is the github branch: https://github.com/jcd/phobos/tree/curl-wrapper And the generated docs: http://freeze.steamwinter.com/D/web/phobos/etc_curl.html I've made the changes as suggested from your comments and pushed to the github branch above. Changes: * Change and delete individual headers when using static convenience methods * Make keep-alive work when using static convenience methods * Add as extra modifiable parameters on follow requests (keep-alive): headers, method, url, postData * Add verbose property to Protocol * No dummy bool in constructors Comments are welcome /Jonas Oh, and btw are you reading the phobos-dev mailing list? Seems like there are no code reviews planned right now, so if you think etc.curl was ready for review, you could propose it now? It don't know anything about a phobos-dev mailing list. The only reviews I've seen so far is on this D newsgroup. How do I get on that mailing list? Here's the web interface: http://lists.puremagic.com/mailman/listinfo (the 'phobos' list) Reviews are always done on the newsgroup, but some phobos related discussion is also happening on the mailing list. Thanks! I'll think It'll soon be ready for an official code review though. /Jonas
Re: Curl wrapper round two
Den 22-06-2011 17:34, Jose Armando Garcia skrev:
Em 22/06/2011, às 07:02, Jonas Drewsen escreveu:
On 22/06/11 06.11, Jose Armando Garcia wrote:
Hi Jonas,
Was reading your implementation but I had to context switch. Only go
to line 145 :(. I see that you are refcounting by sharing a uint* but
what about all the other private fields? What happens if you pass the
Curl object around functions and those values are modified?
Thanks,
-Jose
The refcounting is actually just needed to keep the "handle" alive
under
construction of the Curl object using "Curl()". I'm using "Curl()" by
defining opCall on Curl in order not to have a struct constructor
with a
dummy parameter (structs cannot have a default constructor defined).
After that the Curl instance will always be assigned to a member
variable of
Http/Ftp classes. Instances of Http/Ftp are not to be copied because
they
are used for RAII.
Http/Ftp are structs not classes. Let me try to understand this. You
mean to say that the Http and Ftp struct are not to be passed to other
functions? Are you expecting the user to do all their IO in one scope?
This is unnecessarily limiting.
They are structs because they use a Curl instance which in turn uses
RAII in order not to leak curl handles. If they were classes it would
be easy to leak because you never know when the GC is coming around to
call the destructor and release the curl handle.
Your can pass Http/Ftp structs to other function either by givin them
a reference to a local instance or you could simply do a:
auto http = new Http("http://google.com";);
passItOn(http);
/Jonas
Passing refs around doesn't work if the function wants to store it on
the heap for later.
Have you taken a look at how std.stdio.File and std.typecons.RefCounted
do reference counting?
I think I'll change it to match std.stdio.File in the name of consistency.
Thanks
/Jonas
Re: Curl wrapper round two
Den 22-06-2011 12:08, Dmitry Olshansky skrev:
On 22.06.2011 14:02, Jonas Drewsen wrote:
On 22/06/11 06.11, Jose Armando Garcia wrote:
Hi Jonas,
Was reading your implementation but I had to context switch. Only go
to line 145 :(. I see that you are refcounting by sharing a uint* but
what about all the other private fields? What happens if you pass the
Curl object around functions and those values are modified?
Thanks,
-Jose
The refcounting is actually just needed to keep the "handle" alive
under
construction of the Curl object using "Curl()". I'm using "Curl()" by
defining opCall on Curl in order not to have a struct constructor
with a
dummy parameter (structs cannot have a default constructor defined).
After that the Curl instance will always be assigned to a member
variable of
Http/Ftp classes. Instances of Http/Ftp are not to be copied because
they
are used for RAII.
Http/Ftp are structs not classes. Let me try to understand this. You
mean to say that the Http and Ftp struct are not to be passed to other
functions? Are you expecting the user to do all their IO in one scope?
This is unnecessarily limiting.
They are structs because they use a Curl instance which in turn uses
RAII in order not to leak curl handles. If they were classes it would
be easy to leak because you never know when the GC is coming around to
call the destructor and release the curl handle.
Your can pass Http/Ftp structs to other function either by givin them
a reference to a local instance or you could simply do a:
auto http = new Http("http://google.com";);
I suspect that http is on _heap_ just like class, and it's destructor is
never called ATM. Even when that's fixed it's still non-deterministic.
Maybe refcounting? e.g. std.typecons.RefCounted
passItOn(http);
/Jonas
As I replied a minute ago to Jose I'll change it to match the
std.stdio.File behavior.
/Jonas
Re: RAII implementation for Socket and Selector
Den 20-06-2011 22:38, Jose Armando Garcia skrev: Hi everyone, For the past few of days I have been working on a RAII implementation for Socket and Selector. Sockets are a ref counted wrapper around the socket handle which closes the handle once the ref count goes to zero. It provides safe methods for bind, connect, listen, accept, recv, send and close. The module also provides helper methods for creating sockets for a tcp server, tcp client, udp server and udp client. The helper method used the Address struct which is basically wrapper around getaddrinfo. As of right now the module provides support for ipv4 and ipv6. On top of Socket we have Selector which can be used to safely wait on more than one socket. To register on a selector call the register method in Socket. Use Socket.unregister to unregister the socket. Sockets are automatically unregistered when closed. The current implementation for selector only support epoll (sorry Windows and BSD users) but I am highly confident that it can be ported to other platforms. I plan to it at a future date but there are currently some serious issues with DMD and druntime that invalidate the strong/weak ref counting design. It works well enough to pass the unittests but I had to do a lot of hacks which I hope I can remove once DMD and druntime are fixed. I should also mention that the design was influenced by Java's NIO and Ruby's socket implementation. Here is the code: https://github.com/jsancio/phobos/blob/socket/std/net/socket.d. It doesn't have any documentation right now. I wont be able to work on it for the next couple of weeks but comments are welcome. Thanks! -Jose Isn't there a lot of overlap with std.socket? Could it be based on std.socket instead? Maybe by extending std.socket itself with the any missing features. /Jonas
Re: [OT] GitHub pull request automation
Den 23-06-2011 12:05, Johann MacDonagh skrev: Found this on reddit: http://www.splitbrain.org/blog/2011-06/19-automate_github_pull_requests Adds a new "git pull-request" command. Without any arguments, it lists all the pull requests for your repo. If you give it the pull request number, it will automatically create a new branch and pull down the request. You can then test it out, merge it into master (maybe test after you merge ;) ), and push. It may help out with the large number of pull requests that can't be automatically merged that the GitHub projects seem to be getting. Very nice!
Expected behavior?
I was just about to send the curl wrapper I've been working on for RFC
and did a git pull from upstream to be in sync. After that I got a new
error that may be a dmd bug.
The following is some minimal code that triggers the error.
private mixin template A() {
void foo() {}
}
void bar() {
static struct C {
mixin A!();
};
}
testd.d(6): Error: struct testd.bar.C no size yet for forward reference
testd.d(7): Error: mixin testd.bar.C.A!() error instantiating
If I move the struct C out of the function body it compiles ok. Note
that this did actually work before I did a pull from upstream.
Is this a known issue?
/Jonas
Re: Expected behavior?
Den 05-07-2011 13:21, Jacob Carlborg skrev: On 2011-07-05 09:51, Jonas Drewsen wrote: Found a matching bug report. http://d.puremagic.com/issues/show_bug.cgi?id=2458 Doesn't seem to have caught any attention so it seems I have to hack around it :( /Jonas Just declare the struct outside of the function? That will handle the case as specified in the bug but unfortunately not my actual problem. I'll figure it out - but thanks for the idea. /Jonas
Review
Hi, I believe the curl wrapper is ready for a review now. Is there anything else in the review queue or can we schedule the curl wrapper now? /Jonas
Re: Orbit - Package Manager - Specification/ideas
Den 13-07-2011 21:19, Jacob Carlborg skrev: I've written a more formal specification of my ideas for a package manager for D. https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D Note that I am exploring the possibility of using D as the language for all the files mentioned in the link above. The current status is that building packages and installing them works, but quite limited. No dependency tracking or central repository so far. Please comment and suggest. Nice work! Orb - tool section: Describe what the "use" command does. I guess it simply adds a required orb to the Orbfile in the current directory? Orbfile section: The "orb" command that accepts git/hg/svn repositories should also allow for a tag/commit parameter I think. The "orb" commands second parameter could also be a list of serveral repositories to try in order for fallback. I guess a user configuration file in ~/.orb could contain "source" commands as well. Orb package section: I think the versioning scheme should be set in stone actually. Most other packaging systems does that. It makes your life much easier. Central repository section: Please let us settle for one format for the metadata.xxx file. My vote is for json or yaml. XML is too verbose for my taste. I also think that it should be compressed e.g. metadata.json.bzip since it will quickly grow quite large and the packaging system has to be fast. Maybe add the build revision on /orb/-_ since it is quite common reupload the same package version with at simple build fix. Additionally the architecture should be added to the name: /orb/-_- Now it is just like how debian files look like :) Maybe put the file in an arch subdir /orb//-_ I'm really in favor of doing this in D instead of ruby though. /Jonas
Re: Prototype buildsystem "Drake"
Den 13-07-2011 03:02, Nick Sabalausky skrev:
The recent discussions about package managers and buildsystems has prompted
me to get off my ass (or rather, *on* my ass as per how programming is
usually performed...) and start on the D-based rake-inspired buildtool I've
been meaning to make for awhile now. It's not actually usable yet, but
there's a sample drakefile demonstrating everything, and it does actually
compile and run (but just doesn't build the dependency tree or actually run
any of the build steps). *Should* work on Posix, but I only tested on
Windows, so I may have fucked up the Posix version...
Apologies to Jacob Carlborg for the name being so close to "dake". Didn't
really know what else to call it ("Duck", maybe?) Like dake, it's inspired
by Ruby's Rake. But unlike dake, the buildscript is written in D instead of
Ruby, which was my #1 reason for making a Rake alternative.
Before I go implemeting everything, I'd like to get input on it. Is it
something that could be a major D tool?
Overview of Drake and links to all the code are here:
https://bitbucket.org/Abscissa256/drake/wiki/Home
A good start I think. The first thing that I think should be fixed is
the naming.
Drake should be called dbuild
Orb should be called dpack
You read the name and you have an idea of what it does - nothing fancy.
People coming from ruby may recognize the drake name, but I guess many
c++ peoply have no idea.
Orb is not derived from anything I guess.
Regarding drake itself:
I really like how general it is but at the same time miss some
simplicity. "Make the common task easy and the uncommon possible"
Some steps that you could take in that direction is to predefine
commonly used modes like "build", "clean" etc. And the define
buildMode(), cleanMode() as builtins. That would get rid of the
"immutable modes = ..." lines in simple projects.
Maybe let the wrapper that loads the drakefile insert the "import
drake.all" and also wrap the "void drakefile() { ... }" around the file.
Maybe that is what your have already listed on the todo?
As suggested elsewhere you could get rid of target!Task or target!File
and just defined the target() and file() functions.
Also a convenience could be to just include all .d files found for an
exe target if no .d files are explicitly listed.
I guess this would make it possible to have a drakefile as simple as:
task("appA".exe);
It seems to lack the configure part of the build process e.g. locate the
libfoobar.a file for linking or locate a D compiler.
Thanks,
/Jonas
Re: Std.path: Final Vote
Yes!
Re: std.concurrency wrapper over MPI?
Den 06-08-2011 05:51, dsimcha skrev: I've finally bitten the bullet and learned MPI (http://en.wikipedia.org/wiki/Message_passing_interface) for an ultra computationally intensive research project I've been working on lately. I wrote all the MPI-calling code in D against the C API, using a very quick-and-dirty (i.e. not releasable) translation of the parts of the header I needed. I'm halfway-thinking of writing a std.concurrency-like interface on top of MPI in D. A few questions: 1. Is anyone besides me interested in this? 2. Is anyone already working on something similar. 3. Would this be Phobos material even though it would depend on MPI, or would it better be kept as a 3rd party library? I think std.concurrency needs to define a new interface for passing messages out-of-process ie. other process or host. The implementation itself should probably be 3rd party since there are many serialized representations and protocols out there to pick from. 4. std.concurrency in its current incarnation doesn't allow objects with mutable indirection to be passed as messages. This makes sense when passing messages between threads in the same address space. However, for passing between MPI processes, the object is going to be copied anyhow. Should the restriction be kept (for consistency) or removed (because it doesn't serve much of a purpose in the MPI context)? > 5. For passing complex object graphs, serialization would obviously be necessary. What's the current state of the art in serialization in D? I want something that's efficient and general first and foremost. I really don't care about human readability or standards compliance (in other words, no XML or JSON or anything like that). AFAIK David Nadlinger is handling serialization in his GSOC Thrift project that he is working on currently. /Jonas
Re: DB ORM
Den 09-08-2011 21:15, Robert Clipsham skrev:
On 09/08/2011 08:30, Jonas Drewsen wrote:
Just stumbled upon this db orm for c++ that uses the gcc frontend to
rewrite c++ code to make classes suitable for database access.
http://www.codesynthesis.com/products/odb/
They are using pragmas to accomplish this. I guess an equally good
implementation in D would use custom attributes for this once (if) they
are supported.
/Jonas
How ugly! My (far from complete, but good enough to demonstrate what D
can do) ORM is far simpler than that:
https://github.com/mrmonday/serenity (actually a web framework, includes
an incomplete ORM)
struct Post
{
ulong id;
DateTime dt;
string title;
string content;
}
Persister!Post posts;
// Append a post
posts ~= Post(0, someDateTime, "A title", "Some content");
foreach (post; posts[0..5])
{
// Iterate over the first 5 posts
}
post[3] = posts[2];
All SQL is generated at compile time, and the backend database is called
directly - that's less overhead than you'd get from a typical scripting
language accessing a database directly (no need to convert between types
first).
Very nice. I have to give that a look for sure.
/Jonas
DIP11
What is the status of DIP11 http://www.wikiservice.at/d/wiki.cgi?LanguageDevel/DIPs/DIP11 Has anyone started implementing it? Has it been rejected? /Jonas
Re: std.path Accepted 22-0
Den 12-08-2011 14:45, dsimcha skrev: Lars Kyllingstad's new std.path has been approved by the D community by a vote of 22-0. Congratulations, Lars! I look forward to seeing this in the next release Grats to you Lars. /Jonas
Re: Next to review: Curl Wrapper? (Was: Review Queue Update?)
Den 15-08-2011 20:19, dsimcha skrev: I say give Jonas one more day to post the official review request for the Curl wrapper, and if he doesn't then we can move CSV parser ahead. 1. We've got an embarrassment of riches with how long the review queue is getting. It's a shame to have so many bubbles in it. 2. If Jonas is busy/out of town/otherwise not reading the forums, then this isn't a good time to review the curl wrapper anyhow. I'm reading the forums - just been waiting for Jose to reply because he is before the curl wrapper in the review queue. But I guess your post indicates that it is probably ok to go ahead with the curl review. I will post the request tomorrow since I going to bed in soon. /Jonas 3. We'd only be pushing the Curl wrapper back one spot, to behind the CSV parser, so it's not like we'd be severely devaluing it.
Re: CURL review request
Den 16-08-2011 14:57, Jacob Carlborg skrev: On 2011-08-16 13:48, Jonas Drewsen wrote: Hi all, This is a review request for the curl wrapper. Please read the "known issues" in the top of the source file and if possible suggest a solution. We also need somebody for running the review process. Anyone? Code: https://github.com/jcd/phobos/blob/curl-wrapper/etc/curl.d Docs: http://freeze.steamwinter.com/D/web/phobos/etc_curl.html Demolish! /Jonas The ddoc issue: http://d.puremagic.com/issues/show_bug.cgi?id=648 I would very much like to have this fixed. Ahh... thank you for the pointer. That is only one part of the ddoc issues. The other one is that the Protocol mixin (which is mixed into the Http/Ftp/Smtp classes) does not generate documentation in the Http/Ftp/Smtp classes. I guess that is another bug/feature.
Re: CURL review request
Den 16-08-2011 15:13, dsimcha skrev: On 8/16/2011 7:48 AM, Jonas Drewsen wrote: Hi all, This is a review request for the curl wrapper. Please read the "known issues" in the top of the source file and if possible suggest a solution. We also need somebody for running the review process. Anyone? Code: https://github.com/jcd/phobos/blob/curl-wrapper/etc/curl.d Docs: http://freeze.steamwinter.com/D/web/phobos/etc_curl.html Demolish! /Jonas From a quick look, this looks very well thought out. I'll review it more thoroughly when I have more time. A few questions/comments from a quick look at the docs: Does the async stuff use D threads, or does Curl have its own async API? It uses the spawn method from std.concurrency which uses D threads i think. In your examples for postData, you have onReceive a ubyte[] and write it out to console. Did you mean to cast this to some kind of string? Not really. It is just an example and writeln will output a byte array just fine. For onReceive, what's the purpose of the return value? To specify the number of bytes that have been handled. If this is less that the entire buffer the rest of the request will fail. Additionally both the onSend and onReceive callbacks can return CURL_WRITEFUNC_PAUSE pr CURL_READFUNC_PAUSE to pause the transfer. And onSend can return CURL_READFUNC_ABORT to abort as well. I see that the is missing from the docs... will add. If/when this module makes it into Phobos, are we going to start including a libcurl binary with DMD distributions so that std.curl feels truly **standard** and requires zero extra configuration? I think that would be convenient but don't believe it is the correct thing to do. In my opinion D modules that are not pure D or wrapping a system library (which libcurl is definitely not) should be the only ones allowed in the std package. Other wrappers in phobos should be in the etc package and the lib binaries should not be included. Just how I feel about it though. Anyone who disagrees? /Jonas
Re: CURL review request
Den 16-08-2011 18:55, Martin Nowak skrev: On Tue, 16 Aug 2011 15:13:40 +0200, dsimcha wrote: On 8/16/2011 7:48 AM, Jonas Drewsen wrote: Hi all, This is a review request for the curl wrapper. Please read the "known issues" in the top of the source file and if possible suggest a solution. We also need somebody for running the review process. Anyone? Code: https://github.com/jcd/phobos/blob/curl-wrapper/etc/curl.d Docs: http://freeze.steamwinter.com/D/web/phobos/etc_curl.html Demolish! /Jonas From a quick look, this looks very well thought out. I'll review it more thoroughly when I have more time. A few questions/comments from a quick look at the docs: Does the async stuff use D threads, or does Curl have its own async API? In your examples for postData, you have onReceive a ubyte[] and write it out to console. Did you mean to cast this to some kind of string? For onReceive, what's the purpose of the return value? If/when this module makes it into Phobos, are we going to start including a libcurl binary with DMD distributions so that std.curl feels truly **standard** and requires zero extra configuration? I was also wondering about the async handling. In the long-term I'd like to see a bigger picture for async handling in phobos (offering some kind of futures, maybe event-loops etc.). Though this is not a requirement for the curl wrapper now. std.parallelism also has some kind of this stuff and file reading would benefit from it too. This has been discussed before and I also think this is very important. But before that I think some kind of package management should be prioritized (A DIP11 implementaion or a more traditional solution). One thing I spotted at a quick glance, sending to be filled buffers to another thread should not be done by casting to shared not immutable. I'm not entirely sure what you mean. There is no use of shared buffers in the wrapper. I do cast the buffer between mutable/immutable because only immutable or by value data can be passed using std.concurrency. Since the buffers are only used by the thread that currently has the buffer this is safe. I've previously asked for a non-cast solution (ie. some kind of move between threads semantic for std.concurrency) but was advised that this was the way to do it. martin
Re: CURL review request
Den 16-08-2011 15:54, kennytm skrev: Jonas Drewsen wrote: Hi all, This is a review request for the curl wrapper. Please read the "known issues" in the top of the source file and if possible suggest a solution. We also need somebody for running the review process. Anyone? Code: https://github.com/jcd/phobos/blob/curl-wrapper/etc/curl.d Docs: http://freeze.steamwinter.com/D/web/phobos/etc_curl.html Demolish! /Jonas 1) Why 'Http' and 'Ftp' but 'SMTP' is in all caps? Will fix 2) Some links in the doc says 'std.curl' I know. This is a limitation of the ddoc config file. I guess I have to make a patch for that one as well. Thanks Jonas
