Re: Issues with std.net.curl on Win 10 x64
On Monday, 25 March 2019 at 16:25:37 UTC, cptgrok wrote:
Am I doing something wrong or is there some issue with curl or
something else? I'm pretty new to D and I'm not sure if I need
to go right down to raw sockets and re-invent the wheel or if
there is some other library that can help. If I get this
working, it could potentially save myself and many others hours
per week.
There is a limit of 50 concurrent messages per thread [1] in
byLineAsync also the transmitBuffers argument takes part in.
So using multiple byLineAsync at same time/thread is going to
block the process, I'm not sure if this is a bug or is by design.
You could use download() in a parallel foreach, something like
this:
import std.stdio;
import std.parallelism;
import std.net.curl;
import std.typecons;
void main()
{
auto connections = 3; // 3 parallel downloads
defaultPoolThreads(connections - 1);
auto retries = 4; // try up to 4 times if it fails
auto logList = [
tuple("dlang.org", "log1.txt"),
tuple("dlang.org", "log2.txt"),
tuple("dlang.org", "log3.txt"),
tuple("dlang.org", "log4.txt"),
tuple("dlang.org", "log5.txt"),
tuple("dlang.org", "log6.txt")];
foreach (log; parallel(logList, 1))
{
HTTP conn = HTTP();
foreach (i; 0 .. retries)
{
try
{
writeln("Downloading ", log[0]);
download(log[0], log[1], conn);
if(conn.statusLine.code == 200)
{
writeln("File ", log[1], " created.");
break;
}
}
catch (CurlException e)
{
writeln("Retrying ", log[0]);
}
}
}
}
[1]
https://github.com/dlang/phobos/blob/master/std/net/curl.d#L1679
Re: Issues with std.net.curl on Win 10 x64
On Monday, 25 March 2019 at 19:02:18 UTC, cptgrok wrote: On Monday, 25 March 2019 at 16:44:12 UTC, Andre Pany wrote: First idea, please switch to x86_64 if possible. This will also be the default of Dub in the next dmd release or the release after. Kind regards Andrew Figured out --arch=x86_64, thanks! Sadly I don't see any change. I'm not having luck finding known curl issues similar to what I am experiencing. I have a sneaking suspicion that the web service I am using is doing some nonsense in the background. Might try a packet capture to better see what's up. Alternatively, you could always give requests a shot: https://code.dlang.org/packages/requests It's the unofficial successor of std.net.curl.
Re: Issues with std.net.curl on Win 10 x64
On Monday, 25 March 2019 at 16:44:12 UTC, Andre Pany wrote: First idea, please switch to x86_64 if possible. This will also be the default of Dub in the next dmd release or the release after. Kind regards Andrew Figured out --arch=x86_64, thanks! Sadly I don't see any change. I'm not having luck finding known curl issues similar to what I am experiencing. I have a sneaking suspicion that the web service I am using is doing some nonsense in the background. Might try a packet capture to better see what's up.
Re: Issues with std.net.curl on Win 10 x64
On Monday, 25 March 2019 at 16:25:37 UTC, cptgrok wrote: I need to review syslogs for over 160 systems monthly, and I am trying to write a utility to automate bulk downloads from a custom web service where they are hosted. I need to calculate a date range for the prior month, add start and end date and a serial number to the query string for each system, which is easy, and in a foreach(system; systems) loop in main() I call a function passing the string url in to download and write a log to file. For a small number of systems, it works. [...] First idea, please switch to x86_64 if possible. This will also be the default of Dub in the next dmd release or the release after. Kind regards Andrew
Issues with std.net.curl on Win 10 x64
I need to review syslogs for over 160 systems monthly, and I am trying to write a utility to automate bulk downloads from a custom web service where they are hosted. I need to calculate a date range for the prior month, add start and end date and a serial number to the query string for each system, which is easy, and in a foreach(system; systems) loop in main() I call a function passing the string url in to download and write a log to file. For a small number of systems, it works. My trouble is, using std.net.curl, if I use get(URL) to get the entire text in a single call and write to file, memory usage spirals out of control immediately and within 20 or so calls, gets to about 1.3 GB and the program crashes. If I use byLineAsync(URL), then foreach(line; range) write the lines to file one at a time the memory usage never gets above 5MB but it just hangs always at the 51st call in the loop regardless of what parameters are in the query string, or how much data I have downloaded. The program never terminates, even after hours, but I can't see ANY activity on the process, CPU, mem or network. I can break my download jobs into <=50 systems (and it seems to work), but that seems like sweeping something under the rug, probably leading to future issues. I'm using the 32bit binary from libcurl-7.64.0-WinSSL-zlib-x86-x64.zip on the release archive, and DMD 2.085.0. I've tried curl 7.63 and 7.57 but the behavior is the same. Am I doing something wrong or is there some issue with curl or something else? I'm pretty new to D and I'm not sure if I need to go right down to raw sockets and re-invent the wheel or if there is some other library that can help. If I get this working, it could potentially save myself and many others hours per week.
