Re: [go-nuts] Detecting / Avoiding network bottleneck when sending concurrent HTTP requests to many hosts.

2023-03-07 Thread nohehf
So I unfortunately answered to this via personal email so here's a quick summary of the exchange here (in case anybody else can help): The thing that is weird is that N is pretty low (even with ~20/30 workers it will start to have trouble working, missing some fingerprints). I'll try to netstat,

Re: [go-nuts] Detecting / Avoiding network bottleneck when sending concurrent HTTP requests to many hosts.

2023-03-06 Thread Robert Engels
Have you used netstat to check how many open connections you have? You probably need to forcibly close them so they don’t end up in time wait states. Also, you don’t say what N is? Lastly you will get different errors for a dns or host error than a resources exhausted. If you configure the max

[go-nuts] Detecting / Avoiding network bottleneck when sending concurrent HTTP requests to many hosts.

2023-03-06 Thread nohehf
Hi! I'm building a tool that needs to concurrently send request to multiple hosts (for fingerprinting purposes). I'm using github.com/valyala/fasthttp for the client but it works the same with net/http. The main program basically starts n workers and those will take endpoints to scan from a co