from:"Wojciech Kaczmarek"

[go-nuts] Re: Performance degradation when reading STDIN to multiple buffers?

2023-03-09 Thread Wojciech Kaczmarek


Hey!

May I ask, what would be the practical usage of 500x 1MB buffers? What kind 
of problem are you solving?

Best,
-W
niedziela, 5 marca 2023 o 22:13:57 UTC+1 Kursk napisał(a):

> Oh, interesting enough this performance degradation shows only in some 
> Linux boxes (heavily optimised) from a LAB... I guess then this behaviour 
> might be related to the specific Linux OS settings that those boxes have.
>
> On Sunday 5 March 2023 at 20:36:08 UTC Kursk wrote:
>
>> Hi,
>> I've been recently experimenting with system pipes performance using Go, 
>> trying to measure how fast I could read from STDIN to a buffer, then do 
>> something with it.
>>
>> During that experiment I noticed a significant throughput degradation 
>> when reading into the same buffer vs different (pre-allocated) buffers.
>>
>> To make things more interesting, this performance degradation seems to 
>> happen on my Linux system only, when done in a Mac laptop throughput is 
>> consistent regardless of the count of buffers I am writing to which is what 
>> I was initially expecting.
>>
>> To remove throughput variance from cache misses, I've pinned both 
>> processes to the same virtual core.
>>
>> My question is why is this happening? and, is there a way to avoid it?
>>
>> Below code snipped and command lines
>> ---
>>
>> $ taskset 0x1 cat 10GiB | taskset 0x1 ./stdin-to-buffer -buffer-count=*1*
>> Start time: 2023-03-05T12:26:21-08:00
>> Total bytes read: 10737418240
>> Chunks: 10240
>> *Throughput: 2903 MiB/s*
>> Total time: 3.527284627s
>> Finish time: 2023-03-05T12:26:25-08:00
>>
>> $ taskset 0x1 cat 10GiB | taskset 0x1 ./stdin-to-buffer -buffer-count=
>> *25*
>> Start time: 2023-03-05T12:26:29-08:00
>> Total bytes read: 10737418240
>> Chunks: 10240
>> *Throughput: 2531 MiB/s*
>> Total time: 4.045169972s
>> Finish time: 2023-03-05T12:26:34-08:00
>>
>> $ taskset 0x1 cat 10GiB | taskset 0x1 ./stdin-to-buffer -buffer-count=
>> *50*
>> Start time: 2023-03-05T12:26:39-08:00
>> Total bytes read: 10737418240
>> Chunks: 10240
>> *Throughput: 2215 MiB/s*
>> Total time: 4.623220632s
>> Finish time: 2023-03-05T12:26:43-08:00
>>
>> $ taskset 0x1 cat 10GiB | taskset 0x1 ./stdin-to-buffer -buffer-count=
>> *150*
>> Start time: 2023-03-05T12:26:49-08:00
>> Total bytes read: 10737418240
>> Chunks: 10240
>> *Throughput: 1916 MiB/s*
>> Total time: 5.344739086s
>> Finish time: 2023-03-05T12:26:54-08:00
>>
>> $ taskset 0x1 cat 10GiB | taskset 0x1 ./stdin-to-buffer -buffer-count=
>> *250*
>> Start time: 2023-03-05T12:26:59-08:00
>> Total bytes read: 10737418240
>> Chunks: 10240
>> *Throughput: 1732 MiB/s*
>> Total time: 5.910793462s
>> Finish time: 2023-03-05T12:27:05-08:00
>>
>> $ taskset 0x1 cat 10GiB | taskset 0x1 ./stdin-to-buffer -buffer-count=
>> *500*
>> Start time: 2023-03-05T12:27:13-08:00
>> Total bytes read: 10737418240
>> Chunks: 10240
>> *Throughput: 1757 MiB/s*
>> Total time: 5.827294797s
>> Finish time: 2023-03-05T12:27:19-08:00
>>
>> ---
>> Code for stdin-to-buffer.go follows.
>> ---
>> package main
>>
>> import (
>> "bytes"
>> "flag"
>> "fmt"
>> "io"
>> "log"
>> "math"
>> "os"
>> "time"
>> )
>>
>> var (
>> bufCount = flag.Int("buffer-count", 8, "The number of buffers")
>> bufSize = flag.Int("buffer-size-bytes", 1048576 /*1MiB*/, "The 
>> buffer size in bytes")
>> )
>>
>> func main() {
>> flag.Parse()
>>
>> var nBytes, nChunks int
>>
>> bufPool := make([][]byte, *bufCount)
>> for i := 0; i < *bufCount; i++ {
>> bufPool[i] = make([]byte, *bufSize)
>> }
>>
>> var startTime = time.Now()
>> var loopIteration = -1
>>
>> loop:
>> for {
>> loopIteration++
>> buf := bufPool[loopIteration%*bufCount]
>> n, err := io.ReadFull(os.Stdin, buf)
>>
>> if n > 0 {
>> nChunks++
>> nBytes += n
>> }
>>
>> switch err {
>> case nil:
>> continue
>> case io.EOF, io.ErrUnexpectedEOF:
>> break loop
>> }
>> log.Fatal(err)
>> }
>>
>> finishTime := time.Now()
>> totalTime := finishTime.Sub(startTime)
>> fmt.Fprintf(os.Stderr, "Start time: %v\n", startTime.Format(time.
>> RFC3339))
>> fmt.Fprintf(os.Stderr, "Total bytes read: %v\n", nBytes)
>> fmt.Fprintf(os.Stderr, "Chunks: %v\n", nChunks)
>> fmt.Fprintf(os.Stderr, "Throughput: %v MiB/s\n", math.Round(
>> float64(nBytes)/math.Pow(1024, 2)/totalTime.Seconds()))
>> fmt.Fprintf(os.Stderr, "Total time: %v\n", totalTime)
>> fmt.Fprintf(os.Stderr, "Finish time: %v\n", finishTime.Format(
>> time.RFC3339))
>> }
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit

Re: [go-nuts] HTTP client - streaming POST body?

2023-03-08 Thread Wojciech Kaczmarek

You are right, I overlooked the "bigfile" name clearly suggesting it's 
about the big content being uploaded. My bad.

Anyway, what's important for the OP is that net/http is well-designed and 
will let you do what you need in this case.

cheers,
-W.


wtorek, 7 marca 2023 o 23:07:15 UTC+1 Christian Stewart napisał(a):

No... The previous example streams the file as the post request. The only 
difference in your example is that you're using stdin.

It's not that complicated, files also implement io.reader. 

On Tue, Mar 7, 2023, 1:41 PM Wojciech Kaczmarek  wrote:

Hello,

I think this code example rather streams back the response the server sent 
from its POST handler
and it is not exactly what the OP requested ;) (not to mention that the 
typical result of a POST is
an immediate redirect, possibly without body at all).

I believe what Jim needs is constructing http.Request with a body in a form 
of an io.Reader, then executing it.
Here's the minimal example showing that it can be streamed:

```go

func uploadStdin(url string) error {
req, err := http.NewRequest("POST", url, os.Stdin)
if err != nil {
return err
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
return fmt.Errorf("bad status: %s", resp.Status)
}
_, err = io.Copy(os.Stdout, resp.Body)
return err
}
```

I just tested it and indeed, it gets uploaded after I give some input via 
stdin and hit ctrl+D.

Best,
Wojtek



wtorek, 7 marca 2023 o 06:36:38 UTC+1 Amnon napisał(a):

As Bruno said.

Try something like this:

func uploadBig(url string ) error {
file, err := os.Open("bigfile.txt")
if err != nil {
return err
}
defer file.Close()
resp, err := http.Post(url, "text/plain", file)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
return fmt.Errorf("Bad Status %s", resp.Status)
}
_, err = io.Copy(os.Stdout, resp.Body)
return err
}


On Monday, 6 March 2023 at 15:17:36 UTC Bruno Albuquerque wrote:

The body is just an io.Reader. You just need to write one that materializes 
and streams the data you want instead of using a buffer with all of it. The 
entire design is to allow things like what you want to do. :)

-Bruno


On Mon, Mar 6, 2023 at 10:08 AM 'Jim Smart' via golang-nuts <
golan...@googlegroups.com> wrote:

Hi all,

I'm looking for an HTTP client that allows my code to write the body 
content down the pipe while doing a POST.

The use case here is that I'm wishing to send very large UPDATE/INSERT 
queries/commands to an HTTP endpoint, and the body content of those 
queries/commands is actually generated from a database. So it would be 
better, memory- and performance- wise, if I could write that straight down 
the pipe during the request, as opposed to manifesting the whole of the 
query/command into a buffer before doing the POST.

Is this possible at all with the stdlib http.Client? Perhaps with an 
io.Pipe as the request body?

I did look at the code for http.Client a little, but there's a lot of it, 
and it's not the simplest of code to follow. I do have a base understanding 
of the HTTP 1.1 protocol, but there's a lot more to http.Client than just 
that, of course.

I've also tried looking for existing examples showing similar 
functionality, but did not seem to find anything. Which is partly what 
makes me wonder if perhaps the stdlib http.Client cannot operate like this.

If I can't do this with the stdlib http.Client, and I have to roll-my-own 
client of sorts, are there any parts of the existing http package that I 
should be making use of?

Any tips / pointers / info greatly appreciated.

Thanks,
/Jim

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an 
email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/545d0597-66a0-4412-8ca7-3d447f88ccafn%40googlegroups.com
 
<https://groups.google.com/d/msgid/golang-nuts/545d0597-66a0-4412-8ca7-3d447f88ccafn%40googlegroups.com?utm_medium=email_source=footer>
.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an 
email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/f909420c-6662-4ae3-866a-77b2faba9c76n%40googlegroups.com
 
<https://groups.google.com/d/msgid/golang-nuts/f909420c-6662-4ae3-866a-77b2faba9c76n%40googlegroups.com?utm_medium=email_source=footer>
.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving em

Re: [go-nuts] HTTP client - streaming POST body?

2023-03-07 Thread Wojciech Kaczmarek

PS. My choice of http.Request with client.Do was misleading, http.Post can 
be used as well:

func uploadStdin(url string) error {
resp, err := http.Post(url, "text/plain", os.Stdin)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
return fmt.Errorf("bad status: %s", resp.Status)
}
_, err = io.Copy(os.Stdout, resp.Body)
return err
}

wtorek, 7 marca 2023 o 22:41:39 UTC+1 Wojciech Kaczmarek napisał(a):

> Hello,
>
> I think this code example rather streams back the response the server sent 
> from its POST handler
> and it is not exactly what the OP requested ;) (not to mention that the 
> typical result of a POST is
> an immediate redirect, possibly without body at all).
>
> I believe what Jim needs is constructing http.Request with a body in a 
> form of an io.Reader, then executing it.
> Here's the minimal example showing that it can be streamed:
>
> ```go
>
> func uploadStdin(url string) error {
> req, err := http.NewRequest("POST", url, os.Stdin)
>
> if err != nil {
> return err
> }
> resp, err := http.DefaultClient.Do(req)
>
> if err != nil {
> return err
> }
> defer resp.Body.Close()
> if resp.StatusCode >= 400 {
> return fmt.Errorf("bad status: %s", resp.Status)
>
> }
> _, err = io.Copy(os.Stdout, resp.Body)
> return err
> }
> ```
>
> I just tested it and indeed, it gets uploaded after I give some input via 
> stdin and hit ctrl+D.
>
> Best,
> Wojtek
>
>
>
> wtorek, 7 marca 2023 o 06:36:38 UTC+1 Amnon napisał(a):
>
> As Bruno said.
>
> Try something like this:
>
> func uploadBig(url string ) error {
> file, err := os.Open("bigfile.txt")
> if err != nil {
> return err
> }
> defer file.Close()
> resp, err := http.Post(url, "text/plain", file)
> if err != nil {
> return err
> }
> defer resp.Body.Close()
> if resp.StatusCode >= 400 {
> return fmt.Errorf("Bad Status %s", resp.Status)
> }
> _, err = io.Copy(os.Stdout, resp.Body)
> return err
> }
>
>
> On Monday, 6 March 2023 at 15:17:36 UTC Bruno Albuquerque wrote:
>
> The body is just an io.Reader. You just need to write one that 
> materializes and streams the data you want instead of using a buffer with 
> all of it. The entire design is to allow things like what you want to do. :)
>
> -Bruno
>
>
> On Mon, Mar 6, 2023 at 10:08 AM 'Jim Smart' via golang-nuts <
> golan...@googlegroups.com> wrote:
>
> Hi all,
>
> I'm looking for an HTTP client that allows my code to write the body 
> content down the pipe while doing a POST.
>
> The use case here is that I'm wishing to send very large UPDATE/INSERT 
> queries/commands to an HTTP endpoint, and the body content of those 
> queries/commands is actually generated from a database. So it would be 
> better, memory- and performance- wise, if I could write that straight down 
> the pipe during the request, as opposed to manifesting the whole of the 
> query/command into a buffer before doing the POST.
>
> Is this possible at all with the stdlib http.Client? Perhaps with an 
> io.Pipe as the request body?
>
> I did look at the code for http.Client a little, but there's a lot of it, 
> and it's not the simplest of code to follow. I do have a base understanding 
> of the HTTP 1.1 protocol, but there's a lot more to http.Client than just 
> that, of course.
>
> I've also tried looking for existing examples showing similar 
> functionality, but did not seem to find anything. Which is partly what 
> makes me wonder if perhaps the stdlib http.Client cannot operate like this.
>
> If I can't do this with the stdlib http.Client, and I have to roll-my-own 
> client of sorts, are there any parts of the existing http package that I 
> should be making use of?
>
> Any tips / pointers / info greatly appreciated.
>
> Thanks,
> /Jim
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/545d0597-66a0-4412-8ca7-3d447f88ccafn%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/golang-nuts/545d0597-66a0-4412-8ca7-3d447f88ccafn%40googlegroups.com?utm_medium=email_source=footer>
> .
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/93606431-b8f8-4032-b7e5-a4ffa0898ac5n%40googlegroups.com.

Re: [go-nuts] HTTP client - streaming POST body?

2023-03-07 Thread Wojciech Kaczmarek

Hello,

I think this code example rather streams back the response the server sent 
from its POST handler
and it is not exactly what the OP requested ;) (not to mention that the 
typical result of a POST is
an immediate redirect, possibly without body at all).

I believe what Jim needs is constructing http.Request with a body in a form 
of an io.Reader, then executing it.
Here's the minimal example showing that it can be streamed:

```go

func uploadStdin(url string) error {
req, err := http.NewRequest("POST", url, os.Stdin)
if err != nil {
return err
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
return fmt.Errorf("bad status: %s", resp.Status)
}
_, err = io.Copy(os.Stdout, resp.Body)
return err
}
```

I just tested it and indeed, it gets uploaded after I give some input via 
stdin and hit ctrl+D.

Best,
Wojtek



wtorek, 7 marca 2023 o 06:36:38 UTC+1 Amnon napisał(a):

As Bruno said.

Try something like this:

func uploadBig(url string ) error {
file, err := os.Open("bigfile.txt")
if err != nil {
return err
}
defer file.Close()
resp, err := http.Post(url, "text/plain", file)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
return fmt.Errorf("Bad Status %s", resp.Status)
}
_, err = io.Copy(os.Stdout, resp.Body)
return err
}


On Monday, 6 March 2023 at 15:17:36 UTC Bruno Albuquerque wrote:

The body is just an io.Reader. You just need to write one that materializes 
and streams the data you want instead of using a buffer with all of it. The 
entire design is to allow things like what you want to do. :)

-Bruno


On Mon, Mar 6, 2023 at 10:08 AM 'Jim Smart' via golang-nuts <
golan...@googlegroups.com> wrote:

Hi all,

I'm looking for an HTTP client that allows my code to write the body 
content down the pipe while doing a POST.

The use case here is that I'm wishing to send very large UPDATE/INSERT 
queries/commands to an HTTP endpoint, and the body content of those 
queries/commands is actually generated from a database. So it would be 
better, memory- and performance- wise, if I could write that straight down 
the pipe during the request, as opposed to manifesting the whole of the 
query/command into a buffer before doing the POST.

Is this possible at all with the stdlib http.Client? Perhaps with an 
io.Pipe as the request body?

I did look at the code for http.Client a little, but there's a lot of it, 
and it's not the simplest of code to follow. I do have a base understanding 
of the HTTP 1.1 protocol, but there's a lot more to http.Client than just 
that, of course.

I've also tried looking for existing examples showing similar 
functionality, but did not seem to find anything. Which is partly what 
makes me wonder if perhaps the stdlib http.Client cannot operate like this.

If I can't do this with the stdlib http.Client, and I have to roll-my-own 
client of sorts, are there any parts of the existing http package that I 
should be making use of?

Any tips / pointers / info greatly appreciated.

Thanks,
/Jim

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an 
email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/545d0597-66a0-4412-8ca7-3d447f88ccafn%40googlegroups.com
 

.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/f909420c-6662-4ae3-866a-77b2faba9c76n%40googlegroups.com.

[go-nuts] Re: [security] Go 1.19.3 and Go 1.18.8 are released

2022-11-02 Thread Wojciech Kaczmarek

Thank you Heschi, it worked.

W.

wtorek, 1 listopada 2022 o 19:03:46 UTC+1 Heschi Kreinick napisał(a):

> Thanks, good catch. We accidentally signed the installer twice, uploaded 
> the first one, and posted the checksum for the second one. We'll correct 
> the checksum.
>
> On Tuesday, November 1, 2022 at 1:28:49 PM UTC-4 Wojciech Kaczmarek wrote:
>
>> I think it's important to report that  go1.18.8.windows-amd64.msi 
>> <https://go.dev/dl/go1.18.8.windows-amd64.msi> doesn't match the 
>> published sha256 checksum. 
>>
>>
>> wtorek, 1 listopada 2022 o 18:01:35 UTC+1 anno...@golang.org napisał(a):
>>
>>> Hello gophers,
>>>
>>> We have just released Go versions 1.19.3 and 1.18.8, minor point 
>>> releases.
>>>
>>> These minor releases include 1 security fixes following the security 
>>> policy <https://go.dev/security>:
>>>
>>>- 
>>>
>>>syscall, os/exec: unsanitized NUL in environment variables
>>>
>>>On Windows, syscall.StartProcess and os/exec.Cmd did not properly 
>>>check for invalid environment variable values. A malicious environment 
>>>variable value could exploit this behavior to set a value for a 
>>> different 
>>>environment variable. For example, the environment variable string 
>>>"A=B\x00C=D" set the variables "A=B" and "C=D".
>>>
>>>Thanks to RyotaK (https://twitter.com/ryotkak) for reporting this 
>>>issue.
>>>
>>>This is CVE-2022-41716 and Go issue https://go.dev/issue/56284.
>>>
>>> View the release notes for more information:
>>> https://go.dev/doc/devel/release#go1.19.3
>>>
>>> You can download binary and source distributions from the Go website:
>>> https://go.dev/dl/
>>>
>>> To compile from source using a Git clone, update to the release with
>>> git checkout go1.19.3 and build as usual.
>>>
>>> Thanks to everyone who contributed to the releases.
>>>
>>> Cheers,
>>> Matthew and Heschi for the Go team
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/965bbe27-06b2-44d6-86bf-9b076625ccban%40googlegroups.com.

[go-nuts] Re: [security] Go 1.19.3 and Go 1.18.8 are released

2022-11-01 Thread Wojciech Kaczmarek

I think it's important to report that  go1.18.8.windows-amd64.msi 
 doesn't match the published 
sha256 checksum. 


wtorek, 1 listopada 2022 o 18:01:35 UTC+1 anno...@golang.org napisał(a):

> Hello gophers,
>
> We have just released Go versions 1.19.3 and 1.18.8, minor point releases.
>
> These minor releases include 1 security fixes following the security 
> policy :
>
>- 
>
>syscall, os/exec: unsanitized NUL in environment variables
>
>On Windows, syscall.StartProcess and os/exec.Cmd did not properly 
>check for invalid environment variable values. A malicious environment 
>variable value could exploit this behavior to set a value for a different 
>environment variable. For example, the environment variable string 
>"A=B\x00C=D" set the variables "A=B" and "C=D".
>
>Thanks to RyotaK (https://twitter.com/ryotkak) for reporting this 
>issue.
>
>This is CVE-2022-41716 and Go issue https://go.dev/issue/56284.
>
> View the release notes for more information:
> https://go.dev/doc/devel/release#go1.19.3
>
> You can download binary and source distributions from the Go website:
> https://go.dev/dl/
>
> To compile from source using a Git clone, update to the release with
> git checkout go1.19.3 and build as usual.
>
> Thanks to everyone who contributed to the releases.
>
> Cheers,
> Matthew and Heschi for the Go team
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/b3a092a1-38a2-452b-9451-ceda7ab0cdcdn%40googlegroups.com.

[go-nuts] Re: Performance degradation when reading STDIN to multiple buffers?

Re: [go-nuts] HTTP client - streaming POST body?

Re: [go-nuts] HTTP client - streaming POST body?

Re: [go-nuts] HTTP client - streaming POST body?

[go-nuts] Re: [security] Go 1.19.3 and Go 1.18.8 are released

[go-nuts] Re: [security] Go 1.19.3 and Go 1.18.8 are released

6 matches

Site Navigation

Mail list logo

Footer information