Re: [go-nuts] HTTP client - streaming POST body?

['Jim Smart' via golang-nuts]

Ok, cool, thanks — that was what I asked in my OP :)

> Is this possible at all with the stdlib http.Client? Perhaps with an io.Pipe 
> as the request body?

/J

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/6C7031DD-0E1D-4848-BAA5-0028962AF707%40jimsmart.org.

Re: [go-nuts] HTTP client - streaming POST body?

[burak serdar]

On Wed, Mar 22, 2023 at 8:32 PM 'Christian Stewart' via golang-nuts <
golang-nuts@googlegroups.com> wrote:

> you can achieve this using an io.Pipe. The io.Pipe function returns a
> connected pair of *PipeReader and *PipeWriter, where writes to the
> *PipeWriter are directly read from the *PipeReader. The Write call on the
> *PipeWriter will block until the data is read from the *PipeReader.
>
> pr, pw := io.Pipe()
>
> Pass the pr as the body and write to the pw.
>

And when you are done generating the content, close pw.

>
> On Wed, Mar 22, 2023, 7:06 PM 'Jim Smart' via golang-nuts <
> golang-nuts@googlegroups.com> wrote:
>
>> The issue here, isn’t that I am uploading a big file — that’s easy.
>>
>> As I said in my initial post:
>>
>> > The use case here is that I'm wishing to send very large UPDATE/INSERT
>> queries/commands to an HTTP endpoint, and the body content of those
>> queries/commands is actually generated from a database.
>>
>> The content I wish to push to the server, is /generated/ content. So
>> really what I want is a something I can directly write into.
>>
>> I am trying to avoid generating my upload content into a buffer first.
>> Because the data can be very large.
>>
>> — It’s easy to say “write a reader” but writing it as a reader involves
>> doing complete inversion of control on my code, and isn’t really feasible.
>> I can’t easily make the code I have which has complex logic to build the
>> upload data using Writes, into something that is then driven by Reads.
>>
>> Which is why I asked if it was possible to somehow Write straight down
>> the connection.
>>
>>
>> — Thanks for the suggestions all the same.
>>
>> /J
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/C5B2823D-2458-4F91-A09D-6B12F74CD8B3%40jimsmart.org
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/CA%2Bh8R2pTUXsWPpywxhRJi_Bs3ZP0hmVP%3DkqZMe6FwJWJS-3BDg%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAMV2RqoO26VS-G05Y6U-b4VKJMUQPvMMun7SMZtbU0djJMQqEw%40mail.gmail.com.

Re: [go-nuts] HTTP client - streaming POST body?

['Christian Stewart' via golang-nuts]

you can achieve this using an io.Pipe. The io.Pipe function returns a
connected pair of *PipeReader and *PipeWriter, where writes to the
*PipeWriter are directly read from the *PipeReader. The Write call on the
*PipeWriter will block until the data is read from the *PipeReader.

pr, pw := io.Pipe()

Pass the pr as the body and write to the pw.

On Wed, Mar 22, 2023, 7:06 PM 'Jim Smart' via golang-nuts <
golang-nuts@googlegroups.com> wrote:

> The issue here, isn’t that I am uploading a big file — that’s easy.
>
> As I said in my initial post:
>
> > The use case here is that I'm wishing to send very large UPDATE/INSERT
> queries/commands to an HTTP endpoint, and the body content of those
> queries/commands is actually generated from a database.
>
> The content I wish to push to the server, is /generated/ content. So
> really what I want is a something I can directly write into.
>
> I am trying to avoid generating my upload content into a buffer first.
> Because the data can be very large.
>
> — It’s easy to say “write a reader” but writing it as a reader involves
> doing complete inversion of control on my code, and isn’t really feasible.
> I can’t easily make the code I have which has complex logic to build the
> upload data using Writes, into something that is then driven by Reads.
>
> Which is why I asked if it was possible to somehow Write straight down the
> connection.
>
>
> — Thanks for the suggestions all the same.
>
> /J
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/C5B2823D-2458-4F91-A09D-6B12F74CD8B3%40jimsmart.org
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CA%2Bh8R2pTUXsWPpywxhRJi_Bs3ZP0hmVP%3DkqZMe6FwJWJS-3BDg%40mail.gmail.com.

Re: [go-nuts] HTTP client - streaming POST body?

['Jim Smart' via golang-nuts]

The issue here, isn’t that I am uploading a big file — that’s easy.

As I said in my initial post:

> The use case here is that I'm wishing to send very large UPDATE/INSERT 
> queries/commands to an HTTP endpoint, and the body content of those 
> queries/commands is actually generated from a database.

The content I wish to push to the server, is /generated/ content. So really 
what I want is a something I can directly write into.

I am trying to avoid generating my upload content into a buffer first. Because 
the data can be very large.

— It’s easy to say “write a reader” but writing it as a reader involves doing 
complete inversion of control on my code, and isn’t really feasible. I can’t 
easily make the code I have which has complex logic to build the upload data 
using Writes, into something that is then driven by Reads.

Which is why I asked if it was possible to somehow Write straight down the 
connection.


— Thanks for the suggestions all the same.

/J


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/C5B2823D-2458-4F91-A09D-6B12F74CD8B3%40jimsmart.org.

Re: [go-nuts] HTTP client - streaming POST body?

[Wojciech Kaczmarek]

You are right, I overlooked the "bigfile" name clearly suggesting it's 
about the big content being uploaded. My bad.

Anyway, what's important for the OP is that net/http is well-designed and 
will let you do what you need in this case.

cheers,
-W.


wtorek, 7 marca 2023 o 23:07:15 UTC+1 Christian Stewart napisał(a):

No... The previous example streams the file as the post request. The only 
difference in your example is that you're using stdin.

It's not that complicated, files also implement io.reader. 

On Tue, Mar 7, 2023, 1:41 PM Wojciech Kaczmarek  wrote:

Hello,

I think this code example rather streams back the response the server sent 
from its POST handler
and it is not exactly what the OP requested ;) (not to mention that the 
typical result of a POST is
an immediate redirect, possibly without body at all).

I believe what Jim needs is constructing http.Request with a body in a form 
of an io.Reader, then executing it.
Here's the minimal example showing that it can be streamed:

```go

func uploadStdin(url string) error {
req, err := http.NewRequest("POST", url, os.Stdin)
if err != nil {
return err
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
return fmt.Errorf("bad status: %s", resp.Status)
}
_, err = io.Copy(os.Stdout, resp.Body)
return err
}
```

I just tested it and indeed, it gets uploaded after I give some input via 
stdin and hit ctrl+D.

Best,
Wojtek



wtorek, 7 marca 2023 o 06:36:38 UTC+1 Amnon napisał(a):

As Bruno said.

Try something like this:

func uploadBig(url string ) error {
file, err := os.Open("bigfile.txt")
if err != nil {
return err
}
defer file.Close()
resp, err := http.Post(url, "text/plain", file)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
return fmt.Errorf("Bad Status %s", resp.Status)
}
_, err = io.Copy(os.Stdout, resp.Body)
return err
}


On Monday, 6 March 2023 at 15:17:36 UTC Bruno Albuquerque wrote:

The body is just an io.Reader. You just need to write one that materializes 
and streams the data you want instead of using a buffer with all of it. The 
entire design is to allow things like what you want to do. :)

-Bruno


On Mon, Mar 6, 2023 at 10:08 AM 'Jim Smart' via golang-nuts <
golan...@googlegroups.com> wrote:

Hi all,

I'm looking for an HTTP client that allows my code to write the body 
content down the pipe while doing a POST.

The use case here is that I'm wishing to send very large UPDATE/INSERT 
queries/commands to an HTTP endpoint, and the body content of those 
queries/commands is actually generated from a database. So it would be 
better, memory- and performance- wise, if I could write that straight down 
the pipe during the request, as opposed to manifesting the whole of the 
query/command into a buffer before doing the POST.

Is this possible at all with the stdlib http.Client? Perhaps with an 
io.Pipe as the request body?

I did look at the code for http.Client a little, but there's a lot of it, 
and it's not the simplest of code to follow. I do have a base understanding 
of the HTTP 1.1 protocol, but there's a lot more to http.Client than just 
that, of course.

I've also tried looking for existing examples showing similar 
functionality, but did not seem to find anything. Which is partly what 
makes me wonder if perhaps the stdlib http.Client cannot operate like this.

If I can't do this with the stdlib http.Client, and I have to roll-my-own 
client of sorts, are there any parts of the existing http package that I 
should be making use of?

Any tips / pointers / info greatly appreciated.

Thanks,
/Jim

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an 
email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/545d0597-66a0-4412-8ca7-3d447f88ccafn%40googlegroups.com
 

.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an 
email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/f909420c-6662-4ae3-866a-77b2faba9c76n%40googlegroups.com
 

.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 

Re: [go-nuts] HTTP client - streaming POST body?

[Wojciech Kaczmarek]

PS. My choice of http.Request with client.Do was misleading, http.Post can 
be used as well:

func uploadStdin(url string) error {
resp, err := http.Post(url, "text/plain", os.Stdin)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
return fmt.Errorf("bad status: %s", resp.Status)
}
_, err = io.Copy(os.Stdout, resp.Body)
return err
}

wtorek, 7 marca 2023 o 22:41:39 UTC+1 Wojciech Kaczmarek napisał(a):

> Hello,
>
> I think this code example rather streams back the response the server sent 
> from its POST handler
> and it is not exactly what the OP requested ;) (not to mention that the 
> typical result of a POST is
> an immediate redirect, possibly without body at all).
>
> I believe what Jim needs is constructing http.Request with a body in a 
> form of an io.Reader, then executing it.
> Here's the minimal example showing that it can be streamed:
>
> ```go
>
> func uploadStdin(url string) error {
> req, err := http.NewRequest("POST", url, os.Stdin)
>
> if err != nil {
> return err
> }
> resp, err := http.DefaultClient.Do(req)
>
> if err != nil {
> return err
> }
> defer resp.Body.Close()
> if resp.StatusCode >= 400 {
> return fmt.Errorf("bad status: %s", resp.Status)
>
> }
> _, err = io.Copy(os.Stdout, resp.Body)
> return err
> }
> ```
>
> I just tested it and indeed, it gets uploaded after I give some input via 
> stdin and hit ctrl+D.
>
> Best,
> Wojtek
>
>
>
> wtorek, 7 marca 2023 o 06:36:38 UTC+1 Amnon napisał(a):
>
> As Bruno said.
>
> Try something like this:
>
> func uploadBig(url string ) error {
> file, err := os.Open("bigfile.txt")
> if err != nil {
> return err
> }
> defer file.Close()
> resp, err := http.Post(url, "text/plain", file)
> if err != nil {
> return err
> }
> defer resp.Body.Close()
> if resp.StatusCode >= 400 {
> return fmt.Errorf("Bad Status %s", resp.Status)
> }
> _, err = io.Copy(os.Stdout, resp.Body)
> return err
> }
>
>
> On Monday, 6 March 2023 at 15:17:36 UTC Bruno Albuquerque wrote:
>
> The body is just an io.Reader. You just need to write one that 
> materializes and streams the data you want instead of using a buffer with 
> all of it. The entire design is to allow things like what you want to do. :)
>
> -Bruno
>
>
> On Mon, Mar 6, 2023 at 10:08 AM 'Jim Smart' via golang-nuts <
> golan...@googlegroups.com> wrote:
>
> Hi all,
>
> I'm looking for an HTTP client that allows my code to write the body 
> content down the pipe while doing a POST.
>
> The use case here is that I'm wishing to send very large UPDATE/INSERT 
> queries/commands to an HTTP endpoint, and the body content of those 
> queries/commands is actually generated from a database. So it would be 
> better, memory- and performance- wise, if I could write that straight down 
> the pipe during the request, as opposed to manifesting the whole of the 
> query/command into a buffer before doing the POST.
>
> Is this possible at all with the stdlib http.Client? Perhaps with an 
> io.Pipe as the request body?
>
> I did look at the code for http.Client a little, but there's a lot of it, 
> and it's not the simplest of code to follow. I do have a base understanding 
> of the HTTP 1.1 protocol, but there's a lot more to http.Client than just 
> that, of course.
>
> I've also tried looking for existing examples showing similar 
> functionality, but did not seem to find anything. Which is partly what 
> makes me wonder if perhaps the stdlib http.Client cannot operate like this.
>
> If I can't do this with the stdlib http.Client, and I have to roll-my-own 
> client of sorts, are there any parts of the existing http package that I 
> should be making use of?
>
> Any tips / pointers / info greatly appreciated.
>
> Thanks,
> /Jim
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/545d0597-66a0-4412-8ca7-3d447f88ccafn%40googlegroups.com
>  
> 
> .
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/93606431-b8f8-4032-b7e5-a4ffa0898ac5n%40googlegroups.com.

Re: [go-nuts] HTTP client - streaming POST body?

['Christian Stewart' via golang-nuts]

No... The previous example streams the file as the post request. The only
difference in your example is that you're using stdin.

It's not that complicated, files also implement io.reader.

On Tue, Mar 7, 2023, 1:41 PM Wojciech Kaczmarek  wrote:

> Hello,
>
> I think this code example rather streams back the response the server sent
> from its POST handler
> and it is not exactly what the OP requested ;) (not to mention that the
> typical result of a POST is
> an immediate redirect, possibly without body at all).
>
> I believe what Jim needs is constructing http.Request with a body in a
> form of an io.Reader, then executing it.
> Here's the minimal example showing that it can be streamed:
>
> ```go
>
> func uploadStdin(url string) error {
> req, err := http.NewRequest("POST", url, os.Stdin)
> if err != nil {
> return err
> }
> resp, err := http.DefaultClient.Do(req)
> if err != nil {
> return err
> }
> defer resp.Body.Close()
> if resp.StatusCode >= 400 {
> return fmt.Errorf("bad status: %s", resp.Status)
> }
> _, err = io.Copy(os.Stdout, resp.Body)
> return err
> }
> ```
>
> I just tested it and indeed, it gets uploaded after I give some input via
> stdin and hit ctrl+D.
>
> Best,
> Wojtek
>
>
>
> wtorek, 7 marca 2023 o 06:36:38 UTC+1 Amnon napisał(a):
>
> As Bruno said.
>
> Try something like this:
>
> func uploadBig(url string ) error {
> file, err := os.Open("bigfile.txt")
> if err != nil {
> return err
> }
> defer file.Close()
> resp, err := http.Post(url, "text/plain", file)
> if err != nil {
> return err
> }
> defer resp.Body.Close()
> if resp.StatusCode >= 400 {
> return fmt.Errorf("Bad Status %s", resp.Status)
> }
> _, err = io.Copy(os.Stdout, resp.Body)
> return err
> }
>
>
> On Monday, 6 March 2023 at 15:17:36 UTC Bruno Albuquerque wrote:
>
> The body is just an io.Reader. You just need to write one that
> materializes and streams the data you want instead of using a buffer with
> all of it. The entire design is to allow things like what you want to do. :)
>
> -Bruno
>
>
> On Mon, Mar 6, 2023 at 10:08 AM 'Jim Smart' via golang-nuts <
> golan...@googlegroups.com> wrote:
>
> Hi all,
>
> I'm looking for an HTTP client that allows my code to write the body
> content down the pipe while doing a POST.
>
> The use case here is that I'm wishing to send very large UPDATE/INSERT
> queries/commands to an HTTP endpoint, and the body content of those
> queries/commands is actually generated from a database. So it would be
> better, memory- and performance- wise, if I could write that straight down
> the pipe during the request, as opposed to manifesting the whole of the
> query/command into a buffer before doing the POST.
>
> Is this possible at all with the stdlib http.Client? Perhaps with an
> io.Pipe as the request body?
>
> I did look at the code for http.Client a little, but there's a lot of it,
> and it's not the simplest of code to follow. I do have a base understanding
> of the HTTP 1.1 protocol, but there's a lot more to http.Client than just
> that, of course.
>
> I've also tried looking for existing examples showing similar
> functionality, but did not seem to find anything. Which is partly what
> makes me wonder if perhaps the stdlib http.Client cannot operate like this.
>
> If I can't do this with the stdlib http.Client, and I have to roll-my-own
> client of sorts, are there any parts of the existing http package that I
> should be making use of?
>
> Any tips / pointers / info greatly appreciated.
>
> Thanks,
> /Jim
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/545d0597-66a0-4412-8ca7-3d447f88ccafn%40googlegroups.com
> 
> .
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/f909420c-6662-4ae3-866a-77b2faba9c76n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 

Re: [go-nuts] HTTP client - streaming POST body?

[Wojciech Kaczmarek]

Hello,

I think this code example rather streams back the response the server sent 
from its POST handler
and it is not exactly what the OP requested ;) (not to mention that the 
typical result of a POST is
an immediate redirect, possibly without body at all).

I believe what Jim needs is constructing http.Request with a body in a form 
of an io.Reader, then executing it.
Here's the minimal example showing that it can be streamed:

```go

func uploadStdin(url string) error {
req, err := http.NewRequest("POST", url, os.Stdin)
if err != nil {
return err
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
return fmt.Errorf("bad status: %s", resp.Status)
}
_, err = io.Copy(os.Stdout, resp.Body)
return err
}
```

I just tested it and indeed, it gets uploaded after I give some input via 
stdin and hit ctrl+D.

Best,
Wojtek



wtorek, 7 marca 2023 o 06:36:38 UTC+1 Amnon napisał(a):

As Bruno said.

Try something like this:

func uploadBig(url string ) error {
file, err := os.Open("bigfile.txt")
if err != nil {
return err
}
defer file.Close()
resp, err := http.Post(url, "text/plain", file)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
return fmt.Errorf("Bad Status %s", resp.Status)
}
_, err = io.Copy(os.Stdout, resp.Body)
return err
}


On Monday, 6 March 2023 at 15:17:36 UTC Bruno Albuquerque wrote:

The body is just an io.Reader. You just need to write one that materializes 
and streams the data you want instead of using a buffer with all of it. The 
entire design is to allow things like what you want to do. :)

-Bruno


On Mon, Mar 6, 2023 at 10:08 AM 'Jim Smart' via golang-nuts <
golan...@googlegroups.com> wrote:

Hi all,

I'm looking for an HTTP client that allows my code to write the body 
content down the pipe while doing a POST.

The use case here is that I'm wishing to send very large UPDATE/INSERT 
queries/commands to an HTTP endpoint, and the body content of those 
queries/commands is actually generated from a database. So it would be 
better, memory- and performance- wise, if I could write that straight down 
the pipe during the request, as opposed to manifesting the whole of the 
query/command into a buffer before doing the POST.

Is this possible at all with the stdlib http.Client? Perhaps with an 
io.Pipe as the request body?

I did look at the code for http.Client a little, but there's a lot of it, 
and it's not the simplest of code to follow. I do have a base understanding 
of the HTTP 1.1 protocol, but there's a lot more to http.Client than just 
that, of course.

I've also tried looking for existing examples showing similar 
functionality, but did not seem to find anything. Which is partly what 
makes me wonder if perhaps the stdlib http.Client cannot operate like this.

If I can't do this with the stdlib http.Client, and I have to roll-my-own 
client of sorts, are there any parts of the existing http package that I 
should be making use of?

Any tips / pointers / info greatly appreciated.

Thanks,
/Jim

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an 
email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/545d0597-66a0-4412-8ca7-3d447f88ccafn%40googlegroups.com
 

.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/f909420c-6662-4ae3-866a-77b2faba9c76n%40googlegroups.com.

Re: [go-nuts] HTTP client - streaming POST body?

[Yaw Boakye]

I could be misleading you here, but my immediate thought is that whether you
can stream the body/payload depends on what form you're receiving the query
result in? if it's an io.Reader then it looks like you could use it
directly. otherwise
i'm not sure what you could do to the HTTP client to change that (since the
query result would have already materialized).

On Tue, Mar 7, 2023 at 5:36 AM Amnon  wrote:

> As Bruno said.
>
> Try something like this:
>
> func uploadBig(url string ) error {
> file, err := os.Open("bigfile.txt")
> if err != nil {
> return err
> }
> defer file.Close()
> resp, err := http.Post(url, "text/plain", file)
> if err != nil {
> return err
> }
> defer resp.Body.Close()
> if resp.StatusCode >= 400 {
> return fmt.Errorf("Bad Status %s", resp.Status)
> }
> _, err = io.Copy(os.Stdout, resp.Body)
> return err
> }
>
>
> On Monday, 6 March 2023 at 15:17:36 UTC Bruno Albuquerque wrote:
>
> The body is just an io.Reader. You just need to write one that
> materializes and streams the data you want instead of using a buffer with
> all of it. The entire design is to allow things like what you want to do. :)
>
> -Bruno
>
>
> On Mon, Mar 6, 2023 at 10:08 AM 'Jim Smart' via golang-nuts <
> golan...@googlegroups.com> wrote:
>
> Hi all,
>
> I'm looking for an HTTP client that allows my code to write the body
> content down the pipe while doing a POST.
>
> The use case here is that I'm wishing to send very large UPDATE/INSERT
> queries/commands to an HTTP endpoint, and the body content of those
> queries/commands is actually generated from a database. So it would be
> better, memory- and performance- wise, if I could write that straight down
> the pipe during the request, as opposed to manifesting the whole of the
> query/command into a buffer before doing the POST.
>
> Is this possible at all with the stdlib http.Client? Perhaps with an
> io.Pipe as the request body?
>
> I did look at the code for http.Client a little, but there's a lot of it,
> and it's not the simplest of code to follow. I do have a base understanding
> of the HTTP 1.1 protocol, but there's a lot more to http.Client than just
> that, of course.
>
> I've also tried looking for existing examples showing similar
> functionality, but did not seem to find anything. Which is partly what
> makes me wonder if perhaps the stdlib http.Client cannot operate like this.
>
> If I can't do this with the stdlib http.Client, and I have to roll-my-own
> client of sorts, are there any parts of the existing http package that I
> should be making use of?
>
> Any tips / pointers / info greatly appreciated.
>
> Thanks,
> /Jim
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/545d0597-66a0-4412-8ca7-3d447f88ccafn%40googlegroups.com
> 
> .
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/d9e4e00f-2251-453c-88f2-5e2bb8e61c8bn%40googlegroups.com
> 
> .
>


-- 
*Curried programmer*
*Homepage*: http://yawboakye.com
I'm tweeting  when I'm not coding
 when I'm not holding my niece.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAPJoGXsHvNT%2BsMxVs5w9Z%3DSYdyCOrDBYEHh9s6tWui_gVU%3Do4g%40mail.gmail.com.

Re: [go-nuts] HTTP client - streaming POST body?

[Amnon]

As Bruno said.

Try something like this:

func uploadBig(url string ) error {
file, err := os.Open("bigfile.txt")
if err != nil {
return err
}
defer file.Close()
resp, err := http.Post(url, "text/plain", file)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
return fmt.Errorf("Bad Status %s", resp.Status)
}
_, err = io.Copy(os.Stdout, resp.Body)
return err
}


On Monday, 6 March 2023 at 15:17:36 UTC Bruno Albuquerque wrote:

The body is just an io.Reader. You just need to write one that materializes 
and streams the data you want instead of using a buffer with all of it. The 
entire design is to allow things like what you want to do. :)

-Bruno


On Mon, Mar 6, 2023 at 10:08 AM 'Jim Smart' via golang-nuts <
golan...@googlegroups.com> wrote:

Hi all,

I'm looking for an HTTP client that allows my code to write the body 
content down the pipe while doing a POST.

The use case here is that I'm wishing to send very large UPDATE/INSERT 
queries/commands to an HTTP endpoint, and the body content of those 
queries/commands is actually generated from a database. So it would be 
better, memory- and performance- wise, if I could write that straight down 
the pipe during the request, as opposed to manifesting the whole of the 
query/command into a buffer before doing the POST.

Is this possible at all with the stdlib http.Client? Perhaps with an 
io.Pipe as the request body?

I did look at the code for http.Client a little, but there's a lot of it, 
and it's not the simplest of code to follow. I do have a base understanding 
of the HTTP 1.1 protocol, but there's a lot more to http.Client than just 
that, of course.

I've also tried looking for existing examples showing similar 
functionality, but did not seem to find anything. Which is partly what 
makes me wonder if perhaps the stdlib http.Client cannot operate like this.

If I can't do this with the stdlib http.Client, and I have to roll-my-own 
client of sorts, are there any parts of the existing http package that I 
should be making use of?

Any tips / pointers / info greatly appreciated.

Thanks,
/Jim

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an 
email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/545d0597-66a0-4412-8ca7-3d447f88ccafn%40googlegroups.com
 

.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/d9e4e00f-2251-453c-88f2-5e2bb8e61c8bn%40googlegroups.com.

Re: [go-nuts] HTTP client - streaming POST body?

[Bruno Albuquerque]

The body is just an io.Reader. You just need to write one that materializes
and streams the data you want instead of using a buffer with all of it. The
entire design is to allow things like what you want to do. :)

-Bruno


On Mon, Mar 6, 2023 at 10:08 AM 'Jim Smart' via golang-nuts <
golang-nuts@googlegroups.com> wrote:

> Hi all,
>
> I'm looking for an HTTP client that allows my code to write the body
> content down the pipe while doing a POST.
>
> The use case here is that I'm wishing to send very large UPDATE/INSERT
> queries/commands to an HTTP endpoint, and the body content of those
> queries/commands is actually generated from a database. So it would be
> better, memory- and performance- wise, if I could write that straight down
> the pipe during the request, as opposed to manifesting the whole of the
> query/command into a buffer before doing the POST.
>
> Is this possible at all with the stdlib http.Client? Perhaps with an
> io.Pipe as the request body?
>
> I did look at the code for http.Client a little, but there's a lot of it,
> and it's not the simplest of code to follow. I do have a base understanding
> of the HTTP 1.1 protocol, but there's a lot more to http.Client than just
> that, of course.
>
> I've also tried looking for existing examples showing similar
> functionality, but did not seem to find anything. Which is partly what
> makes me wonder if perhaps the stdlib http.Client cannot operate like this.
>
> If I can't do this with the stdlib http.Client, and I have to roll-my-own
> client of sorts, are there any parts of the existing http package that I
> should be making use of?
>
> Any tips / pointers / info greatly appreciated.
>
> Thanks,
> /Jim
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/545d0597-66a0-4412-8ca7-3d447f88ccafn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAEd86TywTP1FMMdcBLH-CN-Nq0uCErM1r2yUZ211YC-%3D4v5mFA%40mail.gmail.com.