You could always use os.Pipe (which unlike io.Pipe is a standard posix
pipe), and wrap one end or the other with bufio.Reader/Writer. Or if all
you want to do is buffer the response from http.Get you can just wrap the
http response reader with bufio.Reader. Or am I missing something here
To give a little bit more background on what I was doing - I have an API
endpoint which is used to download large amounts of data that is backed by
S3 in AWS. Using an io.Pipe to simply proxy from S3 tops out at around
30MB/sec due to single threaded S3 performance and setup time. I wrote an
I would just copy and paste the stdlib pipe.go and change the ctor to take a
size for the channel. I’m pretty sure that is all that is needed to fix the
problem.
> On Nov 21, 2019, at 11:18 PM, Marcin Romaszewicz wrote:
>
>
> I am in fact replacing an io.Pipe implementation, because I need
I am in fact replacing an io.Pipe implementation, because I need to buffer
some data. io.Pipe doesn't buffer, it just matches up read and writes with
each other.
What I'm effectively doing is producing chunks of data at a certain rate
from one server, and forwarding them to another. Due to the
Yes, I already addressed this but I know people often reply as they read.
This is not really a pipe in the traditional posix sense. A bad name, or bad
implementation imo.
> On Nov 21, 2019, at 10:53 PM, Jamil Djadala wrote:
>
> On Thu, 21 Nov 2019 22:37:37 -0600
> Robert Engels wrote:
>
On Thu, 21 Nov 2019 22:37:37 -0600
Robert Engels wrote:
> The OP specifically requested io.Reader/Writer interfaces.
>
> A pipe is what he wants. Not a ring buffer. (A pipe essentially has a
> ring buffer in the implementation though).
from https://golang.org/pkg/io/#Pipe :
The data is
Not sure why the go pipe doesn’t take a size for the inner channel to make the
synchronous nature optional.
> On Nov 21, 2019, at 10:44 PM, Robert Engels wrote:
>
> Sorry in reading the implementation of a pipe in Go, it may not have the
> desired semantics, especially with a large write
Sorry in reading the implementation of a pipe in Go, it may not have the
desired semantics, especially with a large write buffer since writes and reads
are 1 to 1 (very weird impl - this is not a traditional pipe). Dan’s library is
a better choice.
> On Nov 21, 2019, at 10:37 PM, Robert
The package Dan referenced will work too - but it doesn’t look different than
using a pipe with a large write buffer.
> On Nov 21, 2019, at 10:20 PM, Dan Kortschak wrote:
>
> There is this: https://godoc.org/bitbucket.org/ausocean/utils/ring
>
> It has been used in production fairly
The OP specifically requested io.Reader/Writer interfaces.
A pipe is what he wants. Not a ring buffer. (A pipe essentially has a ring
buffer in the implementation though).
> On Nov 21, 2019, at 10:20 PM, Dan Kortschak wrote:
>
> There is this:
There is this: https://godoc.org/bitbucket.org/ausocean/utils/ring
It has been used in production fairly extensively.
On Thu, 2019-11-21 at 19:47 -0800, Marcin Romaszewicz wrote:
> Hi All,
>
> Before I reinvent the wheel, and because this wheel is particularly
> tricky
> to get right, I was
On Thu, Nov 21, 2019 at 7:57 PM Robert Engels wrote:
> Just use a pipe. Part of the std lib.
>
I haven't written any production Go code but am a grey beard. Looking at
the docs, https://golang.org/pkg/io/#Pipe, a io.Pipe provides somewhat
different semantics compared to a traditional ring
Just use a pipe. Part of the std lib.
> On Nov 21, 2019, at 9:47 PM, Marcin Romaszewicz wrote:
>
>
> Hi All,
>
> Before I reinvent the wheel, and because this wheel is particularly tricky to
> get right, I was wondering if anyone was aware of a a library providing
> something like this
>
Hi All,
Before I reinvent the wheel, and because this wheel is particularly tricky
to get right, I was wondering if anyone was aware of a a library providing
something like this
- conforms to io.Reader
- conforms to io.Writer
- Contains a buffer of fixed size, say, 64MB. If you try to write when
14 matches
Mail list logo
