Re: [go-nuts] Is anyone aware of a blocking ring buffer implementation?
You could always use os.Pipe (which unlike io.Pipe is a standard posix pipe), and wrap one end or the other with bufio.Reader/Writer. Or if all you want to do is buffer the response from http.Get you can just wrap the http response reader with bufio.Reader. Or am I missing something here about this use case? On Saturday, November 23, 2019 at 8:51:51 AM UTC-8, Marcin Romaszewicz wrote: > > To give a little bit more background on what I was doing - I have an API > endpoint which is used to download large amounts of data that is backed by > S3 in AWS. Using an io.Pipe to simply proxy from S3 tops out at around > 30MB/sec due to single threaded S3 performance and setup time. I wrote an > adapter around it which downloads increasingly more S3 chunks in parallel, > until my producer rate starts to exceed the consumer rate - now I'm getting > upwards of a 1GB/sec on really large files. The only tricky bit is > sequencing chunks for feeding into the io.Pipe, but there's few of them, so > an array with a mutex around it works just fine. I didn't have to write the > ring buffer in the end. > > Working in Go is a joy at times like this - having a multi threaded > chunked fetcher hiding behind a single threaded HTTP GET all the while not > buffering too much data is a fiendishly hard thing to do in other languages > due to the nature of their threading libraries and the sparse library > support for asynchronous primitives. I've done stuff like this in the past > in the IRIX kernel (don't laugh), or in C/C++ and pthreads, and each of > those would have been days of work to get right instead of half a day of > writing simple code to glue together available library functions. > > -- Marcin > > On Thu, Nov 21, 2019 at 9:48 PM Robert Engels > wrote: > >> I would just copy and paste the stdlib pipe.go and change the ctor to >> take a size for the channel. I’m pretty sure that is all that is needed to >> fix the problem. >> >> On Nov 21, 2019, at 11:18 PM, Marcin Romaszewicz > > wrote: >> >> >> I am in fact replacing an io.Pipe implementation, because I need to >> buffer some data. io.Pipe doesn't buffer, it just matches up read and >> writes with each other. >> >> What I'm effectively doing is producing chunks of data at a certain rate >> from one server, and forwarding them to another. Due to the latency of >> issuing the read from the server, I need to do some pre-fetch into a >> buffer, but I don't want to prefetch to much. >> >> With the pipe implementation, my reader had many stalls waiting for the >> server to produce the next chunk. The server can produce data a lot >> quicker than the reader can read, however, setup time is high. It's a >> complicated mess :) >> >> On Thu, Nov 21, 2019 at 8:37 PM Robert Engels > > wrote: >> >>> The OP specifically requested io.Reader/Writer interfaces. >>> >>> A pipe is what he wants. Not a ring buffer. (A pipe essentially has a >>> ring buffer in the implementation though). >>> >>> > On Nov 21, 2019, at 10:20 PM, Dan Kortschak >> > wrote: >>> > >>> > There is this: https://godoc.org/bitbucket.org/ausocean/utils/ring >>> > >>> > It has been used in production fairly extensively. >>> > >>> >> On Thu, 2019-11-21 at 19:47 -0800, Marcin Romaszewicz wrote: >>> >> Hi All, >>> >> >>> >> Before I reinvent the wheel, and because this wheel is particularly >>> >> tricky >>> >> to get right, I was wondering if anyone was aware of a a library >>> >> providing >>> >> something like this >>> >> >>> >> - conforms to io.Reader >>> >> - conforms to io.Writer >>> >> - Contains a buffer of fixed size, say, 64MB. If you try to write >>> >> when the >>> >> buffer is too full, write blocks. When you try to read from an empty >>> >> one, >>> >> read blocks. >>> >> >>> >> This describes the behavior of make(chan byte, 64 * MB), however, it >>> >> doesn't seem to be efficient to do this with a channel. Say I'm >>> >> transferring a few hundred GB via this mechanism. A chan of byte >>> >> would need >>> >> a few hundred billion byte writes, and a few hundred billion reads. >>> >> Doesn't >>> >> sound like it could be efficient at all. You can't implement this >>> >> with a >>> >> channel of []byte, because you'd violate your buffering limit if you >>> >> pass >>> >> too large of a byte array, so you'd have to chop things up into >>> >> blocks, but >>> >> maybe that's simpler than a full fledged blocking ring buffer. >>> >> >>> >> Anyhow, I've implemented such things in various OS level plumbing, so >>> >> I >>> >> know I can do it in Go much more easily, just hoping to avoid it :) >>> >> >>> >> Thanks for any advice, >>> >> -- Marcin >>> >> >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> Groups "golang-nuts" group. >>> > To unsubscribe from this group and stop receiving emails from it, send >>> an email to golan...@googlegroups.com . >>> > To view this discussion on the web visit >>> https://groups.google.com
[go-nuts] Re: ParseFiles equivalent for strings?
You can, if you wish, get the best of both worlds by using go generate and
a tool like https://github.com/shurcooL/vfsgen to automatically generate go
source from those files.
On Sunday, August 5, 2018 at 7:33:41 AM UTC-7, buc...@gmail.com wrote:
>
> After changing from embedding my .html/css code in the body of my .go
> program, to making a separate, stand-alone .html/css file and including a
> template.Must(template.ParseFiles("filename.html"))' line in the .go source
> file, I can see that the go creators are a lot smarter than I am.
>
> MUCH easier to work with the separate files than having them combined.
>
> Thanks to all who waded in on this pretty silly pondering of mine.
>
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[go-nuts] Re: Gift wrap errors or not?
Errors in Go aren't really used the same was as exceptions in other languages, and that's a good thing! Most of the time, all you care about is whether the error is non-nil or not. Occasionally you care about the specific type of error, in which case the package emitting the error should define a test method or constant to compare to (as with io.IsNotExit()). Beyond that, erros are just things you throw into your debug log. If it's useful to you to include stack information in the error message, you can do so. fmt.Errorf() is my usual method of formatting errors for such things. I hesitate to mention this, since it leads to a lot of bad practice in my opinion, but if you want something that acts more like other languages' versions of exceptions, you can always use panic/recover... On Thursday, March 1, 2018 at 12:19:16 PM UTC-8, Tad Vizbaras wrote: > > It has been almost two years since this post: > https://dave.cheney.net/2016/06/12/stack-traces-and-the-errors-package > > What is current best practice? Should I use some package to gift wrap > errors in order to get stack trace attached to them? > > Standard library errors package is really "bare bones". Some guidance > would be appreciated. > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[go-nuts] Re: Build started to fail with 1.10
I've run into this as well. It looks somewhat related to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=314435 As the maintainers explained there, nanosleep isn't part of the c99 standard (it is part of the posix standard). When you set -std=c99, that turns off -D_GNU_SOURCE, which turns off _POSIX_C_SOURCE in features.h, which turns off nanosleep, amongst other things. Not clear on exactly why this didn't matter in go 1.9, however. On Friday, February 23, 2018 at 8:53:46 AM UTC-8, Vladislav Mitov wrote: > > Interesting fact is that if I remove "-std=c99" from CGO_CFLAGS it works > locally, fails in Travis though. > > On Friday, February 23, 2018 at 6:32:29 PM UTC+2, Vladislav Mitov wrote: >> >> Not sure, I'l building in golang:latest >> https://github.com/miracl/gomiracl/blob/master/Dockerfile#L23. >> >> On Friday, February 23, 2018 at 12:26:27 PM UTC+2, Dave Cheney wrote: >>> >>> The failing line was added in december last year, >>> >>> >>> https://github.com/golang/go/commit/7cba779cea5#diff-56c7df71bce32f8e50115128ae30941eR13 >>> >>> This also adds a dependency on time.h. Is time.h available in your build >>> container? >>> >>> On Friday, 23 February 2018 20:09:02 UTC+11, Владислав Митов wrote: Nah, it's not that - https://travis-ci.org/miracl/gomiracl/jobs/345158452. If fails also in golang:latest where the yaml gotcha it's not a thing. Solid suggestion though, thanks. On Friday, February 23, 2018 at 11:02:59 AM UTC+2, Владислав Митов wrote: > > Ah, I read about that but ti says `go version go1.10 linux/amd64`. > I'll try putting it in quotes now. > > On Friday, February 23, 2018 at 1:57:53 AM UTC+2, Nic Pottier wrote: >> >> This is almost certainly because your .travis file is specifying your >> version as `1.10` vs `"1.10"` >> >> So your build is building with go 1.1 instead of 1.10. >> >> Ran into this myself this morning. >> >> -Nic >> >> On Thursday, February 22, 2018 at 11:38:43 AM UTC-5, Владислав Митов >> wrote: >>> >>> Hey guys, >>> >>> One of my build started failing with 1.10. Basically I'm building a >>> wrapper around a C library and it fails with: >>> >>> # runtime/cgo >>> gcc_libinit.c: In function '_cgo_try_pthread_create': >>> gcc_libinit.c:97:18: error: storage size of 'ts' isn't known >>> struct timespec ts; >>> ^~ >>> gcc_libinit.c:110:3: error: implicit declaration of function ' >>> nanosleep' [-Werror=implicit-function-declaration] >>>nanosleep(&ts, nil); >>>^ >>> gcc_libinit.c:97:18: error: unused variable 'ts' >>> [-Werror=unused-variable] >>> struct timespec ts; >>> ^~ >>> cc1: all warnings being treated as errors >>> https://travis-ci.org/miracl/gomiracl/jobs/344782123 >>> >>> Any hint's what's causing this? >>> >>> Cheers, >>> Vladi >>> >> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[go-nuts] Getting the current process signal disposition
TL;DR it would be nice if there were a public API to find the current signal disposition of your process. The runtime already stores that information in runtime.sigtable, but doesn't expose it. I ran into a gotcha with handling SIGHUP the other day. I need to handle SIGHUP in order to shut down gracefully. However, nohup works by setting the process's signal disposition to ignore SIGHUP, and adding a handler un-ignores the signal, meaning the process shuts down when it shouldn't. Figuring out whether it was ignored, however, proved unpleasantly difficult. I know this would be straightforward enough to solve with cgo, but that opens us up to deployment difficulties for customers with older libc. It's a bit aggravating that the runtime already has the information I want but doesn't give me a way to see it. For the moment I'm just making the syscall but this seems like information the os/signal package should expose, for people who for example want to know what signal.Reset is resetting things *to*. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[go-nuts] Re: why do we need generic thread.
Personally I don't think a completely, um, generic version of generics is
needed. But something like a list comprehension, possibly specifically
limited to "select"/"map" type comprehensions which do not change slice or
map length, would be invaluable. There's already a lot of special magic
around slices and maps, so adding more wouldn't be as world-changing as
adding it everywhere else (in both good and bad ways), while enabling one
to write a lot of useful list-processing utilities once instead of several
times. Since most of real-life software engineering turns into list
processing at some point, that would be super nice.
On Friday, August 4, 2017 at 9:30:43 PM UTC-7, Lucio wrote:
>
> We know how to deal with that without generics. But how would you define
> generics to address that more elegantly than you would with already
> available Go constructs (type assertions)?
>
> Lucio.
>
> On Saturday, 5 August 2017 01:48:28 UTC+2, T L wrote:
>>
>> A case:
>>
>> func iterateAndCopy(slice []T, p *T) {
>> for _, v := range slice {
>> *p = v
>> }
>> }
>>
>> func Benchmark_CopyCosts(b *testing.B) {
>> b.Run("bool-4", func(b *testing.B){
>> iterateAndCopy(make([]bool, 4), new(bool))
>> })
>> b.Run("int32-4", func(b *testing.B){
>> iterateAndCopy(make([]int32, 4), new(int32))
>> })
>> b.Run("int64-4", func(b *testing.B){
>> iterateAndCopy(make([]int64, 4), new(int64))
>> })
>> b.Run("[128]int-4", func(b *testing.B){
>> iterateAndCopy(make([][128]int64, 4), new([128]int64))
>> })
>> b.Run("struct{a,b,c,d int32}-100", func(b *testing.B){
>> iterateAndCopy(make([]struct{a,b,c,d int32}, 100),
>> new(struct{a,b,c,d int32}))
>> })
>> b.Run("bool-100", func(b *testing.B){
>> iterateAndCopy(make([]bool, 100), new(bool))
>> })
>> b.Run("int32-100", func(b *testing.B){
>> iterateAndCopy(make([]int32, 100), new(int32))
>> })
>> b.Run("int64-100", func(b *testing.B){
>> iterateAndCopy(make([]int64, 100), new(int64))
>> })
>> b.Run("[128]int-100", func(b *testing.B){
>> iterateAndCopy(make([][128]int64, 100), new([128]int64))
>> })
>> b.Run("struct{a,b,c,d int32}-100", func(b *testing.B){
>> iterateAndCopy(make([]struct{a,b,c,d int32}, 100),
>> new(struct{a,b,c,d int32}))
>> })
>> }
>>
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
