[go-nuts] Re: Performance of unique package

The unique package itself has a few benchmarks you can run if you'd like, though they're somewhat limited. They exist primarily to validate that the implementation has

Re: [go-nuts] Memory operations hung in semacquire state, GC routine parked while holding semaphore?

The example goroutine in the original post is parked waiting for some other goroutine to finish up the GC cycle. Somewhere, a goroutine is getting stuck trying to finish it up, which could possibly be a deadlock. (I am especially suspicious of a deadlock bug because most threads are stopped the

[go-nuts] Re: discrepancy between goal heap size and NextGC

838103616 bytes is 799 MiB (Mebibytes, the power-of-two-based prefix), which matches GC 13. (Note that the computation for the heap goal just divides by 1<<20, or 1 MiB.) The discrepancy in GC count is probably due to the fact that the GC count value reported by the GC trace is *after* the incr

[go-nuts] Re: array index not starting at 0

I see that you're on Windows. The clock granularity on Windows is really coarse (15 ms) because AFAIK there are no fast fine-grained OS-level clocks available. There are fine-grained clocks, but they're heavyweight. They're too heavyweight for many situations in the runtime, but honestly, they'

Re: [go-nuts] Re: Passing 2d arrays from Go to C using runtime.Pinner

Thanks for the reproducer. I think I've puzzled out what's going wrong and it's pretty subtle. TL;DR: You can work around this by either calling `calloc` or just allocating `inRows` as Go memory and pinning that as well. The latter will be safer and faster overall. It's not totally clear to me at

[go-nuts] Re: Passing 2d arrays from Go to C using runtime.Pinner

Ignoring more efficient ways to pass memory to C for a moment, superficially I do think your code using Pinner should work. Do you have a full reproducer? It's hard to tell from just looking at your code if your code is the problem, or its just enough to trigger some other cgo issue elsewhere i

[go-nuts] Re: garbage collections on Windows and Mac

On Tuesday, January 9, 2024 at 10:21:26 PM UTC-5 Michael Mitchell wrote: Hi, I have noticed on my macbook pro that HeapInUse is always slightly above NextGC when the garbage collection occurs. According to mstats.go, the "garbage collector's goal is to keep HeapAlloc less than NextGC" and

[go-nuts] Re: Low memory utilization when using soft memory limit with GOGC=off

On Tuesday, December 5, 2023 at 1:06:19 AM UTC-5 Zhihui Jiang wrote: Hi there, We are running a large scale recommendation system using Golang and we are working on some GC related improvement recently. One of the changes we are trying to apply is to use soft memory limit with GOGC=off as su

Re: [go-nuts] Go/Python interop

On Thu, Nov 23, 2023 at 11:52 AM Sebastien Binet wrote: > On Thu Nov 23, 2023 at 15:23 CET, 'Michael Knyszek' via golang-nuts wrote: > > Also, I'd like to clarify: > > > > - [David]: is it a good idea to use cgo for Go-Python interop? > > - [Michael

Re: [go-nuts] Go/Python interop

Also, I'd like to clarify: - [David]: is it a good idea to use cgo for Go-Python interop? - [Michael]: no. better with pipe or RPC I'm wrong about this. A conversation after the meeting clarified a misunderstanding I had about Go->Python calls specifically. Both Go->Python and Python->Go with c

[go-nuts] Re: Why golang's garbage collector didn't free obj when an finalizer is associate with it？

On Tuesday, November 7, 2023 at 2:32:28 AM UTC-5 Jan wrote: Btw, I don't follow the sentence: "the GC necessarily has to keep referents of the object-to-be-finalized live even if the object isn't referenced anymore" That is true for objects not in cycles that need to be finalized as well. I'

[go-nuts] Re: Why golang's garbage collector didn't free obj when an finalizer is associate with it？

Yes, cycles containing a finalizer aren't guaranteed to be freed. As others have pointed out, this is documented. SetFinalizer is really designed for a narrow set of use-cases, so concessions were made for overall GC performance. This case is one of them. IIUC, the core problem is a combination

Re: [go-nuts] Suggestions on optimizing Go GC

I second Jason's message, and +1 to off-heap memory as a last resort. Here are a few more details: For a starting point on how to reduce memory allocations directly, see https://go.dev/doc/gc-guide#Optimization_guide. Note that this may require restructuring your program in places. (e.g. passin

Re: [go-nuts] Question about runtime.Pinner

On Thursday, October 12, 2023 at 9:55:12 AM UTC-4 Ian Lance Taylor wrote: On Thu, Oct 12, 2023 at 5:06 AM Pascal Costanza wrote: > > I have a question about runtime.Pinner: The documentation states that you can only pin an object when it’s the result of calling new, taking the address of a

[go-nuts] Re: Question about runtime.Pinner

On Thursday, October 12, 2023 at 8:07:02 AM UTC-4 Pascal Costanza wrote: Hi, I have a question about runtime.Pinner: The documentation states that you can only pin an object when it’s the result of calling new, taking the address of a composite literal, or taking the address of a local variab

[go-nuts] Re: go 1.21 tuned gc: what's it about?

I saw this post a while back and forgot to reply; sorry about that. It's about this issue but also a little bit about this issue . Both of those problems can be summarized as the GC pacing itself for the exac

[go-nuts] Re: Maybe a Bug? The Go compiler stores a stack pointer into a global object

That line (the full sentence is "The garbage collector now includes non-heap sources of garbage collector work (e.g., stack scanning) when determining how frequently to run.") is unrelated. It only refers to a change in accounting for what gets included in the GOGC calculation, not a change in

[go-nuts] Re: about "hybrid barrier"

On Friday, July 28, 2023 at 8:05:03 AM UTC-4 metronome wrote: Hi, I came across two questions regarding the hybrid barrier that I am hoping someone can help with. 1. Chang https://go-review.googlesource.com/c/go/+/31765 says: *"It's unconditional for now because barriers on channel operati

[go-nuts] Re: Possible Go Compiler or Runtime bug?

On Thursday, July 27, 2023 at 6:16:57 PM UTC-4 Michael Knyszek wrote: I believe this is working as intended, because I don't think the spec makes any guarantees about the capacity of the slice you get back from a conversion. (Other than the fact that the

[go-nuts] Re: Possible Go Compiler or Runtime bug?

The number 32 for the capacity comes from this constant , and the returned byte slice in that case is likely coming from a compiler-inserted call to stringtoslicebyte

[go-nuts] Go 1.20 is released

Hello gophers, We have just released Go 1.20. To find out what has changed in Go 1.20, read the release notes: https://go.dev/doc/go1.20 You can download binary and source distributions from our download page: https://go.dev/dl/#go1.20 If you have Go installed already, an easy way to try go1.20

Re: [go-nuts] Create a 1GB ballast but it takes up RSS and pages are Dirty?

On Wed, Nov 16, 2022 at 10:19 AM Jie Zhang wrote: > Actually, we used go1.16.5 before go1.19 released. Now we're consider > using go1.19 GOMEMLIMIT instead. > > But GOMEMLIMIT is a soft limit, if you deploy multiple services in the > same host (not per container per service), there will be proble

Re: [go-nuts] Create a 1GB ballast but it takes up RSS and pages are Dirty?

That's correct. The runtime has a simple heuristic for avoiding zeroing but it's far from perfect. As a result, a ballast is inherently always going to be a little risky. This is especially true on some platforms, such as Windows, since there's no way to avoid marking the memory as committed (W

[go-nuts] Re: Go allocator: allocates arenas, reclaims pages?

Just to clarify, when I said "publicly visible" I meant via blog posts and talks. There are a few design documents

[go-nuts] Re: Go allocator: allocates arenas, reclaims pages?

Thanks for the question. The scavenger isn't as publicly visible as other parts of the runtime. You've got it mostly right, but I'm going to repeat some things you've already said to make it clear what's different. The Go runtime maps new heap memory (specifically: a new virtual memory mapping

Re: [go-nuts] TotalAlloc dropped in runtime/metrics

Hi Marco, https://go-review.googlesource.com/c/go/+/312431 just landed and should resolve the issue for Go 1.17. On Tue, Apr 20, 2021 at 3:28 PM Michael Knyszek wrote: > Oh, actually, you can compute Mallocs and Frees from allocs-by-size and > frees-by-size (summing the total # of samples). You

Re: [go-nuts] TotalAlloc dropped in runtime/metrics

Oh, actually, you can compute Mallocs and Frees from allocs-by-size and frees-by-size (summing the total # of samples). You can only estimate TotalAlloc though, which is genuinely missing. On Tue, Apr 20, 2021 at 3:14 PM Michael Knyszek wrote: > Oh gosh, I think TotalAlloc, Mallocs, and Frees ar

Re: [go-nuts] TotalAlloc dropped in runtime/metrics

Oh gosh, I think TotalAlloc, Mallocs, and Frees are actually an oversight on my part. Sorry about that. They're very easy to add and I can probably even add them for this release. Please do use the new runtime/metrics package! Most of the other metrics should be there in some form (e.g. the divis

[go-nuts] Re: super large virtual memory allocation for simplest golang apps, why?

On Thursday, January 14, 2021 at 12:27:25 PM UTC-5 laos...@gmail.com wrote: > I will add info to issues 39400 in the future. > > while Golang is run at data centers, we still need be memory efficient no > matter how cheap memory is, especially when you want to run thousands of > them in parall

[go-nuts] Re: super large virtual memory allocation for simplest golang apps, why?

This change came in Go 1.14 as part of an allocator scalability improvement (a specific data structure benefits from having a large memory reservation), but with it also came an effort to mitigate the (potential) negative effects you mention. I think all of the possible problematic cases you li

Re: [go-nuts] Does recent bug in debug.FreeOSMemory also affect automatic background scavenging?

Hi John, If you're referring to scavenging that kicked in every 2.5 minutes and returned spans which had been unused for 5 minutes, that's been replaced with an asynchronous, continuous mechanism as of Go 1.13 ( https://github.com/golang/go/issues/30333). This policy is generally more aggressive t

Re: [go-nuts] Change in virtual memory patterns in Go 1.12

I'm glad to hear it! What went into Go 1.12.5 should supersede that patch anyway. On Wed, May 8, 2019 at 3:48 AM Rémy Oudompheng wrote: > Hello, > > I didn't have time to try the patch, but as Go 1.12.5 is out with another > fix, I can confirm with high certainty that the problem is now gone, an

Re: [go-nuts] Change in virtual memory patterns in Go 1.12

Hey Rémy, If you have a chance, could you please try out this patch ? It's been known to help the other application Austin mentioned with virtual memory footprint and it should patch cleanly onto the go1.12. Let me know what you see! It'd help u