[go-nuts] Re: goroutine priority
Just realized I now to use taskset to make sure all the goroutine run on
the same CPU
sudo taskset -c 7 ./sche -r 0.3s
and It's working now
On Thursday, September 26, 2024 at 11:49:53 AM UTC+8 Zhao Weng wrote:
> Hi gophers,
> I'm doing a research on how to prioritise some goroutines over others.So I
> can allocate more CPU resource to more important part of the program.
> I try to do it by calling runtime.LockOSThread to assign the goroutine
> needs to be prioritise to a designated OS thread, and unix.SchedSetAttr to
> prioritise that OS thread. Here is the code:
>
> ```go
> package main
>
> import (
> "flag"
> "fmt"
> "runtime"
> "sync"
> "time"
>
> "golang.org/x/sys/unix"
> )
>
> var (
> nPrioritizedGoroutines = flag.Int("p", 1, "# of prioritized goroutines")
> nNormalGoroutines = flag.Int("n", 1, "# of normal goroutines")
> restDuration = flag.Duration("r", 0, "rest for a certain amount of time
> between works")
> )
>
> func prioritizeThread() {
> // set thread priority to the highest
> a := unix.SchedAttr{
> Size: unix.SizeofSchedAttr,
> Policy: 1,
> Priority: 99,
> }
>
> if err := unix.SchedSetAttr(0, &a, 0); err != nil {
> panic(err)
> }
> }
>
> func doWorks(workerId int) {
> t := time.Now()
> for i := 0; i < 100; i++ {
> st := time.Now()
> res := 0
> for ii := 0; ii < 1e9; ii++ {
> res += ii
> }
> fmt.Printf("%d@%d, timecost: %s, res: %d \n", workerId, unix.Gettid(),
> time.Since(st), res)
>
> // sleep for a while to simulate gaps between requests.
> if *restDuration > 0 {
> time.Sleep(*restDuration)
> }
> }
> fmt.Printf("total execute time for worker: %d is %s\n", workerId, time.
> Since(t))
> }
>
> func main() {
> flag.Parse()
>
> runtime.GOMAXPROCS(*nPrioritizedGoroutines)
> var wg sync.WaitGroup
>
> workerId := 0
> for i := 0; i < *nPrioritizedGoroutines; i++ {
> wg.Add(1)
> go func(workerId int) {
> // assign goroutine to a designated thread
> runtime.LockOSThread()
> // prioritize this thread
> prioritizeThread()
>
> defer wg.Done()
> doWorks(workerId)
> }(workerId)
> workerId++
> }
>
> for i := 0; i < *nNormalGoroutines; i++ {
> wg.Add(1)
> go func(workerId int) {
> defer wg.Done()
> doWorks(workerId)
> }(workerId)
> workerId++
> }
>
> wg.Wait()
> }
> ```
> compile on linux, and run with command `sudo ./sche`, it seems not
> working, CPU resource is shared by two thread, and two goroutine execute `
> doWorks` in similar timecost(1.5 seconds).
>
> sudo ./sche
> 1@255429, timecost: 1.475347182s, res: 45
> 0@255425, timecost: 1.517077413s, res: 45
> 1@255429, timecost: 1.473167148s, res: 45
> 0@255425, timecost: 1.515322146s, res: 45
> 1@255429, timecost: 1.494751901s, res: 45
> 0@255425, timecost: 1.532692691s, res: 45
>
> while with 1 goroutine only, the timecost will be 0.75 seconds
>
> sudo ./sche -n 0
> 0@257072, timecost: 751.18938ms, res: 45
> 0@257072, timecost: 747.364725ms, res: 45
> 0@257072, timecost: 745.362553ms, res: 45
> 0@257072, timecost: 748.353778ms, res: 45
>
> Am I doing anything wrong here?
>
> Zhao Weng
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/4eb06017-ec55-4198-ad60-365bb6947728n%40googlegroups.com.
[go-nuts] goroutine priority
Hi gophers,
I'm doing a research on how to prioritise some goroutines over others.So I
can allocate more CPU resource to more important part of the program.
I try to do it by calling runtime.LockOSThread to assign the goroutine
needs to be prioritise to a designated OS thread, and unix.SchedSetAttr to
prioritise that OS thread. Here is the code:
```go
package main
import (
"flag"
"fmt"
"runtime"
"sync"
"time"
"golang.org/x/sys/unix"
)
var (
nPrioritizedGoroutines = flag.Int("p", 1, "# of prioritized goroutines")
nNormalGoroutines = flag.Int("n", 1, "# of normal goroutines")
restDuration = flag.Duration("r", 0, "rest for a certain amount of time
between works")
)
func prioritizeThread() {
// set thread priority to the highest
a := unix.SchedAttr{
Size: unix.SizeofSchedAttr,
Policy: 1,
Priority: 99,
}
if err := unix.SchedSetAttr(0, &a, 0); err != nil {
panic(err)
}
}
func doWorks(workerId int) {
t := time.Now()
for i := 0; i < 100; i++ {
st := time.Now()
res := 0
for ii := 0; ii < 1e9; ii++ {
res += ii
}
fmt.Printf("%d@%d, timecost: %s, res: %d \n", workerId, unix.Gettid(), time.
Since(st), res)
// sleep for a while to simulate gaps between requests.
if *restDuration > 0 {
time.Sleep(*restDuration)
}
}
fmt.Printf("total execute time for worker: %d is %s\n", workerId, time.Since
(t))
}
func main() {
flag.Parse()
runtime.GOMAXPROCS(*nPrioritizedGoroutines)
var wg sync.WaitGroup
workerId := 0
for i := 0; i < *nPrioritizedGoroutines; i++ {
wg.Add(1)
go func(workerId int) {
// assign goroutine to a designated thread
runtime.LockOSThread()
// prioritize this thread
prioritizeThread()
defer wg.Done()
doWorks(workerId)
}(workerId)
workerId++
}
for i := 0; i < *nNormalGoroutines; i++ {
wg.Add(1)
go func(workerId int) {
defer wg.Done()
doWorks(workerId)
}(workerId)
workerId++
}
wg.Wait()
}
```
compile on linux, and run with command `sudo ./sche`, it seems not working,
CPU resource is shared by two thread, and two goroutine execute `doWorks`
in similar timecost(1.5 seconds).
sudo ./sche
1@255429, timecost: 1.475347182s, res: 45
0@255425, timecost: 1.517077413s, res: 45
1@255429, timecost: 1.473167148s, res: 45
0@255425, timecost: 1.515322146s, res: 45
1@255429, timecost: 1.494751901s, res: 45
0@255425, timecost: 1.532692691s, res: 45
while with 1 goroutine only, the timecost will be 0.75 seconds
sudo ./sche -n 0
0@257072, timecost: 751.18938ms, res: 45
0@257072, timecost: 747.364725ms, res: 45
0@257072, timecost: 745.362553ms, res: 45
0@257072, timecost: 748.353778ms, res: 45
Am I doing anything wrong here?
Zhao Weng
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/a1c6c11b-7f85-4807-90b5-609f40421ff6n%40googlegroups.com.
[go-nuts] strange perf sample when trying to understand the cause of high minor page fault
Hi gophers, We have a web service running with high minor page fault and I'm trying to understand why. I start by using linux perf: ```bash sudo perf record -e minor-faults -c 1 -C 0-2 -ag -- sleep 15 sudo perf script > perf.out ``` and in the output file perf.out, I found many strange trace records: ``` tsproducer-prod 390772 [002] 2661398.756621: 1 minor-faults: ea8a10 memcpy+0x24 (/usr/local/bin/tsproducer-prod) ``` and they are the majority of all records. these records surprise me that it contains no stack trace between, seem like it's calling memcpy from nowhere, and i can not understand how this could happen base one my knowledge. I would expect something like "runtime.goexit.abi0" or "runtime.asmcgocall.abi0" before "memcpy". By the way, this code is compiled using go 1.22.2 I should attach a flame graph generated base on perf.out. please advise me on how to further understand this, maybe I miss some doc? Thanks a lot, Zhao Weng -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/2cc16e0f-176b-4f33-8798-bba239a4ec1cn%40googlegroups.com.
Re: [go-nuts] Some questions about GC and map[int]int
nuse_objects.
>>>>>
>>>>> For context, I have a long running service which has a long-lived
>>>>> map[int]int in memory, we constantly add new data to it and maintain its
>>>>> size by calling delete() when oversize, we assume there should not
>>>>> result
>>>>> in GC overhead growth as time goes by.
>>>>>
>>>>> Thanks
>>>>> On Friday, November 24, 2023 at 5:54:24 PM UTC+8 Zhao Weng wrote:
>>>>>
>>>>>> Sorry about the redundant spam like mail I send to the mail box, I do
>>>>>> delete the conversation on the google groups, I send later emails for
>>>>>> correcting typos and replace screenshot to text. I'm so sorry for that.
>>>>>>
>>>>>> On Friday, November 24, 2023 at 3:13:04 PM UTC+8 Kurtis Rader wrote:
>>>>>>
>>>>>>> You only need to ask for help once. You sent essentially the same
>>>>>>> message twice. Also, your example program doesn't provide any time for
>>>>>>> the
>>>>>>> GC to run in any meaningful manner as far as I can tell. So I am
>>>>>>> confused
>>>>>>> how your trivial program relates to a real world program. That is,
>>>>>>> while
>>>>>>> simple reproductions of a problem are always a good idea that does not
>>>>>>> seem
>>>>>>> to be the case with your example reproduction. No real program would
>>>>>>> instantiate a huge map then immediately terminate. I think you need to
>>>>>>> provide more context and a more realistic program to reproduce the
>>>>>>> problem.
>>>>>>>
>>>>>>> On Thu, Nov 23, 2023 at 10:10 PM 'Zhao Weng' via golang-nuts <
>>>>>>> golan...@googlegroups.com> wrote:
>>>>>>>
>>>>>>>> Dear Gophers:
>>>>>>>>
>>>>>>>> I have some questions about GC and map[int]int, please help.
>>>>>>>>
>>>>>>>> consider the following program:
>>>>>>>>
>>>>>>>> ```go
>>>>>>>> package main
>>>>>>>>
>>>>>>>> import (
>>>>>>>> "fmt"
>>>>>>>> "os"
>>>>>>>> "runtime/pprof"
>>>>>>>> )
>>>>>>>>
>>>>>>>> func main() {
>>>>>>>> f, err := os.OpenFile("memory.pb.gz",
>>>>>>>> os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0666)
>>>>>>>> if err != nil {
>>>>>>>> panic(err)
>>>>>>>> }
>>>>>>>>
>>>>>>>> m := make(map[int]int)
>>>>>>>> for i := 0; i < 10e6; i++ {
>>>>>>>> m[i] = 2 * i
>>>>>>>> }
>>>>>>>>
>>>>>>>> if err := pprof.WriteHeapProfile(f); err != nil {
>>>>>>>> panic(err)
>>>>>>>> }
>>>>>>>> }
>>>>>>>> ```
>>>>>>>>
>>>>>>>> after inserting 10 million key/value pair, the result of this
>>>>>>>> memory profile show inuse_objects count ~ 100k
>>>>>>>>
>>>>>>>> ```bash
>>>>>>>> go tool pprof -inuse_objects memory.pb.gz
>>>>>>>> Type: inuse_objects
>>>>>>>> Time: Nov 23, 2023 at 3:40pm (CST)
>>>>>>>> Entering interactive mode (type "help" for commands, "o" for
>>>>>>>> options)
>>>>>>>> (pprof) top
>>>>>>>> Showing nodes accounting for 127450, 100% of 127450 total
>>>>>>>> flat flat% sum%cum cum%
>>>>>>>> 127450 100% 100% 127450 100% main.main
>>>>>>>>
>>>>>>>> ```
>>>>>>>>
>>>>>>>> in my view this inuse_objects is found by GC by following the
>>>>>>>> bucket and old_buckets pointer of hmap
>>>>>>>>
>>>>>>>> ```go
>>>>>>>> // A header for a Go map.
>>>>>>>> type hmap struct {
>>>>>>>> // Note: the format of the hmap is also encoded in
>>>>>>>> cmd/compile/internal/reflectdata/reflect.go.
>>>>>>>> // Make sure this stays in sync with the compiler's definition.
>>>>>>>> count int // # live cells == size of map. Must be first (used by
>>>>>>>> len() builtin)
>>>>>>>> flags uint8
>>>>>>>> B uint8 // log_2 of # of buckets (can hold up to loadFactor * 2^B
>>>>>>>> items)
>>>>>>>> noverflow uint16 // approximate number of overflow buckets; see
>>>>>>>> incrnoverflow for details
>>>>>>>> hash0 uint32 // hash seed
>>>>>>>>
>>>>>>>> buckets unsafe.Pointer // array of 2^B Buckets. may be nil if
>>>>>>>> count==0.
>>>>>>>> oldbuckets unsafe.Pointer // previous bucket array of half the
>>>>>>>> size, non-nil only when growing
>>>>>>>> nevacuate uintptr // progress counter for evacuation (buckets less
>>>>>>>> than this have been evacuated)
>>>>>>>>
>>>>>>>> extra *mapextra // optional fields
>>>>>>>> }
>>>>>>>> ```
>>>>>>>>
>>>>>>>> but according to this doc:
>>>>>>>> https://go101.org/optimizations/6-map.html
>>>>>>>>
>>>>>>>> If the key type and element type of a map both don't contain
>>>>>>>> pointers, then in the scan phase of a GC cycle, the garbage collector
>>>>>>>> will
>>>>>>>> not scan the entries of the map. This could save much time.
>>>>>>>>
>>>>>>>> My question is:
>>>>>>>> maybe GC should not follow the bucket and old_buckets pointer of
>>>>>>>> hmap?
>>>>>>>> or the doc above just forbid GC from scan the entries not the
>>>>>>>> bucket and old_buckets pointer of hmap, and larger map[int]int do
>>>>>>>> increase
>>>>>>>> GC scan overhead by having more buckets and old_buckets?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "golang-nuts" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to golang-nuts...@googlegroups.com.
>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/golang-nuts/b7049ce2-d5a8-49bc-ae44-7a73777494f4n%40googlegroups.com
>>>>>>>>
>>>>>>>> <https://groups.google.com/d/msgid/golang-nuts/b7049ce2-d5a8-49bc-ae44-7a73777494f4n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>> .
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Kurtis Rader
>>>>>>> Caretaker of the exceptional canines Junior and Hank
>>>>>>>
>>>>>>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/b0d0feef-fbe2-49b5-94d6-1a2e3fc216f7n%40googlegroups.com.
[go-nuts] Some questions about GC and map[int]int, please help
Dear Gophers:
I have some questions about GC and map[int]int, please help.
consider the following program:
```go
package main
import (
"fmt"
"os"
"runtime/pprof"
)
func main() {
f, err := os.OpenFile("memory.pb.gz", os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0666
)
if err != nil {
panic(err)
}
m := make(map[int]int)
for i := 0; i < 10e6; i++ {
m[i] = 2 * i
}
if err := pprof.WriteHeapProfile(f); err != nil {
panic(err)
}
}
```
after inserting 10 million key/value pair, the result of this memory
profile show inuse_objects count ~ 100k
```bash
go tool pprof -inuse_objects memory.pb.gz
Type: inuse_objects
Time: Nov 23, 2023 at 3:40pm (CST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 127450, 100% of 127450 total
flat flat% sum%cum cum%
127450 100% 100% 127450 100% main.main
0 0% 100% 127450 100% runtime.main
(pprof)
```
in my view this inuse_objects is found by GC by following the bucket and
old_buckets pointer of hmap
```go
// A header for a Go map.
type hmap struct {
// Note: the format of the hmap is also encoded in
cmd/compile/internal/reflectdata/reflect.go.
// Make sure this stays in sync with the compiler's definition.
count int // # live cells == size of map. Must be first (used by len()
builtin)
flags uint8
B uint8 // log_2 of # of buckets (can hold up to loadFactor * 2^B items)
noverflow uint16 // approximate number of overflow buckets; see
incrnoverflow for details
hash0 uint32 // hash seed
buckets unsafe.Pointer // array of 2^B Buckets. may be nil if count==0.
oldbuckets unsafe.Pointer // previous bucket array of half the size,
non-nil only when growing
nevacuate uintptr // progress counter for evacuation (buckets less than
this have been evacuated)
extra *mapextra // optional fields
}
```
but according to this doc: https://go101.org/optimizations/6-map.html
If the key type and element type of a map both don't contain pointers, then
in the scan phase of a GC cycle, the garbage collector will not scan the
entries of the map. This could save much time.
My question is:
maybe GC should not follow the bucket and old_buckets pointer of hmap?
or the doc above just forbid GC from scanning the entries not the bucket
and old_buckets pointer of hmap, and larger map[int]int do increase GC scan
overhead by having more buckets and old_buckets?
Thanks
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/36fec7cb-6808-4a47-85ba-754963a6ddc5n%40googlegroups.com.
[go-nuts] Some questions about GC and map[int]int
Dear Gophers:
I have some questions about GC and map[int]int, please help.
consider the following program:
```go
package main
import (
"fmt"
"os"
"runtime/pprof"
)
func main() {
f, err := os.OpenFile("memory.pb.gz", os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0666
)
if err != nil {
panic(err)
}
m := make(map[int]int)
for i := 0; i < 10e6; i++ {
m[i] = 2 * i
}
if err := pprof.WriteHeapProfile(f); err != nil {
panic(err)
}
}
```
after inserting 10 million key/value pair, the result of this memory
profile show inuse_objects count ~ 100k
```bash
go tool pprof -inuse_objects memory.pb.gz
Type: inuse_objects
Time: Nov 23, 2023 at 3:40pm (CST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 127450, 100% of 127450 total
flat flat% sum%cum cum%
127450 100% 100% 127450 100% main.main
```
in my view this inuse_objects is found by GC by following the bucket and
old_buckets pointer of hmap
```go
// A header for a Go map.
type hmap struct {
// Note: the format of the hmap is also encoded in
cmd/compile/internal/reflectdata/reflect.go.
// Make sure this stays in sync with the compiler's definition.
count int // # live cells == size of map. Must be first (used by len()
builtin)
flags uint8
B uint8 // log_2 of # of buckets (can hold up to loadFactor * 2^B items)
noverflow uint16 // approximate number of overflow buckets; see
incrnoverflow for details
hash0 uint32 // hash seed
buckets unsafe.Pointer // array of 2^B Buckets. may be nil if count==0.
oldbuckets unsafe.Pointer // previous bucket array of half the size,
non-nil only when growing
nevacuate uintptr // progress counter for evacuation (buckets less than
this have been evacuated)
extra *mapextra // optional fields
}
```
but according to this doc: https://go101.org/optimizations/6-map.html
If the key type and element type of a map both don't contain pointers, then
in the scan phase of a GC cycle, the garbage collector will not scan the
entries of the map. This could save much time.
My question is:
maybe GC should not follow the bucket and old_buckets pointer of hmap?
or the doc above just forbid GC from scan the entries not the bucket and
old_buckets pointer of hmap, and larger map[int]int do increase GC scan
overhead by having more buckets and old_buckets?
Thanks
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/b7049ce2-d5a8-49bc-ae44-7a73777494f4n%40googlegroups.com.
[go-nuts] Some questions about GC and map[int]int
Dear Gophers:
I have some questions about GC and map[int]int, please help.
consider the following program:
```go
package main
import (
"fmt"
"os"
"runtime/pprof"
)
func main() {
f, err := os.OpenFile("memory.pb.gz", os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0666
)
if err != nil {
panic(err)
}
m := make(map[int]int)
for i := 0; i < 10e6; i++ {
m[i] = 2 * i
}
if err := pprof.WriteHeapProfile(f); err != nil {
panic(err)
}
}
```
after inserting 10 million key/value pair, the result of this memory
profile show inuse_objects count ~ 100k
[image: inuse_objects.png]
in my view this inuse_objects is found by GC by following the bucket and
old_buckets pointer of hmap
```go
// A header for a Go map.
type hmap struct {
// Note: the format of the hmap is also encoded in
cmd/compile/internal/reflectdata/reflect.go.
// Make sure this stays in sync with the compiler's definition.
count int // # live cells == size of map. Must be first (used by len()
builtin)
flags uint8
B uint8 // log_2 of # of buckets (can hold up to loadFactor * 2^B items)
noverflow uint16 // approximate number of overflow buckets; see
incrnoverflow for details
hash0 uint32 // hash seed
buckets unsafe.Pointer // array of 2^B Buckets. may be nil if count==0.
oldbuckets unsafe.Pointer // previous bucket array of half the size,
non-nil only when growing
nevacuate uintptr // progress counter for evacuation (buckets less than
this have been evacuated)
extra *mapextra // optional fields
}
```
but according to this doc: https://go101.org/optimizations/6-map.html
If the key type and element type of a map both don't contain pointers, then
in the scan phase of a GC cycle, the garbage collector will not scan the
entries of the map. This could save much time.
My question is:
maybe GC should not follow the bucket and old_buckets pointer of hmap?
or the doc above just forbid GC from scan the entries not the bucket and
old_buckets pointer of hmap, and larger map[int]int do increase GC scan
overhead by having more buckets and old_buckets?
Thanks
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/db9507eb-fd64-46c0-867a-f4eaf5c78b38n%40googlegroups.com.
[go-nuts] Some question about GC and map[int]int
consider the following program:
```go
package main
import (
"fmt"
"os"
"runtime/pprof"
)
func main() {
f, err := os.OpenFile("memory.pb.gz", os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0666
)
if err != nil {
panic(err)
}
m := make(map[int]int)
for i := 0; i < 10e6; i++ {
m[i] = 2 * i
}
if err := pprof.WriteHeapProfile(f); err != nil {
panic(err)
}
}
```
after inserting 10 million key/value pair, the result of this memory
profile show inuse_objects count ~ 100k[image: inuse_objects.png]
in my view this inuse_objects is found by GC by following the bucket and
old_buckets pointer of hmap
```go
// A header for a Go map.
type hmap struct {
// Note: the format of the hmap is also encoded in
cmd/compile/internal/reflectdata/reflect.go.
// Make sure this stays in sync with the compiler's definition.
count int // # live cells == size of map. Must be first (used by len()
builtin)
flags uint8
B uint8 // log_2 of # of buckets (can hold up to loadFactor * 2^B items)
noverflow uint16 // approximate number of overflow buckets; see
incrnoverflow for details
hash0 uint32 // hash seed
buckets unsafe.Pointer // array of 2^B Buckets. may be nil if count==0.
oldbuckets unsafe.Pointer // previous bucket array of half the size,
non-nil only when growing
nevacuate uintptr // progress counter for evacuation (buckets less than
this have been evacuated)
extra *mapextra // optional fields
}
```
but according to this doc: https://go101.org/optimizations/6-map.html
If the key type and element type of a map both don't contain pointers, then
in the scan phase of a GC cycle, the garbage collector will not scan the
entries of the map. This could save much time.
This tip is also valid for other kinds of container in Go, such as slices,
arrays and channels.
My question is:
maybe GC should not follow the bucket and old_buckets pointer of hmap?
or the doc above just forbid GC from scan the entries not the bucket and
old_buckets pointer of hmap, and larger map[int]int do increase GC scan
overhead by having more buckets and old_buckets?
Thanks
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/32427592-36e4-4262-8904-e2d68b14f7e6n%40googlegroups.com.
[go-nuts] Re: Go 1.17.1 runtime seems retain too much memory from OS
sorry for being not specific. my question is: according to my understanding about memstat ( correct me if I'm wrong) with GOGC set to 50, I can see from the graph above that GCGoal is ~ 26GB and heapIdle - heapReleased is how much memory go runtime retain from OS for later usage, how can it (heapIdle - heapReleased) be 35GB which is 1.3x GCGoal ? it seem to me is something stop scavenger from releasing those memory back to OS. and by reading the code: // HeapIdle is bytes in idle (unused) spans. // // Idle spans have no objects in them. These spans could be // (and may already have been) returned to the OS, or they can // be reused for heap allocations, or they can be reused as // stack memory. // // HeapIdle minus HeapReleased estimates the amount of memory // that could be returned to the OS, but is being retained by // the runtime so it can grow the heap without requesting more // memory from the OS. If this difference is significantly // larger than the heap size, it indicates there was a recent // transient spike in live heap size. HeapIdle uint64 and from mgcscavenge.go: // That goal is defined as: // (retainExtraPercent+100) / 100 * (heapGoal / lastHeapGoal) * last_heap_inuse it seems (heapIdle - heapReleased) should be like 3GB (which is retainExtraPercent/100 * heapInUse ~ 10% * 26GB ~ 3GB) and i'm really confused by this situation that (heapIdle - heapReleased) be 35GB On Thursday, November 24, 2022 at 3:09:10 PM UTC+8 Amnon wrote: > Have a look at https://tip.golang.org/doc/gc-guide > > and > https://opensourcelive.withgoogle.com/events/go-day-2022/watch?talk=talk4 > > On Thursday, 24 November 2022 at 05:08:55 UTC Zhao Weng wrote: > >> [image: memory.jpg] >> memstats show heapIdle - heapReleased > heapAlloc or heapInUse >> >> I try to search the code and found the below comment: >> >> // HeapIdle is bytes in idle (unused) spans. >> // >> // Idle spans have no objects in them. These spans could be >> // (and may already have been) returned to the OS, or they can >> // be reused for heap allocations, or they can be reused as >> // stack memory. >> // >> // HeapIdle minus HeapReleased estimates the amount of memory >> // that could be returned to the OS, but is being retained by >> // the runtime so it can grow the heap without requesting more >> // memory from the OS. If this difference is significantly >> // larger than the heap size, it indicates there was a recent >> // transient spike in live heap size. >> HeapIdle uint64 >> >> and from mgcscavenge.go: >> >> // That goal is defined as: >> // (retainExtraPercent+100) / 100 * (heapGoal / lastHeapGoal) * >> last_heap_inuse >> >> it seem to me heapIdle - heapReleased should be ~ heapInUse * 10% >> >> can anybody help me understand this? >> >> many thanks!! >> > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/08146e99-5ea8-49ed-9a65-4b04c362f4d3n%40googlegroups.com.
