Re: [go-nuts] weird "index out of range" in strconv.formatBits

[metronome]

We waited for two weeks, but the panic never resurfaced, will provide 
further updates if it reoccurs or as soon as we have more information.

Thanks Dan and Kurtis for looking into it.

On Friday, August 18, 2023 at 3:18:10 PM UTC+8 Dan Kortschak wrote:

> On Thu, 2023-08-17 at 23:32 -0700, metronome wrote:
> > > > Have you built with CGO_ENABLED=0?
> > Building with CGO_ENABLED=0 succeeded, does that mean the binary's
> > runtime behavior has nothing to do with CGO, deploying
> > a CGO_ENABLED=0 binary online is not an option as well, for now (We
> > are trying, but not sure if we can make it happen).
>
> Do you get the same behaviour with CGO_ENABLED=0?
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/55c5fc8d-aa5e-4a7e-8ea6-812c1462fd36n%40googlegroups.com.

Re: [go-nuts] weird "index out of range" in strconv.formatBits

['Dan Kortschak' via golang-nuts]

On Thu, 2023-08-17 at 23:32 -0700, metronome wrote:
> > > Have you built with CGO_ENABLED=0?
> Building with CGO_ENABLED=0 succeeded, does that mean the binary's
> runtime behavior has nothing to do with CGO, deploying
> a CGO_ENABLED=0 binary online is not an option as well, for now (We
> are trying, but not sure if we can make it happen).

Do you get the same behaviour with CGO_ENABLED=0?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/35aff4ec8fd97582dcfec804f1d8714c4e55c26c.camel%40kortschak.io.

Re: [go-nuts] weird "index out of range" in strconv.formatBits

[metronome]

Thanks.

>> Have you eliminated the possibility of races?
We're trying to extract a smaller case for reproducing (perhaps 
impractical, as the frequency of the panic is way low) then apply the race 
detector, enabling race check online is not an option now, sadly. 

>> Have you built with CGO_ENABLED=0?
Building with CGO_ENABLED=0 succeeded, does that mean the binary's runtime 
behavior has nothing to do with CGO, deploying
a CGO_ENABLED=0 binary online is not an option as well, for now (We are 
trying, but not sure if we can make it happen).

On Thursday, August 17, 2023 at 2:46:16 PM UTC+8 Dan Kortschak wrote:

> On Wed, 2023-08-16 at 23:43 -0700, metronome wrote:
> > Thanks for commenting, a few supplements.
> > 
> > # 1. Version of Go?
> > We observed the issue with both go1.20.5 and go1.18.10 on linux/amd64
> > (centos)
> > 
> > # 2. Context?
> > All panics we observed so far are from either 
> >  strconv.FormatInt -> strconv.formatBits chain, or
> >  strconv.FormatUint -> strconv.formatBits chain
> > where the base is always 10.
> > 
> > // typical call site, toId is an "*int64".
> > if com_count > 1 {
> > com_string = anchor + "," + strconv.FormatInt(*toId, 10)
> > }
> > 
> > # 3. If your program using pure Go (statically linked) or Cgo?
> > Binary was built with CGO_ENABLED=1 and -buildmode=exe.
> > All panic call sites we observed so far are "pure go", that is, no C
> > calling go path.
> > 
> > # 4. panic stack trace
> > panic: runtime error: index out of range [18446744073708732603] with
> > length 200
> > 
> > goroutine 1 [running]:
> > strconv.formatBits({0x0?, 0x0?, 0x0?}, 0xc09e00b750?, 0x1?, 0x1?,
> > 0x0?)
> > /usr/lib/go-1.20/src/strconv/itoa.go:140 +0x4b9
> > strconv.FormatInt(0x0?, 0xc07393df80?)
> > /usr/lib/go-1.20/src/strconv/itoa.go:29 +0xa5
> >   ...
>
>
> Have you eliminated the possibility of races? Have you built with
> CGO_ENABLED=0?
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/bc3cde3e-3ed5-4402-b778-a7bd8d7a22c4n%40googlegroups.com.

Re: [go-nuts] weird "index out of range" in strconv.formatBits

['Dan Kortschak' via golang-nuts]

On Wed, 2023-08-16 at 23:43 -0700, metronome wrote:
> Thanks for commenting, a few supplements.
> 
> # 1. Version of Go?
> We observed the issue with both go1.20.5 and go1.18.10 on linux/amd64
> (centos)
> 
> # 2. Context?
> All panics we observed so far are from either 
>                  strconv.FormatInt -> strconv.formatBits chain, or
>                  strconv.FormatUint -> strconv.formatBits chain
> where the base is always 10.
> 
> // typical call site, toId is an "*int64".
> if com_count > 1 {
> com_string = anchor + "," + strconv.FormatInt(*toId, 10)
> }
> 
> # 3. If your program using pure Go (statically linked) or Cgo?
> Binary was built with CGO_ENABLED=1 and -buildmode=exe.
> All panic call sites we observed so far are "pure go", that is, no C
> calling go path.
> 
> # 4. panic stack trace
> panic: runtime error: index out of range [18446744073708732603] with
> length 200
> 
> goroutine 1 [running]:
> strconv.formatBits({0x0?, 0x0?, 0x0?}, 0xc09e00b750?, 0x1?, 0x1?,
> 0x0?)
>   /usr/lib/go-1.20/src/strconv/itoa.go:140 +0x4b9
> strconv.FormatInt(0x0?, 0xc07393df80?)
>   /usr/lib/go-1.20/src/strconv/itoa.go:29 +0xa5
>                   ...


Have you eliminated the possibility of races? Have you built with
CGO_ENABLED=0?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/79322d9278fab127f2a75f51986cc784a8c8666c.camel%40kortschak.io.

Re: [go-nuts] weird "index out of range" in strconv.formatBits

[metronome]

Thanks for commenting, a few supplements.

*# 1. Version of Go?*
We observed the issue with both go1.20.5 and go1.18.10 on linux/amd64 
(centos)

*# 2. Context?*
All panics we observed so far are from either 
 *strconv.FormatInt -> strconv.formatBits* chain, or
 *strconv.FormatUint -> strconv.formatBits* chain
where the base is always 10.

// typical call site, toId is an "*int64".

if com_count > 1 {

com_string = anchor + "," + strconv.FormatInt(*toId, 10)

}


*# 3. If your program using pure Go (statically linked) or Cgo?*

Binary was built with CGO_ENABLED=1 and -buildmode=exe.

All panic call sites we observed so far are "pure go", that is, no C 
calling go path.


*# 4. panic stack trace*
panic: runtime error: index out of range [18446744073708732603] with length 
200

goroutine 1 [running]:
strconv.formatBits({0x0?, 0x0?, 0x0?}, 0xc09e00b750?, 0x1?, 0x1?, 0x0?)
/usr/lib/go-1.20/src/strconv/itoa.go:140 +0x4b9
strconv.FormatInt(0x0?, 0xc07393df80?)
/usr/lib/go-1.20/src/strconv/itoa.go:29 +0xa5

  ...


On Thursday, August 17, 2023 at 12:56:17 PM UTC+8 Kurtis Rader wrote:

> Insufficient information. Version of Go? Since formatBits is private we 
> need to see the actual code of a call to a public API that resulted in the 
> call to formatBits that failed. Also, show us the literal panic stack. 
> Showing us the assembly code with no context is not useful. If your program 
> using pure Go (statically linked) or Cgo? Wild guesses, what I used to call 
> SWAGS (silly wild ass guesses) as a Unix support engineer, are seldom 
> useful. If your guess is scientifically informed that is a different matter 
> but you should be able to articulate why you think your guess is more 
> likely to be true than a random coin flip.
>
> On Wed, Aug 16, 2023 at 8:31 PM metronome  wrote:
>
>> Hi,
>>
>> We ran into a weird *out of range* issue of *strconv.formatBits*, hope 
>> someone can shed a light on what could be the root cause, any comment is 
>> highly appreciated.
>>
>> problem description:
>> *  random out of range at code 
>> ,
>>  
>> most of the time the indexing is a huge int but we observed at least one 
>> exception (#2).*
>>
>> *  #1: runtime error: index out of range [18446744073709449339] with 
>> length 200*
>> * #2: runtime error: index out of range [102511] with length 200*
>>
>> Wild guesses:
>> 1. The machine code seems to suggest it's unlikely a data race or memory 
>> corruption? But perhaps 
>> relevant registers, like R10, had been saved and restored, then it might 
>> be due to stack corruption?
>> Given that R12 is scratch reg, is it possible that R12 is clobbered 
>> somehow, say, by signal handling?
>>
>> ===
>>0x00495b0a<+810>:mov%rdi,%r10
>>
>>0x00495b0d<+813>:shr%rdi
>>
>>0x00495b10<+816>:mov%rax,%rsi
>>
>>0x00495b13<+819>:movabs $0xa3d70a3d70a3d70b,%rax
>>
>>0x00495b1d<+829>:mov%rdx,%r11
>>
>>0x00495b20<+832>:mul%rdi
>>
>>0x00495b23<+835>:shr$0x5,%rdx
>>
>> *   0x00495b27<+839>:imul   $0x64,%rdx,%r12*
>>
>> *   0x00495b2b<+843>:sub%r12,%r10*
>>
>> *   0x00495b2e<+846>:lea(%r10,%r10,1),%rax*
>>
>>0x00495b32<+850>:lea0x1(%rax),%rax
>>
>>0x00495b36<+854>:nopw   0x0(%rax,%rax,1)
>>
>>0x00495b3f<+863>:nop
>>
>>0x00495b40<+864>:cmp$0xc8,%rax
>>
>>0x00495b46<+870>:jae0x495c8f 
>> 
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/ff804d4c-24ee-480d-8ed1-219f9b8d7cbcn%40googlegroups.com
>>  
>> 
>> .
>>
>
>
> -- 
> Kurtis Rader
> Caretaker of the exceptional canines Junior and Hank
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/003ec4ce-443e-48ec-9ac8-22aa57b0c006n%40googlegroups.com.

Re: [go-nuts] weird "index out of range" in strconv.formatBits

[Kurtis Rader]

Insufficient information. Version of Go? Since formatBits is private we
need to see the actual code of a call to a public API that resulted in the
call to formatBits that failed. Also, show us the literal panic stack.
Showing us the assembly code with no context is not useful. If your program
using pure Go (statically linked) or Cgo? Wild guesses, what I used to call
SWAGS (silly wild ass guesses) as a Unix support engineer, are seldom
useful. If your guess is scientifically informed that is a different matter
but you should be able to articulate why you think your guess is more
likely to be true than a random coin flip.

On Wed, Aug 16, 2023 at 8:31 PM metronome  wrote:

> Hi,
>
> We ran into a weird *out of range* issue of *strconv.formatBits*, hope
> someone can shed a light on what could be the root cause, any comment is
> highly appreciated.
>
> problem description:
> *  random out of range at code
> ,
> most of the time the indexing is a huge int but we observed at least one
> exception (#2).*
>
> *  #1: runtime error: index out of range [18446744073709449339] with
> length 200*
> * #2: runtime error: index out of range [102511] with length 200*
>
> Wild guesses:
> 1. The machine code seems to suggest it's unlikely a data race or memory
> corruption? But perhaps
> relevant registers, like R10, had been saved and restored, then it might
> be due to stack corruption?
> Given that R12 is scratch reg, is it possible that R12 is clobbered
> somehow, say, by signal handling?
>
> ===
>0x00495b0a<+810>:mov%rdi,%r10
>
>0x00495b0d<+813>:shr%rdi
>
>0x00495b10<+816>:mov%rax,%rsi
>
>0x00495b13<+819>:movabs $0xa3d70a3d70a3d70b,%rax
>
>0x00495b1d<+829>:mov%rdx,%r11
>
>0x00495b20<+832>:mul%rdi
>
>0x00495b23<+835>:shr$0x5,%rdx
>
> *   0x00495b27<+839>:imul   $0x64,%rdx,%r12*
>
> *   0x00495b2b<+843>:sub%r12,%r10*
>
> *   0x00495b2e<+846>:lea(%r10,%r10,1),%rax*
>
>0x00495b32<+850>:lea0x1(%rax),%rax
>
>0x00495b36<+854>:nopw   0x0(%rax,%rax,1)
>
>0x00495b3f<+863>:nop
>
>0x00495b40<+864>:cmp$0xc8,%rax
>
>0x00495b46<+870>:jae0x495c8f
> 
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/ff804d4c-24ee-480d-8ed1-219f9b8d7cbcn%40googlegroups.com
> 
> .
>


-- 
Kurtis Rader
Caretaker of the exceptional canines Junior and Hank

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CABx2%3DD_t6vWM1gHojz_waK%3Ddq1wRDh8yWrvGFPSy9QteZV18pg%40mail.gmail.com.

Re: [go-nuts] weird "index out of range" in strconv.formatBits

['Dan Kortschak' via golang-nuts]

On Wed, 2023-08-16 at 20:31 -0700, metronome wrote:
> Hi,
> 
> We ran into a weird out of range issue of strconv.formatBits, hope
> someone can shed a light on what could be the root cause, any comment
> is highly appreciated.
> 
> problem description:
>       random out of range at code, most of the time the indexing is a
> huge int but we observed at least one exception (#2).
>       #1: runtime error: index out of range [18446744073709449339]
> with length 200
>   #2: runtime error: index out of range [102511] with length 200
> 
> Wild guesses:
> 1. The machine code seems to suggest it's unlikely a data race or
> memory corruption? But perhaps 
> relevant registers, like R10, had been saved and restored, then it
> might be due to stack corruption?
> Given that R12 is scratch reg, is it possible that R12 is clobbered
> somehow, say, by signal handling?
> 
> ===
>    0x00495b0a<+810>:mov    %rdi,%r10
>    0x00495b0d<+813>:shr    %rdi
>    0x00495b10<+816>:mov    %rax,%rsi
>    0x00495b13<+819>:movabs $0xa3d70a3d70a3d70b,%rax
>    0x00495b1d<+829>:mov    %rdx,%r11
>    0x00495b20<+832>:mul    %rdi
>    0x00495b23<+835>:shr    $0x5,%rdx
>    0x00495b27<+839>:imul   $0x64,%rdx,%r12
>    0x00495b2b<+843>:sub    %r12,%r10
>    0x00495b2e<+846>:lea    (%r10,%r10,1),%rax
>    0x00495b32<+850>:lea    0x1(%rax),%rax
>    0x00495b36<+854>:nopw   0x0(%rax,%rax,1)
>    0x00495b3f<+863>:nop
>    0x00495b40<+864>:cmp    $0xc8,%rax
>    0x00495b46<+870>:jae    0x495c8f
> 

Go version?
Function invocation context?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/c888ef0205e12ec6ef15a50ffc7a26d89313636c.camel%40kortschak.io.