Krzysztof, thanks for that information - I believe I know what might be causing that crash now.
On Thursday, August 30, 2018 at 4:43:15 PM UTC-4, Krzysztof Kowalczyk wrote: > > The second issue is a bug in runtime, which might be fixed in 1.11 ( > https://github.com/golang/go/issues/24925 and > https://github.com/golang/go/issues/25499 and > https://github.com/golang/go/issues/24634 look like might be the issue). > > The original issue is that storage library calls Client.Do() on nil > *http.Client which eventually crashes. > > This could be a bug in the library but more likely a bug in how you use > the library. > > Skimming the code of storage library, it can be triggered if you call > client.Close() before calling Download(). > > https://github.com/GoogleCloudPlatform/google-cloud-go/blob/master/storage/storage.go#L114 > (this clears c.hc which is *http.Client on which Do() is called which > causes a crash). > > -- kjk > > On Thursday, August 30, 2018 at 12:22:39 PM UTC-7, Ewan Walker wrote: >> >> I've been trying to trace a crash (seemingly from a dependency) for quite >> some time: >> >> panic: runtime error: invalid memory address or nil pointer dereference >> [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x656492] >> >> >> goroutine 86 [running]: >> net/http.(*Client).deadline(0x0, 0xc42e2dc0b0, 0xc420456be0, 0x0) >> /usr/local/go/src/net/http/client.go:189 +0x22 >> net/http.(*Client).Do(0x0, 0xc42054c800, 0xc43631e237, 0x1, 0xc450328900) >> /usr/local/go/src/net/http/client.go:500 +0x86 >> vendor/ >> cloud.google.com/go/storage.(*ObjectHandle).NewRangeReader.func2.1(0xc450328988, >> >> 0x5da16f) >> /src/vendor/cloud.google.com/go/storage/reader.go:100 +0x64 >> vendor/cloud.google.com/go/storage.runWithRetry.func1(0x0, 0x0, >> 0xc450328a20) >> /src/vendor/cloud.google.com/go/storage/invoke.go:27 +0x2a >> vendor/cloud.google.com/go/internal.retry(0x7ff670cac2a8, 0xc45032cde0, >> 0x0, 0x0, 0x0, 0x0, 0xc450328aa8, 0xd1b648, 0xc42005dc00, 0x688335) >> /src/vendor/cloud.google.com/go/internal/retry.go:39 +0x4c >> vendor/cloud.google.com/go/internal.Retry(0x7ff670cac2a8, 0xc45032cde0, >> 0x0, 0x0, 0x0, 0x0, 0xc450328aa8, 0x80, 0x80) >> /src/vendor/cloud.google.com/go/internal/retry.go:32 +0x82 >> vendor/cloud.google.com/go/storage.runWithRetry(0x7ff670cac2a8, >> 0xc45032cde0, 0xc450328b88, 0xc42005dc00, 0xc4205ca800) >> /src/vendor/cloud.google.com/go/storage/invoke.go:26 +0x73 >> vendor/ >> cloud.google.com/go/storage.(*ObjectHandle).NewRangeReader.func2(0x0, >> 0xc420456b80, 0x0, 0x0) >> /src/vendor/cloud.google.com/go/storage/reader.go:99 +0x125 >> vendor/ >> cloud.google.com/go/storage.(*ObjectHandle).NewRangeReader(0xc445af0210, >> 0x7ff670cac2a8, 0xc45032cde0, 0x0, 0xffffffffffffffff, 0x0, 0xc43b62a600, >> 0x7ff670c5f1d0) >> /src/vendor/cloud.google.com/go/storage/reader.go:129 +0x5bf >> vendor/cloud.google.com/go/storage.(*ObjectHandle).NewReader(0xc445af0210, >> 0x7ff670c5f1d0, 0xc43b62a660, 0x7ff670c5f1d0, 0xc43b62a660, 0xc43cbc4690) >> /src/vendor/cloud.google.com/go/storage/reader.go:42 +0x51 >> project/storage/cloud.(*Client).Download(0xc4204f69f0, 0xc429129f00, >> 0x13, 0xcea887, 0x6, 0xc4205ff798, 0x8, 0x0, 0xc420335080, 0x18) >> /src/project/storage/cloud/client.go:306 +0x3e2 >> project/storage/retention.insert.download(0xc420018600, 0x0, 0x0, >> 0xbed7eea2f2ff5d40, 0x98b539fed9f, 0x11f7e60) >> /src/project/storage/retention/insert.go:303 +0x96 >> project/storage/retention.(*insert).Run(0xc4220f60c0, 0x0, 0x0) >> /src/project/storage/retention/insert.go:82 +0x2a7 >> project/storage/retention.(*repair).Run(0xc42000e580, 0x98b539bca2f, >> 0x11f7e60) >> /src/project/storage/retention/repair.go:62 +0x7e9 >> project/storage/retention.(*Client).sync(0xc42052e150, 0xc420250000, >> 0x1f, 0x20, 0x0, 0x0) >> /src/project/storage/retention/client.go:371 +0x2cb >> project/storage/retention.(*Client).Run(0xc42052e150, 0x0, 0x0) >> /src/project/storage/retention/client.go:222 +0x137 >> project/storage.Engine.retention(0xc4200b9d80, 0xc4201d9e00, >> 0xc42052e150, 0xc4200b7630) >> /src/project/storage/engine.go:90 +0x43 >> created by project/storage.New >> /src/project/storage/engine.go:55 +0x270 >> >> >> To prevent the problem I've added a defer'd recover to try to catch the >> exception, however now I'm getting: >> >> >> runtime: invalid pc-encoded table f=net/http.(*conn).readRequest >> pc=0x6934d3 targetpc=0x6934da tab=[0/0]0x0 >> value=0 until pc=0x692812 >> value=304 until pc=0x692bcb >> value=0 until pc=0x692bcc >> value=304 until pc=0x692c76 >> value=0 until pc=0x692c77 >> value=304 until pc=0x692f96 >> value=0 until pc=0x692f97 >> value=304 until pc=0x693114 >> value=0 until pc=0x693115 >> value=304 until pc=0x693154 >> value=0 until pc=0x693155 >> value=304 until pc=0x6931bc >> value=0 until pc=0x6931bd >> value=304 until pc=0x6932c1 >> value=0 until pc=0x6932c2 >> value=304 until pc=0x693316 >> value=0 until pc=0x693317 >> value=304 until pc=0x693338 >> value=0 until pc=0x693339 >> value=304 until pc=0x6933ef >> value=0 until pc=0x6933f0 >> value=304 until pc=0x6934c8 >> value=0 until pc=0x6934d3 >> fatal error: invalid runtime symbol table >> >> goroutine 0 [idle]: >> runtime: unexpected return pc for runtime.sigtramp called from >> 0x7f406d73a390 >> stack: frame={sp:0xc42050fa68, fp:0xc42050fac0} >> stack=[0xc420508000,0xc420510000) >> 000000c42050f968: 0000000000000000 0000000000000000 >> 000000c42050f978: 0000000000000000 0000000000000001 >> 000000c42050f988: 000000c4205d0760 000000c4201ed610 >> 000000c42050f998: 000000c42050fbf0 000000c42050fac0 >> 000000c42050f9a8: 000000c42050fa58 0000000000441dfc >> <runtime.sigtrampgo+540> >> 000000c42050f9b8: 000000c40000001b 000000c42050fbf0 >> 000000c42050f9c8: 000000c42050fac0 000000c42031be00 >> 000000c42050f9d8: 0000000000000000 000000000081e810 <vendor/ >> google.golang.org/grpc.(*clientStream).(vendor/google.golang.org/grpc.commitAttemptLocked)-fm+0 >> > >> 000000c42050f9e8: 0000000000000000 0000000000000000 >> 000000c42050f9f8: 0000000000000000 0000000000000000 >> 000000c42050fa08: 0000000000000000 000000c42031be00 >> 000000c42050fa18: 000000c4201ed560 0000000000c928e0 >> 000000c42050fa28: 0000000000000000 0000000000000000 >> 000000c42050fa38: 0000000000000000 0000000000000000 >> 000000c42050fa48: 0000000000000000 0000000000000000 >> 000000c42050fa58: 000000c42050fab0 000000000045b433 >> <runtime.sigtramp+67> >> 000000c42050fa68: <000000000000001b 000000c42050fbf0 >> 000000c42050fa78: 000000c42050fac0 0000000000000008 >> 000000c42050fa88: 00007f406db216c8 0000000000000000 >> 000000c42050fa98: 0000000000001840 000000c42050fab0 >> 000000c42050faa8: 0000000000002000 00007f40689f4c78 >> 000000c42050fab8: !00007f406d73a390 >0000000000000001 >> 000000c42050fac8: 0000000000000000 000000c420508000 >> 000000c42050fad8: 000000c400000000 0000000000008000 >> 000000c42050fae8: 0000000000000080 00000000011ae8a0 >> 000000c42050faf8: 0000000000000040 0000000000000000 >> 000000c42050fb08: 0000000000001840 0000000000000000 >> 000000c42050fb18: 00007f406db216c8 0000000000000008 >> 000000c42050fb28: 0000000000455350 <runtime.(*mheap).freeSpan.func1+0> >> 0000000000001cfe >> 000000c42050fb38: 00007f40689f4c78 0000000000002000 >> 000000c42050fb48: 000000c420000f00 000000c42031be00 >> 000000c42050fb58: 000000c4204ce000 00007f40689f4c60 >> 000000c42050fb68: 000000000045b2bb <runtime.nanotime+43> >> 0000000000000206 >> 000000c42050fb78: 0000000000000033 0000000000000000 >> 000000c42050fb88: 0000000000000000 0000000000000000 >> 000000c42050fb98: 0000000000000000 000000c42050fc80 >> 000000c42050fba8: 000000000043e2b0 <runtime.selectgo+3664> >> 0000000000d1a128 >> 000000c42050fbb8: 0000000000000000 >> runtime.throw(0xcfae73, 0x1c) >> /usr/local/go/src/runtime/panic.go:616 +0x81 >> runtime.pcvalue(0xf11b08, 0x11b2100, 0xc4001668b7, 0x6934da, >> 0xc42050f498, 0x6fe201, 0xc400000000) >> /usr/local/go/src/runtime/symtab.go:790 +0x529 >> runtime.funcspdelta(0xf11b08, 0x11b2100, 0x6934da, 0xc42050f498, >> 0x7f4000000000) >> /usr/local/go/src/runtime/symtab.go:842 +0x5f >> runtime.gentraceback(0x45b2bb, 0x7f40689f4c60, 0x0, 0xc42031be00, 0x0, >> 0xc42050f6f8, 0x40, 0x0, 0x0, 0x6, ...) >> /usr/local/go/src/runtime/traceback.go:204 +0x17e1 >> runtime.sigprof(0x45b2bb, 0x7f40689f4c60, 0x0, 0xc42031be00, 0xc4204ce000) >> /usr/local/go/src/runtime/proc.go:3712 +0x3e9 >> runtime.sighandler(0xc40000001b, 0xc42050fbf0, 0xc42050fac0, 0xc42031be00) >> /usr/local/go/src/runtime/signal_sighandler.go:33 +0x678 >> runtime.sigtrampgo(0x1b, 0xc42050fbf0, 0xc42050fac0) >> /usr/local/go/src/runtime/signal_unix.go:349 +0x21c >> runtime: unexpected return pc for runtime.sigtramp called from >> 0x7f406d73a390 >> stack: frame={sp:0xc42050fa68, fp:0xc42050fac0} >> stack=[0xc420508000,0xc420510000) >> 000000c42050f968: 0000000000000000 0000000000000000 >> 000000c42050f978: 0000000000000000 0000000000000001 >> 000000c42050f988: 000000c4205d0760 000000c4201ed610 >> 000000c42050f998: 000000c42050fbf0 000000c42050fac0 >> 000000c42050f9a8: 000000c42050fa58 0000000000441dfc >> <runtime.sigtrampgo+540> >> 000000c42050f9b8: 000000c40000001b 000000c42050fbf0 >> 000000c42050f9c8: 000000c42050fac0 000000c42031be00 >> 000000c42050f9d8: 0000000000000000 000000000081e810 <vendor/ >> google.golang.org/grpc.(*clientStream).(vendor/google.golang.org/grpc.commitAttemptLocked)-fm+0 >> > >> 000000c42050f9e8: 0000000000000000 0000000000000000 >> 000000c42050f9f8: 0000000000000000 0000000000000000 >> 000000c42050fa08: 0000000000000000 000000c42031be00 >> 000000c42050fa18: 000000c4201ed560 0000000000c928e0 >> 000000c42050fa28: 0000000000000000 0000000000000000 >> 000000c42050fa38: 0000000000000000 0000000000000000 >> 000000c42050fa48: 0000000000000000 0000000000000000 >> 000000c42050fa58: 000000c42050fab0 000000000045b433 >> <runtime.sigtramp+67> >> 000000c42050fa68: <000000000000001b 000000c42050fbf0 >> 000000c42050fa78: 000000c42050fac0 0000000000000008 >> 000000c42050fa88: 00007f406db216c8 0000000000000000 >> 000000c42050fa98: 0000000000001840 000000c42050fab0 >> 000000c42050faa8: 0000000000002000 00007f40689f4c78 >> 000000c42050fab8: !00007f406d73a390 >0000000000000001 >> 000000c42050fac8: 0000000000000000 000000c420508000 >> 000000c42050fad8: 000000c400000000 0000000000008000 >> 000000c42050fae8: 0000000000000080 00000000011ae8a0 >> 000000c42050faf8: 0000000000000040 0000000000000000 >> 000000c42050fb08: 0000000000001840 0000000000000000 >> 000000c42050fb18: 00007f406db216c8 0000000000000008 >> 000000c42050fb28: 0000000000455350 <runtime.(*mheap).freeSpan.func1+0> >> 0000000000001cfe >> 000000c42050fb38: 00007f40689f4c78 0000000000002000 >> 000000c42050fb48: 000000c420000f00 000000c42031be00 >> 000000c42050fb58: 000000c4204ce000 00007f40689f4c60 >> 000000c42050fb68: 000000000045b2bb <runtime.nanotime+43> >> 0000000000000206 >> 000000c42050fb78: 0000000000000033 0000000000000000 >> 000000c42050fb88: 0000000000000000 0000000000000000 >> 000000c42050fb98: 0000000000000000 000000c42050fc80 >> 000000c42050fba8: 000000000043e2b0 <runtime.selectgo+3664> >> 0000000000d1a128 >> 000000c42050fbb8: 0000000000000000 >> runtime.sigtramp(0x1, 0x0, 0xc420508000, 0xc400000000, 0x8000, 0x80, >> 0x11ae8a0, 0x40, 0x0, 0x1840, ...) >> /usr/local/go/src/runtime/sys_linux_amd64.s:352 +0x43 >> >> >> Could anyone help provide some insight as to what might be going on? >> >> Goversion: go version go1.10.1 linux/amd64 >> > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
