[go-nuts] Re: What is the behavior of an HTTP persistent connection in Go?

[golangnutter]

Also do you have any recommendations for deriving appropriate values for 
Transport.MaxIdleConns 
or Transport.MaxIdleConnsPerHost?


On Monday, August 20, 2018 at 12:59:07 AM UTC-7, golang...@gmail.com wrote:
>
> The http.Transport caches connections for future re-use
> https://golang.org/pkg/net/http/#Transport
>
> Are requests sent serially by which I mean only one request and response 
> can be handled by this persistent connection at a time, so that other 
> concurrent requests to the same host are blocked until a response is 
> received and closed?
>
> If I forget to Close() a Response.Body, does that leave the connection 
> open so that it can't be shared? Do other concurrent requests open a new 
> connection during that time? 
>
> And if I do forget, by default the Transport.IdleConnTimeout will close 
> any connections automatically for me right?
>
> I understand that persistent connections save on protocol overhead, but if 
> it can only handle one request at a time wouldn't in some cases it be 
> better to have a pool of clients each with their own persistent connection 
> to the same host to allow for parallel requests?
>
> Alternatively, I could increase the Transport.MaxIdleConns or 
> Transport.MaxIdleConnsPerHost right? Which would trade throughput for 
> server resources to maintain those connections.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Garbage collector and Slices

[keith . randall]

There is no conservativeness to unsafe.Pointer. The type information for 
any object in the heap is recorded at allocation time, and is unrelated to 
the type of the reference currently held to it (be it unsafe.Pointer, 
[]byte, *int64, or whatever).
So for 2.2, the answer is no, you are incorrect.

You can allocate your backing store with make([]int64, (numBits+63)/64) and 
then store [0] in your BitSlice type (assuming numBits>0).  You can type 
that as *int64 or unsafe.Pointer.  I would use the latter just because it 
is clearer to the reader that shenanigans are in progress.

On Monday, August 20, 2018 at 4:56:00 PM UTC-7, Kevin Malachowski wrote:
>
> That information is old now - Go has a precise GC now. There is some sort 
> of metadata stored for each variable related to whether the type contains 
> pointers (as Jan alluded to earlier in the thread).
>
> On Monday, August 20, 2018 at 7:28:51 AM UTC-7, Florian Uekermann wrote:
>>
>> From the GC POV, the only interesting thing about an unsafe.Pointer is if 
>>> its value falls anywhere into a GC managed memory block for liveness 
>>> analysis, AFAIK.
>>>
>> And then it will check that memory block for stuff that looks like a 
>> pointer (not caring what the actual type being pointed to is) afaik. That 
>> is what I meant with conservative GC.
>> I found a thread from  2013, where Ian says the same 
>> https://groups.google.com/forum/#!topic/golang-nuts/yNis7bQG_rY.
>> Maybe my information is outdated, I haven't followed the GC evolution in 
>> recent years that closely.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] internal compilation error while compiling gccgo from source solaris 10

[Ian Lance Taylor]

On Sun, Aug 19, 2018 at 9:39 PM, Amandeep Gautam
 wrote:

> gmake[3]: Entering directory
> '/export/home/amandeep/gccgo-obj/sparc-sun-solaris2.10/libgo'
> gmake[4]: Entering directory
> '/export/home/amandeep/gccgo-obj/sparc-sun-solaris2.10/libgo'
> /opt/csw/bin/gmkdir -p .; files=`echo
> /export/home/amandeep/gccgo-src/libgo/go/fmt/doc.go
> /export/home/amandeep/gccgo-src/libgo/go/fmt/format.go
> /export/home/amandeep/gccgo-src/libgo/go/fmt/print.go
> /export/home/amandeep/gccgo-src/libgo/go/fmt/scan.go errors.gox io.gox
> math.gox os.gox reflect.gox strconv.gox sync.gox unicode/utf8.gox | sed -e
> 's/[^ ]*\.gox//g' -e 's/[^ ]*\.dep//'`; /bin/bash ./libtool --tag GO
> --mode=compile /export/home/amandeep/gccgo-obj/./gcc/gccgo
> -B/export/home/amandeep/gccgo-obj/./gcc/
> -B/usr/gnu/sparc-sun-solaris2.10/bin/ -B/usr/gnu/sparc-sun-solaris2.10/lib/
> -isystem /usr/gnu/sparc-sun-solaris2.10/include -isystem
> /usr/gnu/sparc-sun-solaris2.10/sys-include   -O2 -g -I . -c
> -fgo-pkgpath=`echo fmt.lo | sed -e 's/.lo$//' -e
> 's|golang_org|vendor/golang_org|'`  -o fmt.lo $files
> libtool: compile:  /export/home/amandeep/gccgo-obj/./gcc/gccgo
> -B/export/home/amandeep/gccgo-obj/./gcc/
> -B/usr/gnu/sparc-sun-solaris2.10/bin/ -B/usr/gnu/sparc-sun-solaris2.10/lib/
> -isystem /usr/gnu/sparc-sun-solaris2.10/include -isystem
> /usr/gnu/sparc-sun-solaris2.10/sys-include -O2 -g -I . -c -fgo-pkgpath=fmt
> /export/home/amandeep/gccgo-src/libgo/go/fmt/doc.go
> /export/home/amandeep/gccgo-src/libgo/go/fmt/format.go
> /export/home/amandeep/gccgo-src/libgo/go/fmt/print.go
> /export/home/amandeep/gccgo-src/libgo/go/fmt/scan.go  -fPIC -o .libs/fmt.o
> go1: internal compiler error: Segmentation Fault
> 0x6b9757 crash_signal
> /export/home/amandeep/gccgo-src/gcc/toplev.c:325
> Please submit a full bug report,
> with preprocessed source if appropriate.
> Please include the complete backtrace with any bug report.
> See  for instructions.
> gmake[4]: *** [Makefile:3324: fmt.lo] Error 1


Thanks.  Unfortunately, I don't know what could be causing this.
Everything looks fine, except that the compiler is crashing.  I'm not
seeing this on Solaris 11.  I think you will have to debug the
compiler.  To see more about what is happening, run the "compile:"
command by itself.  It should crash in the same way.  Then run it with
the -v option to see how the go1 program is being run.  Running the
go1 program with those arguments should crash.  That is what you need
to debug; presumably there is a NULL pointer dereference or something.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Garbage collector and Slices

['Kevin Malachowski' via golang-nuts]

That information is old now - Go has a precise GC now. There is some sort 
of metadata stored for each variable related to whether the type contains 
pointers (as Jan alluded to earlier in the thread).

On Monday, August 20, 2018 at 7:28:51 AM UTC-7, Florian Uekermann wrote:
>
> From the GC POV, the only interesting thing about an unsafe.Pointer is if 
>> its value falls anywhere into a GC managed memory block for liveness 
>> analysis, AFAIK.
>>
> And then it will check that memory block for stuff that looks like a 
> pointer (not caring what the actual type being pointed to is) afaik. That 
> is what I meant with conservative GC.
> I found a thread from  2013, where Ian says the same 
> https://groups.google.com/forum/#!topic/golang-nuts/yNis7bQG_rY.
> Maybe my information is outdated, I haven't followed the GC evolution in 
> recent years that closely.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Garbage collector and Slices

['Florian Uekermann' via golang-nuts]

>
> From the GC POV, the only interesting thing about an unsafe.Pointer is if 
> its value falls anywhere into a GC managed memory block for liveness 
> analysis, AFAIK.
>
And then it will check that memory block for stuff that looks like a 
pointer (not caring what the actual type being pointed to is) afaik. That 
is what I meant with conservative GC.
I found a thread from  2013, where Ian says the same 
https://groups.google.com/forum/#!topic/golang-nuts/yNis7bQG_rY.
Maybe my information is outdated, I haven't followed the GC evolution in 
recent years that closely.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Garbage collector and Slices

[Jan Mercl]

On Mon, Aug 20, 2018 at 3:48 PM 'Florian Uekermann' via golang-nuts <
golang-nuts@googlegroups.com> wrote:


>> The garbage collector does not care about the values of items of a []T
when T is not a pointer type nor a type containing transitively any
pointers.
>
> Just to make sure I understand you correctly. Are you saying I could hold
an unsafe.Pointer to an array of uint64 and the GC will still not care
about what is inside that array?
> Because that is my concern here.

unsafe.Pointer does not point to a particular type, that's why it cannot be
dereferenced. From the GC POV, the only interesting thing about an
unsafe.Pointer is if its value falls anywhere into a GC managed memory
block for liveness analysis, AFAIK. Additionally, there are some runtime
checks for obviously invalid unsafe.Pointer values, IIRC.

-- 

-j

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Garbage collector and Slices

['Florian Uekermann' via golang-nuts]

>
> The garbage collector does not care about the values of items of a []T 
> when T is not a pointer type nor a type containing transitively any 
> pointers.
>
Just to make sure I understand you correctly. Are you saying I could hold 
an unsafe.Pointer to an array of uint64 and the GC will still not care 
about what is inside that array? Because that is my concern here.
 

> However, do not play games with fake lengths and/or capacities of a slice, 
> namely setting them to a larger than actual value. Go supports interior 
> pointers pointing inside allocated memory blocks and the mechanism 
> implementing this feature could be easily tripped by such invalid values in 
> the general case, event though possibly not for a particular Go compiler.
>
Thanks for the warning. This disqualifies option 1.
 

> An easy way how to avoid any nasty interactions with the runtime is to 
> manually allocate/deallocate memory holding the bit array. See for example: 
> http://github.com/cznic/memory. I'd use the uintptr returning variants 
> making GC absolutely uninterested in the value. Any for the pointer 
> arithmetic you have to perform the calculations on uintptrs anyway.
>
Well, I would have to track the memory myself then (BitSlice should support 
slicing similar to normal slices) and can't work with memory supplied by 
the user or give the user references to it (such as using a user provided 
[]uint64 for constructing the BitSlice or returning a []uint64 to the 
user). I'm afraid this is not viable. I didn't state any of these 
requirements initially, so you couldn't have known.

However, do you already have a positive proof that the bound checks are 
> substantially expensive in your use case?
>
Yes absolutely, the performance difference is significant. But this isn't 
about removing bounds checks. We can already skip those by forcing early 
bounds checks for the last word, so the compiler can remove the rest. The 
idea here is to reduce the size of the BitSlice struct by a word or two.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Garbage collector and Slices

[Jan Mercl]

On Mon, Aug 20, 2018 at 2:49 PM 'Florian Uekermann' via golang-nuts <
golang-nuts@googlegroups.com> wrote:

> ... which would put unnecessary strain on the GC iirc, because the memory
pointed to would be scanned conservatively (basically assuming everything
in there could be a pointer).

The garbage collector does not care about the values of items of a []T when
T is not a pointer type nor a type containing transitively any pointers.
However, do not play games with fake lengths and/or capacities of a slice,
namely setting them to a larger than actual value. Go supports interior
pointers pointing inside allocated memory blocks and the mechanism
implementing this feature could be easily tripped by such invalid values in
the general case, event though possibly not for a particular Go compiler.

An easy way how to avoid any nasty interactions with the runtime is to
manually allocate/deallocate memory holding the bit array. See for example:
http://github.com/cznic/memory. I'd use the uintptr returning variants
making GC absolutely uninterested in the value. Any for the pointer
arithmetic you have to perform the calculations on uintptrs anyway.

However, do you already have a positive proof that the bound checks are
substantially expensive in your use case? I'm not saying that's not
possible, but I'd guess it's not much probable in many, if not most
realistic applications. Otherwise the code would have to access the slice
most of the time doing basically nothing else. That may happen in some
benchmarks, of course.

-- 

-j

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Garbage collector and Slices

['Florian Uekermann' via golang-nuts]

Thanks Axel. 

Good to know the first part. I have a couple of comments regarding your 
answer to the second part.

In regards to your second question: That would require having an array type 
> for every possible size, which isn't really tenable.
>
The idea was to use the same array type (`[(^uint(0)) >> 17]uint64`) 
regardless of the actual size. I would use `[0]` just like you suggest, 
but avoid the use of `unsafe.Pointer`, which would put unnecessary strain 
on the GC iirc, because the memory pointed to would be scanned 
conservatively (basically assuming everything in there could be a pointer). 
However, I realize now that I over-complicated the type def. I guess I 
should just store a pointer to the first element lik you suggest, but not 
as an unsafe.Pointer:
type BitSlice {
  data *uint64
  len int
  cap int
}
That surely doesn't break any rules as a data type, and I guess there is no 
chance the GC will start playing games with the underlying array any time 
soon.
 

> It does also mean, however, that you have to very carefully and tediously 
> re-implement a bunch of language primitives using unsafe

This is actually relatively straightforward and mostly unavoidable, since I 
already need to do those checks to properly panic when someone tries to 
access bits padding bits if the BitSlice has a non-multiple of 64 length.
It'll boil down to something like `if uint(i) >= uint(b.len) { panic(...) 
}` in most cases, which isn't all that bad.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Garbage collector and Slices

['Axel Wagner' via golang-nuts]

I'm not 100% sure, but… pretty sure? that the GC does not look at the
len/cap fields of slices. Reason being that that would require significant
alias-analysis, as two slices of the same underlying array can have
different lengths and capacities. So treating the underlying array not as
"one object" would require global knowledge about all slices of that array.
I think you can assume that right now, the GC will treat the underlying
array as one object and every slice as a pointer to that object (regardless
of len/cap). That being said, I don't think it's advisable doing that. The
GC may change and none of this is guaranteed by the language.

In regards to your second question: That would require having an array type
for every possible size, which isn't really tenable. The memory usage of
your program would explode even more, probably, than if you just use a
slice. But you *can* use `make` to get a slice and then use `[0]` to get
a pointer to its first element. i.e. something like this should work:
https://play.golang.org/p/CWiyNzz3nYx
AFAICT this conforms to the rules of unsafe pointers and thus works with
the guarantees provided by the language. It does also mean, however, that
you have to very carefully and tediously re-implement a bunch of language
primitives using unsafe, so be sure that the performance gains are actually
needed and worth it here.

On Mon, Aug 20, 2018 at 2:13 PM 'Florian Uekermann' via golang-nuts <
golang-nuts@googlegroups.com> wrote:

> Mistake in the second struct. The data field should have the type:
> *[(^uint(0)) >> 17]uint64
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Garbage collector and Slices

['Florian Uekermann' via golang-nuts]

Mistake in the second struct. The data field should have the type: 
*[(^uint(0)) >> 17]uint64

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Garbage collector and Slices

['Florian Uekermann' via golang-nuts]

Dear all,

I have some questions concerning the implementation details of the garbage 
collector regarding slices and arrays.
I'm implementing bit slices. The current approach is to use the following 
struct:
type BitSlice {
  data []uint64
  len int
}
Given that the slices length and capacity are fairly useless to me (doing 
my own bounds checks on bit resolution anyway), I am considering two kinds 
of space optimizations using "unsafe". Unfortunately I don't know enough 
about how the garbage collector is implemented to judge if these 
optimizations are safe or not.

Option 1: Use the length or capacity to store the length of the bit slice. 
This is easy to do, but I worry about how the garbage collector interacts 
with len and cap.
Question 1.1: Do I have to expect problems if I set the capacity to a 
larger value than the underlying array size or a negative value.
Question 1.2: Is len used for anything but bounds checks? I assume no, so I 
could use that any way I want, as long as i don't perform an access that 
triggers a bounds check.

Option 2: It may actually be easiest to use an array pointer instead. Like 
this:
type BitSlice {
  data *[1<<((^uint(0))>>8)]uint64
  len int
  cap int
}
In this case I would simply allocate using make([]uint64, desiredLength) 
and store a pointer to the underlying array in the data field.
Question 2.1: Assuming I take care of bounds checks myself. Do I have to 
expect problems when I am storing a pointer to an array as a pointer to a 
larger array?
Question 2.2: using an unsafe.Pointer as data field would trigger 
conservative garbage collection, correct?

If anyone has any insights or smart ideas, I am all ears. Especially some 
opinions about option two would be appreciated, since I am favoring that at 
the moment.

Best regards,
Florian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Function not working like I think it should?

[rob . shelby]

That was it. Thanks!

On Saturday, August 18, 2018 at 10:27:25 PM UTC-4, rob.s...@team-rehab.com 
wrote:
>
> Hi all! I'm hoping any of you could checkout my Golang question on Stack?
>
> https://stackoverflow.com/questions/51911363/passing-results-in-go-and-wxgo
>
> I really didn't know what to search for on this one. Thank you!!
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Confusion between Pointer or No Pointer for Nested array struct

[Tamás Gulácsi]

It depends. For values (no pointers) that 40 may be slower to copy than the 
dereference of the pointer. But pointers allow modifications, so the extra cost 
may be balanced by the gained safety and less headache. And less nil pointers.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] [Bug?] Error when running "go tool cover" on package name contains dot

[Dave Cheney]

Thanks for confirming this is a regression.

On 20 August 2018 at 18:41, Shulhan  wrote:
> On Sun, 19 Aug 2018 16:13:00 -0700 (PDT)
> Dave Cheney  wrote:
>
>> Point of clarification, the package name doesn’t contain a dot, that
>> is not permitted by the syntax of the package declaration. The name
>> of the directory you placed the file in ends in .go and this is
>> confusing the tool.
>>
>
> Sorry, I write a misleading words in previous message.  I wrote
> "package name" where it should be "directory name".
>
>> If this is a regression from an earlier version of Go, please raise
>> an issue.
>>
>
> There is no problem when tested with Go v1.10.3.  I will report it
> later, thank you for checking this.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] [Bug?] Error when running "go tool cover" on package name contains dot

[Shulhan]

On Sun, 19 Aug 2018 16:13:00 -0700 (PDT)
Dave Cheney  wrote:

> Point of clarification, the package name doesn’t contain a dot, that
> is not permitted by the syntax of the package declaration. The name
> of the directory you placed the file in ends in .go and this is
> confusing the tool. 
> 

Sorry, I write a misleading words in previous message.  I wrote
"package name" where it should be "directory name".

> If this is a regression from an earlier version of Go, please raise
> an issue. 
> 

There is no problem when tested with Go v1.10.3.  I will report it
later, thank you for checking this.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] What is the behavior of an HTTP persistent connection in Go?

[golangnutter]

The http.Transport caches connections for future re-use
https://golang.org/pkg/net/http/#Transport

Are requests sent serially by which I mean only one request and response 
can be handled by this persistent connection at a time, so that other 
concurrent requests to the same host are blocked until a response is 
received and closed?

If I forget to Close() a Response.Body, does that leave the connection open 
so that it can't be shared? Do other concurrent requests open a new 
connection during that time? 

And if I do forget, by default the Transport.IdleConnTimeout will close any 
connections automatically for me right?

I understand that persistent connections save on protocol overhead, but if 
it can only handle one request at a time wouldn't in some cases it be 
better to have a pool of clients each with their own persistent connection 
to the same host to allow for parallel requests?

Alternatively, I could increase the Transport.MaxIdleConns or 
Transport.MaxIdleConnsPerHost right? Which would trade throughput for 
server resources to maintain those connections.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Confusion between Pointer or No Pointer for Nested array struct

[R Sharma]

Hi Tamás Gulácsi

If my User struct has 40 fields, will the decision be still the same?

I minified my `User` struct. It has 40 fields like "city,state,country, 
address,email,IP,ua,status,verified.etc.,"  

Will still `Case B` is better?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.