Re: [go-nuts] Re: Explosion in memory usage when compiling a big file

2017-01-09 Thread Vincent Rischmann
Thanks for the responses.



I took a look at the generated code and go-bindata does in fact use the
form "var _data = []byte(`my giant string`)".


But there's something strange happening: I can't reproduce the problem
on my Macbook Pro. It compiles just fine here.




On Mon, Jan 9, 2017, at 12:50 AM, Pierre Durand wrote:

> This option ?
> https://github.com/jteeuwen/go-bindata#lower-memory-footprint
> 

> Le dimanche 8 janvier 2017 21:46:15 UTC+1, Dave Cheney a écrit :

>> This is a somewhat known issue. Each token in a parsered .go file is
>> represented by a Node structure inside the program. The Node
>> structure is large, especially on 64 bit systems.
>> Normally this is not a problem, but in th e case where code has large
>> tables of data memory usage when compiling can be unexpectedly high.
>> This problem is being worked on, but not solution exists in a
>> shipping version of Go yet.
>> The mitigation to this problem is to reduce the number of parsed
>> tokens, so, rather than
>> var data = []byte{ 65, 66, 67, 68, 69, 70}



>> Do



>> const data = "abcdef"



>> The latter produces O(1) tokens per declaration vs O(N) tokens for
>> the former.
>> If the data cannot be represented as text, compressing and base64
>> encoding can help.
>> I'm not sure what strategy go-bindata uses, but you can check this
>> yourself looking at its generated output.
>> Dave



>> 



>> 



>> 



>> 



>> 



>> 



>> 



>> 



>> 



> 



> --

> You received this message because you are subscribed to the Google
> Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[go-nuts] Re: Explosion in memory usage when compiling a big file

2017-01-08 Thread Pierre Durand
This option ? https://github.com/jteeuwen/go-bindata#lower-memory-footprint

Le dimanche 8 janvier 2017 21:46:15 UTC+1, Dave Cheney a écrit :
>
> This is a somewhat known issue. Each token in a parsered .go file is 
> represented by a Node structure inside the program. The Node structure is 
> large, especially on 64 bit systems. 
>
> Normally this is not a problem, but in th e case where code has large 
> tables of data memory usage when compiling can be unexpectedly high. 
>
> This problem is being worked on, but not solution exists in a shipping 
> version of Go yet.
>
> The mitigation to this problem is to reduce the number of parsed tokens, 
> so, rather than
>
> var data = []byte{ 65, 66, 67, 68, 69, 70}
>
> Do
>
> const data = "abcdef"
>
> The latter produces O(1) tokens per declaration vs O(N) tokens for the 
> former. 
>
> If the data cannot be represented as text, compressing and base64 encoding 
> can help. 
>
> I'm not sure what strategy go-bindata uses, but you can check this 
> yourself looking at its generated output. 
>
> Dave
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.