Re: [go-nuts] Compiling a really really large package

[chad . retz]

Good deal. Issue opened at https://github.com/golang/go/issues/18602 and I 
attached code to replicate.

On Tuesday, January 10, 2017 at 3:31:11 PM UTC-6, Ian Lance Taylor wrote:
>
> On Tue, Jan 10, 2017 at 9:47 AM,   
> wrote: 
> > 
> > Is there any value in me reporting it? If helpful, I may be able to toss 
> a 
> > large tarball up on Dropbox or something. It's not really high priority 
> for 
> > me either tbh because I know it would be difficult to stream the 
> compilation 
> > inside the same package w/ circular refs and therefore may not be fixed 
> for 
> > years. Thanks for your help. 
>
> Yes, it's useful to report it at https://golang.org/issue.  It's 
> useful to have real test cases.  Thanks. 
>
> Ian 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Compiling a really really large package

[Ian Lance Taylor]

On Tue, Jan 10, 2017 at 9:47 AM,   wrote:
>
> Is there any value in me reporting it? If helpful, I may be able to toss a
> large tarball up on Dropbox or something. It's not really high priority for
> me either tbh because I know it would be difficult to stream the compilation
> inside the same package w/ circular refs and therefore may not be fixed for
> years. Thanks for your help.

Yes, it's useful to report it at https://golang.org/issue.  It's
useful to have real test cases.  Thanks.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Compiling a really really large package

[chad . retz]

Is there any value in me reporting it? If helpful, I may be able to toss a 
large tarball up on Dropbox or something. It's not really high priority for 
me either tbh because I know it would be difficult to stream the 
compilation inside the same package w/ circular refs and therefore may not 
be fixed for years. Thanks for your help.

On Tuesday, January 10, 2017 at 11:35:32 AM UTC-6, Ian Lance Taylor wrote:
>
> On Tue, Jan 10, 2017 at 9:22 AM,   
> wrote: 
> > 
> > Just as an update, Go 1.8 doesn't help much, and the compile flags do 
> reduce 
> > memory usage which only extends the inevitable time it runs out of 
> memory. I 
> > guess I am at a loss here. I suppose this is not considered a bug, 
> correct? 
> > Just a practical limitation of the compiler to use more memory 
> proportional 
> > to same-package code size? 
>
> Oh, no, it's definitely a bug.  It's just very hard to fix and rare in 
> practice, so it's not high priority. 
>
> Ian 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Compiling a really really large package

[chad . retz]

In my case, many of the functions are very small (many only a single line 
that I'm hoping will be inlined). This is a transpiler from another 
language (Java) akin to Grumpy (Python) and many of the functions are 
single-line dispatch methods to support OOP. The transpiler is at 
https://github.com/cretz/goahead if you're curious ("sbt buildRt" w/ the 
right env vars builds the 150MB stdlib). It has no documentation yet and is 
early in its life.

On Monday, January 9, 2017 at 1:02:14 PM UTC-6, Than McIntosh wrote:
>
> One thing to keep in mind: generated-code compilation time issues can 
> sometimes be due to a large function (or functions) as opposed just the 
> total volume of code in the package.
>
> For example, https://github.com/golang/go/issues/16407 demonstrates a 
> compile-time problem that sounds a bit like what you are seeing. The 
> problem can be avoided by tweaking the generator so that it creates a 
> collection of smaller functions as opposed to a single giant function. See 
> https://github.com/zhenjl/xparse/pull/2.
>
> Depending on how your generator works (and whether you have control over 
> it) maybe this is something you can consider.
>
>
> On Mon, Jan 9, 2017 at 1:28 PM,  wrote:
>
>> It does matter for my use case, but not for these first steps. Thanks. I 
>> think still, practically, I need to reduce the code size unfortunately.
>>
>> On Monday, January 9, 2017 at 12:24:36 PM UTC-6, Ian Lance Taylor wrote:
>>
>>> On Mon, Jan 9, 2017 at 9:00 AM,   wrote: 
>>> > I have a really really large package of code that was generated via a 
>>> code 
>>> > generator. Granted the main code that references it I expect to remove 
>>> a lot 
>>> > via DCE or something so the binaries wouldn't be extreme. The code is 
>>> > 
>>> > 140MB in the single package which I know sounds extreme. Let's ignore 
>>> > practical solutions like reducing code size. I have attempted to 
>>> compile on 
>>> > my Windows machine and the compile process just runs out of memory and 
>>> is 
>>> > unable to allocate anymore. I tried on a 4G VM w/ 8G swap and after 
>>> almost 
>>> > two hours it just gets killed (e.g. "go build example.com/pkg: 
>>> > /usr/local/go/pkg/tool/linux_amd64/compile: signal: killed"). 
>>> > 
>>> > This is with Go 1.7, I have not tested with Go 1.8 but will shortly. I 
>>> was 
>>> > hoping the compiler would be able to scale, even on a single package, 
>>> where 
>>> > it could stream the compilation. Are there any flags I should pass to 
>>> go 
>>> > build to make it use less RAM? Is there an effective upper limit on 
>>> package 
>>> > size or any plans to make the compiler not use linearly-more memory 
>>> based on 
>>> > code size? I can give instructions on how to build this extreme amount 
>>> of 
>>> > code too if anyone else wants to try. 
>>>
>>> There has been some work on improving the compilation of very large 
>>> packages in Go 1.8.  I expect that more work needs to be done. 
>>>
>>> You can try compiling with -gcflags=-N to disable the optimizers and 
>>> -gcflags=-l to disable the inliner.  That may reduce the memory 
>>> requirements, though of course the generated code will be worse.  That 
>>> may not matter for your case. 
>>>
>>> Ian 
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Compiling a really really large package

['Than McIntosh' via golang-nuts]

One thing to keep in mind: generated-code compilation time issues can
sometimes be due to a large function (or functions) as opposed just the
total volume of code in the package.

For example, https://github.com/golang/go/issues/16407 demonstrates a
compile-time problem that sounds a bit like what you are seeing. The
problem can be avoided by tweaking the generator so that it creates a
collection of smaller functions as opposed to a single giant function. See
https://github.com/zhenjl/xparse/pull/2.

Depending on how your generator works (and whether you have control over
it) maybe this is something you can consider.


On Mon, Jan 9, 2017 at 1:28 PM,  wrote:

> It does matter for my use case, but not for these first steps. Thanks. I
> think still, practically, I need to reduce the code size unfortunately.
>
> On Monday, January 9, 2017 at 12:24:36 PM UTC-6, Ian Lance Taylor wrote:
>
>> On Mon, Jan 9, 2017 at 9:00 AM,   wrote:
>> > I have a really really large package of code that was generated via a
>> code
>> > generator. Granted the main code that references it I expect to remove
>> a lot
>> > via DCE or something so the binaries wouldn't be extreme. The code is >
>> > 140MB in the single package which I know sounds extreme. Let's ignore
>> > practical solutions like reducing code size. I have attempted to
>> compile on
>> > my Windows machine and the compile process just runs out of memory and
>> is
>> > unable to allocate anymore. I tried on a 4G VM w/ 8G swap and after
>> almost
>> > two hours it just gets killed (e.g. "go build example.com/pkg:
>> > /usr/local/go/pkg/tool/linux_amd64/compile: signal: killed").
>> >
>> > This is with Go 1.7, I have not tested with Go 1.8 but will shortly. I
>> was
>> > hoping the compiler would be able to scale, even on a single package,
>> where
>> > it could stream the compilation. Are there any flags I should pass to
>> go
>> > build to make it use less RAM? Is there an effective upper limit on
>> package
>> > size or any plans to make the compiler not use linearly-more memory
>> based on
>> > code size? I can give instructions on how to build this extreme amount
>> of
>> > code too if anyone else wants to try.
>>
>> There has been some work on improving the compilation of very large
>> packages in Go 1.8.  I expect that more work needs to be done.
>>
>> You can try compiling with -gcflags=-N to disable the optimizers and
>> -gcflags=-l to disable the inliner.  That may reduce the memory
>> requirements, though of course the generated code will be worse.  That
>> may not matter for your case.
>>
>> Ian
>>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Compiling a really really large package

[chad . retz]

It does matter for my use case, but not for these first steps. Thanks. I 
think still, practically, I need to reduce the code size unfortunately.

On Monday, January 9, 2017 at 12:24:36 PM UTC-6, Ian Lance Taylor wrote:
>
> On Mon, Jan 9, 2017 at 9:00 AM,   wrote: 
> > I have a really really large package of code that was generated via a 
> code 
> > generator. Granted the main code that references it I expect to remove a 
> lot 
> > via DCE or something so the binaries wouldn't be extreme. The code is > 
> > 140MB in the single package which I know sounds extreme. Let's ignore 
> > practical solutions like reducing code size. I have attempted to compile 
> on 
> > my Windows machine and the compile process just runs out of memory and 
> is 
> > unable to allocate anymore. I tried on a 4G VM w/ 8G swap and after 
> almost 
> > two hours it just gets killed (e.g. "go build example.com/pkg: 
> > /usr/local/go/pkg/tool/linux_amd64/compile: signal: killed"). 
> > 
> > This is with Go 1.7, I have not tested with Go 1.8 but will shortly. I 
> was 
> > hoping the compiler would be able to scale, even on a single package, 
> where 
> > it could stream the compilation. Are there any flags I should pass to go 
> > build to make it use less RAM? Is there an effective upper limit on 
> package 
> > size or any plans to make the compiler not use linearly-more memory 
> based on 
> > code size? I can give instructions on how to build this extreme amount 
> of 
> > code too if anyone else wants to try. 
>
> There has been some work on improving the compilation of very large 
> packages in Go 1.8.  I expect that more work needs to be done. 
>
> You can try compiling with -gcflags=-N to disable the optimizers and 
> -gcflags=-l to disable the inliner.  That may reduce the memory 
> requirements, though of course the generated code will be worse.  That 
> may not matter for your case. 
>
> Ian 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.