[pollen] Re: parallel rendering fixes

[Matthew Butterick]

Eh, don't bother. I just found another bug in it.

> On Jun 13, 2019, at 6:22 PM, Matthew Butterick  wrote:
> 
> I recommend that anyone who enjoys `raco pollen render -p` update to the 
> newest Pollen. In the last week I've discovered (and hopefully corrected) a 
> couple bugs that were causing it to silently ignore files during a parallel 
> render.
> 
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Pollen" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pollenpub+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pollenpub/672D0CA6-9D69-4C82-9548-9CFF6ABC1EE7%40mbtype.com.
For more options, visit https://groups.google.com/d/optout.

[pollen] Re: parallel rendering fixes

[Matthew Butterick]

I'm afraid the parallel-processing problem is going to require more scrutiny 
about how it can be done safely. For the time being I've disabled parallel 
processing (you can still ask for `raco pollen render -p`, but it won't be 
parallel).


> On Jun 13, 2019, at 7:33 PM, Matthew Butterick  wrote:
> 
> Eh, don't bother. I just found another bug in it.
> 
>> On Jun 13, 2019, at 6:22 PM, Matthew Butterick  wrote:
>> 
>> I recommend that anyone who enjoys `raco pollen render -p` update to the 
>> newest Pollen. In the last week I've discovered (and hopefully corrected) a 
>> couple bugs that were causing it to silently ignore files during a parallel 
>> render.
>> 
>> 
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Pollen" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pollenpub+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pollenpub/2E3701BA-B6E8-4781-AD6C-5DDEC962D559%40mbtype.com.
For more options, visit https://groups.google.com/d/optout.

[pollen] Re: parallel rendering fixes

[Joel Dueck]

I haven’t had any trouble and it’s been saving me a *lot* of time, so I’ll 
hold off updating for now. Thank you for tackling this!

On Thursday, June 13, 2019 at 10:01:15 PM UTC-5, Matthew Butterick wrote:
>
> I'm afraid the parallel-processing problem is going to require more 
> scrutiny about how it can be done safely. For the time being I've disabled 
> parallel processing (you can still ask for `raco pollen render -p`, but it 
> won't be parallel). 
>
>
> > On Jun 13, 2019, at 7:33 PM, Matthew Butterick  > wrote: 
> > 
> > Eh, don't bother. I just found another bug in it. 
> > 
> >> On Jun 13, 2019, at 6:22 PM, Matthew Butterick  > wrote: 
> >> 
> >> I recommend that anyone who enjoys `raco pollen render -p` update to 
> the newest Pollen. In the last week I've discovered (and hopefully 
> corrected) a couple bugs that were causing it to silently ignore files 
> during a parallel render. 
> >> 
> >> 
> > 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Pollen" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pollenpub+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pollenpub/925f4f60-9743-429e-a049-f86a370cd55c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pollen] Re: parallel rendering fixes

[Matthew Butterick]

Armed with more knowledge, I have reinstated parallel rendering. `raco 
pollen setup` is also faster.


On Thursday, June 13, 2019 at 8:01:15 PM UTC-7, Matthew Butterick wrote:
>
> I'm afraid the parallel-processing problem is going to require more 
> scrutiny about how it can be done safely. For the time being I've disabled 
> parallel processing (you can still ask for `raco pollen render -p`, but it 
> won't be parallel). 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Pollen" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pollenpub+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pollenpub/9e5e8d34-2b8a-4718-85dd-6a6168858e64%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pollen] Re: parallel rendering fixes

[Joel Dueck]

Awesome, thank you!

On Wednesday, June 26, 2019 at 1:29:31 PM UTC-5, Matthew Butterick wrote:
>
> Armed with more knowledge, I have reinstated parallel rendering. `raco 
> pollen setup` is also faster.
>
>
> On Thursday, June 13, 2019 at 8:01:15 PM UTC-7, Matthew Butterick wrote:
>>
>> I'm afraid the parallel-processing problem is going to require more 
>> scrutiny about how it can be done safely. For the time being I've disabled 
>> parallel processing (you can still ask for `raco pollen render -p`, but it 
>> won't be parallel). 
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Pollen" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pollenpub+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pollenpub/a047207d-1a85-4021-a899-4f86738e0fde%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pollen] Re: parallel rendering fixes

[Shrutarshi Basu]

This is great. I'm wondering how this should affect Makefile-driven setups?
Currently I have a project with a bunch of directories, and a Makefile that
runs `raco pollen render` for all the directories, and then publish at the
toplevel directory. I've been running it `make -j 4` (since that's how many
cores I have), but maybe there's a better way to do it? Should I just use
`raco pollen render --recursive` at the toplevel and let pollen figure
everything out?

Here's a gist of my current Makefile, if it helps:
https://gist.github.com/basus/e524fed3a305fe4235ba03eab68c634d

-- 
Shrutarshi Basu
Basus.me
The ByteBaker  -- computer science is not about
computers
@basus 

-- 
You received this message because you are subscribed to the Google Groups 
"Pollen" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pollenpub+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pollenpub/CA%2BYT8Whdf8sCAzu4r0jhxYwXP7ZeqszQPu8rr2opvx0YGv2NJA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pollen] Re: parallel rendering fixes

[Joel Dueck]

Take a look at one of my recent Pollen makefiles:  
https://thelocalyarn.com/cgi-bin/yarncode/artifact/7a3d32d6264f1bcb

I define my dependencies such that if one of the core files (like 
pollen.rkt) has changed, it triggers a re-render of all documents with 
`raco pollen setup` and then `raco pollen render -p ...`.

The later dependencies check to see if individual Pollen articles 
(.poly.pm) have changed since their .html file was last rendered. If a 
render-of-everything has already been triggered above, then these will 
already be up to date at this point. But if not, any changed articles will 
be re-rendered singly, without parallel processing. (I don't use the -j 
switch when running make.)

So the basic idea is that I give make a very specific dependency tree, and 
rely on it to check those dependencies in a specific order, bringing in the 
new parallel setup and render when it's clear a complete rebuild is needed.

You also have to make sure that the makefile and the Pollen setup cache 
watchlist agree precisely on which are the "core" files that would trigger 
a complete rebuild.

Because I have some Pollen articles that depend on others, I also use 
tricks like "touching" the dependent articles during the evaluation of the 
`root` tag.
See 
https://thelocalyarn.com/cgi-bin/yarncode/artifact?udc=1&ln=91-92&name=d9318d9822c88d35&udc=1
   and 
https://thelocalyarn.com/cgi-bin/yarncode/artifact?udc=1&ln=101-111&name=30ceaf46d62ff4c0



On Thursday, June 27, 2019 at 3:23:37 PM UTC-5, Shrutarshi Basu wrote:
>
>
> This is great. I'm wondering how this should affect Makefile-driven 
> setups? Currently I have a project with a bunch of directories, and a 
> Makefile that runs `raco pollen render` for all the directories, and then 
> publish at the toplevel directory. I've been running it `make -j 4` (since 
> that's how many cores I have), but maybe there's a better way to do it? 
> Should I just use `raco pollen render --recursive` at the toplevel and let 
> pollen figure everything out?
>
> Here's a gist of my current Makefile, if it helps: 
> https://gist.github.com/basus/e524fed3a305fe4235ba03eab68c634d
>
> -- 
> Shrutarshi Basu
> Basus.me
> The ByteBaker  -- computer science is not about 
> computers
> @basus 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Pollen" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pollenpub+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pollenpub/d52ce987-16bf-4df1-be52-0b67450c8d22%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pollen] Re: parallel rendering fixes

[Shrutarshi Basu]

Thank you! This is very helpful. I'm still trying to figure out what the
good balance between Makefiles and render -p. Some experimenting is in
order.

On Fri, Jun 28, 2019 at 12:47 PM Joel Dueck  wrote:

> Take a look at one of my recent Pollen makefiles:
> https://thelocalyarn.com/cgi-bin/yarncode/artifact/7a3d32d6264f1bcb
>
> I define my dependencies such that if one of the core files (like
> pollen.rkt) has changed, it triggers a re-render of all documents with
> `raco pollen setup` and then `raco pollen render -p ...`.
>
> The later dependencies check to see if individual Pollen articles (.
> poly.pm) have changed since their .html file was last rendered. If a
> render-of-everything has already been triggered above, then these will
> already be up to date at this point. But if not, any changed articles will
> be re-rendered singly, without parallel processing. (I don't use the -j
> switch when running make.)
>
> So the basic idea is that I give make a very specific dependency tree, and
> rely on it to check those dependencies in a specific order, bringing in the
> new parallel setup and render when it's clear a complete rebuild is needed.
>
> You also have to make sure that the makefile and the Pollen setup cache
> watchlist agree precisely on which are the "core" files that would trigger
> a complete rebuild.
>
> Because I have some Pollen articles that depend on others, I also use
> tricks like "touching" the dependent articles during the evaluation of the
> `root` tag.
> See
> https://thelocalyarn.com/cgi-bin/yarncode/artifact?udc=1&ln=91-92&name=d9318d9822c88d35&udc=1
>and
> https://thelocalyarn.com/cgi-bin/yarncode/artifact?udc=1&ln=101-111&name=30ceaf46d62ff4c0
>
>
>
> On Thursday, June 27, 2019 at 3:23:37 PM UTC-5, Shrutarshi Basu wrote:
>>
>>
>> This is great. I'm wondering how this should affect Makefile-driven
>> setups? Currently I have a project with a bunch of directories, and a
>> Makefile that runs `raco pollen render` for all the directories, and then
>> publish at the toplevel directory. I've been running it `make -j 4` (since
>> that's how many cores I have), but maybe there's a better way to do it?
>> Should I just use `raco pollen render --recursive` at the toplevel and let
>> pollen figure everything out?
>>
>> Here's a gist of my current Makefile, if it helps:
>> https://gist.github.com/basus/e524fed3a305fe4235ba03eab68c634d
>>
>> --
>> Shrutarshi Basu
>> Basus.me
>> The ByteBaker  -- computer science is not about
>> computers
>> @basus 
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Pollen" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pollenpub+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pollenpub/d52ce987-16bf-4df1-be52-0b67450c8d22%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Shrutarshi Basu
Basus.me
The ByteBaker  -- computer science is not about
computers
@basus 

-- 
You received this message because you are subscribed to the Google Groups 
"Pollen" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pollenpub+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pollenpub/CA%2BYT8Wip70pTRvr0KV0YL_hGPETRQAUEvEXJ2aKmZiNC01V%3Dbg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.