Re: [racket-users] Racket slower than Chez Scheme on interpreter benchmark, potential low hanging fruit?

[Sorawee Porncharoenwase]

With the recent improvements by Phil, the rank of the syntax object variant
moves up from 26th to the second (what?!?), losing only to c++ / g++.
Moreover, it's significantly faster than the third place.

On Fri, Mar 5, 2021 at 3:29 AM philngu...@gmail.com <
philnguyen0...@gmail.com> wrote:

> Oh I see. So one problem is here that `match-define` expands to `match`
> with an implicit error case, which at the low level, isn't distinguished
> from a user-written second case, and the tag check can't just be eliminated.
>
> On Thursday, March 4, 2021 at 9:40:22 AM UTC-8 Sam Tobin-Hochstadt wrote:
>
>> I think there are two reasons that code gets slower.
>>
>> 1. The `match-define` includes pair and struct checks, which are
>> omitted for plain accessor uses because of the unsafe declaration.
>> 2. That use of `match` expands to `define-values` which ends up as a
>> `call-with-values` and a `case-lambda` at the chez layer and is not
>> removed.
>>
>> `match` could recognize that it's being compiled with unsafe mode and
>> omit these checks, although it's not that straightforward. Also
>> schemify (or Chez) could do more to eliminate the use of multiple
>> values, although that's hard without eliminating the failure cases.
>>
>> Sam
>>
>> On Thu, Mar 4, 2021 at 3:23 AM philngu...@gmail.com
>>  wrote:
>> >
>> > Thanks for the tip about PLT_CS_COMPILE_LIMIT! I submitted a revision
>> to the syntax object variant that incorporated sleepnova's and yjqww6's
>> improvements.
>> >
>> > Also, I never knew about `(#%declare #:unsafe)` until I saw yjqww6's
>> pull request. It makes a noticeable difference. One unsatisfying thing is
>> that in one place, if I replace the 4 separate define clauses with just
>> `(match-define (cons (op o val) rst) parsed)`, the benchmarks are more than
>> twice slower.
>> >
>> > On Wednesday, March 3, 2021 at 11:12:30 AM UTC-8 Sam Tobin-Hochstadt
>> wrote:
>> >>
>> >> First, there's no longer a difference because yjqww6 just had a PR
>> >> merged that improves the Racket performance.
>> >>
>> >> The performance difference that was there was mostly because the Chez
>> >> code was run with `--optimize-level 3` which turns off safety. If that
>> >> was changed to `--optimize-level 2` the timing became much slower.
>> >>
>> >> Sam
>> >>
>> >> On Mon, Mar 1, 2021 at 2:39 AM philngu...@gmail.com
>> >>  wrote:
>> >> >
>> >> > There’s this benchmark on BF interpreter where the Racket and Chez
>> Scheme implementations are very similar, but Chez Scheme is much faster
>> than Racket 8.0 at interpreting bench.b (3s vs 8s) and mandel.b (40s vs
>> 136s).
>> >> >
>> >> > There’s the “Racket (Syntax Object)” variant that directly parses
>> BF’s syntax into Racket syntax object, which is faster (3.7s for bench, 82s
>> for mandel), but still significantly behind Chez Scheme’s naive
>> interpreter.
>> >> >
>> >> > Profiling doesn’t give very illuminating results, saying most of the
>> cost is from interpreting BF’s loop instruction.
>> >> >
>> >> > Given that Racket is on Chez, could this benchmark reveal some low
>> hanging fruit for improving Racket’s performance?
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> Groups "Racket Users" group.
>> >> > To unsubscribe from this group and stop receiving emails from it,
>> send an email to racket-users...@googlegroups.com.
>> >> > To view this discussion on the web visit
>> https://groups.google.com/d/msgid/racket-users/83b2819d-8295-4769-944d-fa0c155976dan%40googlegroups.com.
>>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups "Racket Users" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> an email to racket-users...@googlegroups.com.
>> > To view this discussion on the web visit
>> https://groups.google.com/d/msgid/racket-users/a5e77286-68b8-481a-8dea-0f547c5ce968n%40googlegroups.com.
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to racket-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/racket-users/f771a13a-9fc4-4b71-9e47-3a83eb5290e7n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-users/CADcuegvSFZvD9SaTz2uakJp%3D6tmoEmVbGJKhxwzdeHsse8jVkQ%40mail.gmail.com.

Re: [racket-users] Racket slower than Chez Scheme on interpreter benchmark, potential low hanging fruit?

[philngu...@gmail.com]

Oh I see. So one problem is here that `match-define` expands to `match` 
with an implicit error case, which at the low level, isn't distinguished 
from a user-written second case, and the tag check can't just be eliminated.

On Thursday, March 4, 2021 at 9:40:22 AM UTC-8 Sam Tobin-Hochstadt wrote:

> I think there are two reasons that code gets slower.
>
> 1. The `match-define` includes pair and struct checks, which are
> omitted for plain accessor uses because of the unsafe declaration.
> 2. That use of `match` expands to `define-values` which ends up as a
> `call-with-values` and a `case-lambda` at the chez layer and is not
> removed.
>
> `match` could recognize that it's being compiled with unsafe mode and
> omit these checks, although it's not that straightforward. Also
> schemify (or Chez) could do more to eliminate the use of multiple
> values, although that's hard without eliminating the failure cases.
>
> Sam
>
> On Thu, Mar 4, 2021 at 3:23 AM philngu...@gmail.com
>  wrote:
> >
> > Thanks for the tip about PLT_CS_COMPILE_LIMIT! I submitted a revision to 
> the syntax object variant that incorporated sleepnova's and yjqww6's 
> improvements.
> >
> > Also, I never knew about `(#%declare #:unsafe)` until I saw yjqww6's 
> pull request. It makes a noticeable difference. One unsatisfying thing is 
> that in one place, if I replace the 4 separate define clauses with just 
> `(match-define (cons (op o val) rst) parsed)`, the benchmarks are more than 
> twice slower.
> >
> > On Wednesday, March 3, 2021 at 11:12:30 AM UTC-8 Sam Tobin-Hochstadt 
> wrote:
> >>
> >> First, there's no longer a difference because yjqww6 just had a PR
> >> merged that improves the Racket performance.
> >>
> >> The performance difference that was there was mostly because the Chez
> >> code was run with `--optimize-level 3` which turns off safety. If that
> >> was changed to `--optimize-level 2` the timing became much slower.
> >>
> >> Sam
> >>
> >> On Mon, Mar 1, 2021 at 2:39 AM philngu...@gmail.com
> >>  wrote:
> >> >
> >> > There’s this benchmark on BF interpreter where the Racket and Chez 
> Scheme implementations are very similar, but Chez Scheme is much faster 
> than Racket 8.0 at interpreting bench.b (3s vs 8s) and mandel.b (40s vs 
> 136s).
> >> >
> >> > There’s the “Racket (Syntax Object)” variant that directly parses 
> BF’s syntax into Racket syntax object, which is faster (3.7s for bench, 82s 
> for mandel), but still significantly behind Chez Scheme’s naive interpreter.
> >> >
> >> > Profiling doesn’t give very illuminating results, saying most of the 
> cost is from interpreting BF’s loop instruction.
> >> >
> >> > Given that Racket is on Chez, could this benchmark reveal some low 
> hanging fruit for improving Racket’s performance?
> >> >
> >> > --
> >> > You received this message because you are subscribed to the Google 
> Groups "Racket Users" group.
> >> > To unsubscribe from this group and stop receiving emails from it, 
> send an email to racket-users...@googlegroups.com.
> >> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/racket-users/83b2819d-8295-4769-944d-fa0c155976dan%40googlegroups.com
> .
> >
> > --
> > You received this message because you are subscribed to the Google 
> Groups "Racket Users" group.
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to racket-users...@googlegroups.com.
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/racket-users/a5e77286-68b8-481a-8dea-0f547c5ce968n%40googlegroups.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-users/f771a13a-9fc4-4b71-9e47-3a83eb5290e7n%40googlegroups.com.

Re: [racket-users] Racket slower than Chez Scheme on interpreter benchmark, potential low hanging fruit?

[Sam Tobin-Hochstadt]

I think there are two reasons that code gets slower.

1. The `match-define` includes pair and struct checks, which are
omitted for plain accessor uses because of the unsafe declaration.
2. That use of `match` expands to `define-values` which ends up as a
`call-with-values` and a `case-lambda` at the chez layer and is not
removed.

`match` could recognize that it's being compiled with unsafe mode and
omit these checks, although it's not that straightforward. Also
schemify (or Chez) could do more to eliminate the use of multiple
values, although that's hard without eliminating the failure cases.

Sam

On Thu, Mar 4, 2021 at 3:23 AM philngu...@gmail.com
 wrote:
>
> Thanks for the tip about PLT_CS_COMPILE_LIMIT! I submitted a revision to the 
> syntax object variant that incorporated sleepnova's and yjqww6's improvements.
>
> Also, I never knew about `(#%declare #:unsafe)` until I saw yjqww6's pull 
> request. It makes a noticeable difference. One unsatisfying thing is that in 
> one place, if I replace the 4 separate define clauses with just 
> `(match-define (cons (op o val) rst) parsed)`, the benchmarks are more than 
> twice slower.
>
> On Wednesday, March 3, 2021 at 11:12:30 AM UTC-8 Sam Tobin-Hochstadt wrote:
>>
>> First, there's no longer a difference because yjqww6 just had a PR
>> merged that improves the Racket performance.
>>
>> The performance difference that was there was mostly because the Chez
>> code was run with `--optimize-level 3` which turns off safety. If that
>> was changed to `--optimize-level 2` the timing became much slower.
>>
>> Sam
>>
>> On Mon, Mar 1, 2021 at 2:39 AM philngu...@gmail.com
>>  wrote:
>> >
>> > There’s this benchmark on BF interpreter where the Racket and Chez Scheme 
>> > implementations are very similar, but Chez Scheme is much faster than 
>> > Racket 8.0 at interpreting bench.b (3s vs 8s) and mandel.b (40s vs 136s).
>> >
>> > There’s the “Racket (Syntax Object)” variant that directly parses BF’s 
>> > syntax into Racket syntax object, which is faster (3.7s for bench, 82s for 
>> > mandel), but still significantly behind Chez Scheme’s naive interpreter.
>> >
>> > Profiling doesn’t give very illuminating results, saying most of the cost 
>> > is from interpreting BF’s loop instruction.
>> >
>> > Given that Racket is on Chez, could this benchmark reveal some low hanging 
>> > fruit for improving Racket’s performance?
>> >
>> > --
>> > You received this message because you are subscribed to the Google Groups 
>> > "Racket Users" group.
>> > To unsubscribe from this group and stop receiving emails from it, send an 
>> > email to racket-users...@googlegroups.com.
>> > To view this discussion on the web visit 
>> > https://groups.google.com/d/msgid/racket-users/83b2819d-8295-4769-944d-fa0c155976dan%40googlegroups.com.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to racket-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/racket-users/a5e77286-68b8-481a-8dea-0f547c5ce968n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-users/CAK%3DHD%2BbSp%3Dg-POJ7v9HUksw5WivY6_RbsPSnGv8B6PwfSYZs1A%40mail.gmail.com.

Re: [racket-users] Racket slower than Chez Scheme on interpreter benchmark, potential low hanging fruit?

[philngu...@gmail.com]

Thanks for the tip about PLT_CS_COMPILE_LIMIT! I submitted a revision to 
the syntax object variant that incorporated sleepnova's and yjqww6's 
improvements.

Also, I never knew about `(#%declare #:unsafe)` until I saw yjqww6's pull 
request. It makes a noticeable difference. One unsatisfying thing is that 
in one place, if I replace the 4 separate define clauses 

 
with just `(match-define (cons (op o val) rst) parsed)`, the benchmarks are 
more than twice slower.

On Wednesday, March 3, 2021 at 11:12:30 AM UTC-8 Sam Tobin-Hochstadt wrote:

> First, there's no longer a difference because yjqww6 just had a PR
> merged that improves the Racket performance.
>
> The performance difference that was there was mostly because the Chez
> code was run with `--optimize-level 3` which turns off safety. If that
> was changed to `--optimize-level 2` the timing became much slower.
>
> Sam
>
> On Mon, Mar 1, 2021 at 2:39 AM philngu...@gmail.com
>  wrote:
> >
> > There’s this benchmark on BF interpreter where the Racket and Chez 
> Scheme implementations are very similar, but Chez Scheme is much faster 
> than Racket 8.0 at interpreting bench.b (3s vs 8s) and mandel.b (40s vs 
> 136s).
> >
> > There’s the “Racket (Syntax Object)” variant that directly parses BF’s 
> syntax into Racket syntax object, which is faster (3.7s for bench, 82s for 
> mandel), but still significantly behind Chez Scheme’s naive interpreter.
> >
> > Profiling doesn’t give very illuminating results, saying most of the 
> cost is from interpreting BF’s loop instruction.
> >
> > Given that Racket is on Chez, could this benchmark reveal some low 
> hanging fruit for improving Racket’s performance?
> >
> > --
> > You received this message because you are subscribed to the Google 
> Groups "Racket Users" group.
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to racket-users...@googlegroups.com.
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/racket-users/83b2819d-8295-4769-944d-fa0c155976dan%40googlegroups.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-users/a5e77286-68b8-481a-8dea-0f547c5ce968n%40googlegroups.com.

Re: [racket-users] Racket slower than Chez Scheme on interpreter benchmark, potential low hanging fruit?

[Sam Tobin-Hochstadt]

First, there's no longer a difference because yjqww6 just had a PR
merged that improves the Racket performance.

The performance difference that was there was mostly because the Chez
code was run with `--optimize-level 3` which turns off safety. If that
was changed to `--optimize-level 2` the timing became much slower.

Sam

On Mon, Mar 1, 2021 at 2:39 AM philngu...@gmail.com
 wrote:
>
> There’s this benchmark on BF interpreter where the Racket and Chez Scheme 
> implementations are very similar, but Chez Scheme is much faster than Racket 
> 8.0 at interpreting bench.b (3s vs 8s) and mandel.b (40s vs 136s).
>
> There’s the “Racket (Syntax Object)” variant that directly parses BF’s syntax 
> into Racket syntax object, which is faster (3.7s for bench, 82s for mandel), 
> but still significantly behind Chez Scheme’s naive interpreter.
>
> Profiling doesn’t give very illuminating results, saying most of the cost is 
> from interpreting BF’s loop instruction.
>
> Given that Racket is on Chez, could this benchmark reveal some low hanging 
> fruit for improving Racket’s performance?
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to racket-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/racket-users/83b2819d-8295-4769-944d-fa0c155976dan%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-users/CAK%3DHD%2BY6e%3D7QXJZhOhF8-QbNc5HGu%2BUcrGzr1HhctSp1sA470g%40mail.gmail.com.

Re: [racket-users] Racket slower than Chez Scheme on interpreter benchmark, potential low hanging fruit?

[Dexter Lagan]

  For what it's worth, I ran my own benchmark on Racket 8.0 and Racket 
8.0.10 (current), and current is between 50 and 100% faster for certain 
operations. There must have been some optimizations done recently to 
current.

Dex
On Tuesday, March 2, 2021 at 9:37:29 AM UTC+1 wanp...@gmail.com wrote:

> I previously came out with a version that converts a BF program directly 
> into a module, which has received some optimization contributions from 
> Matthew Flatt.
> Got some pretty good results.
>
> Execute bench.b in cpu time: 1542 ms, (2.2s include compile time).
>
> And mandel.b in cpu time: 3851 ms, (7s include compile time). But you have 
> to manually "export PLT_CS_COMPILE_LIMIT=10" environment variable in 
> order to get compiled and executed properly.
>
> https://gist.github.com/sleepnova/92d7b2a7f077e7de76da4ce31f60335e
>
> philngu...@gmail.com  於 2021年3月1日 週一 下午3:39寫道：
>
>> There’s this benchmark on BF interpreter where the Racket 
>>  and Chez 
>> Scheme  
>> implementations are very similar, but Chez Scheme is much faster than 
>> Racket 8.0 at interpreting bench.b 
>>  (3s vs 8s) and mandel.b 
>>  (40s vs 136s).
>>
>> There’s the “Racket (Syntax Object) 
>> ”
>>  
>> variant that directly parses BF’s syntax into Racket syntax object, which 
>> is faster (3.7s for bench, 82s for mandel), but still significantly behind 
>> Chez Scheme’s naive interpreter.
>>
>> Profiling doesn’t give very illuminating results, saying most of the cost 
>> is from interpreting BF’s loop instruction.
>>
>> Given that Racket is on Chez, could this benchmark reveal some low 
>> hanging fruit for improving Racket’s performance?
>>
>> -- 
>>
> You received this message because you are subscribed to the Google Groups 
>> "Racket Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to racket-users...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/racket-users/83b2819d-8295-4769-944d-fa0c155976dan%40googlegroups.com
>>  
>> 
>> .
>>
>
>
> -- 
> - sleepnova
> 呼叫小黃創辦人 & CEO
>

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-users/281c3e71-028e-4709-bc77-6f9956afadf1n%40googlegroups.com.

Re: [racket-users] Racket slower than Chez Scheme on interpreter benchmark, potential low hanging fruit?

[sleepnova]

I previously came out with a version that converts a BF program directly
into a module, which has received some optimization contributions from
Matthew Flatt.
Got some pretty good results.

Execute bench.b in cpu time: 1542 ms, (2.2s include compile time).

And mandel.b in cpu time: 3851 ms, (7s include compile time). But you have
to manually "export PLT_CS_COMPILE_LIMIT=10" environment variable in
order to get compiled and executed properly.

https://gist.github.com/sleepnova/92d7b2a7f077e7de76da4ce31f60335e

philngu...@gmail.com  於 2021年3月1日 週一 下午3:39寫道：

> There’s this benchmark on BF interpreter where the Racket
>  and Chez
> Scheme 
> implementations are very similar, but Chez Scheme is much faster than
> Racket 8.0 at interpreting bench.b
>  (3s vs 8s) and mandel.b
>  (40s vs 136s).
>
> There’s the “Racket (Syntax Object)
> ”
> variant that directly parses BF’s syntax into Racket syntax object, which
> is faster (3.7s for bench, 82s for mandel), but still significantly behind
> Chez Scheme’s naive interpreter.
>
> Profiling doesn’t give very illuminating results, saying most of the cost
> is from interpreting BF’s loop instruction.
>
> Given that Racket is on Chez, could this benchmark reveal some low hanging
> fruit for improving Racket’s performance?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to racket-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/racket-users/83b2819d-8295-4769-944d-fa0c155976dan%40googlegroups.com
> 
> .
>


-- 
- sleepnova
呼叫小黃創辦人 & CEO

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-users/CABa2-7PQp4OnagRZk9QtNtLx67mX0huL3f%3DM71XB_ggxRX-U3w%40mail.gmail.com.

Re: [racket-users] Racket slower than Chez Scheme on interpreter benchmark, potential low hanging fruit?

[Sam Tobin-Hochstadt]

When Chez is faster than Racket CS, the usual culprits are either:
- mutable pairs
- very large code size that causes Racket CS to interpret the outer module

However, neither of those seem to be happening here.

Sam

On Mon, Mar 1, 2021 at 2:39 AM philngu...@gmail.com
 wrote:
>
> There’s this benchmark on BF interpreter where the Racket and Chez Scheme 
> implementations are very similar, but Chez Scheme is much faster than Racket 
> 8.0 at interpreting bench.b (3s vs 8s) and mandel.b (40s vs 136s).
>
> There’s the “Racket (Syntax Object)” variant that directly parses BF’s syntax 
> into Racket syntax object, which is faster (3.7s for bench, 82s for mandel), 
> but still significantly behind Chez Scheme’s naive interpreter.
>
> Profiling doesn’t give very illuminating results, saying most of the cost is 
> from interpreting BF’s loop instruction.
>
> Given that Racket is on Chez, could this benchmark reveal some low hanging 
> fruit for improving Racket’s performance?
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to racket-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/racket-users/83b2819d-8295-4769-944d-fa0c155976dan%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-users/CAK%3DHD%2BY5p%3DW2%2BMrg_QpvVQZVXY%2B3sHW-c%3DGcK1KrTT%3D0hRC9-g%40mail.gmail.com.