Re: Completely blank flowgraph takes a minute to generate

[Marcus Müller]

Hi Jameson,

don't overdo it! Johannes' approach might really be the more sensible here!

On 03.10.21 22:06, Jameson Collins wrote:
> So I haven't forgotten about this, it's just turned into more work than 
> expected.  The
> target didn't have perf, and it turns out there was a kernel bug that made it 
> unusable. 
> Fixed that, and then didn't have enough space on the ram filesystem anymore.  
> Fixed that
> and noticed that I don't get function names, just addresses.  Currently 
> working on that
> problem.
> 
> On Wed, Sep 29, 2021 at 1:26 PM Marcus Müller  >
> wrote:
> 
> Hi Jameson,
> 
> yep, and if it's not IO-bound, it's still a lot of YAML files to parse,
> so it's really a nice mini-benchmark for your filesystem and python's
> yaml implementation. By the way, which version of GNU Radio are you 
> running?
> If you want to have a look into what your OS is up to while you run
> grcc, I'd run `sudo perf record -a -g grcc parameters...`
> 
> followed by `perf report`; it's not as useful to profile the Python side
> of things, Johannes' method is the right one here, but in case you need
> to know in which libc function or which kernel function you're stuck
> most of the time: great tool!
> 
> Best regards,
> Marcus
> 
> On 29/09/2021 14.04, Jameson Collins wrote:
> > @Johannes, that's neat.  I'll give that a shot.
> >
> > @Marcus, I can't run GRC on the target as it doesn't have X.  I'm using
> > grcc to generate the block, I should have mentioned that.  When you say
> > IO-bound I'm guessing you mean because it's scanning the disk?  I
> > haven't benchmarked this hardware yet, but the filesystem is a ram disk
> > so I would expect it to be reasonably fast.  But this is something I can
> > look into.
> >
> > Thanks
> >
> > On Wed, Sep 29, 2021 at 6:55 AM Marcus Müller  
> > >> wrote:
> >
> >     But before going into too much depth, make sure the duration isn't 
> just
> >     basically identical to clicking on the "reload block library" 
> button,
> >     which is first IO-bound (which *can* take significant amount of CPU
> >     time
> >     on weaker ARMs) and then parsing-limited.
> >
> >     Best Regards,
> >
> >     Marcus
> >
> >     On 9/29/21 11:21 AM, Johannes Demel wrote:
> >      > Hi,
> >      >
> >      > I used:
> >      > https://docs.python.org/3/library/profile.html#module-cProfile
> 
> >      >
> >      > in the past to locate the problematic lines of code.
> >      >
> >      > ```
> >      > import cProfile
> >      > import pstats
> >      >
> >      > with cProfile.Profile() as pr:
> >      >     run_the_problematic_function_etc()
> >      > stats = pstats.Stats(pr)
> >      > stats.sort_stats('cumtime').reverse_order()
> >      > stats.print_stats()
> >      > ```
> >      > You might want to adopt these lines for your use-case. I started 
> at
> >      > the top-level of my application and then gradually moved the code
> >     to a
> >      > more fitting function.
> >      >
> >      > Cheers
> >      > Johannes
> >      >
> >      > On 28.09.21 23:40, Jameson Collins wrote:
> >      >> I have a completely blank flowgraph (well just the default
> >     'Options'
> >      >> block).
> >      >>
> >      >> On my 8 core, 2 GHz, Intel based virtual machine this flowgraph
> >      >> generates in under a second.
> >      >>
> >      >> On my 4 core, 1GHz, embedded ARM processor it takes 56 seconds 
> and
> >      >> completely maxes out 1 core while it's running.
> >      >>
> >      >> Why might cause this?  How might I troubleshoot it?
> >      >
> >
> 



smime.p7s
Description: S/MIME Cryptographic Signature

Re: Completely blank flowgraph takes a minute to generate

[Jameson Collins]

So I haven't forgotten about this, it's just turned into more work than
expected.  The target didn't have perf, and it turns out there was a kernel
bug that made it unusable.  Fixed that, and then didn't have enough space
on the ram filesystem anymore.  Fixed that and noticed that I don't get
function names, just addresses.  Currently working on that problem.

On Wed, Sep 29, 2021 at 1:26 PM Marcus Müller  wrote:

> Hi Jameson,
>
> yep, and if it's not IO-bound, it's still a lot of YAML files to parse,
> so it's really a nice mini-benchmark for your filesystem and python's
> yaml implementation. By the way, which version of GNU Radio are you
> running?
> If you want to have a look into what your OS is up to while you run
> grcc, I'd run `sudo perf record -a -g grcc parameters...`
>
> followed by `perf report`; it's not as useful to profile the Python side
> of things, Johannes' method is the right one here, but in case you need
> to know in which libc function or which kernel function you're stuck
> most of the time: great tool!
>
> Best regards,
> Marcus
>
> On 29/09/2021 14.04, Jameson Collins wrote:
> > @Johannes, that's neat.  I'll give that a shot.
> >
> > @Marcus, I can't run GRC on the target as it doesn't have X.  I'm using
> > grcc to generate the block, I should have mentioned that.  When you say
> > IO-bound I'm guessing you mean because it's scanning the disk?  I
> > haven't benchmarked this hardware yet, but the filesystem is a ram disk
> > so I would expect it to be reasonably fast.  But this is something I can
> > look into.
> >
> > Thanks
> >
> > On Wed, Sep 29, 2021 at 6:55 AM Marcus Müller  > > wrote:
> >
> > But before going into too much depth, make sure the duration isn't
> just
> > basically identical to clicking on the "reload block library" button,
> > which is first IO-bound (which *can* take significant amount of CPU
> > time
> > on weaker ARMs) and then parsing-limited.
> >
> > Best Regards,
> >
> > Marcus
> >
> > On 9/29/21 11:21 AM, Johannes Demel wrote:
> >  > Hi,
> >  >
> >  > I used:
> >  > https://docs.python.org/3/library/profile.html#module-cProfile
> > 
> >  > in the past to locate the problematic lines of code.
> >  >
> >  > ```
> >  > import cProfile
> >  > import pstats
> >  >
> >  > with cProfile.Profile() as pr:
> >  > run_the_problematic_function_etc()
> >  > stats = pstats.Stats(pr)
> >  > stats.sort_stats('cumtime').reverse_order()
> >  > stats.print_stats()
> >  > ```
> >  > You might want to adopt these lines for your use-case. I started
> at
> >  > the top-level of my application and then gradually moved the code
> > to a
> >  > more fitting function.
> >  >
> >  > Cheers
> >  > Johannes
> >  >
> >  > On 28.09.21 23:40, Jameson Collins wrote:
> >  >> I have a completely blank flowgraph (well just the default
> > 'Options'
> >  >> block).
> >  >>
> >  >> On my 8 core, 2 GHz, Intel based virtual machine this flowgraph
> >  >> generates in under a second.
> >  >>
> >  >> On my 4 core, 1GHz, embedded ARM processor it takes 56 seconds
> and
> >  >> completely maxes out 1 core while it's running.
> >  >>
> >  >> Why might cause this?  How might I troubleshoot it?
> >  >
> >
>
>

Re: Completely blank flowgraph takes a minute to generate

[Marcus Müller]

Hi Jameson,

yep, and if it's not IO-bound, it's still a lot of YAML files to parse, 
so it's really a nice mini-benchmark for your filesystem and python's 
yaml implementation. By the way, which version of GNU Radio are you running?
If you want to have a look into what your OS is up to while you run 
grcc, I'd run `sudo perf record -a -g grcc parameters...`


followed by `perf report`; it's not as useful to profile the Python side 
of things, Johannes' method is the right one here, but in case you need 
to know in which libc function or which kernel function you're stuck 
most of the time: great tool!


Best regards,
Marcus

On 29/09/2021 14.04, Jameson Collins wrote:

@Johannes, that's neat.  I'll give that a shot.

@Marcus, I can't run GRC on the target as it doesn't have X.  I'm using 
grcc to generate the block, I should have mentioned that.  When you say 
IO-bound I'm guessing you mean because it's scanning the disk?  I 
haven't benchmarked this hardware yet, but the filesystem is a ram disk 
so I would expect it to be reasonably fast.  But this is something I can 
look into.


Thanks

On Wed, Sep 29, 2021 at 6:55 AM Marcus Müller > wrote:


But before going into too much depth, make sure the duration isn't just
basically identical to clicking on the "reload block library" button,
which is first IO-bound (which *can* take significant amount of CPU
time
on weaker ARMs) and then parsing-limited.

Best Regards,

Marcus

On 9/29/21 11:21 AM, Johannes Demel wrote:
 > Hi,
 >
 > I used:
 > https://docs.python.org/3/library/profile.html#module-cProfile

 > in the past to locate the problematic lines of code.
 >
 > ```
 > import cProfile
 > import pstats
 >
 > with cProfile.Profile() as pr:
 >     run_the_problematic_function_etc()
 > stats = pstats.Stats(pr)
 > stats.sort_stats('cumtime').reverse_order()
 > stats.print_stats()
 > ```
 > You might want to adopt these lines for your use-case. I started at
 > the top-level of my application and then gradually moved the code
to a
 > more fitting function.
 >
 > Cheers
 > Johannes
 >
 > On 28.09.21 23:40, Jameson Collins wrote:
 >> I have a completely blank flowgraph (well just the default
'Options'
 >> block).
 >>
 >> On my 8 core, 2 GHz, Intel based virtual machine this flowgraph
 >> generates in under a second.
 >>
 >> On my 4 core, 1GHz, embedded ARM processor it takes 56 seconds and
 >> completely maxes out 1 core while it's running.
 >>
 >> Why might cause this?  How might I troubleshoot it?
 >





smime.p7s
Description: S/MIME Cryptographic Signature

Re: Completely blank flowgraph takes a minute to generate

[Jameson Collins]

@Johannes, that's neat.  I'll give that a shot.

@Marcus, I can't run GRC on the target as it doesn't have X.  I'm using
grcc to generate the block, I should have mentioned that.  When you say
IO-bound I'm guessing you mean because it's scanning the disk?  I haven't
benchmarked this hardware yet, but the filesystem is a ram disk so I would
expect it to be reasonably fast.  But this is something I can look into.

Thanks

On Wed, Sep 29, 2021 at 6:55 AM Marcus Müller  wrote:

> But before going into too much depth, make sure the duration isn't just
> basically identical to clicking on the "reload block library" button,
> which is first IO-bound (which *can* take significant amount of CPU time
> on weaker ARMs) and then parsing-limited.
>
> Best Regards,
>
> Marcus
>
> On 9/29/21 11:21 AM, Johannes Demel wrote:
> > Hi,
> >
> > I used:
> > https://docs.python.org/3/library/profile.html#module-cProfile
> > in the past to locate the problematic lines of code.
> >
> > ```
> > import cProfile
> > import pstats
> >
> > with cProfile.Profile() as pr:
> > run_the_problematic_function_etc()
> > stats = pstats.Stats(pr)
> > stats.sort_stats('cumtime').reverse_order()
> > stats.print_stats()
> > ```
> > You might want to adopt these lines for your use-case. I started at
> > the top-level of my application and then gradually moved the code to a
> > more fitting function.
> >
> > Cheers
> > Johannes
> >
> > On 28.09.21 23:40, Jameson Collins wrote:
> >> I have a completely blank flowgraph (well just the default 'Options'
> >> block).
> >>
> >> On my 8 core, 2 GHz, Intel based virtual machine this flowgraph
> >> generates in under a second.
> >>
> >> On my 4 core, 1GHz, embedded ARM processor it takes 56 seconds and
> >> completely maxes out 1 core while it's running.
> >>
> >> Why might cause this?  How might I troubleshoot it?
> >
>
>

Re: Completely blank flowgraph takes a minute to generate

[Marcus Müller]

But before going into too much depth, make sure the duration isn't just 
basically identical to clicking on the "reload block library" button, 
which is first IO-bound (which *can* take significant amount of CPU time 
on weaker ARMs) and then parsing-limited.


Best Regards,

Marcus

On 9/29/21 11:21 AM, Johannes Demel wrote:

Hi,

I used:
https://docs.python.org/3/library/profile.html#module-cProfile
in the past to locate the problematic lines of code.

```
import cProfile
import pstats

with cProfile.Profile() as pr:
    run_the_problematic_function_etc()
stats = pstats.Stats(pr)
stats.sort_stats('cumtime').reverse_order()
stats.print_stats()
```
You might want to adopt these lines for your use-case. I started at 
the top-level of my application and then gradually moved the code to a 
more fitting function.


Cheers
Johannes

On 28.09.21 23:40, Jameson Collins wrote:
I have a completely blank flowgraph (well just the default 'Options' 
block).


On my 8 core, 2 GHz, Intel based virtual machine this flowgraph 
generates in under a second.


On my 4 core, 1GHz, embedded ARM processor it takes 56 seconds and 
completely maxes out 1 core while it's running.


Why might cause this?  How might I troubleshoot it?

Re: Completely blank flowgraph takes a minute to generate

[Johannes Demel]

Hi,

I used:
https://docs.python.org/3/library/profile.html#module-cProfile
in the past to locate the problematic lines of code.

```
import cProfile
import pstats

with cProfile.Profile() as pr:
run_the_problematic_function_etc()
stats = pstats.Stats(pr)
stats.sort_stats('cumtime').reverse_order()
stats.print_stats()
```
You might want to adopt these lines for your use-case. I started at the 
top-level of my application and then gradually moved the code to a more 
fitting function.


Cheers
Johannes

On 28.09.21 23:40, Jameson Collins wrote:

I have a completely blank flowgraph (well just the default 'Options' block).

On my 8 core, 2 GHz, Intel based virtual machine this flowgraph 
generates in under a second.


On my 4 core, 1GHz, embedded ARM processor it takes 56 seconds and 
completely maxes out 1 core while it's running.


Why might cause this?  How might I troubleshoot it?

Completely blank flowgraph takes a minute to generate

[Jameson Collins]

I have a completely blank flowgraph (well just the default 'Options' block).

On my 8 core, 2 GHz, Intel based virtual machine this flowgraph generates
in under a second.

On my 4 core, 1GHz, embedded ARM processor it takes 56 seconds and
completely maxes out 1 core while it's running.

Why might cause this?  How might I troubleshoot it?