Re: [libbsd 00/22] Remove FreeBSD file descriptors and avoid VFS

2022-10-21 Thread Sebastian Huber 

On 21/10/2022 16:07, Joel Sherrill wrote:
On Thu, Oct 20, 2022 at 7:30 PM Chris Johns > wrote:


On 20/10/2022 7:01 pm, Sebastian Huber wrote:
 > On 29/08/2022 10:27, Sebastian Huber wrote:
 >> Hello Chris,
 >>
 >> On 25/07/2022 08:12, Sebastian Huber wrote:
 >>> On 11/07/2022 15:04, Sebastian Huber wrote:
  On 24/06/2022 08:33, Sebastian Huber wrote:
 > This patch set removes the FreeBSD file descriptors.  The VFS
is no longer
 > used
 > if only the USB, SD/MMC, network, PCI, and NVMe support is
used by the
 > application.  This change significantly reduce the memory
usage of LibBSD for
 > these applications.  Using the media01 test case for the
arm/lpc32xx BSP as a
 > benchmark, the heap usage dropped from 14.3MiB to 10.2MiB. 
The "_BSD

 > bufdaemon", "_BSD vnlru", "_BSD syncer", and "_BSD
bufspacedaemon-" tasks are
 > no longer present in media01.  The code size is reduced by
about 8KiB.  The
 > data size is reduced by about 30KiB.  The throughput with a
simple FTP test
 > increased by about 1%.


Only the decrease in heap size is significant enough to discuss. The 
others are noise

when considered in light of the source transparency we aimed for.

https://freebsdfoundation.org/wp-content/uploads/2016/08/FreeBSD-and-RTEMS-Unix-in-a-Real-Time-Operating-System.pdf
 


This guideline which is intended to increase compatibility and lower 
long-term

maintenance is at odds with this set of patches.


It would be really nice if you could review the patch set in detail 
before you write something like this. Did you ever update a FreeBSD 
baseline in one of the libbsd branches? It is also a nice exercise to 
step through for example sendto() from the file descriptor until you get 
the socket object.




That leaves us with the 4MB of heap. That alone is your problem. What is
allocating it? Is there a hint or sysctl value that can turn it down? Is it
representative of a single feature that could just be disabled in your
low memory situation [1].

[1] I agree with Chris' statements that there are no requirements for
a set of functionality to fit within a certain amount of memory. And
if you are this close to the edge, almost anything that happens will
break it again.


With a clean constructor based initialization you can avoid a lot of 
resource issues.


--
embedded brains GmbH
Herr Sebastian HUBER
Dornierstr. 4
82178 Puchheim
Germany
email: sebastian.hu...@embedded-brains.de
phone: +49-89-18 94 741 - 16
fax:   +49-89-18 94 741 - 08

Registergericht: Amtsgericht München
Registernummer: HRB 157899
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier:
https://embedded-brains.de/datenschutzerklaerung/
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [libbsd 00/22] Remove FreeBSD file descriptors and avoid VFS

2022-10-21 Thread Joel Sherrill 
On Thu, Oct 20, 2022 at 7:30 PM Chris Johns  wrote:

> On 20/10/2022 7:01 pm, Sebastian Huber wrote:
> > On 29/08/2022 10:27, Sebastian Huber wrote:
> >> Hello Chris,
> >>
> >> On 25/07/2022 08:12, Sebastian Huber wrote:
> >>> On 11/07/2022 15:04, Sebastian Huber wrote:
>  On 24/06/2022 08:33, Sebastian Huber wrote:
> > This patch set removes the FreeBSD file descriptors.  The VFS is no
> longer
> > used
> > if only the USB, SD/MMC, network, PCI, and NVMe support is used by
> the
> > application.  This change significantly reduce the memory usage of
> LibBSD for
> > these applications.  Using the media01 test case for the arm/lpc32xx
> BSP as a
> > benchmark, the heap usage dropped from 14.3MiB to 10.2MiB.  The "_BSD
> > bufdaemon", "_BSD vnlru", "_BSD syncer", and "_BSD bufspacedaemon-"
> tasks are
> > no longer present in media01.  The code size is reduced by about
> 8KiB.  The
> > data size is reduced by about 30KiB.  The throughput with a simple
> FTP test
> > increased by about 1%.
>

Only the decrease in heap size is significant enough to discuss. The others
are noise
when considered in light of the source transparency we aimed for.

https://freebsdfoundation.org/wp-content/uploads/2016/08/FreeBSD-and-RTEMS-Unix-in-a-Real-Time-Operating-System.pdf

This guideline which is intended to increase compatibility and lower
long-term
maintenance is at odds with this set of patches.

That leaves us with the 4MB of heap. That alone is your problem. What is
allocating it? Is there a hint or sysctl value that can turn it down? Is it
representative of a single feature that could just be disabled in your
low memory situation [1].

[1] I agree with Chris' statements that there are no requirements for
a set of functionality to fit within a certain amount of memory. And
if you are this close to the edge, almost anything that happens will
break it again.


> >
> > The "Remove FreeBSD file descriptors" change removes more lines than
> there are
> > added.
> >
> > This change makes it easier to port the NFS support to the master
> branch since
> > now the changes are more localized.
> 
>  I have a target with only 8MiB of RAM (for code and data). So, this
> patch
>  set is not just a micro optimization.
> >>>
> >>> This pending patch set is currently a blocker for me. I would like to
> work on
> >>> the NTP daemon integration and an update of the libbsd master branch
> to a
> >>> more recent FreeBSD version. This patch set just restores what worked
> well
> >>> for several years.
> >>
> >> did you have time to review this patch set in the meantime? The NTP
> client no
> >> longer crashes with this patch set.
> >
> > This patch set is now pending for about four months. From my point of
> view it is
> > a straight forward fix of some severe issues which prevent the use of
> libbsd on
> > lower end targets.
>
> Is there a repo somewhere with the patches applied? I am being lazy asking
> as I
> would like to avoid downloading the 22 patches and applying them by hand.
>
> It seems the media test cannot grow by more than 9K of code and the heap
> by 5M!
> Is my understanding correct? Anything else we need to understand about the
> low
> end limits?
>
> I have not seen any requirements on the lower end for libbsd before now.
> How do
> we manage the competing requirement of low end support and adding
> functionality
> to libbsd in the future if they clash?
>
> Adding VFS support as FreeBSD implements it has pushed you over the edge
> but
> that edge is not defined so it makes it difficult for anyone other than
> you to
> work on this code base in a major way. I was not aware this was an issue
> and for
> me the extra 9K of code is not a problem and a 5M change in heap is
> acceptable.
>

Again that increased heap usage is clearly a memory allocation and there is
code
which is doing it. Can it be disabled and we avoid this entire set of
invasive patches?


>
> > It is not really good if our internal libbsd branches diverge
> > too much from the RTEMS mainline.
>
> Yes I agree. It is not in anyone's interest and that only benefits you.
>
> I would like to reiterate we have an obligation to maintain the code base
> as
> close to FreeBSD as possible. Some of these changes push that boundary. We
> need
> to find a suitable path forward.
>
> Reviewing the changes:
>
> 1. The break up of the syscalls file. The related comments are:
>
>  "Collecting all system calls in a single translation unit is not good
>   due to the library initialization through linker sets."
>
> Could you please explain what the linker set issue is?
>
> Why the need to change FreeBSD calls to static? Is it solely a performance
> optimisation?
>
> We are required to use function and data sections so I assume the syscall
> file
> will be broken up by the linker where possible?
>
> I cannot tell from the patches if the conditional syscall trace support is
> still
> present?
>
> I do

Re: [libbsd 00/22] Remove FreeBSD file descriptors and avoid VFS

2022-10-21 Thread Sebastian Huber 

On 21/10/2022 02:30, Chris Johns wrote:

On 20/10/2022 7:01 pm, Sebastian Huber wrote:

On 29/08/2022 10:27, Sebastian Huber wrote:

Hello Chris,

On 25/07/2022 08:12, Sebastian Huber wrote:

On 11/07/2022 15:04, Sebastian Huber wrote:

On 24/06/2022 08:33, Sebastian Huber wrote:

This patch set removes the FreeBSD file descriptors.  The VFS is no longer
used
if only the USB, SD/MMC, network, PCI, and NVMe support is used by the
application.  This change significantly reduce the memory usage of LibBSD for
these applications.  Using the media01 test case for the arm/lpc32xx BSP as a
benchmark, the heap usage dropped from 14.3MiB to 10.2MiB.  The "_BSD
bufdaemon", "_BSD vnlru", "_BSD syncer", and "_BSD bufspacedaemon-" tasks are
no longer present in media01.  The code size is reduced by about 8KiB.  The
data size is reduced by about 30KiB.  The throughput with a simple FTP test
increased by about 1%.

The "Remove FreeBSD file descriptors" change removes more lines than there are
added.

This change makes it easier to port the NFS support to the master branch since
now the changes are more localized.


I have a target with only 8MiB of RAM (for code and data). So, this patch
set is not just a micro optimization.


This pending patch set is currently a blocker for me. I would like to work on
the NTP daemon integration and an update of the libbsd master branch to a
more recent FreeBSD version. This patch set just restores what worked well
for several years.


did you have time to review this patch set in the meantime? The NTP client no
longer crashes with this patch set.


This patch set is now pending for about four months. From my point of view it is
a straight forward fix of some severe issues which prevent the use of libbsd on
lower end targets.


Is there a repo somewhere with the patches applied? I am being lazy asking as I
would like to avoid downloading the 22 patches and applying them by hand.


Thanks for having a look at this work. You find a branch for testing here:

https://github.com/sebhub/rtems-libbsd/tree/filedesc

You need this RTEMS patch:

https://lists.rtems.org/pipermail/devel/2022-June/072055.html



It seems the media test cannot grow by more than 9K of code and the heap by 5M!
Is my understanding correct? Anything else we need to understand about the low
end limits?


If you have only 16M or 32M of RAM for code and data than adding 5M for 
nothing is a big deal. The libbsd started as a libusb. Not every 
application uses the network stuff.




I have not seen any requirements on the lower end for libbsd before now. How do
we manage the competing requirement of low end support and adding functionality
to libbsd in the future if they clash?

Adding VFS support as FreeBSD implements it has pushed you over the edge but
that edge is not defined so it makes it difficult for anyone other than you to
work on this code base in a major way. I was not aware this was an issue and for
me the extra 9K of code is not a problem and a 5M change in heap is acceptable.


The problem is not the VFS support. The problem is that you added the 
FreeBSD file descriptors and concentrated all the system calls in one 
translation unit.





It is not really good if our internal libbsd branches diverge
too much from the RTEMS mainline.


Yes I agree. It is not in anyone's interest and that only benefits you.

I would like to reiterate we have an obligation to maintain the code base as
close to FreeBSD as possible. Some of these changes push that boundary. We need
to find a suitable path forward.


Being as close as possible to FreeBSD should not result in being 
FreeBSD. RTEMS has its own infrastructure and one part is the file 
descriptor handling. There are always trade-offs to consider when 
porting new components from FreeBSD to RTEMS.




Reviewing the changes:

1. The break up of the syscalls file. The related comments are:

  "Collecting all system calls in a single translation unit is not good
   due to the library initialization through linker sets."

Could you please explain what the linker set issue is?


The libbsd uses a constructor based initialization. The constructors are 
pulled in through the configuration and indirectly by using API level 
functions. If you concentrate the API level function in one file, then 
everything related to all of the API level functions is pulled in and 
initialized. Therefore, the API level functions need to be placed in 
separate translation units so that you get only what you need in your 
executable. For example, if you just want to use USB and sysctl(), then 
you should not end up with stuff related to socket().




Why the need to change FreeBSD calls to static? Is it solely a performance
optimisation?


It is also easier to maintain if you have the RTEMS glue code attached 
to the FreeBSD system call implementation (sys_* and kern_*). For 
example, in uipc_syscalls.c the glue code was placed in white space 
between functions. Such code areas are absolutely no

Re: [PATCH rtems-docs v2] user/bsps: Update riscv for PolarFire SoC

2022-10-21 Thread Chris Johns 
On 21/10/2022 4:04 pm, padmarao.beg...@microchip.com wrote:
> The Patch v1 was reviewed by Alan, update few comments, and submitted Patch 
> v2.
>  https://lists.rtems.org/pipermail/devel/2022-October/073469.html - Patch v1
> 
> https://lists.rtems.org/pipermail/devel/2022-October/073528.html - Patch v2
> 
> Can you please review or merge it to rtems-docs git?

Sorry about the delay. I have merged the patch based on Alan's review.

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel