Phyllis et al,
While I have not tested the current state of the git repository, I thought I might add a few comments about testing performance in general from the perspective of a retired systems programmer. I am a fan of Cinelerra-GG and use it for burning bluray disks so I hope to be able to test the new code before long.
I realize that the posts to which I am referring are not intended as exhaustive, definitive test suites, but it can be useful to know how the results can be affected. On most operating systems, memory requested by an application is not returned to the system wide free pool when the application releases it with a call to free or delete. You can see this by running a memory intensive application in one window and the gnome-system-monitor (or equivalent) in another window. Unless there is a great deal of demand for free memory by other applications, this memory will remain assigned to the first application until that application is closed. Therefore, each test should be run from a fresh invocation of the test application. Unfortunately, this doesn't guarantee that the file system cache is flushed.
Any settings within a given application, eg cache or pre-roll, will need to be measured in a controlled environment which means rebooting the computer to clear the file system file cache. Otherwise, any request for data that has been previously read may be satisfied from the operating systems file cache regardless of the settings in the application. That said, the effect of the file system cache loading will be most or exclusively felt in short tests that fit completely or mostly within the file system cache. The tests should be run with as few other applications loaded as possible to minimize CPU contention and disk access. If there is a way to run the test from the command line, this will eliminate additional video card specific rendering issues, by which I mean presenting the images on the screen. I am not referring to using GPU's used to render the video to the files on disk that will be read during playback. Of course, if what you are trying to measure is the onscreen presentation rate, then a command line test is not appropriate.
The terminal commands free and top can be used to see how much memory is being allocated to caches. If you run these before and after a test, you should see an increase in the memory allocated to the file caches.
You can use the command: cat /proc/meminfo to see how much memory is being allocated to file system caches. Note: these values are for an older laptop with only 8 GB of physical ram and an 8 GB physical swap partition.
cat /proc/meminfo
MemTotal: 7802112 kB
MemFree: 2199052 kB
MemAvailable: 5405988 kB
Buffers: 175436 kB
Cached: 3361856 kB
SwapCached: 0 kB
Active: 1470940 kB
Inactive: 3556264 kB
Active(anon): 4204 kB
Inactive(anon): 1616348 kB
Active(file): 1466736 kB
Inactive(file): 1939916 kB
swapon
NAME TYPE SIZE USED PRIO
/dev/sda3 partition 7.9G 0B -2
/dev/zram0 partition 7.4G 0B 100
See the URL below for a description of the zram0 swap partition.
https://www.kernel.org/doc/html/v5.3/admin-guide/blockdev/zram.html
It goes without saying that any use of swap space will radically degrade performance unless that swap space is on a solid state drive and the swap partition is quite generous in size. Recent versions of Fedora, ie 33 and newer, seem to favor a compressed swap space and/or using memory compression, which can have a large impact on memory intensive operations. My recommendation is to use a dedicated, on disk (SSD) swap partition that is twice the size of physical memory or larger depending on the memory requirements of your application. I use the Fotoxx graphics application to make very large panoramas and have frequently used up to 40 GB of swap space on a system with 16 GB of physical ram. Cinerlerra is nowhere near that memory hungry, but it is essential to avoid running into situations where the kernel will have to evict code pages or data from the cache. The output of the swapon command indicates that the compressed zram0 block device has a much higher priority than the physical disk based swap partition. This can negatively impact file caching if your on disk swap partition is fast and your memory is older, slower memory. It may be worth trying to limit the size of your compressed swap device as described in the URL above to see if that helps or hurts performance. If the kernel is trying to compress already compressed data, it may make more sense to force it to be written to the SSD. This can only be determined by testing.
A very good tool to analyze memory usage during any given run is heaptrack by Milan Wolf
https://milianw.de/blog/heaptrack-a-heap-memory-profiler-for-linux.html
It is available for most Linux distributions and works with heaptrack_gui to display memory leaks and memory usage at each point in an application run. If your distribution channel does not have a current package for heaptrack, you can download the source and compile it fairly easily.
Richard Nolde
While I have not tested the current state of the git repository, I thought I might add a few comments about testing performance in general from the perspective of a retired systems programmer. I am a fan of Cinelerra-GG and use it for burning bluray disks so I hope to be able to test the new code before long.
I realize that the posts to which I am referring are not intended as exhaustive, definitive test suites, but it can be useful to know how the results can be affected. On most operating systems, memory requested by an application is not returned to the system wide free pool when the application releases it with a call to free or delete. You can see this by running a memory intensive application in one window and the gnome-system-monitor (or equivalent) in another window. Unless there is a great deal of demand for free memory by other applications, this memory will remain assigned to the first application until that application is closed. Therefore, each test should be run from a fresh invocation of the test application. Unfortunately, this doesn't guarantee that the file system cache is flushed.
Any settings within a given application, eg cache or pre-roll, will need to be measured in a controlled environment which means rebooting the computer to clear the file system file cache. Otherwise, any request for data that has been previously read may be satisfied from the operating systems file cache regardless of the settings in the application. That said, the effect of the file system cache loading will be most or exclusively felt in short tests that fit completely or mostly within the file system cache. The tests should be run with as few other applications loaded as possible to minimize CPU contention and disk access. If there is a way to run the test from the command line, this will eliminate additional video card specific rendering issues, by which I mean presenting the images on the screen. I am not referring to using GPU's used to render the video to the files on disk that will be read during playback. Of course, if what you are trying to measure is the onscreen presentation rate, then a command line test is not appropriate.
The terminal commands free and top can be used to see how much memory is being allocated to caches. If you run these before and after a test, you should see an increase in the memory allocated to the file caches.
You can use the command: cat /proc/meminfo to see how much memory is being allocated to file system caches. Note: these values are for an older laptop with only 8 GB of physical ram and an 8 GB physical swap partition.
cat /proc/meminfo
MemTotal: 7802112 kB
MemFree: 2199052 kB
MemAvailable: 5405988 kB
Buffers: 175436 kB
Cached: 3361856 kB
SwapCached: 0 kB
Active: 1470940 kB
Inactive: 3556264 kB
Active(anon): 4204 kB
Inactive(anon): 1616348 kB
Active(file): 1466736 kB
Inactive(file): 1939916 kB
swapon
NAME TYPE SIZE USED PRIO
/dev/sda3 partition 7.9G 0B -2
/dev/zram0 partition 7.4G 0B 100
See the URL below for a description of the zram0 swap partition.
https://www.kernel.org/doc/html/v5.3/admin-guide/blockdev/zram.html
It goes without saying that any use of swap space will radically degrade performance unless that swap space is on a solid state drive and the swap partition is quite generous in size. Recent versions of Fedora, ie 33 and newer, seem to favor a compressed swap space and/or using memory compression, which can have a large impact on memory intensive operations. My recommendation is to use a dedicated, on disk (SSD) swap partition that is twice the size of physical memory or larger depending on the memory requirements of your application. I use the Fotoxx graphics application to make very large panoramas and have frequently used up to 40 GB of swap space on a system with 16 GB of physical ram. Cinerlerra is nowhere near that memory hungry, but it is essential to avoid running into situations where the kernel will have to evict code pages or data from the cache. The output of the swapon command indicates that the compressed zram0 block device has a much higher priority than the physical disk based swap partition. This can negatively impact file caching if your on disk swap partition is fast and your memory is older, slower memory. It may be worth trying to limit the size of your compressed swap device as described in the URL above to see if that helps or hurts performance. If the kernel is trying to compress already compressed data, it may make more sense to force it to be written to the SSD. This can only be determined by testing.
A very good tool to analyze memory usage during any given run is heaptrack by Milan Wolf
https://milianw.de/blog/heaptrack-a-heap-memory-profiler-for-linux.html
It is available for most Linux distributions and works with heaptrack_gui to display memory leaks and memory usage at each point in an application run. If your distribution channel does not have a current package for heaptrack, you can download the source and compile it fairly easily.
Richard Nolde
-- Cin mailing list Cin@lists.cinelerra-gg.org https://lists.cinelerra-gg.org/mailman/listinfo/cin
