Hello all
I don't want to contribute to the heated thread but, I was looking into
the performance Issues in IPS for quite some time and with help of
others. Here is a list of known factoids in no particular order.
- The OpenIndiana Repository metadata is between 4MB and 200MB in size.
In all my experiments with either Rust/Go/C++ simply reading those files
requires deep understanding of the languages streaming and memory saving
techniques. In Rust this is Zerocopy in C++ this is streaming parser, in
Go this is Just in time parsing. All of the Techniques basically try to
eliminate memory allocation. That is just for reading.
- Every attempt to merge these publisher created JSON files into the
Overall JSON file will result in massive execution times. This is also
true with the OmniOS IPS.
- Even though all three IPS have diverged a bit the OpenIndiana
performance problem is mainly on the JSON metadata front. Neither OmniOS
nor Solaris have to solve this particular problem. This issue with the
size and complexity of the JSON files for the usecase of a rolling
release Distrubution.
- Thus far the only way I could come up with based on my knowledge is to
introduce code into the merging step that skips all "userland
consolidation" packages (or maybe a configurable set of packages).
However that would need to be done on the client and every once in while
the old consolidations also need to be deleted from the main JSON again
if they are not needed.
- Zypper dnf and other circumvent this problem by using Random access
Databases as backing store for metadata. SQLite in most cases but any
embedded key-value or NoSQL store will do. LLMDB, RocksDB or SQLite come
to mind here. If we replace the main merged JSON with that i would
expect drastic impreovments for search and other features that alsways
need to skip the 3k entries for userland consolidation.
- SAT solver performence is the same over all three platforms. One can
see this by developing for ARM which uses the Host IPS to generate an
ARM ZFS pool. This is equally fast when using an OmniOS host as when
using OpenIndiana.
- The amount of userland consolidation packages is based on the Build
process of OpenIndiana and unfortunately not changeable as it turned out
in the last months.
Hope this sheds some clarity on where the main problem lies. As
mentioned I have some ideas how to address this but no initiative was
successful thus far. But the problem is understood. It is this JSON
based metadata and merging process. (Or any file that does not easily
allow random access)
-Till
_______________________________________________
openindiana-discuss mailing list
openindiana-discuss@openindiana.org
https://openindiana.org/mailman/listinfo/openindiana-discuss
