On 22.02.22 16:50, Ralph Siemsen wrote:
Hi Konrad,
On Fri, Jan 7, 2022 at 4:59 AM Konrad Weihmann <kweihm...@outlook.com
<mailto:kweihm...@outlook.com>> wrote:
On 07.01.22 10:48, Konrad Weihmann wrote:
> this should prevent running into the very rare error
> sqlite3.OperationalError: attempt to write a readonly database
It's also possible that check_same_thread (that defaults to True ) in
the sqlite3.connect causes this (see
https://docs.python.org/3/library/sqlite3.html#sqlite3.connect
<https://docs.python.org/3/library/sqlite3.html#sqlite3.connect>) -
I have
too little experience with sqlite to provide any further input.
I'd be happy to get some feedback from users that do have that
experience with sqlite in a highly multithreaded/multiprocess
environment. Or by people that use cvecheck class in larger setups than
just core.
Afraid I am not well versed in the internals of sqlite, so I cannot
comment on that part.
However, I do run cve-check on a larger setup than core, and after this
commit recently landed in dunfell branch, I noticed that do_cve_check
step is taking considerably longer than it used to. I have not
benchmarked it carefully, but visually from watching the console during
the build, it seems to be somewhere from 2 to 5 times slower. This is
running a native build on an ext4 filesystem (not using containers etc).
This is somehow expected from my side tbh - as the lock file disables
any kind of parallelism - so just one instance of cve-check-task can run
at a time.
One thing that came to my mind is to move the created CVE_CHECK-sqlite
db into the recipe-sysroot before running the task (currently it resides
in DL_DIR), so each task would run on its own copy of the db - that
would avoid having multiple access call to the same file while enabling
us to remove the arbitrary task lock.
Are others seeing similar behaviour?
While I am writing, I should also mention that the CVE JSON feeds are
apparently going to be changing (again).
https://www.cve.org/Media/News/item/news/2022/01/11/Changes-Coming-to-CVE-Record
<https://www.cve.org/Media/News/item/news/2022/01/11/Changes-Coming-to-CVE-Record>
I wonder if we should perhaps move instead to using the git repo:
https://github.com/CVEProject/cvelist
<https://github.com/CVEProject/cvelist> (existing v4 format)
https://github.com/CVEProject/cvelistV5
<https://github.com/CVEProject/cvelistV5> (preview of upcoming v5 format)
This would not solve the sqlite locking issue, though perhaps we could
sidestep it and just search the json directly?
I guess I should really start a separate thread about this ;-)
Regards,
Ralph
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#162179):
https://lists.openembedded.org/g/openembedded-core/message/162179
Mute This Topic: https://lists.openembedded.org/mt/88258164/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
