Some file systems, like ZFS, are very slow at appending to existing
files. Due to Copy-On-Write nature, they create a new copy of a file
each time we do ">>" in a shell script. This becomes very noticeable
if shell script does lots and lots of appends, like sanitize_fragment()
function in kconf_check. On my setup, do_kernel_configcheck task takes
literally hours to complete.
To fix this issue, we can store sanitized_list and fragment_errors.txt
files on tmpfs, which is extremely fast at writing. As most distros
use tmpfs for /tmp, logical step is to use `mktemp` to create
temporary files.
After completing writing to temporary locations, we can move those two
files back to ${LOGDIR}.
Also, function 'cleanup' was added to remove temporary files in case
of abnormal exit.
With this patch, do_kernel_configcheck task completes in ~2 minutes on
my setup, which is a great improvement.
Signed-off-by: Volodymyr Babchuk <volodymyr_babc...@epam.com>
---
tools/kconf_check | 20 +++++++++++++++++++-
1 file changed, 19 insertions(+), 1 deletion(-)
diff --git a/tools/kconf_check b/tools/kconf_check
index 57d3132..5f25f17 100755
--- a/tools/kconf_check
+++ b/tools/kconf_check
@@ -195,6 +195,21 @@ function classify_options() {
fi
}
+# Temporary files to store sanitized list and fragment errors.
+# Create those files in /tmp because /tmp in most cases is stored in RAM
+# This greatly increases performance, as we will append data to those files
+# and some file systems (like ZFS) are very slow at appends.
+FRAGMENT_ERRORS_TMP=`mktemp`
+SANITIZED_LIST_TMP=`mktemp`
+
+# Clean temporary files from /tmp on exit
+function cleanup {
+ rm -f $FRAGMENT_ERRORS_TMP
+ rm -f $SANITIZED_LIST_TMP
+}
+
+trap cleanup EXIT
+
# step #1.
# run merge_config.sh (without regenerating the .config), and capture its
output.
# This gets us redefined options, and a combined (but unprocessed) config.
@@ -295,7 +310,7 @@ SED_CONFIG_EXP="s/^\(# \)\{0,1\}\(CONFIG_[a-zA-Z0-9_]*\)[=
].*/\2/p"
rm -f ${LOGDIR}/fragment_errors.txt
declare -A CONFIGMAP
for c in ${configs_abs}; do
- sanitize_fragment ${c} ${LOGDIR}/sanitized_list >>
${LOGDIR}/fragment_errors.txt
+ sanitize_fragment ${c} $SANITIZED_LIST_TMP >> $FRAGMENT_ERRORS_TMP
cfg_list=$(sed -n "$SED_CONFIG_EXP" ${c})
for option in ${cfg_list}; do
if [ -z "${CONFIGMAP[${option}]}" ]; then
@@ -310,6 +325,9 @@ for c in ${configs_abs}; do
fi
done
+mv $FRAGMENT_ERRORS_TMP ${LOGDIR}/fragment_errors.txt
+mv $SANITIZED_LIST_TMP ${LOGDIR}/sanitized_list
+
# If we had meta data, we can use the classifications to group the options.
if [ -n "${metadata}" ]; then
rm -f $LOGDIR/avail_hardware.cfg
--
2.38.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#11847):
https://lists.yoctoproject.org/g/linux-yocto/message/11847
Mute This Topic: https://lists.yoctoproject.org/mt/94817916/21656
Group Owner: linux-yocto+ow...@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/linux-yocto/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
