On Wed, Jan 16, 2019 at 3:28 PM Richard Purdie <richard.pur...@linuxfoundation.org> wrote: > The data is in the codeparser cache which is first populated at parse > time so its enough just to parse the machine+recipe in question, not > build it. I think that explains the answer to a ew of your questions > above.
Yes, thanks. > Sorry for asking so many questions btw, I'd just really love to be able > to reproduce this issue! Thanks for trying to help answer them too! > > Is the bitbake-cookerdeamon.log file still there for this build (in the > top level build directory)? I don't seem to have this file in any of my OpenXT builds. I still have the "bad" bb_codeparser.dat file. It is 30MB whereas the new one is only 6.5MB. I thought it may be excessively large, but I actually have an 80MB one in a different build directory. Anyway, it has 4 different entries that look like core2-32 python-async do_install()-s 3c6fe664c51d2f793f8fd0eb103d68cb - reproduces currently 3df9018676de219bb3e46e88eea09c98 - one matching binutils core2-64 do_install 382871fb17743ba9635d7efc4db7d993 ee6850bdcf70ba63dea37e09c78c599f They all have frozenset({'[', 'mv', 'test', '/home/build/openxt-compartments/build/tmp-glibc/work/core2-32-oe-linux/python-async/0.6.2-r0/recipe-sysroot-native/usr/bin/python-native/python', 'sed', 'install', 'bbfatal_log', 'find', 'rm', 'rmdir'}) Eyeballing distutils_do_install, I don't see what could produce so many variations. Going into the new, clean build container, I can see those last two hashes with different entries: >>> d['ee6850bdcf70ba63dea37e09c78c599f'] frozenset({'tr', 'rm', 'sed', 'ln', 'cd', 'oe_multilib_header', 'autotools_do_install', 'echo', 'basename', 'install'}) >>> d['382871fb17743ba9635d7efc4db7d993'] frozenset({'tr', 'rm', 'sed', 'ln', 'cd', 'oe_multilib_header', 'autotools_do_install', 'echo', 'basename', 'install'}) and the expected core2-32 python-async do_install >>> d['3c6fe664c51d2f793f8fd0eb103d68cb'] frozenset({'bbfatal_log', 'rm', 'test', 'sed', '[', 'rmdir', '/home/build/openxt-compartments/build/tmp-glibc/work/core2-32-oe-linux/python-async/0.6.2-r0/recipe-sysroot-native/usr/bin/python-native/python', 'find', 'install', 'mv'}) I've only run one core2-32 build in the fresh container, so no 64bit binutils at the original collision 3df9018676de219bb3e46e88eea09c98 Ok, I hacked up a script to check two bb_codeparser.dat files for collisions. Compare the current one with the "bad" one: $ ./pickle-cmp.py cache/bb_codeparser.dat cache/bb_codeparser.dat.old-bad-one Collision ee6850bdcf70ba63dea37e09c78c599f frozenset({'echo', 'rm', 'autotools_do_install', 'tr', 'oe_multilib_header', 'cd', 'basename', 'sed', 'ln', 'install'}) frozenset({'find', 'test', 'rm', 'bbfatal_log', '[', 'sed', 'mv', '/home/build/openxt-compartments/build/tmp-glibc/work/core2-32-oe-linux/python-async/0.6.2-r0/recipe-sysroot-native/usr/bin/python-native/python', 'rmdir', 'install'}) Collision 382871fb17743ba9635d7efc4db7d993 frozenset({'echo', 'rm', 'autotools_do_install', 'tr', 'oe_multilib_header', 'cd', 'basename', 'sed', 'ln', 'install'}) frozenset({'find', 'test', 'rm', 'bbfatal_log', '[', 'sed', 'mv', '/home/build/openxt-compartments/build/tmp-glibc/work/core2-32-oe-linux/python-async/0.6.2-r0/recipe-sysroot-native/usr/bin/python-native/python', 'rmdir', 'install'}) Collision 5254083eac08e32fc68bc9421d7df287 frozenset({'autotools_do_install', 'rm', 'sed', 'touch', 'install'}) frozenset({'/etc/init.d/xenclient-boot-sound', 'true', ':', '['}) Collision d0701fd5c05175aeafc06d8ce34d3532 frozenset({'create-cracklib-dict', 'autotools_do_install'}) frozenset({'/etc/init.d/gateone', 'true', ':', '['}) Collision ec332415bd96520823ba383494e7a9a7 frozenset({'ln', 'popd', ':', 'pushd'}) frozenset({'DEPLOY_DIR', 'useradd_preinst', 'perform_useradd', 'PKGD', 'PKGDEST', 'pkg_preinst', 'MLPREFIX', 'perform_groupadd', 'PN', 'perform_groupmems', 'PACKAGES', 'NOAUTOPACKAGEDEBUG', 'USERADD_PACKAGES', 'WORKDIR'}) Collision 3df9018676de219bb3e46e88eea09c98 frozenset({'echo', 'rm', 'autotools_do_install', 'tr', 'oe_multilib_header', 'cd', 'basename', 'sed', 'ln', 'install'}) frozenset({'find', 'test', 'rm', 'bbfatal_log', '[', 'sed', 'mv', '/home/build/openxt-compartments/build/tmp-glibc/work/core2-32-oe-linux/python-async/0.6.2-r0/recipe-sysroot-native/usr/bin/python-native/python', 'rmdir', 'install'}) Collision 0aa15eb469ad8854cda0b0675217b8f6 frozenset({'find', 'test', 'rm', 'bbfatal_log', '[', 'sed', 'mv', '/home/build/openxt-compartments/build/tmp-glibc/work/core2-32-oe-linux/python-mock/2.0.0-r0/recipe-sysroot-native/usr/bin/python-native/python', 'rmdir', 'install'}) frozenset({'oe_runmake', 'find', 'true', 'test', 'echo', 'chmod', 'rm', 'mkdir', '[', 'oe_multilib_header', 'cd', 'lnr', 'basename', 'continue', 'mv', 'ln', 'local', 'install'}) Compare the current one with the fresh one from the other container (build4): $ ./pickle-cmp.py cache/bb_codeparser.dat build4-codeparser.dat Collision d0701fd5c05175aeafc06d8ce34d3532 frozenset({'create-cracklib-dict', 'autotools_do_install'}) frozenset({'[', ':', '/etc/init.d/gateone', 'true'}) Collision 5254083eac08e32fc68bc9421d7df287 frozenset({'touch', 'install', 'sed', 'autotools_do_install', 'rm'}) frozenset({'[', '/etc/init.d/xenclient-boot-sound', ':', 'true'}) I figured I can reproduce the hash collisions for d0701fd5c05175aeafc06d8ce34d3532, but I cannot. gateone update-rc.d updatercd_prerm matches d0701fd5c05175aeafc06d8ce34d3532 Below, it should be a tab before /etc/init.d/gateone , but I can't insert that because of gmail) ''' # Begin section update-rc.d if true && [ -z "$D" -a -x "/etc/init.d/gateone" ]; then /etc/init.d/gateone stop || : fi # End section update-rc.d ''' cracklib do_install is only two lines, but I cannot reproduce. (Below, it should be a tab before create-cracklib-dict) $ sed -n '/^do_install/,/^}/p' tmp-glibc/work/core2-32-oe-linux/cracklib/2.9.5-r0/temp/run.do_install | head -n -7 | tail -n 2 | tee /dev/stderr | md5sum autotools_do_install create-cracklib-dict -o /home/build/openxt-compartments/build/tmp-glibc/work/core2-32-oe-linux/cracklib/2.9.5-r0/image/usr/share/cracklib/pw_dict /home/build/openxt-compartments/build/tmp-glibc/work/core2-32-oe-linux/cracklib/2.9.5-r0/image/usr/share/cracklib/cracklib-small 8ca3daacb394a11f38c851a8d71ed4de - #current $ ./i-m-in-a-pickle.py cache/bb_codeparser.dat cracklib 3933316 '8ca3daacb394a11f38c851a8d71ed4de': frozenset({'create-cracklib-dict', 'autotools_do_install'}) 5050048 'b50633e8f154817d7cc3ec94c6405379': frozenset({'create-cracklib-dict', 'autotools_do_install'}) 5641024 'd0701fd5c05175aeafc06d8ce34d3532': frozenset({'create-cracklib-dict', 'autotools_do_install'}) #old bad $ ./i-m-in-a-pickle.py cache/bb_codeparser.dat.old cracklib 13503349 '8ca3daacb394a11f38c851a8d71ed4de': frozenset({'autotools_do_install', 'create-cracklib-dict'}) 39348111 'b50633e8f154817d7cc3ec94c6405379': frozenset({'autotools_do_install', 'create-cracklib-dict'}) #fresh $ ./i-m-in-a-pickle.py build4-codeparser.dat cracklib 2085662 '8ca3daacb394a11f38c851a8d71ed4de': frozenset({'create-cracklib-dict', 'autotools_do_install'}), b50633e8f154817d7cc3ec94c6405379 is the core2-64 cracklib do_install $ sed -n '/^do_install/,/^}/p' tmp-glibc/work/core2-64-oe-linux/cracklib/2.9.5-r0/temp/run.do_install | head -n -7 | tail -n 2 | tee /dev/stderr | md5sum autotools_do_install create-cracklib-dict -o /home/build/openxt-compartments/build/tmp-glibc/work/core2-64-oe-linux/cracklib/2.9.5-r0/image/usr/share/cracklib/pw_dict /home/build/openxt-compartments/build/tmp-glibc/work/core2-64-oe-linux/cracklib/2.9.5-r0/image/usr/share/cracklib/cracklib-small b50633e8f154817d7cc3ec94c6405379 - But where did d0701fd5c05175aeafc06d8ce34d3532 come from? When pickling the cache for writing to disk, could a dangling pointer/index/reference be left such that a given hash's frozenset entry changes? An aside - entries include the standard linux utilities & shell builtins, but those are always available. If a given shell cache entry only relies on those utilities, it could collide and still run. It's only an issue when a shell function or special utility is needed. Regards, Jason -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core