It's a lot less obvious from a CPU designer perspective. One will make the speculative bits, declare 'all the actions I do are rolled back, so this is perfectly safe!' and someone else making the cache doesn't realize that the reads were speculative and their effects should have been rolled back.
People were talking about timing attacks for a while, but somehow it hadn't clicked that: array[*malicious_address & 1] Actually leaks (via timing) the content of the first bit of malicious_address. Or maybe it's just hadn't for me.