org-entities contains the line:("not" "\\textlnot{}" nil "¬" "[angled dash]" "¬" "¬") This should be ("neg" "\\neg" t "¬" "[angled dash]" "¬" "¬") The first line fails in math mode (i.e. "\not" is not replaced at all and negates/slashes the next symbol); the second works in either case.
OK, so org doesn't touch anything inside a math environment at all. Which makes sense in a way. The confusion arises with org-pretty-entities because what you see is definitely not what you get.
