I have recreated self-signed certificate with propper CN inside VisualSVN (by default it was computer hostname), and now --non-interactive --trust-server-cert is working as expected (accepts without prompting or error).
Self-signed is also untrusted as one with non matching hostname, but maybe it's SVN policy to ignore "--trust-server-cert" for hostname missmatch specificlally as for "more dangerous" case?