If you think you're burning up stack unreasonably quickly, it might be a good idea to see if you can reduce the problem to relatively simple testcase and post a Bugzilla report so we can examine exactly what it's doing and determine whether something unreasonable is going on.
But it's possible that your combination of document and stylesheet really is recursing quite deeply. You may want to try adjusting the JVM's available stack space and see if that's sufficient to get you over the hump.
