Ok, I figured it out. Hadoop Streaming adds the entire stream.shipped.hadoopstreaming directory to the jar file. For me, I wasn't setting it and it was defaulting to /tmp. That means my entire /tmp directory was getting added to the jar.
I set that directory to the location of my hadoop streaming jar directory and it seemed to work fine. Sorry for the noise.