A few comments: It shouldn't be necessary to call malloc or to copy the file name except in the special case where the first top level is found.
Don't store string lengths in 'int'. Use size_t, in case the string length exceeds INT_MAX. Tar already prevents users from misusing "..", so why does this code do that as well? How does it interact with existing ".." code? I don't understand why the code is worried about ".". The documentation makes it sound like if the archive contains names like "./foo1/x" and "./foo1/y", then that's an error, because the "." is duplicated. If that's intended, it should be stated more clearly. The documentation should state how the new option interacts with the --strip-components, --transform, and --absolute-file-names options. That is, do we insist on a single top level before or after these other options transform the file name? Also, please look at all the other options and see if there are other interactions that I've missed. The option name should start with --one, not with --single, by analogy with --one-file-system.