On second thought (and in the light of Geoffs reply) I probably won't. If you do care, the following change to the loop in question will at least preserve all input:
while((c = rchr(f))!=EOF){ if(strchr(stop, c)){ if(s!=wd){ *s='\0'; v = newword(wd, v); s = wd; } } else if(s==ewd){ *s='\0'; v = newword(wd, v); s = wd; *s++=c; } else *s++=c; } With a dynamic buffer the tokenisation could be prevented, but in your example the lexical scanner would quite likely bail afterwards. (I remember a discussion some time ago about this.)