[Citadel Development] Re:SVN commit log: revision 5376

[IGnatius T Foobar]

 >so |foo|bar|bla   
 >   
 >gimme number 3    
 >   
 >will give you nada?   
   
 The tokenizer understands zero-length tokens.  So in the example you 
gave:  
   
 Token 0 is ""  
 Token 1 is "foo"  
 Token 2 is "bar"  
 Token 3 is "bla"  
   
 And if you supply it a string that doesn't contain the delimiter, it is 
considered to have a single token (token 0).  
   
 I guarantee that if you change any of this behavior, you will introduce 
hundreds of new bugs into Citadel.