TY wrote: > Hi, > > Can someone point me to a Howto or Tutorial for tokenize module for a > newbie? I read the documentation but it doesn't have a lot of info... > Thanks! > Hi there
I don't know if I got your requirement. But I used a own version of tokenizer which can use more than one delimiters. It is below. Hopefully of some use to you. def gettokens(dat): delims={'=':'',';':',','=':'','(':'',')':'',':':'','[':'',']':''} wlist=[] appended=0 for n in string.split(dat,'\n'): word='' for m in n: appended=0 if delims.has_key(m): wlist.append(word) wlist.append(m) word='' appended=1 else: word=word+m if appended==0: wlist.append(word) appended=1 wlist.append("\n") return wlist I am sure there are plenty of ways to write it better than that. Good luck sree -- http://mail.python.org/mailman/listinfo/python-list