Today at 1:09am, Richard Miller said: >My mobile phone provider, Nextel, allows me to read text messages sent >to or from my phone at Nextel.com. I must login to read them. I would >like a script to login for me and scrape all the new messages, which it >could then provide in an RSS feed or save to a database, etc. > >Here are a few concerns that come to mind: > >- Should my script act like a normal user, "typing in" a name and >password to become authenticated and redirected to the Message Inbox, >or should I, the programmer, figure out which cookies or POST values >nextel.com is looking for and supply them?
Both ways can usually work, but it depends on their setup. The most reliable way to get it to work is to mimic as closely as possible what would happen if you were using a browser to do it, since that's what their site is expecting. But it isn't always easy... it means you have to parse their page, identify the form you need to "fill out", etc. If I were you, I'd probably try to figure out the cookies/POST method first, then try the other if that doesn't work. One thing that may throw a wrench in the works is that there might be some server-side stuff they do as part of the login that is triggered by you loading the login page or something like that, so skipping it wouldn't let you commplete a full login. >- Are there any methods or tools for helping me through the hassle of >discovering which cookies and POST values are passed between pages >while authenticating? > >- (Is there a browser or plugin that will allow me to see, for testing >purposes, which POST values have been sent to the server or are >currently in effect?) The answer to both of these questions is tcpdump, or something like it, like httpgrabber, tcpshow, etc. Even wget or curl can be helpful in this regard. They can let you watch the network traffic being sent back and forth, including cookies, post variables, etc, to see what interactions are taking place. For POST variables, load the page in a browser, and view the source, and look for the <form> and <input> tags to find out what they're expecting. For cookies, most browsers have a way to view cookies. Just find the ones for the nextel site, and see what they're set to. They may or may not be required for login. >- IN GENERAL, which tools should one use to produce a script that acts >like a normal user in a browser, "typing in" names and passwords, >submitting forms, working with cookies, etc.? I'd really like to >automate a lot of this process. You can use wget, and maybe curl, for a lot of this. IIRC, they have support for forms (GET and POST), cookies, SSL, etc., and can easily be made to do what you want. Other things you could possibly try are lynx, links, w3m, or other text based browsers. Mac -- Mac Newbold MNE - Mac Newbold Enterprises, LLC [EMAIL PROTECTED] http://www.macnewbold.com/ ____________________ BYU Unix Users Group http://uug.byu.edu/ ___________________________________________________________________ List Info: http://uug.byu.edu/cgi-bin/mailman/listinfo/uug-list
