Re: [Tutor] Retrieve data from log file
Abe Miranda wrote: Hi there, I'm sorry to ask this, I tried and example that was posted in 2009 but I couldn't make it work. I'm trying to parse a log file and retrieve what I just need. Hope you can help me with this, this is part of the log file: It repeats itself for every transaction and I need to extract from this lines: [2011-10-12 21:02:43:383] DEBUG [BaseQualitasDaoWS: 86] [http-80-1][clientThread-Thread-3] -wscall - hours:0.0 minutes:0.0 seconds:3.0 [2011-10-12 21:02:43:462] DEBUG [RestController : 81] [http-80-1][clientThread-Thread-3] - hours:0.0 minutes:0.0 seconds:16.0 this information: 2011-10-12 21:02:43:383 clientThread-Thread-3 hours:0.0 minutes:0.0 seconds:3.0 2011-10-12 21:02:43:462 clientThread-Thread-3 hours:0.0 minutes:0.0 seconds:16.0 Can this be done? I'd start with a few simple tests like with open(test.log) as instream: for line in instream: if line.startswith([): parts = line.replace([, ]).split(]) duration = parts[8].strip( \n) if hours in duration and minutes in duration: duration = duration[duration.index(hours):] time = parts[1] thread = parts[7] print %s\t%s\t%s % (time, thread, duration) and see if you get false positives/negatives. Refine as necessary. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
[Tutor] Retrieve data from log file
Hi there, I'm sorry to ask this, I tried and example that was posted in 2009 but I couldn't make it work. I'm trying to parse a log file and retrieve what I just need. Hope you can help me with this, this is part of the log file: [2011-10-11 20:43:54:307] INFO [QuoteCompareDaoWSImpl: 24] Se va a invocar el ws de qualitas para comparar cotizaciones [2011-10-11 20:43:55:899] DEBUG [BaseQualitasDaoWS: 79] [http-80-3] wscall - hours:0.0 minutes:0.0 seconds:1.0 [2011-10-11 21:09:42:604] DEBUG [QuoteCompareDaoWSImpl: 25] [http-80-3]Se va a invocar el ws de qualitas para comparar cotizaciones [2011-10-11 21:09:42:708] DEBUG [BaseQualitasDaoWS: 71] [http-80-3] -request: ?xml version=1.0 encoding=ISO-8859-1? Movimientos Movimiento TipoMovimiento=2 NoNegocio=63 NoPoliza= NoCotizacion= NoEndoso= TipoEndoso= NoOtra= DatosAsegurado NoAsegurado= Nombre/ Direccion/ Colonia/ Poblacion/ Estado3519/Estado CodigoPostal/ NoEmpleado/ Agrupador/ /DatosAsegurado [2011-10-12 21:02:43:380] DEBUG [BaseQualitasDaoWS: 78] [http-80-1][clientThread-Thread-3] -response: ?xml version=1.0 encoding=UTF-8? Movimientos Movimiento TipoMovimiento=2 NoNegocio=63 NoPoliza= NoCotizacion=0018333748 NoEndoso= TipoEndoso=21 NoOtra=5470013P DatosAsegurado NoAsegurado= Nombre/Nombre Direccion/Direccion Colonia/Colonia Poblacion/Poblacion Estado3519/Estado CodigoPostal0/CodigoPostal NoEmpleado0/NoEmpleado Agrupador/Agrupador /DatosAsegurado DatosVehiculo NoInciso=0001 ClaveAmis7650/ClaveAmis Modelo2002/Modelo DescripcionVehiculoVOLKSWAGEN DERBY STD., 05 OCUP./DescripcionVehiculo Uso1/Uso Servicio1/Servicio Paquete4/Paquete Motor/Motor Serie/Serie Coberturas NoCobertura=4 SumaAsegurada125/SumaAsegurada TipoSuma2/TipoSuma Deducible/Deducible Prima1091.24/Prima /Coberturas Coberturas NoCobertura=5 SumaAsegurada20/SumaAsegurada TipoSuma2/TipoSuma Deducible0/Deducible Prima279.26/Prima /Coberturas Coberturas NoCobertura=7 SumaAsegurada50/SumaAsegurada TipoSuma2/TipoSuma Deducible0/Deducible Prima310.0/Prima /Coberturas Coberturas NoCobertura=14 SumaAsegurada8560/SumaAsegurada TipoSuma2/TipoSuma Deducible0/Deducible Prima260.0/Prima /Coberturas Coberturas NoCobertura=34 SumaAsegurada120/SumaAsegurada TipoSuma2/TipoSuma Deducible0/Deducible Prima0.0/Prima /Coberturas ConsideracionesAdicionalesDV NoConsideracion=37 TipoRegla0/TipoRegla ValorReglaN/ValorRegla /ConsideracionesAdicionalesDV /DatosVehiculo DatosGenerales FechaEmision2011-10-12/FechaEmision FechaInicio2011-10-06/FechaInicio FechaTermino2011-10-09/FechaTermino Moneda0/Moneda Agente17065/Agente FormaPagoS/FormaPago TarifaValores1109/TarifaValores TarifaCuotas1109/TarifaCuotas TarifaDerechos1109/TarifaDerechos Plazo14/Plazo Agencia/Agencia Contrato/Contrato PorcentajeDescuento0.0/PorcentajeDescuento ConsideracionesAdicionalesDG NoConsideracion=1 TipoRegla1/TipoRegla ValorRegla0/ValorRegla /ConsideracionesAdicionalesDG ConsideracionesAdicionalesDG NoConsideracion=5 TipoRegla0/TipoRegla ValorRegla14/ValorRegla /ConsideracionesAdicionalesDG ConsideracionesAdicionalesDG NoConsideracion=8 TipoRegla12/TipoRegla ValorRegla1100.0/ValorRegla /ConsideracionesAdicionalesDG ConsideracionesAdicionalesDG NoConsideracion=14 TipoRegla0/TipoRegla ValorReglaX01109/ValorRegla /ConsideracionesAdicionalesDG ConsideracionesAdicionalesDG NoConsideracion=15 TipoRegla69/TipoRegla ValorRegla79/ValorRegla /ConsideracionesAdicionalesDG ConsideracionesAdicionalesDG NoConsideracion=20 TipoRegla0/TipoRegla ValorRegla/ValorRegla /ConsideracionesAdicionalesDG ConsideracionesAdicionalesDG NoConsideracion=26 TipoRegla0/TipoRegla ValorRegla62/ValorRegla /ConsideracionesAdicionalesDG ConsideracionesAdicionalesDG NoConsideracion=35 TipoRegla2/TipoRegla ValorRegla0.0/ValorRegla /ConsideracionesAdicionalesDG ConsideracionesAdicionalesDG NoConsideracion=55 TipoRegla0/TipoRegla ValorRegla/ValorRegla /ConsideracionesAdicionalesDG /DatosGenerales Primas PrimaNeta1940.5/PrimaNeta Derecho400/Derecho Recargo46.57/Recargo Impuesto262.58/Impuesto PrimaTotal2649.65/PrimaTotal Comision15/Comision /Primas Recibos NoRecibo=1 FechaInicio2011-10-06/FechaInicio FechaTermino2011-10-09/FechaTermino PrimaNeta1940.5/PrimaNeta
[Tutor] Retrieve data
Hello everyone, I would to retrieve data, and especially the temperature and the weather from http://www.nytimes.com/weather. And I don't know how to do so. Thank you, Louis Sent from my BlackBerry® on the MetroPCS Network ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Retrieve data
You cand use urllib to connect that web page and then use a custom html parser to get exactly what you want, even I think you could use regex; just look source code from the webpage, check for the data you are interested in and get it. 2011/4/12 l.leicht...@gmail.com Hello everyone, I would to retrieve data, and especially the temperature and the weather from http://www.nytimes.com/weather. And I don't know how to do so. Thank you, Louis Sent from my BlackBerry® on the MetroPCS Network ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Retrieve data
NYtimes has an API http://developer.nytimes.com/ http://developer.nytimes.com/No clue on how well it works or even if the weather is in the API. Looks like a lot of fluff, like comments and such. At any rate, maybe you can request they add weather to their API. On Tue, Apr 12, 2011 at 12:36 PM, l.leicht...@gmail.com wrote: Hello everyone, I would to retrieve data, and especially the temperature and the weather from http://www.nytimes.com/weather. And I don't know how to do so. Thank you, Louis Sent from my BlackBerry® on the MetroPCS Network ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Retrieve data
l.leicht...@gmail.com wrote: Hello everyone, I would to retrieve data, and especially the temperature and the weather from http://www.nytimes.com/weather. And I don't know how to do so. Consider whether the NY Times terms and conditions permit such automated scraping of their web site. Be careful you do not abuse their hospitality by hammering their web site unnecessarily (say, by checking the weather eighty times a minute). Consider whether they have a public API for downloading data directly. If so, use that. Otherwise: Use the urlib2 and urlib modules to download the raw HTML source of the page you are interested in. You may need to use them to login, to set cookies, set the referer [sic], submit data via forms, change the user-agent... it's a PITA. Better to use an API if the web site offers one. Use the htmllib module to parse the source looking for the information you are after. If their HTML is crap, as it so often is with commercial websites that should know better, download and install BeautifulSoup, and use that for parsing the HTML. Don't be tempted to use regexes for parsing the HTML. That is the wrong solution. Regexes *seem* like a good idea for parsing HTML, and for simple tasks they are quick to program, but they invariably end up being ten times as much work as a proper HTML parser. If the content you are after requires Javascript, you're probably out of luck. -- Steven ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] retrieve data from an online database without knowing the url
I was wondering whether it was possible to write a program which could directly write some word in a box and click 'search' on a typical online database without using the url. (e.g. is there a way to write a program which would write some keyword, say 'tomato' on google and click 'google search' automatically and copy the page without having to know the url 'http://www.google.com/search? hl=ensa=Xoi=spellresnum=0ct=resultcd=1q=tomatospell=1') Thank you very much for any help you could provide, I would use urllib urllib2 (google around for some example code), but then you'll still have to know - the names of the input search box and search button - whether it's a get or post form Some of this info can be gotten by first downloading the form and examine it for tags (such as 'action'), but possibly not all. Other than that, I wouldn't really know. ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] retrieve data from an online database without knowing the url
On Thu, Nov 15, 2007 at 04:06:31AM +0100, pileux systeme wrote:
Hello,
I was wondering whether it was possible to write a program which could
directly write some word in a box and click 'search' on a typical online
database without using the url. (e.g. is there a way to write a program which
would write some keyword, say 'tomato' on google and click 'google search'
automatically and copy the page without having to know the url
'http://www.google.com/search?hl=ensa=Xoi=spellresnum=0ct=resultcd=1q=tomatospell=1')
Thank you very much for any help you could provide,
N.
Yes, this is possible. Have a look at urllib
(http://docs.python.org/lib/module-urllib.html) and within that page see
urlencode. The idea is that first you make a connection to google.com,
and then post the search values to the form. If you look at the form on
google.com (from source) you see the following:
form action=/search name=f
--snip--
input name=hl value=nl type=hidden
input maxlength=2048 name=q size=55 title=Google zoeken value=
input name=btnG value=Google zoeken type=submitinput name=btnI
value=Ik doe een gok type=submit
--snip--
/form
For example:
import urllib
search_string = 'cars'
encoded_search_string = urllib.urlencode({'q': search_string})
reponse = urllib.urlopen('www.google.com', encoded_search_string)
To search for cars
Good luck!
--
Yoram Hekma
Unix Systems Administrator
CICT Department
AOES Netherlands B.V.
Haagse Schouwweg 6G
2332 KG Leiden, The Netherlands
Phone: +31 (0)71 5795588
Fax:+31 (0)71 5721277
e-mail: [EMAIL PROTECTED]
http://www.aoes.com
signature.asc
Description: Digital signature
___
Tutor maillist - Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] retrieve data from an online database without knowing the url
pileux systeme wrote: Hello, I was wondering whether it was possible to write a program which could directly write some word in a box and click 'search' on a typical online database without using the url. (e.g. is there a way to write a program which would write some keyword, say 'tomato' on google and click 'google search' automatically and copy the page without having to know the url 'http://www.google.com/search?hl=ensa=Xoi=spellresnum=0ct=resultcd=1q=tomatospell=1') Maybe ClientForm is what you want: http://wwwsearch.sourceforge.net/ClientForm/ Kent ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
[Tutor] retrieve data from an online database without knowing the url
Hello, I was wondering whether it was possible to write a program which could directly write some word in a box and click 'search' on a typical online database without using the url. (e.g. is there a way to write a program which would write some keyword, say 'tomato' on google and click 'google search' automatically and copy the page without having to know the url 'http://www.google.com/search?hl=ensa=Xoi=spellresnum=0ct=resultcd=1q=tomatospell=1') Thank you very much for any help you could provide, N. - Ne gardez plus qu'une seule adresse mail ! Copiez vos mails vers Yahoo! Mail ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
