Re: [Tutor] General question rgrd. usage of libraries
Hum i also suggest you get more experience with python
think of a project and learn while doing it. thus you'll get motivation
while at the same time doing something useful which you could reuse in the
future.
else,
me for bs4 i googled what i needed. I also put an increasing variable to
ease web scraping tasks. like
var =0
...
print(var, element)
also, i suggest you decode to unicode as you'll get crazy hex stuffs if you
don't
.decode("utf-8")
i scrape websites written in french, so i always need unicode.
else the
.text is very helpful
like 'p' gives you the element
but
'p.text' gives you the content
To find suitable libraries i suggest you become good at doing the desired
task by hand as far as possible, so you'll know your job well. Then you
identify boring, impossible or tiring tasks. Then you google like .. python
module or just python how to and see how they did it
or what module they used to do it.
Hope it helps,
Abdur-Rahmaan Janhangeer,
Mauritius
On 6 May 2017 00:56, "Jim" wrote:
> On 05/05/2017 08:45 AM, Rafael Knuth wrote:
>
>> Hi there,
>>
>> I just recently learned how to build a basic web scraper with Python
>> 3.5 (I am learning Python for data analytics purposes). Being new to
>> coding, I have a question:
>>
>> How do I know which libraries I need to perform a certain task?
>> For example, in case of this web scraper (which I built with help of a
>> tutorial on YouTube) I need to have urrlib and Beautiful Soup
>>
>> import urllib
>> import urllib.request
>> from bs4 import BeautifulSoup
>>
>> theurl = "https://twitter.com/rafaelknuth;
>> thepage = urllib.request.urlopen(theurl)
>> soup = BeautifulSoup(thepage, "html.parser")
>>
>> print(soup.title.text)
>>
>> i = 1
>> for tweets in soup.findAll("div",{"class":"content"}):
>> print(i)
>> print(tweets.find("p").text)
>> i = i + 1
>>
>> Is there a way I can figure out which libraries I need when drafting my
>> code?
>> Can you share your experiences? Right now, if I wanted for example to
>> populate a Google Sheet with my scraped web content - how would I know
>> which libraries I would need to actually make this happen? I am trying
>> wondering if there is a process to figure out what I exactly need
>> library-wise.
>>
>>
>>
> There is a Python API to google sheets but when I had a look, it seemed
> fairly complex. I haven't tried it yet but depending on what you need to do
> this library may be what you need:
> https://pypi.python.org/pypi/gspread.
>
> Regards, Jim
>
>
> ___
> Tutor maillist - Tutor@python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] General question rgrd. usage of libraries
On 05/05/2017 08:45 AM, Rafael Knuth wrote:
Hi there,
I just recently learned how to build a basic web scraper with Python
3.5 (I am learning Python for data analytics purposes). Being new to
coding, I have a question:
How do I know which libraries I need to perform a certain task?
For example, in case of this web scraper (which I built with help of a
tutorial on YouTube) I need to have urrlib and Beautiful Soup
import urllib
import urllib.request
from bs4 import BeautifulSoup
theurl = "https://twitter.com/rafaelknuth;
thepage = urllib.request.urlopen(theurl)
soup = BeautifulSoup(thepage, "html.parser")
print(soup.title.text)
i = 1
for tweets in soup.findAll("div",{"class":"content"}):
print(i)
print(tweets.find("p").text)
i = i + 1
Is there a way I can figure out which libraries I need when drafting my code?
Can you share your experiences? Right now, if I wanted for example to
populate a Google Sheet with my scraped web content - how would I know
which libraries I would need to actually make this happen? I am trying
wondering if there is a process to figure out what I exactly need
library-wise.
There is a Python API to google sheets but when I had a look, it seemed
fairly complex. I haven't tried it yet but depending on what you need to
do this library may be what you need:
https://pypi.python.org/pypi/gspread.
Regards, Jim
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] General question rgrd. usage of libraries
On 05/05/17 14:45, Rafael Knuth wrote: > How do I know which libraries I need to perform a certain task? Mostly youn learn by experience, but otherwise google (or other search engine) is your friend. It doesn't do any harm to read the Library Reference document on the python.org web site too, it gives a good overview of the most common ones. But for stuff outside the standard library its pretty much down to google and asking on fora like the tutor list or a more topic related one. > populate a Google Sheet with my scraped web content One thing that will definitely help when asking on forums is to identify the correct (or at least the common) terminology. For example I've no idea what a "Google Sheet" is - that may be the correct term - but but I recognize the term I might be able to offer help. If I don't recognize the term I'll assume I don't know the answer. So some research before asking definitely helps. And don't forget PyPI which has libraries categorised to make browsing/searching easier. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] General question rgrd. usage of libraries
Hi there,
I just recently learned how to build a basic web scraper with Python
3.5 (I am learning Python for data analytics purposes). Being new to
coding, I have a question:
How do I know which libraries I need to perform a certain task?
For example, in case of this web scraper (which I built with help of a
tutorial on YouTube) I need to have urrlib and Beautiful Soup
import urllib
import urllib.request
from bs4 import BeautifulSoup
theurl = "https://twitter.com/rafaelknuth;
thepage = urllib.request.urlopen(theurl)
soup = BeautifulSoup(thepage, "html.parser")
print(soup.title.text)
i = 1
for tweets in soup.findAll("div",{"class":"content"}):
print(i)
print(tweets.find("p").text)
i = i + 1
Is there a way I can figure out which libraries I need when drafting my code?
Can you share your experiences? Right now, if I wanted for example to
populate a Google Sheet with my scraped web content - how would I know
which libraries I would need to actually make this happen? I am trying
wondering if there is a process to figure out what I exactly need
library-wise.
Thank you,
Rafael
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
