Re: Repeatedly crawl website every 1 min

[Iuri]

Unless you are authorized, don't do it. It literally costs a lot of money
to the website you are crawling, in CPU and bandwidth.

Hundreds of concurrent requests can even kill a small server (with bad
configuration).

Look scrapy package, it is great for scraping, but be friendly with the
websites you are crawling.

Em 10 de mai de 2017 23:22,  escreveu:

> Hi Everyone,
>
> Thanks for stoping by. I am working on a feature to crawl website content
> every 1 min. I am curious to know if there any good open source project for
> this specific scenario.
>
> Specifically, I have many urls, and I want to maintain a thread pool so
> that each thread will repeatedly crawl content from the given url. It could
> be a hundreds thread at the same time.
>
> Your help is greatly appreciated.
>
> ;)
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Horizontal Scalability in python

[Iuri]

You can do it with nginx.

Tornado has some docs about it:
http://tornado.readthedocs.org/en/latest/guide/running.html#running-behind-a-load-balancer

Also, nginx has extensive documentation about the topic;
https://www.nginx.com/resources/admin-guide/load-balancer/

Cheers!

On Tue, Sep 15, 2015 at 8:54 AM, AliReza Firuzabadi  wrote:

> I am new in tornado and I want to make web application with tornado and
> want to scale one instance of tornado application to many in separated
> servers. I should have load balancer and it send client requests to
> servers. please tell me how can I do it in the best way.
> thank you for helping me.
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: SQLAlchemy - web framework ?

[Iuri]

I don't know what exactly you mean with "wanted to not build it all
myself", but Flask is great with SQLAlchemy. You have the Flask-SQLAlchemy
extension and it has a lot of other integrations, like Flask-Admin.

You don't have to fear flask to bigger projects. To be honest, I prefer it
instead of Django because I can grow the project the way I want, without
framework strings.




On Mon, May 12, 2014 at 8:34 PM, flebber  wrote:

> If I want to use SQLAlchemy as my ORM what would be the best option for a
> web framework?
>
> It appears the general advice regarding Django is to do it the Django way
> and use the django ORM and change it out for SQLAlchemy.
>
> That to me limited knowledge leaves flask, pyramid and turbogears 2. So if
> I wanted to not build it all myself as with flask then potentially pyramid,
> turbogears is the best option?
>
> Is this true? I have completed the TG2 intro tutorial and have built
> several small things with flask although I feel offput by doing anything
> bigger in flask.
>
> See what I have done is got my python knowledge to a fair point where I
> can do useful things, good knowledge of web HTML/CSS, built a few small
> projects in flask to get an idea for python web, completed django
> tutorials, turogears tutorials and now looking to design out a bigger
> project I want to set myself and i am trying to compile the parts so I can
> see what I will need to use and gather info to cover what othe things I
> will need to know.
>
> Do I have a false fear of flask and doing bigger projects?
>
> So at this point I know I want SQLAlchemy, will use postgres(although
> mysql/maria would work fine).
>
> Any pratical advice warmly welcomed, I think I am thining too much
> aimlessly maybe.
>
> http://turbogears.org/
> http://www.pylonsproject.org/
> http://flask.pocoo.org/
> https://www.djangoproject.com/
> http://www.tornadoweb.org/en/stable/
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is It Bug?

[Iuri]

This way, it will replace '\' with '\', so nothing change.

>>> 'Hello, World'.replace('', '\\')
'Hello, \\World'
>>> print 'Hello, World'.replace('', '\\')
Hello, \World


On Sat, Dec 7, 2013 at 10:58 PM, Mahan Marwat  wrote:

> Why this is not working.
>
> >>> 'Hello, World'.replace('\\', '\\')
>
> To me, Python will interpret '' to '\\'. And the replace method will
> replace '\\' with '\'. So, the result will be 'Hello, \World'. But it's
> give me 'Hello, World'.
>
> The result I want form the code is 'Hello, \World'.
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

imp.load_module error in Python 3.3

[Iuri]

I want to load a file with an invalid module name (with a dash).

filename = '...' # something like /my/path/to/ejtp-crypto
with open(filename, 'rb') as fp:
my_module = imp.load_module('crypto', fp, 'ejtp-crypto', ('.py',
'rb', imp.PY_SOURCE))

It works to all Python >= 2.5, except 3.3. Some behaviour changed to break
it or is it a regression bug?

I know imp.load_module is deprecated in 3.3, but I didn't found anything
related with behaviour changes.

Also, I didn't understanded how exactly change my code to use importlib,
can you help me with this change?

Thanks!
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: _mysql_exceptions.OperationalError: (2005, "Unknown MySQL server host

[Iuri]

MySQLdb.connect does not accept a "connection string" parameter.

On Wed, Aug 15, 2012 at 11:53 AM, nepaul  wrote:

> 在 2012年8月15日星期三UTC+8下午10时23分24秒，Mark Lawrence写道：
> > On 15/08/2012 14:30, nepaul wrote:
> >
> > > The code:
> >
> > > import MySQLDB
> >
> > > strCmd = "user = 'root', passwd = '123456', db = 'test', host =
> 'localhost'"
> >
> > >
> >
> > > _mysql_exceptions.OperationalError: (2005, "Unknown MySQL server host
> 'user = 'root', passwd = '123456', db = 'test', host = 'localhost''
> (11004)")
> >
> > >
> >
> >
> >
> > My highly paid team of consultants suggest that you're trying to connect
> >
> > to a MySQL server host that doesn't exist.  However I'm sure that people
> >
> > on this list with more knowledge of MySQLDB than my consultants will be
> >
> > able to give you more detailed data.
> >
> >
> >
> > --
> >
> > Cheers.
> >
> >
> >
> > Mark Lawrence.
>
> if i use MySQLdb.connect(strCmd) -> wrong, but if use MySQLdb.connect(user
> = 'root', passwd = '123456', db = 'test', host = 'localhost'),it work. and
> the strCmd is read from a xml file.
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: To start a simple server

[Iuri]

Something is running in your port 8080. Change the port and try again.

On Mon, Oct 3, 2011 at 3:15 PM, Amelia T Cat  wrote:

> On Mon, 03 Oct 2011 10:09:59 -0700, sillyou su wrote:
>
> > 啊！！
> >
> > I should use 127.0.0.1 instance of  0.0.0.0
>
> Theoretically either one should be fine. If you use 127.0.0.1 it will
> only expose the service to your local machine. If you use 0.0.0.0 ut will
> expose the service to other computers on the same network as the one
> running the service.
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: program organization question for web development with python

[Iuri]

Another pretty web framework is Bottle (http://bottle.paws.de). It is very
easy to setup and use. I use it to tasks like this one you want.

[]s
iuri


On Thu, Sep 16, 2010 at 3:01 PM, MRAB  wrote:

> On 16/09/2010 18:19, Hans wrote:
>
>> On Sep 15, 5:33 pm, James Mills  wrote:
>>
>>> On Thu, Sep 16, 2010 at 10:14 AM, Hans  wrote:
>>>
>>>> I'm new to this area. Please allow me to ask some (maybe stupid)
>>>> questions.
>>>>
>>>
>>> Without reading the rest of your post too much. Designs are up to you,
>>> I can't comment.
>>>
>>> I can only share in a fairly common view, and that is, we'd encourage
>>> you to use a web framework
>>> as opposed to plain old CGI.
>>>
>>> cheers
>>> james
>>>
>>> --
>>> -- James Mills
>>> --
>>> -- "Problems are solved by method"
>>>
>>
>> Hi James,
>>
>> Thanks for response.
>> Maybe I did not make my question clear. I never tried python web
>> programing before, so I want to start from CGI.
>>
>> I read something about web framework like django, but seems it's a
>> little bit complicated. my task is actually very simple: get search
>> string from input, and then search database, print search result. I
>> thought CGI should be good enough to do this.
>>
>> I don't have any idea about how to organize those cgi codes, so what
>> I'm asking is:
>>
>> 1. do I have to have each single file for each hyper-link? Can I put
>> them together? how?
>> 2. how can I pass a db_cursor to another file? can I use db_cursor as
>> a parameter?
>>
>>  I recently wrote a web-based program using CherryPy, which was very
> straightforward. That might suit your needs.
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: compile() error

[Iuri]

Steven, it works fine to some cases.

I have problem only when last line is a comment and before it I have an
indentation. It is a specific case, and it is not a common case.

>>> compile("for i in [1,2,3]:\n pass\n#end", "test_file.py", "exec")

What I understanded about Benjamin's answer is compile() without a new line
at the end has an indetermined behaviour, so in some cases it works.

Iuri de Silvio

On Wed, May 19, 2010 at 9:52 PM, Steven D'Aprano <
steve-remove-t...@cybersource.com.au> wrote:

> On Wed, 19 May 2010 22:31:31 +, Benjamin Peterson wrote:
>
> > Iuri  gmail.com> writes:
> >> Any ideas about what is happening?
> >
> > Until Python 2.7/3.2, compile() does't like sources which don't end in a
> > newline.
>
>
> Are you sure about that?
>
>
> >>> x = compile("print __import__('sys').version, 'see ma, no newline'",
> "", "single")
> >>> exec x
> 2.5 (r25:51908, Nov  6 2007, 16:54:01)
> [GCC 4.1.2 20070925 (Red Hat 4.1.2-27)] see ma, no newline
>
>
> The original post isn't showing up for me, so I don't know the full
> context, but it seems to me that compile can deal with the lack of
> newline.
>
>
> >>> x = compile("""import sys
> ... print sys.version
> ... print 'see ma, still no final newline!'""", "", "exec")
> >>> exec x
> 2.5 (r25:51908, Nov  6 2007, 16:54:01)
> [GCC 4.1.2 20070925 (Red Hat 4.1.2-27)]
> see ma, still no final newline!
>
>
>
>
> --
> Steven
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: compile() error

[Iuri]

Thanks, Benjamin.

I used Python 2.6 to these tests.

[]s
iuri


On Wed, May 19, 2010 at 7:31 PM, Benjamin Peterson wrote:

> Iuri  gmail.com> writes:
> > Any ideas about what is happening?
>
> Until Python 2.7/3.2, compile() does't like sources which don't end in a
> newline.
>
>
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-- 
http://mail.python.org/mailman/listinfo/python-list

compile() error

[Iuri]

>>> compile("for i in [1,2,3]:\n pass\n#end\n", "test_file.py", "exec")
 at 0x266a378, file "test_file.py", line 1>
>>> compile("for i in [1,2,3]:\n pass\n#end", "test_file.py", "exec")
Traceback (most recent call last):
File "", line 1, in 
File "", line 2, in cptest
File "test_file.py", line 3
#end
^
SyntaxError: invalid syntax

I guess these commands explain the problem.

Any ideas about what is happening?

Regards,
iuri
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: def method with variable no of parameters file.writeStuff(n, a1, a2, ...an)

[Iuri]

What you need is var-args:

def func(*args):
  for arg in args: print arg

func(1,2,3,4)



On Fri, Apr 2, 2010 at 8:25 AM, vlad_fig  wrote:

> Hello all,
>
> I would like some help with setting up a method that would allow me to
> change its number of parameters. For example:
>
> #-
> class createfile(object):
>
> def __init__(self,
> modelName = None,
> someLines = None):
>
> self.modelName = modelName
>
> if someLines is None:
> self.someLines = []
> else:
> self.someLines = someLines
>
> def writeStuff(self,
> numberParameters = None,
> Parameter1 = None,... ??? )
> self.someLines .append("yes, we can %s" % self.Parameter1)
> #-
> file = createfile('file')
>
> file.writeStuff(2,a1,a2)
> file.writeStuff(3,a1,a2,a3)
> 
> file.writeStuff(n,a1,a2,...an)
>
> ---
> so i want a method i can call based on the number of parameters n , and
> that allows me to add these extra parameters based on n
>
> Thank you,
> Vicnic
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: DreamPie - The Python shell you've always dreamed about!

[Iuri]

I tested it in Windows Vista.

When I type single or double quotes, I get a unicode character, different of
python's quotes, and it break my code.

But i liked this tool! Thanks!

[]s
iuri



On Sun, Feb 21, 2010 at 3:13 PM, Chris Colbert  wrote:

> This is bloody fantastic! I must say, this fixes everything I hate about
> Ipython and gives me the feature I wished it had (with a few minor
> exceptions).
>
> I confirm this working on Kubuntu 9.10 using the ppa listed on the sites
> download page.
>
> I also confirm that it works interactively with PyQt4 and PyGtk (as to be
> expected since these toolkits use the PyOS_inputhook for the mainloop).
> However, it does not work interactively with wx (again, this is as expected
> since wx doesn't use the PyOS_inputhook). In short, the gui toolkit support
> is the same as in Ipython if you dont use any of the magic threading
> switches, which are now deprecated anyway.
>
> Matplotlib does not work interactively for me. Is there a special switch
> that needs to be used? or should a pick a non-wx backend? (i'm thinking the
> latter is more likely)
>
> A couple of things I would like to see (and will help implement if I can
> find the time):
> 1) A shortcut to show the docstring of an object. Something like Ipython's
> `?`. i.e.  `object.foo?` translates to `help(object.foo)`
> 2) How do I change the color of the blinking cursor at the bottom? I can't
> see the damn thing!
> 3) line numbers instead of the `>>>` prompt
> 4) a plugin facility where we can define our own `magic` commands. I use
> Ipython's %timeit ALL the time.
> 5) Double-click to re-fold the output section as well.
>
> Thanks for making this
>
> Cheers,
>
> Chris
>
>
> On Sun, Feb 21, 2010 at 4:42 AM, Noam Yorav-Raphael wrote:
>
>> I'm pleased to announce DreamPie 1.0 - a new graphical interactive
>> Python shell!
>>
>> Some highlights:
>>
>> * Has whatever you would expect from a graphical Python shell -
>> attribute completion, tooltips which show how to call functions,
>> highlighting of matching parentheses, etc.
>> * Fixes a lot of IDLE nuisances - in DreamPie interrupt always works,
>> history recall and completion works as expected, etc.
>> * Results are saved in the Result History.
>> * Long output is automatically folded so you can focus on what's
>> important.
>> * Jython and IronPython support makes DreamPie a great tool for
>> exploring Java and .NET classes.
>> * You can copy any amount of code and immediately execute it, and you
>> can also copy code you typed interactively into a new file, with the
>> Copy Code Only command. No tabs are used!
>> * Free software licensed under GPL version 3.
>>
>> Check it out at http://dreampie.sourceforge.net/ and tell me what you
>> think!
>>
>> Have fun,
>> Noam
>> --
>> http://mail.python.org/mailman/listinfo/python-list
>>
>
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Sikuli: the coolest Python project I have yet seen...

[Iuri]

I liked this tool! I'm thinking about how I can write acceptance tests with
Sikuli.

[]s
iurisilvio

On Tue, Jan 26, 2010 at 7:49 AM, Jean-Michel Pichavant <
jeanmic...@sequans.com> wrote:

> Terry Reedy wrote:
>
>> On 1/25/2010 9:14 AM, Javier Collado wrote:
>>
>>  I think the site is under maintenance. I tried a couple of hours ago
>>> and it worked fine.
>>>
>>> As an alternative, I found that this link also worked:
>>> http://www.sikuli.org/
>>>
>>
>> This just redirects to the link below
>>
>>  http://sikuli.csail.mit.edu/ I also did this
>

>>  This link is broken!

>>>
>> Worked for me both yesterday and now.
>>
>>  I confirmed it now works.
> The demo is pretty impressive.
>
> However one should not be the kind of guy changing its desktop theme on a
> regular basis. If I got it well, all is based on bitmap recognition (with
> some tolerance though). I'll still give it a try.
>
> JM
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Something More Elegant

[Iuri]

And you should use cursor.fetchall() instead of cursor in list
comprehension:

packageIDs = [itm[0] for itm in cursor.fetchall()]

On Sat, Jan 9, 2010 at 1:01 PM, Gabriel Genellina wrote:

> En Sat, 09 Jan 2010 11:01:25 -0300, Victor Subervi <
> victorsube...@gmail.com> escribió:
>
>> On Sat, Jan 9, 2010 at 8:39 AM, Tim Chase > >wrote:
>>
>>  It would also help if you didn't pass the categoryID as a
>>> string-formatted
>>> value, but as a proper parameter, something like
>>>
>>>  sql = "... where c.categoryid=?" % (store, store)
>>>  cursor.execute(sql, (category_id,))
>>>
>>>
>> I now have the following:
>>
>>  sql = 'select distinct p.ID from %sPackages p join
>> %sCategoriesPackages c where c.CategoryID=?;' % (store, store)
>>  cursor.execute(sql, (categoryID,))
>>  packageIDs = [itm[0] for itm in cursor]
>>
>> It threw this error:
>>
>> TypeError: not all arguments converted during string formatting
>>  args = ('not all arguments converted during string formatting',)
>>
>>  You'd have to check the place-holder character for your particular
>>> back-end:
>>>
>>>  >>> import  as db
>>>  >>> print db.paramstyle
>>>
>>> Printed "format". What's that mean? I use MySQLdb
>>>
>>
> That means, MySQLdb uses %s as a placeholder for parameter substitution --
> same as Python when doing string interpolation. Unfortunately this will
> confuse things. In your code above, the ? near the end should become %s --
> but you don't want THAT %s to be interpreted by Python at that time, instead
> it must remain as a literal %s until the cursor.execute line. You have to
> escape the % by doubling it: %%s
>
>   sql = 'select distinct p.ID from %sPackages p join
> %sCategoriesPackages c where c.CategoryID=%%s;' % (store, store)
>
>   cursor.execute(sql, (categoryID,))
>   packageIDs = [itm[0] for itm in cursor]
>
> --
> Gabriel Genellina
>
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Something More Elegant

[Iuri]

Your code select some ids from database and list distinct ids in packageIDs.
You can use SELECT DISTINCT in your SQL statement.



On Sat, Jan 9, 2010 at 11:23 AM, Victor Subervi wrote:

> Hi;
> The following code works fine. I would like you to suggest something more
> simple and elegant:
>
>   sql = 'select p.ID from %sPackages p join %sCategoriesPackages c
> where c.CategoryID=%s;' % (store, store, categoryID)
>   cursor.execute(sql)
>   tmp = [itm[0] for itm in cursor]
>   packageIDs = []
>   for t in tmp:
> if t not in packageIDs:
>   packageIDs.append(t)
>
> TIA,
> beno
>
> --
> The Logos has come to bear
> http://logos.13gems.com/
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Converting a float to a formatted outside of print command

[Iuri]

You forgot a % simbol in your string:

y = 0.5
x = '*%*.1f' % y

[]s
iurisilvio

On Mon, Nov 23, 2009 at 7:15 PM, stephen_b  wrote:

> I'd like to convert a list of floats to formatted strings. The
> following example raises a TypeError:
>
> y = 0.5
> x = '.1f' % y
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: help wanted with list

[Iuri]

You should use a dictionary.
dic = {'a':1, 'b':2}

Use dic['a'] to get output 1.

[]s
iurisilvio



On Thu, Sep 24, 2009 at 5:31 PM, Ahmed Shamim wrote:

> list = [ 'a', '1', 'b', '2']
> what would be the logic, if I input a to get output 1.
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: regexp help

[Iuri]

You can use r"[+-]?\d+" to get positive and negative integers.

It returns true to these strings: "+123", "-123", "123"



On Thu, Aug 27, 2009 at 3:15 PM, Bakes  wrote:

> If I were using the code:
>
> (?P[0-9]+)
>
> to get an integer between 0 and 9, how would I allow it to register
> negative integers as well?
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-- 
http://mail.python.org/mailman/listinfo/python-list