Re: Psycopg2 pool clarification
On Thu, Jun 8, 2017 at 10:47 AM, Israel Brewster wrote:
>> On Jun 7, 2017, at 10:31 PM, dieter wrote:
>>
>> israel writes:
>>> On 2017-06-06 22:53, dieter wrote:
>>> ...
>>> As such, using psycopg2's pool is essentially
>>> worthless for me (plenty of use for it, i'm sure, just not for me/my
>>> use case).
>>
>> Could you not simply adjust the value for the "min" parameter?
>> If you want at least "n" open connections, then set "min" to "n".
>
> Well, sure, if I didn't care about wasting resources (which, I guess many
> people don't). I could set "n" to some magic number that would always give
> "enough" connections, such that my application never has to open additional
> connections, then adjust that number every few months as usage changes. In
> fact, now that I know how the logic of the pool works, that's exactly what
> I'm doing until I am confident that my caching replacement is solid.
>
> Of course, in order to avoid having to open/close a bunch of connections
> during the times when it is most critical - that is, when the server is under
> heavy load - I have to set that number arbitrarily high. Furthermore, that
> means that much of the time many, if not most, of those connections would be
> idle. Each connection uses a certain amount of RAM on the server, not to
> mention using up limited connection slots, so now I've got to think about if
> my server is sized properly to be able to handle that load not just
> occasionally, but constantly - when reducing server load by reducing the
> frequency of connections being opened/closed was the goal in the first place.
> So all I've done is trade dynamic load for static load - increasing
> performance at the cost of resources, rather than more intelligently using
> the available resources. All-in-all, not the best solution, though it does
> work. Maybe if load was fairly constant it would make more sense though. So
> like I said *my* use case, whi
ch
> is a number of web apps with varying loads, loads that also vary from
> day-to-day and hour-to-hour.
>
> On the other hand, a pool that caches connections using the logic I laid out
> in my original post would avoid the issue. Under heavy load, it could open
> additional connections as needed - a performance penalty for the first few
> users over the min threshold, but only the first few, rather than all the
> users over a certain threshold ("n"). Those connections would then remain
> available for the duration of the load, so it doesn't need to open/close
> numerous connections. Then, during periods of lighter load, the unused
> connections can drop off, freeing up server resources for other uses. A
> well-written pool could even do something like see that the available
> connection pool is running low, and open a few more connections in the
> background, thus completely avoiding the connection overhead on requests
> while never having more than a few "extra" connections at any given time.
> Even if you left of the expiration logic, it would still be an improvement,
> because while unused connections wouldn't
d
> rop, the "n" open connections could scale up dynamically until you have
> "enough" connections, without having to figure out and hard-code that "magic
> number" of open connections.
>
> Why wouldn't I want something like that? It's not like its hard to code -
> took me about an hour and a half to get to a working prototype yesterday.
> Still need to write tests and add some polish, but it works. Perhaps, though,
> the common thought is just "throw more hardware at it and keep a lot of
> connections open at all time?" Maybe I was raised to conservatively, or the
> company I work for is too poor :-D
Psycopg is first and foremost a database adapter. To quote from the
psycopg2.pool module documentation, "This module offers a few pure
Python classes implementing *simple* connection pooling directly in
the client application" (emphasis added). The advertised list of
features at http://initd.org/psycopg/features/ doesn't even mention
connection pooling. In short, you're getting what you paid for.
It sounds like your needs are beyond what the psycopg2.pool module
provides. I suggest looking into a dedicated connection pooler like
PgBouncer. You'll find that it's much more feature-rich and
configurable than psycopg2.pool. It's production-ready, unlike your
prototype. And since it's a proxy, it can take connections from
multiple client apps and tune the pool to your overall load rather
than on an app-by-app basis (and thus risk overloading the backend if
multiple apps unexpectedly peak together).
As for why psycopg2.pool is the way it is, maybe most users don't have
your situation of serving multiple apps with loads varying on
different cycles. Most are probably only serving a single app, or if
serving multiple apps then they likely have common user bases with
similar peak times. You can't dynamically adjust the amount of RAM in
your server, so saving re
Re: How to run self-contained Python scripts who don't need Python installation?
On 06/08/2017 07:40 PM, Juan C. wrote: > 2. I'd like to create a simple BAT to run my Python script, so there > would be only 2 things (a 'dist' folder with everything and a run.bat > to make it clear what should be run), for example: Sure you can. You just have to do it like this (untested; I haven't used Windows in many years): @echo off start %~dp0dist\python-3.6.0-embed-win32\python.exe %~dp0dist\app.py %* I think %* expands to all the commandline parameters, apparently excluding %0, which would be the command name itself. -- https://mail.python.org/mailman/listinfo/python-list
How to run self-contained Python scripts who don't need Python installation?
I need to run some Python 3.6.0 scripts on the users' machines (W7 and W10) in an enterprise environment, but I can't install Python on those machines. I tried looking for those "py to exe", but sadly they don't support Python 3.6.0. Then I found out about "Windows x86/x86-64 embeddable zip file" that Python.org made available since 3.5.0, and I thought it would be the right thing for my case. I have some questions regarding embeddable Python: 1. How can I install custom modules using pip? 2. I'd like to create a simple BAT to run my Python script, so there would be only 2 things (a 'dist' folder with everything and a run.bat to make it clear what should be run), for example: @echo off start %~dp0dist\python-3.6.0-embed-win32\python.exe %~dp0dist\app.py The issue is that I can't parse args to app.py in this case. How would that be possible? In the example above, if Python was fully installed, I would give a file as argument to app.py, generally clicking and dragging the file into the app.py icon so that it would start. 3. Is there a better approach for my case other than embedded Python? -- https://mail.python.org/mailman/listinfo/python-list
Re: New to Python - Career question
On 6/6/17 5:39 PM, [1]pta...@gmail.com wrote: New to Python and have been at it for about a month now. I'm doing well and like it very much. Considering a career change down the road and have been wondering... What are the job prospects for a middle age entry level programmer. Just trying to get a better understanding where I stand career wise. Appreciate all feed back. Thanks! Are you asking about job prospects based on experience with Python vs other languages? If so, here's a site that shows the relative demand for various programming languages over time. Based on its huge database of job listings. Also shows the relative supply of programmers in various languages over time. You can specify which languages or other marketable skills to show in the graph: - [2]https://www.indeed.com/jobtrends/q-python-q-ruby-q-php-q-javascript-q-java-q-perl.html Hope this helps, --Fred -- Fred Stluka -- [3]mailto:f...@bristle.com -- [4]http://bristle.com/~fred/ Bristle Software, Inc -- [5]http://bristle.com -- Glad to be of service! Open Source: Without walls and fences, we need no Windows or Gates. -- References Visible links 1. mailto:pta...@gmail.com 2. https://www.indeed.com/jobtrends/q-python-q-ruby-q-php-q-javascript-q-java-q-perl.html 3. mailto:f...@bristle.com 4. http://bristle.com/~fred/ 5. http://bristle.com/ -- https://mail.python.org/mailman/listinfo/python-list
PyDev 5.8.0: Code Coverage fixes, IronPython debugging
PyDev 5.8.0 Release Highlights - *Important* PyDev now requires Java 8 and Eclipse 4.6 (Neon) onwards. - PyDev 5.2.0 is the last release supporting Eclipse 4.5 (Mars). - *Code Analysis* - Fixed issue getting existing PyLint markers. - There's now an Info and an Ignore level. - *Debugger* - The debugger now provides hooks for clients and provides ways to extend the handling of custom types. See: https://github.com/fabioz/PyDev.Debugger/tree/master/pydevd_plugins/extensions (patch by *Yuli Fiterman*). - Fixed issue where the debugger could end up removing quotes on args. *#PyDev-797* - The debugger now works with IronPython again -- although note that *IronPython* *2.7.6* and *2.7.7* have a critical bug which prevents IronPython from working in PyDev: https://github.com/IronLanguages/main/issues/1663 - *Code Coverage* - Fixed issue getting code-coverage version. *#PyDev-791* - Properly works when running with pytest (provided that pytest-cov is installed). - *Others* - Update .yaml file for google app engine project templates (patch by *JunjieW*). - Upgraded Lucene to 6.1.0 (patch by *Sopot Cela*). - Update docstring from parameters (Ctrl+1 on *def*) properly considers sphinx with types. *#PyDev-787* - Code Completion: Properly finding *__init__* from superclass in inherited classes. *#PyDev-802* - No longer showing icon to start interactive console in toolbar because Eclipse could end up creating multiple entries which were shown forever. *#PyDev-708* - Other minor bugfixes. What is PyDev? PyDev is an open-source Python IDE on top of Eclipse for Python, Jython and IronPython development. It comes with goodies such as code completion, syntax highlighting, syntax analysis, code analysis, refactor, debug, interactive console, etc. Details on PyDev: http://pydev.org Details on its development: http://pydev.blogspot.com What is LiClipse? LiClipse is a PyDev standalone with goodies such as support for Multiple cursors, theming, TextMate bundles and a number of other languages such as Django Templates, Jinja2, Kivy Language, Mako Templates, Html, Javascript, etc. It's also a commercial counterpart which helps supporting the development of PyDev. Details on LiClipse: http://www.liclipse.com/ Cheers, -- Fabio Zadrozny -- Software Developer LiClipse http://www.liclipse.com PyDev - Python Development Environment for Eclipse http://pydev.org http://pydev.blogspot.com PyVmMonitor - Python Profiler http://www.pyvmmonitor.com/ -- https://mail.python.org/mailman/listinfo/python-list
Re: Reg : Wikipedia 1.4 package
On Thursday, June 8, 2017 at 11:10:49 AM UTC-4, Siva Kumar S wrote: > Dear members, > > how to install wikipedia 1.4 package in python 2.7 above without PIP. Why don't you want to use pip? You can probably just download the .tar.gz, unpack it, and run: python setup.py install but pip will be better. --Ned. -- https://mail.python.org/mailman/listinfo/python-list
Re: python certification
On Thu, Jun 8, 2017 at 4:49 AM, Gonzalo V wrote: > hi, > good day. > where can i get a python certification? > > thanks! > -- > https://mail.python.org/mailman/listinfo/python-list > I don't believe there is any official certification. Nor do I believe that a Python certification would be something employers are even looking for. That said, if you really need some document that says you know Python, edX and udacity do offer completion certificates for their courses I believe. -- https://mail.python.org/mailman/listinfo/python-list
Re: python certification
On Jun 8, 2017 7:58 AM, "Gonzalo V" wrote: > > hi, > good day. > where can i get a python certification? I'm not sure there is such a thing. Try Googling. > thanks! > -- > https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: Psycopg2 pool clarification
---
Israel Brewster
Systems Analyst II
Ravn Alaska
5245 Airport Industrial Rd
Fairbanks, AK 99709
(907) 450-7293
---
> On Jun 7, 2017, at 10:31 PM, dieter wrote:
>
> israel writes:
>> On 2017-06-06 22:53, dieter wrote:
>> ...
>> As such, using psycopg2's pool is essentially
>> worthless for me (plenty of use for it, i'm sure, just not for me/my
>> use case).
>
> Could you not simply adjust the value for the "min" parameter?
> If you want at least "n" open connections, then set "min" to "n".
Well, sure, if I didn't care about wasting resources (which, I guess many
people don't). I could set "n" to some magic number that would always give
"enough" connections, such that my application never has to open additional
connections, then adjust that number every few months as usage changes. In
fact, now that I know how the logic of the pool works, that's exactly what I'm
doing until I am confident that my caching replacement is solid.
Of course, in order to avoid having to open/close a bunch of connections during
the times when it is most critical - that is, when the server is under heavy
load - I have to set that number arbitrarily high. Furthermore, that means that
much of the time many, if not most, of those connections would be idle. Each
connection uses a certain amount of RAM on the server, not to mention using up
limited connection slots, so now I've got to think about if my server is sized
properly to be able to handle that load not just occasionally, but constantly -
when reducing server load by reducing the frequency of connections being
opened/closed was the goal in the first place. So all I've done is trade
dynamic load for static load - increasing performance at the cost of resources,
rather than more intelligently using the available resources. All-in-all, not
the best solution, though it does work. Maybe if load was fairly constant it
would make more sense though. So like I said *my* use case, which
is a number of web apps with varying loads, loads that also vary from
day-to-day and hour-to-hour.
On the other hand, a pool that caches connections using the logic I laid out in
my original post would avoid the issue. Under heavy load, it could open
additional connections as needed - a performance penalty for the first few
users over the min threshold, but only the first few, rather than all the users
over a certain threshold ("n"). Those connections would then remain available
for the duration of the load, so it doesn't need to open/close numerous
connections. Then, during periods of lighter load, the unused connections can
drop off, freeing up server resources for other uses. A well-written pool could
even do something like see that the available connection pool is running low,
and open a few more connections in the background, thus completely avoiding the
connection overhead on requests while never having more than a few "extra"
connections at any given time. Even if you left of the expiration logic, it
would still be an improvement, because while unused connections wouldn't d
rop, the "n" open connections could scale up dynamically until you have
"enough" connections, without having to figure out and hard-code that "magic
number" of open connections.
Why wouldn't I want something like that? It's not like its hard to code - took
me about an hour and a half to get to a working prototype yesterday. Still need
to write tests and add some polish, but it works. Perhaps, though, the common
thought is just "throw more hardware at it and keep a lot of connections open
at all time?" Maybe I was raised to conservatively, or the company I work for
is too poor :-D
>
> --
> https://mail.python.org/mailman/listinfo/python-list
--
https://mail.python.org/mailman/listinfo/python-list
Reg : Wikipedia 1.4 package
Dear members, How to install Wikipedia 1.4 package in python 2.7 above without PIP. from, Sivakumar S -- https://mail.python.org/mailman/listinfo/python-list
Reg : Wikipedia 1.4 package
Dear members, how to install wikipedia 1.4 package in python 2.7 above without PIP. from, Sivakumar S -- https://mail.python.org/mailman/listinfo/python-list
Re: Hello from a super noob!
Hello,
you can refactor your code a little bit and learn more about exceptions:
def get_numbers():
first = None
second = None
while True:
try:
if first is None:
first = int(input('Enter your first number: '))
if second is None:
second = int(input('Enter your second number: '))
except ValueError:
print('You have to enter a number')
continue
else:
return first, second
Am 08.06.2017 um 01:56 schrieb CB:
> Hi everyone,
> I am taking a python class and I'm stuck in an exercise.
>
> what am i doing wrong? Can anyone try to run it? Thanks so much!
>
> #Description:Input validation and while loops.
>
>
> import random
> def main(): #main function need in all programs for automated testing
>
>
> #your program goes here
>
> print()
>
>
>
>
> print("This program will help us practice input validation and while
> loops.")
> print("The user will be asked to enter two numbers which will both be
> validated. ")
> print("The sum of the numbers will then be displayed in a complex print
> statement ")
> print("and the user will be asked if they would like to run the program
> again."
> )
> print()
> print()
>
> while True:
> FirstNumber = input ("Please enter the first number: ")
> if FirstNumber.isdigit ():
> FirstNumber = int(FirstNumber)
> break
> else:
> print ("Invalid response. Please enter a whole number. " )
>
> while True:
>
> SecondNumber = input ("Please enter the second number: " )
> if SecondNumber.isdigit():
> SecondNumber= int(SecondNumber)
>
> break
> else:
> print("Invalid response. Please enter a whole number." )
>
> print()
> print (str(FirstNumber) + " + " + str(SecondNumber)+ " = " +
> str(FirstNumber + SecondNumber))
> print()
>
> while True:
>
> ans= input('Would you like to run the program again (Y/N) : ')
> if ans== 'Y' or ans== 'N':
> break
>
> else:
> print(" lnvalid response. Please answer with 'Y' or 'N' ")
>
> if ans== 'N':
> break
>
>
signature.asc
Description: OpenPGP digital signature
--
https://mail.python.org/mailman/listinfo/python-list
EuroPython 2017: Full schedule now online
We are happy to announce the schedule for EuroPython 2017 in Rimini, Italy (July 9-16). The program WG has been working hard trying to fit all the sessions in the last few weeks. With over 200 sessions, over 180 speakers, one day for workshops, 5 days of talks, training, keynotes, lightning talks, posters, help desks and open spaces, followed by 2 days of sprints, EuroPython will be one of the most exciting and vibrant Python events this year: * EuroPython 2017 Schedule * https://ep2017.europython.eu/p3/schedule/ep2017/ The schedule is available in table and list format. Please note that we are still applying small changes to the slots where necessary. Many thanks to everyone who submitted proposals. EuroPython wouldn’t be possible without our speakers ! If you want to join the fun, be sure to get your tickets as soon as possible, since ticket sales usually start picking up quite a bit after we announce the schedule. Aside: If you haven’t done yet, please get your EuroPython 2017 ticket soon. We will switch to on-desk rates later in June, which will cost around 30% more than the regular rates. https://ep2017.europython.eu/en/registration/buy-tickets/ Enjoy, -- EuroPython 2017 Team http://ep2017.europython.eu/ http://www.europython-society.org/ PS: Please forward or retweet to help us reach all interested parties: https://twitter.com/europython/status/872790385055211520 Thanks. -- https://mail.python.org/mailman/listinfo/python-list
python certification
hi, good day. where can i get a python certification? thanks! -- https://mail.python.org/mailman/listinfo/python-list
Re: Hello from a super noob!
On 08/06/17 00:56, CB wrote: Hi everyone, I am taking a python class and I'm stuck in an exercise. what am i doing wrong? Can anyone try to run it? Thanks so much! It helps if you describe what is going wrong. Not just us, either; "Teddy Bear Debugging", explaining to a colleague (or indeed a soft toy) why your code cannot possibly be wrong, is an excellent way of finding bugs. The number of times I've stopped in the middle of an explanation to fix the now glaringly obvious mistake... Try it and see. Here's a hint: be very aware of what your indentation means. -- Rhodri James *-* Kynesim Ltd -- https://mail.python.org/mailman/listinfo/python-list
New release of Scalpl (v0.2.4) ✨🍰✨
Good morning evernyone !
I released a new version (0.2.4) of Scalpl yesterday evening which is
available on PyPI :)
https://github.com/ducdetronquito/scalpl
https://pypi.python.org/pypi/scalpl/
Scalpl is a lightweight wrapper that helps you to operate on nested
dictionaries through the built-in dict API, by using dot-separated
string keys.
It's not a drop-in replacement for your dictionnaries, just syntactic
sugar to avoid this['annoying']['kind']['of']['things'] and
prefer['a.different.approach'].
It aims to keep the power of the standard dict API while being lighter
and faster that Box or Addict.
This new release allows you to traverse nested list in your
dictionnaries:
```
from scalpl import Cut
data = {...}
proxy = Cut(Data)
proxy.update({'users[42][0].skills', 'Python'})
# data['users'][42][0]['skills'] == 'Python
```
It also contains:
* A pretty good refactoring of the code base.
* Better exceptions
* More tests
I also tried to improve the README, with a benchmark section and a FAQ.
I would really appreciate your feedbacks to improve this project !
Have a great day :)
✨🍰✨
--
https://mail.python.org/mailman/listinfo/python-list
CfP: 14th International Conference on Managed Languages & Runtimes
CALL FOR PAPERS - ManLang '17 14th International Conference on Managed Languages & Runtimes (ManLang, formerly PPPJ) September 25-29, 2017, Prague, Czech Republic http://d3s.mff.cuni.cz/conferences/manlang17/ Sponsored by Charles University and Oracle Labs In-cooperation with ACM SIGAPP and SIGPLAN ManLang is a premier forum for presenting and discussing innovations and breakthroughs in the area of programming languages and runtime systems, which form the basis of many modern computing systems, from small scale (embedded and real-time systems) to large-scale (cloud-computing and big-data platforms). --- TOPICS - Virtual machines * Managed runtime systems (JVM, Dalvik VM and Android Runtime (ART), LLVM, .NET CLR, RPython, etc.) * VM design and optimization * VMs for mobile and embedded devices * VMs for real-time applications * Isolation and resource control * Memory management - Languages * Managed languages (Java, Scala, JavaScript, Python, Ruby, C#, F#, Clojure, Groovy, Kotlin, R, Smalltalk, Racket, etc.) * Domain-specific languages * Language design and calculi * Compilers * Language interoperability * Parallelism and concurrency * Modular and aspect-oriented programming * Model-driven development * Frameworks and applications * Teaching - Techniques and tools * Static and dynamic program analysis * Real-time systems * Embedded systems * Testing * Verification * Monitoring and debugging * Security and information flow * Workload characterization and performance evaluation Do not hesitate to contact the Program Chair (Shigeru Chiba ) to clarify if a particular topic falls within the scope of ManLang '17. --- IMPORTANT DATES Abstract submission deadline: June 23, 2017 Submission deadline: June 26, 2017 Author notification: August 7, 2017 Camera-ready papers deadline: August 21, 2017 --- SUBMISSIONS ManLang '17 accepts four types of submissions: - Regular research paper: up to 12 pages - Work-in-progress paper: up to 6 pages - Industry and tool paper: up to 6 pages - Poster (standalone or accompanying paper submission) The conference proceedings will be published as part of the ACM International Conference Proceedings Series and will be disseminated through the ACM Digital Library. Research papers will be judged on their relevance, novelty, technical rigor, and contribution to the state-of-the-art. For work-in-progress research papers, more emphasis will be placed on novelty and the potential of the new idea than on technical rigor and experimental results. Industry and tool papers will be judged on their relevance, usefulness, and results. Suitability for demonstration and availability will also be considered for tool papers. Posters can accompany any submission, in particular to provide additional demonstration and discussion opportunities. Criteria for standalone posters will be similar to criteria for short papers. --- PROGRAM CHAIR * Shigeru Chiba, University of Tokyo, Japan GENERAL CHAIR * Petr Tuma, Charles University, Czech Republic PROGRAM COMMITTEE * Walter Binder, University of Lugano (USI), Switzerland * Christoph Bockisch, University Marburg, Germany * Elisa Gonzalez Boix, Vrije Universiteit Brussel, Belgium * Carl Friedrich Bolz, PyPy * Walter Cazzola, Università degli Studi di Milano, Italy * Yvonne Coady, University of Victoria, Canada * Stéphane Ducasse, Inria Lille Nord Europe, France * Görel Hedin, Lunds University, Sweden * Robert Hirschfeld, HPI, Germany * Tony Hosking, Purdue University, USA * Doug Lea, State University of New York at Oswego, USA * J. Eliot Moss, University of Massachusetts Amherst, USA * Hanspeter Mössenböck, Johannes Kepler University Linz, Austria * Tiark Rompf, Purdue University, USA * Koichi Sasada, Cookpad Inc., Japan * Martin Schoeberl, Technical University of Denmark * Jeremy Singer, University of Glasgow, Scotland * Vincent St-Amour, Northwestern University, USA * Laurence Tratt, King's College London, UK * Jan Vitek, Northeastern University, USA * Christian Wimmer, Oracle Labs, USA * Jianjun Zhao, Kyushu University, Japan -- https://mail.python.org/mailman/listinfo/python-list
testfixtures 5.1.0 released!
Hi All, I'm pleased to announce the release of testfixtures 5.1.0 featuring the following: * Added support for including non-editable fields to the|comparer|used by|compare()|when comparing/django/|Model|instances. The package is on PyPI and a full list of all the links to docs, issue trackers and the like can be found here: https://github.com/Simplistix/testfixtures Any questions, please do ask on the Testing in Python list or on the Simplistix open source mailing list... cheers, Chris -- https://mail.python.org/mailman/listinfo/python-list
How to use win32api package to send to printer a file in memory
I have an Angular application that is communicating via websocket with a python app. Actually, I save the file in Angular, then I pass to the websocket the file path and I print it by win32api.ShellExecute. Anyway, I do not really need to save the file, so would be much better to send the content though the websocket and then then print it. Can anyone send me some code snippet to show: 1. Which is the best data structure to receive the file content via websocket (bytes array?) 2. How to invoke win32api.ShellExecute, or equivalent function, to print the content. Thank you -- https://mail.python.org/mailman/listinfo/python-list
