Re: [Tutor] When do you know you're ready to start applying for jobs?

[Alex Kleider]

On 2017-12-14 16:21, Alan Gauld via Tutor wrote:

On 14/12/17 20:30, Matthew Ngaha wrote:

direction on where I can learn about the principles of system design.
Also why isn't this taught in beginner tutorials, or is it an advanced
concept?


It's advanced compared to programming, but there are lots of
books on the subject, especially OO design. If your local library
has an ordering scheme I'd recommend starting with
Coad & Youdon's two books - OOA and OOD.

Although his notation has been superseded by UML (another
area you could usefully research online) the principles of design
(especially in the yellow design book) are still valid. The books
are short and fairly non-technical, so quite easy to read. (But
very expensive so I don't recommend buying them new! But
I've just noticed they are very cheap 2nd hand on Amazon...)


$4.99 and $6.89 to be exact (amazon prime in the US)




___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] When do you know you're ready to start applying for jobs?

[Alan Gauld via Tutor]

On 14/12/17 20:30, Matthew Ngaha wrote:
> direction on where I can learn about the principles of system design.
> Also why isn't this taught in beginner tutorials, or is it an advanced
> concept?

It's advanced compared to programming, but there are lots of
books on the subject, especially OO design. If your local library
has an ordering scheme I'd recommend starting with
Coad & Youdon's two books - OOA and OOD.

Although his notation has been superseded by UML (another
area you could usefully research online) the principles of design
(especially in the yellow design book) are still valid. The books
are short and fairly non-technical, so quite easy to read. (But
very expensive so I don't recommend buying them new! But
I've just noticed they are very cheap 2nd hand on Amazon...)

Other good design books include Grady Booch's OOD book
(ideally the first edition) where again the notation is now
irrelevant but the principles are sound. And for a slightly
more modern twist look for Design Patterns by Gamma et al.

There are also books on architecture although they are often
domain specific, so you can find stuff on network architecture,
data architectures, web architectures and service oriented
architectures, as examples. Wikipedia is probably a good
place to start on that topic.

> refresher on the os module and I recalled you covered it extensively,
> but sadly I found it's no longer on your site.
The v2 material is still there but I haven't got round to porting it
to v3. (It is covered in my paper book "Python Projects" which is
effectively that whole section of my tutor in dead tree format)
I'm about to do a full revision of the tutor which should include
writing the missing topics in that section.

HTH
-- 

Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] When do you know you're ready to start applying for jobs?

[Matthew Ngaha]

Thank you all for replying, I really appreciate it. It's all I've been
thinking about lately and your responses have really cleared up a lot
of things for me.

On Tue, Dec 12, 2017 at 11:37 PM, Alan Gauld via Tutor  wrote:
>
> But there are things you can learn now that won't ever change.
> And that includes the principles of design - an aspect all too
> often ignored by people learning "programming" But you can't
> build a significant system of any kind without an underlying
> design. And you can't build big systems without understanding architecture.
>

Thanks Alan you gave a really profound reply and I've been digesting
it slowly but surely. Can you give me any tips on how I can improve on
this (where I quoted you). And are there any books or tutorials you
can recommend for me.

> Hopefully you see where I'm going - you never stop learning
> and you can waste a lot of time becoming an expert in things
> that will be of no use in a few years, or you can focus on
> principles that are the very basis of system design and
> construction.
>

Yes I see where you're going. I just need a push in the right
direction on where I can learn about the principles of system design.
Also why isn't this taught in beginner tutorials, or is it an advanced
concept?

Thanks again for your response. I remember years ago your website was
one of the 1st tutorials I read on python. Recently I needed a
refresher on the os module and I recalled you covered it extensively,
but sadly I found it's no longer on your site. It says under
construction. Any ideas of when that will be finished?

Thanks again.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] How to debug a memory leak in a wsgi application?

[Etienne Robillard]

Hi again James,


Le 2017-12-14 à 04:44, James Chapman a écrit :
No, I'm saying you shouldn't need to make any kind of malloc calls 
manually. Python handles memory allocation and deallocation on your 
behalf.
All I did is installing a precompiled Python 2.7 build from the Debian 
repository. I believe it was built with pymalloc and debugging.

Why do you need to call pymalloc?

I have not yet took the time to manually compile Python without pymalloc.

Are you using ctypes?

No.

And if you are I presume this is then to make C-calls into a shared 
library?
I use Cython instead of ctypes. I'm guessing the memory leak was not 
caused by the Cython-generated C code, but from the uWSGI backend.


Cheers,

Etienne





--
James

On 13 December 2017 at 18:30, Etienne Robillard > wrote:


Hi James,

Thank for your reply. Are you suggesting that under Linux the
malloc() glibc library call is more memory efficient than using
pymalloc?

Best regards,

Etienne


Le 2017-12-13 à 12:27, James Chapman a écrit :

Why pymalloc? I presume this means you're using ctypes which
means I have more questions.

If you're allocating your own blocks of memory then you need to
free them too. IE, does each call to pymalloc have a
corresponding call to pyfree?

Is the overhead of pythons built in malloc really a problem?

Are you changing pointers before you've freed the corresponding
block of memory?

There are many ways to create a memory leak, all of them
eliminated by letting python handle your memory allocations.

But, back to your original question, check out "valgrind".

HTH

--
James

On 6 December 2017 at 16:23, Etienne Robillard
mailto:tkad...@yandex.com>> wrote:

Hi Alan,

Thanks for the reply. I use Debian 9 with 2G of RAM and
precompiled Python 2.7 with pymalloc. I don't know if
debugging was enabled for this build and whether I should
enable it to allow memory profiling with guppy... My problem
is that guppy won't show the heap stats for the uWSGI master
process. However I have partially resolved this issue by
enabling --reload-on-rss 200 for the uwsgi process. 
Previously, the htop utility indicated a 42.7% rss memory
usage for 2 uWSGI processes. I have restarted the worker
processes with SIGINT signal. Now my uwsgi command line looks
like:

% uwsgi --reload-on-rss 200 --gevent 100 --socket
localhost:8000 --with-file /path/to/file.uwsgi --threads 2
--processes 4 --master --daemonize /var/log/uwsgi.log

My framework is Django with django-hotsauce 0.8.2 and
werkzeug. The web server is nginx using uWSGI with the gevent
pooling handler.

Etienne

Le 2017-12-06 à 10:00, Alan Gauld via Tutor a écrit :

On 06/12/17 09:21, Etienne Robillard wrote:

Hi

I think my wsgi application is leaking and I would
like to debug it.

What is the best way to profile memory usage in a
running wsgi app?

This is probably a bit advanced for the tutor list, you might
get a better response on the main Python list.

But to get a sensible answer you need to provide more data:
What OS and Python version?
What toolset/framework are you using?
What measurements lead you to suspect a memory leak?



-- 
Etienne Robillard

tkad...@yandex.com 
https://www.isotopesoftware.ca/ 

___
Tutor maillist  - Tutor@python.org 
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor





-- 
Etienne Robillard

tkad...@yandex.com 
https://www.isotopesoftware.ca/ 




--
Etienne Robillard
tkad...@yandex.com
https://www.isotopesoftware.ca/

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] How to debug a memory leak in a wsgi application?

[James Chapman]

​Ah OK, now I understand why you mentioned pymalloc to begin with.

I'm not familiar with uWSGI or cython. That said, why do you think it's
uWSGI causing a leak? It seems unlikely.
Python projects can grow in size if you're not dereferencing objects...
(see https://f0rki.at/hunting-memory-leaks-in-python.html)

If you use valgrind combined with python memory_profiler you should
hopefully be able to get an idea as to where the leak is coming from. It's
probably in your own code and leaks can be incredibly difficult to track
down. Typically while reviewing your own code you end up skipping over the
error time and time again because you become blind to your errors, so it
might help to have someone else peer review it.

These 2 links are a good starting point.
https://github.com/KratosMultiphysics/Kratos/wiki/Checking-memory-usage-with-Valgrind
https://github.com/pythonprofilers/memory_profiler

One last note, if you are doing any of your own memory allocations, then
make sure you're also freeing them:
https://cython.readthedocs.io/en/latest/src/tutorial/memory_allocation.html

But note, if you did this in cython:

cdef double* data
data =  PyMem_Malloc(100 * sizeof(double))
data =  PyMem_Malloc(100 * sizeof(double))
PyMem_Free(data)

You would (probably, you would in C/C++) end up with a leak because
you've changed the pointer. When you go to free it, only the 2nd
allocation will be freed and you'll have no way of freeing the first.

HTH
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] How to debug a memory leak in a wsgi application?

[James Chapman]

No, I'm saying you shouldn't need to make any kind of malloc calls
manually. Python handles memory allocation and deallocation on your behalf.
Why do you need to call pymalloc?
Are you using ctypes?
And if you are I presume this is then to make C-calls into a shared library?

James



--
James

On 13 December 2017 at 18:30, Etienne Robillard  wrote:

> Hi James,
>
> Thank for your reply. Are you suggesting that under Linux the malloc()
> glibc library call is more memory efficient than using pymalloc?
>
> Best regards,
>
> Etienne
>
> Le 2017-12-13 à 12:27, James Chapman a écrit :
>
> Why pymalloc? I presume this means you're using ctypes which means I have
> more questions.
>
> If you're allocating your own blocks of memory then you need to free them
> too. IE, does each call to pymalloc have a corresponding call to pyfree?
>
> Is the overhead of pythons built in malloc really a problem?
>
> Are you changing pointers before you've freed the corresponding block of
> memory?
>
> There are many ways to create a memory leak, all of them eliminated by
> letting python handle your memory allocations.
>
> But, back to your original question, check out "valgrind".
>
> HTH
>
> --
> James
>
> On 6 December 2017 at 16:23, Etienne Robillard  wrote:
>
>> Hi Alan,
>>
>> Thanks for the reply. I use Debian 9 with 2G of RAM and precompiled
>> Python 2.7 with pymalloc. I don't know if debugging was enabled for this
>> build and whether I should enable it to allow memory profiling with
>> guppy... My problem is that guppy won't show the heap stats for the uWSGI
>> master process. However I have partially resolved this issue by enabling
>> --reload-on-rss 200 for the uwsgi process.  Previously, the htop utility
>> indicated a 42.7% rss memory usage for 2 uWSGI processes. I have restarted
>> the worker processes with SIGINT signal. Now my uwsgi command line looks
>> like:
>>
>> % uwsgi --reload-on-rss 200 --gevent 100 --socket localhost:8000
>> --with-file /path/to/file.uwsgi --threads 2 --processes 4 --master
>> --daemonize /var/log/uwsgi.log
>>
>> My framework is Django with django-hotsauce 0.8.2 and werkzeug. The web
>> server is nginx using uWSGI with the gevent pooling handler.
>>
>> Etienne
>>
>> Le 2017-12-06 à 10:00, Alan Gauld via Tutor a écrit :
>>
>>> On 06/12/17 09:21, Etienne Robillard wrote:
>>>
 Hi

 I think my wsgi application is leaking and I would like to debug it.

 What is the best way to profile memory usage in a running wsgi app?

>>> This is probably a bit advanced for the tutor list, you might
>>> get a better response on the main Python list.
>>>
>>> But to get a sensible answer you need to provide more data:
>>> What OS and Python version?
>>> What toolset/framework are you using?
>>> What measurements lead you to suspect a memory leak?
>>>
>>>
>>>
>> --
>> Etienne Robillard
>> tkad...@yandex.com
>> https://www.isotopesoftware.ca/
>>
>> ___
>> Tutor maillist  -  Tutor@python.org
>> To unsubscribe or change subscription options:
>> https://mail.python.org/mailman/listinfo/tutor
>>
>
>
> --
> Etienne Robillardtkadm30@yandex.comhttps://www.isotopesoftware.ca/
>
>
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor