Re: What's wrong on using Popen's communicate method?

[jfong]

Wildman於 2019年7月4日星期四 UTC+8上午11時15分57秒寫道：
> On Thu, 04 Jul 2019 10:36:36 +1000, Cameron Simpson wrote:
> 
> > On 03Jul2019 16:57, Jach Fong  wrote:
> >>I have the test0.py below. I expect to see 'abcd' showing in the notepad 
> >>window:
> >>-
> >>import subprocess as sp
> >>p0 = sp.Popen('notepad.exe', stdin=sp.PIPE)
> >>p0.communicate(input=b'abcd')
> >>-
> >>But nothing happens. The notepad is completely empty. What have I missed?
> >>--Jach
> >>
> >>PS. I am using python 3.4 on Windows Vista
> > 
> > Well I am not a Windows person, but in most GUI environments a desktop 
> > app such as Notepad does not read from its standard input - it reads 
> > keyboard stuff though the GUI interface instead. So it is ignoring your 
> > input data - this approach will generally fail with any desktop app on 
> > any platform.
> > 
> > You may need to investigate generating synthetic keystrokes somehow.
> > 
> > Or alternatively write your data to a text file and hand the name of the 
> > text file to notepad as a file to edit.
> > 
> > Cheers,
> > Cameron Simpson 
> 
> I block googlegroups so I don't see the original post.  I am
> talking to the OP.
> 
> Take a look at the SendKeys module.
> 
> https://pypi.org/project/SendKeys/
> 
> -- 
>  GNU/Linux user #557453
> The cow died so I don't need your bull!

Thank you. But the download there seems is for Python 2.7?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What's wrong on using Popen's communicate method?

[Wildman via Python-list]

On Thu, 04 Jul 2019 10:36:36 +1000, Cameron Simpson wrote:

> On 03Jul2019 16:57, Jach Fong  wrote:
>>I have the test0.py below. I expect to see 'abcd' showing in the notepad 
>>window:
>>-
>>import subprocess as sp
>>p0 = sp.Popen('notepad.exe', stdin=sp.PIPE)
>>p0.communicate(input=b'abcd')
>>-
>>But nothing happens. The notepad is completely empty. What have I missed?
>>--Jach
>>
>>PS. I am using python 3.4 on Windows Vista
> 
> Well I am not a Windows person, but in most GUI environments a desktop 
> app such as Notepad does not read from its standard input - it reads 
> keyboard stuff though the GUI interface instead. So it is ignoring your 
> input data - this approach will generally fail with any desktop app on 
> any platform.
> 
> You may need to investigate generating synthetic keystrokes somehow.
> 
> Or alternatively write your data to a text file and hand the name of the 
> text file to notepad as a file to edit.
> 
> Cheers,
> Cameron Simpson 

I block googlegroups so I don't see the original post.  I am
talking to the OP.

Take a look at the SendKeys module.

https://pypi.org/project/SendKeys/

-- 
 GNU/Linux user #557453
The cow died so I don't need your bull!
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What's wrong on using Popen's communicate method?

[Chris Angelico]

On Thu, Jul 4, 2019 at 11:31 AM  wrote:
>
> Chris Angelico於 2019年7月4日星期四 UTC+8上午8時37分13秒寫道：
> > On Thu, Jul 4, 2019 at 10:01 AM  wrote:
> > >
> > > I have the test0.py below. I expect to see 'abcd' showing in the notepad 
> > > window:
> > > -
> > > import subprocess as sp
> > > p0 = sp.Popen('notepad.exe', stdin=sp.PIPE)
> > > p0.communicate(input=b'abcd')
> > > -
> > > But nothing happens. The notepad is completely empty. What have I missed?
> > >
> >
> > The "communicate" method sends text to the standard input pipe. This
> > has nothing to do with the GUI, and most Windows GUI programs take no
> > notice of it. You'll need something GUI-aware for this.
> >
> > Is Notepad just an example, or are you actually trying to control MS 
> > Notepad?
> >
> > ChrisA
>
> Yes, the notepad is just an example. My real attempt is to operate the 
> external programs through Python. I know there are some "keyboard simulation" 
> packages in Pypi which may work on this situation. But I prefer not bother to 
> install them if Python's build-ins can do it.
>
> By the way, after Popen invokes an external program, is there a way of making 
> it on-foucs when there are multiple Popen instances?
>

Definitely look into GUI manipulation tools. What you're doing is
nothing to do with the subprocess module (at least, not with the way
most Windows apps are built).

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What's wrong on using Popen's communicate method?

[Aldwin Pollefeyt]

This is same as   echo abcd | notepad.exein Command Prompt. You won't
see the abcd neither.

On Thu, Jul 4, 2019 at 8:41 AM Chris Angelico  wrote:

> On Thu, Jul 4, 2019 at 10:01 AM  wrote:
> >
> > I have the test0.py below. I expect to see 'abcd' showing in the notepad
> window:
> > -
> > import subprocess as sp
> > p0 = sp.Popen('notepad.exe', stdin=sp.PIPE)
> > p0.communicate(input=b'abcd')
> > -
> > But nothing happens. The notepad is completely empty. What have I missed?
> >
>
> The "communicate" method sends text to the standard input pipe. This
> has nothing to do with the GUI, and most Windows GUI programs take no
> notice of it. You'll need something GUI-aware for this.
>
> Is Notepad just an example, or are you actually trying to control MS
> Notepad?
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What's wrong on using Popen's communicate method?

[jfong]

Chris Angelico於 2019年7月4日星期四 UTC+8上午8時37分13秒寫道：
> On Thu, Jul 4, 2019 at 10:01 AM  wrote:
> >
> > I have the test0.py below. I expect to see 'abcd' showing in the notepad 
> > window:
> > -
> > import subprocess as sp
> > p0 = sp.Popen('notepad.exe', stdin=sp.PIPE)
> > p0.communicate(input=b'abcd')
> > -
> > But nothing happens. The notepad is completely empty. What have I missed?
> >
> 
> The "communicate" method sends text to the standard input pipe. This
> has nothing to do with the GUI, and most Windows GUI programs take no
> notice of it. You'll need something GUI-aware for this.
> 
> Is Notepad just an example, or are you actually trying to control MS Notepad?
> 
> ChrisA

Yes, the notepad is just an example. My real attempt is to operate the external 
programs through Python. I know there are some "keyboard simulation" packages 
in Pypi which may work on this situation. But I prefer not bother to install 
them if Python's build-ins can do it.

By the way, after Popen invokes an external program, is there a way of making 
it on-foucs when there are multiple Popen instances?

--Jach
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What's wrong on using Popen's communicate method?

[Chris Angelico]

On Thu, Jul 4, 2019 at 10:01 AM  wrote:
>
> I have the test0.py below. I expect to see 'abcd' showing in the notepad 
> window:
> -
> import subprocess as sp
> p0 = sp.Popen('notepad.exe', stdin=sp.PIPE)
> p0.communicate(input=b'abcd')
> -
> But nothing happens. The notepad is completely empty. What have I missed?
>

The "communicate" method sends text to the standard input pipe. This
has nothing to do with the GUI, and most Windows GUI programs take no
notice of it. You'll need something GUI-aware for this.

Is Notepad just an example, or are you actually trying to control MS Notepad?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What's wrong on using Popen's communicate method?

[Cameron Simpson]

On 03Jul2019 16:57, Jach Fong  wrote:

I have the test0.py below. I expect to see 'abcd' showing in the notepad window:
-
import subprocess as sp
p0 = sp.Popen('notepad.exe', stdin=sp.PIPE)
p0.communicate(input=b'abcd')
-
But nothing happens. The notepad is completely empty. What have I missed?
--Jach

PS. I am using python 3.4 on Windows Vista


Well I am not a Windows person, but in most GUI environments a desktop 
app such as Notepad does not read from its standard input - it reads 
keyboard stuff though the GUI interface instead. So it is ignoring your 
input data - this approach will generally fail with any desktop app on 
any platform.


You may need to investigate generating synthetic keystrokes somehow.

Or alternatively write your data to a text file and hand the name of the 
text file to notepad as a file to edit.


Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

What's wrong on using Popen's communicate method?

[jfong]

I have the test0.py below. I expect to see 'abcd' showing in the notepad window:
-
import subprocess as sp
p0 = sp.Popen('notepad.exe', stdin=sp.PIPE)
p0.communicate(input=b'abcd')
-
But nothing happens. The notepad is completely empty. What have I missed?

--Jach

PS. I am using python 3.4 on Windows Vista
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Handle foreign character web input

[Chris Angelico]

On Thu, Jul 4, 2019 at 8:12 AM Igor Korot  wrote:
>
> Hi, Chris,
>
> On Wed, Jul 3, 2019 at 4:41 PM Chris Angelico  wrote:
> >
> > On Thu, Jul 4, 2019 at 7:08 AM Igor Korot  wrote:
> > >
> > > Hi, Thomas,
> > >
> > > On Sat, Jun 29, 2019 at 11:06 AM Thomas Jollans  wrote:
> > > >
> > > > On 28/06/2019 22:25, Tobiah wrote:
> > > > > A guy comes in and enters his last name as RÖnngren.
> > > > With a capital Ö in the middle? That's unusual.
> > > > >
> > > > > So what did the browser really give me; is it encoded
> > > > > in some way, like latin-1?  Does it depend on whether
> > > > > the name was cut and pasted from a Word doc. etc?
> > > > > Should I handle these internally as unicode?  Right
> > > > > now my database tables are latin-1 and things seem
> > > > > to usually work, but not always.
> > > >
> > > >
> > > > If your database is using latin-1, German and French names will work,
> > > > but Croatian and Polish names often won't. Not to mention people using
> > > > other writing systems.
> > > >
> > > > So Günther and François are ok, but Bolesław turns into Boles?aw and
> > > > don't even think about anybody called Владимир or محمد.
> > >
> > > As others pointed out - it is very easy to do transliteration especially 
> > > if
> > > its' not a user registration that will be done.
> > >
> > > But I would simply not do that at all - create your forms in English and
> > > accept English spellings only.
> > > Most people that do computers this days can enter phonetic spelling
> > > of their first/last names (even in Chinese/Japanese/Hebrew).
> > >
> > > And all European names can be transliterated to English.
> > >
> > > Besides as the OP said - if someone comes to him and will
> > > try to enter the non-English name. The OP might not even have the 
> > > appropriate
> > > keyboard layout to input such a name. And if this is an (time consuming) 
> > > event
> > > all (s)he can do is ask for phonetic spelling.
> > >
> > > Thank you.
> > >
> > What you basically just said was "I wish all those ugly foreign names
> > would just go away". Honestly, that's not really an acceptable
> > solution; you assume that you can transliterate any name into
> > "English" in some perfect way, which is acceptable to everyone in the
> > world. And you also assume that this transformation will be completely
> > consistent, so you can ask someone his/her name and always get back
> > the same thing.
> >
> > If you want to do a Latinization and accent strip for the sake of a
> > search, that's fine; but make sure you retain the name as people want
> > it to be retained. Don't be bigoted.
> >
> > https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/
>
> I'm not opposing this, in fact I'm all for keeping the native name somewhere 
> in
> the DB.
>
> But as I said, imaging the following situation:
>
> You are somewhere in Germany and you have a German version of OS
> (any OS)
> .
> You also have a German keyboard (hardware) with German keys.
>
> Now you are assigned to go to some international events where people
> all over the world will be coming to your presentation and they will be
> registering on you machine
>
> Also imagine that the company policy prohibits you from  modifying the
> system settings.
>
> My solution:
> I would probably grab a lot of registering paper and ask people to enter
> English transliteration of the names on the machine so when you come
> back to the office you can properly enter their names using all those
> different keyboards (maybe virtual ones) to associate them with
> their English counterparts.
>
> Just curious - what would you do?
>

I would use a Compose key (if available) or a software input method
(always available, as long as you have an internet connection and
browser, and often available locally too). With your method, how would
you enter it back at the office, if all you have is an English
transliteration? How do you transform it back into the original
characters?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Handle foreign character web input

[Igor Korot]

Hi, Chris,

On Wed, Jul 3, 2019 at 4:41 PM Chris Angelico  wrote:
>
> On Thu, Jul 4, 2019 at 7:08 AM Igor Korot  wrote:
> >
> > Hi, Thomas,
> >
> > On Sat, Jun 29, 2019 at 11:06 AM Thomas Jollans  wrote:
> > >
> > > On 28/06/2019 22:25, Tobiah wrote:
> > > > A guy comes in and enters his last name as RÖnngren.
> > > With a capital Ö in the middle? That's unusual.
> > > >
> > > > So what did the browser really give me; is it encoded
> > > > in some way, like latin-1?  Does it depend on whether
> > > > the name was cut and pasted from a Word doc. etc?
> > > > Should I handle these internally as unicode?  Right
> > > > now my database tables are latin-1 and things seem
> > > > to usually work, but not always.
> > >
> > >
> > > If your database is using latin-1, German and French names will work,
> > > but Croatian and Polish names often won't. Not to mention people using
> > > other writing systems.
> > >
> > > So Günther and François are ok, but Bolesław turns into Boles?aw and
> > > don't even think about anybody called Владимир or محمد.
> >
> > As others pointed out - it is very easy to do transliteration especially if
> > its' not a user registration that will be done.
> >
> > But I would simply not do that at all - create your forms in English and
> > accept English spellings only.
> > Most people that do computers this days can enter phonetic spelling
> > of their first/last names (even in Chinese/Japanese/Hebrew).
> >
> > And all European names can be transliterated to English.
> >
> > Besides as the OP said - if someone comes to him and will
> > try to enter the non-English name. The OP might not even have the 
> > appropriate
> > keyboard layout to input such a name. And if this is an (time consuming) 
> > event
> > all (s)he can do is ask for phonetic spelling.
> >
> > Thank you.
> >
> What you basically just said was "I wish all those ugly foreign names
> would just go away". Honestly, that's not really an acceptable
> solution; you assume that you can transliterate any name into
> "English" in some perfect way, which is acceptable to everyone in the
> world. And you also assume that this transformation will be completely
> consistent, so you can ask someone his/her name and always get back
> the same thing.
>
> If you want to do a Latinization and accent strip for the sake of a
> search, that's fine; but make sure you retain the name as people want
> it to be retained. Don't be bigoted.
>
> https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/

I'm not opposing this, in fact I'm all for keeping the native name somewhere in
the DB.

But as I said, imaging the following situation:

You are somewhere in Germany and you have a German version of OS
(any OS)
.
You also have a German keyboard (hardware) with German keys.

Now you are assigned to go to some international events where people
all over the world will be coming to your presentation and they will be
registering on you machine

Also imagine that the company policy prohibits you from  modifying the
system settings.

My solution:
I would probably grab a lot of registering paper and ask people to enter
English transliteration of the names on the machine so when you come
back to the office you can properly enter their names using all those
different keyboards (maybe virtual ones) to associate them with
their English counterparts.

Just curious - what would you do?

Thank you.

>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Handle foreign character web input

[Chris Angelico]

On Thu, Jul 4, 2019 at 7:08 AM Igor Korot  wrote:
>
> Hi, Thomas,
>
> On Sat, Jun 29, 2019 at 11:06 AM Thomas Jollans  wrote:
> >
> > On 28/06/2019 22:25, Tobiah wrote:
> > > A guy comes in and enters his last name as RÖnngren.
> > With a capital Ö in the middle? That's unusual.
> > >
> > > So what did the browser really give me; is it encoded
> > > in some way, like latin-1?  Does it depend on whether
> > > the name was cut and pasted from a Word doc. etc?
> > > Should I handle these internally as unicode?  Right
> > > now my database tables are latin-1 and things seem
> > > to usually work, but not always.
> >
> >
> > If your database is using latin-1, German and French names will work,
> > but Croatian and Polish names often won't. Not to mention people using
> > other writing systems.
> >
> > So Günther and François are ok, but Bolesław turns into Boles?aw and
> > don't even think about anybody called Владимир or محمد.
>
> As others pointed out - it is very easy to do transliteration especially if
> its' not a user registration that will be done.
>
> But I would simply not do that at all - create your forms in English and
> accept English spellings only.
> Most people that do computers this days can enter phonetic spelling
> of their first/last names (even in Chinese/Japanese/Hebrew).
>
> And all European names can be transliterated to English.
>
> Besides as the OP said - if someone comes to him and will
> try to enter the non-English name. The OP might not even have the appropriate
> keyboard layout to input such a name. And if this is an (time consuming) event
> all (s)he can do is ask for phonetic spelling.
>
> Thank you.
>
What you basically just said was "I wish all those ugly foreign names
would just go away". Honestly, that's not really an acceptable
solution; you assume that you can transliterate any name into
"English" in some perfect way, which is acceptable to everyone in the
world. And you also assume that this transformation will be completely
consistent, so you can ask someone his/her name and always get back
the same thing.

If you want to do a Latinization and accent strip for the sake of a
search, that's fine; but make sure you retain the name as people want
it to be retained. Don't be bigoted.

https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Handle foreign character web input

[Igor Korot]

Hi, Thomas,

On Sat, Jun 29, 2019 at 11:06 AM Thomas Jollans  wrote:
>
> On 28/06/2019 22:25, Tobiah wrote:
> > A guy comes in and enters his last name as RÖnngren.
> With a capital Ö in the middle? That's unusual.
> >
> > So what did the browser really give me; is it encoded
> > in some way, like latin-1?  Does it depend on whether
> > the name was cut and pasted from a Word doc. etc?
> > Should I handle these internally as unicode?  Right
> > now my database tables are latin-1 and things seem
> > to usually work, but not always.
>
>
> If your database is using latin-1, German and French names will work,
> but Croatian and Polish names often won't. Not to mention people using
> other writing systems.
>
> So Günther and François are ok, but Bolesław turns into Boles?aw and
> don't even think about anybody called Владимир or محمد.

As others pointed out - it is very easy to do transliteration especially if
its' not a user registration that will be done.

But I would simply not do that at all - create your forms in English and
accept English spellings only.
Most people that do computers this days can enter phonetic spelling
of their first/last names (even in Chinese/Japanese/Hebrew).

And all European names can be transliterated to English.

Besides as the OP said - if someone comes to him and will
try to enter the non-English name. The OP might not even have the appropriate
keyboard layout to input such a name. And if this is an (time consuming) event
all (s)he can do is ask for phonetic spelling.

Thank you.

>
>
> >
> > Also, what do people do when searching for a record.
> > Is there some way to get 'Ronngren' to match the other
> > possible foreign spellings?
>
> --
> https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Handle foreign character web input

[mm0fmf]

On 30/06/2019 15:04, Chris Angelico wrote:

ut it looks like our old "Py3's
Unicode is buggy" troll is back


Yes... check the from field in the posts for confirmation. But it's 
obvious from the message content anyway.



--
https://mail.python.org/mailman/listinfo/python-list

Re: FCNLT Module in Windows

[Igor Korot]

Hi,

On Wed, Jul 3, 2019 at 12:56 PM Shakeel Ahmad  wrote:
>
>

ACK


>
> Sent from Mail for Windows 10
>
> --
> https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Multiprocessing and memory management

[Peter J. Holzer]

On 2019-07-03 08:37:50 -0800, Israel Brewster wrote:
> 1) Determine the total amount of RAM in the machine (how?), assume an
> average of 10GB per process, and only launch as many processes as
> calculated to fit. Easy, but would run the risk of under-utilizing the
> processing capabilities and taking longer to run if most of the
> processes were using significantly less than 10GB
> 
> 2) Somehow monitor the memory usage of the various processes, and if
> one process needs a lot, pause the others until that one is complete.
> Of course, I’m not sure if this is even possible.

If you use Linux or another unixoid OS, you can pause and resume
processes with the STOP and CONT signals. Just keep in mind that the
paused processes still need virtual memory and cannot complete - so you
need enough swap space.


> 3) Other approaches?

Is the memory usage at all predictable? I.e. can you estimate the usage
from the size of the input data? Or after the process has been running
for a short time? In that case you could monitor the free space and use
your estimates to determine whether you can start another process now or
need to wait until later (until a process terminates or maybe only until
you get a better estimate).

hp

-- 
   _  | Peter J. Holzer| we build much bigger, better disasters now
|_|_) || because we have much more sophisticated
| |   | h...@hjp.at | management tools.
__/   | http://www.hjp.at/ | -- Ross Anderson 


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

EuroPython 2019: Late Bird Rates and Day Passes

[M.-A. Lemburg]

We will be switching to the late bird rates for tickets on Saturday
(July 6), so this is your last chance to get tickets at the regular
rate, which is about 30% less than the late bird rate.

   EuroPython 2019 Tickets

  * https://ep2019.europython.eu/registration/buy-tickets/ *


Late Bird Tickets
-

We will have the following categories of late bird ticket prices for
the conference tickets:

- Business conference ticket: EUR 775.00 excl. VAT, EUR 834.68
  incl. 7.7% Swiss VAT (for people using Python to make a living)

- Personal conference day pass: EUR 520.00 incl. 7.7% Swiss VAT (for
  people enjoying Python from home)

- Student conference ticket: not available (please get day passes or a
  personal ticket instead)

Day Passes
--

As in the past, we will also sell day passes for the conference. These
allow attending the conference for a single day (July 8 - 14; valid on
the day you pick up the day pass badge):

- Business conference day pass: EUR 390.00 excl. VAT, EUR 420.03
  incl. 7.7% Swiss VAT (for people using Python to make a living)

- Personal conference day pass: EUR 260.00 incl. 7.7% Swiss VAT (for
  people enjoying Python from home)

- Student conference day pass: EUR 110.00 incl. 7.7% Swiss VAT (only
  available for pupils, students and postdoctoral researchers; please
  bring your student card or declaration from University, stating your
  affiliation, starting and end dates of your contract)

Day passes are available starting today, Wednesday, July 3.

Please see the registration page for full details of what is included
in the ticket price. Also note that neither late bird tickets, nor day
passes are refundable.


Dates and Venues


EuroPython will be held from July 8-14 2019 in Basel, Switzerland, at
the Congress Center Basel (CCB) for the main conference days (Wed-Fri)
and the FHNW Muttenz for the workshops/trainings/sprints days
(Mon-Tue, Sat-Sun).

The schedule is available at:

https://ep2019.europython.eu/events/schedule/

Tickets can be purchased on our registration page:

https://ep2019.europython.eu/registration/buy-tickets/

For more details, please have a look at our website and the FAQ:

https://ep2019.europython.eu/faq


Help spread the word


Please help us spread this message by sharing it on your social
networks as widely as possible. Thank you !

Link to the blog post:

https://blog.europython.eu/post/186029753397/europython-2019-late-bird-rates-and-day-passes

Tweet:

https://twitter.com/europython/status/1146486458162864130


Enjoy,
--
EuroPython 2019 Team
https://ep2019.europython.eu/
https://www.europython-society.org/

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Multiprocessing and memory management

[Gary Herron]

On 7/3/19 9:37 AM, ijbrews...@alaska.edu wrote:

I have a script that benefits greatly from multiprocessing (it’s generating a 
bunch of images from data). Of course, as expected each process uses a chunk of 
memory, and the more processes there are, the more memory used. The amount used 
per process can vary from around 3 GB (yes, gigabytes) to over 40 or 50 GB, 
depending on the amount of data being processed (usually closer to 10GB, the 
40/50 is fairly rare). This puts me in a position of needing to balance the 
number of processes with memory usage, such that I maximize resource 
utilization (running one process at a time would simply take WAY to long) while 
not overloading RAM (which at best would slow things down due to swap).

Obviously this process will be run on a machine with lots of RAM, but as I 
don’t know how large the datasets that will be fed to it are, I wanted to see 
if I could build some intelligence into the program such that it doesn’t 
overload the memory. A couple of approaches I thought of:

1) Determine the total amount of RAM in the machine (how?), assume an average 
of 10GB per process, and only launch as many processes as calculated to fit. 
Easy, but would run the risk of under-utilizing the processing capabilities and 
taking longer to run if most of the processes were using significantly less 
than 10GB



Try psutil to get information about memory (and cpu usage and lots 
more).  For example:


>>> import psutil
>>> psutil.virtual_memory()
svmem(total=16769519616, available=9151971328, percent=45.4, 
used=7031549952, free=4486520832, active=9026158592, 
inactive=2238566400, buffers=312815616, cached=4938633216, 
shared=234295296, slab=593375232)


Home page: https://github.com/giampaolo/psutil




2) Somehow monitor the memory usage of the various processes, and if one 
process needs a lot, pause the others until that one is complete. Of course, 
I’m not sure if this is even possible.

3) Other approaches?


---
Israel Brewster
Software Engineer
Alaska Volcano Observatory
Geophysical Institute - UAF
2156 Koyukuk Drive
Fairbanks AK 99775-7320
Work: 907-474-5172
cell:  907-328-9145


--
Dr. Gary Herron
Professor of Computer Science
DigiPen Institute of Technology
(425) 895-4418

--
https://mail.python.org/mailman/listinfo/python-list

FCNLT Module in Windows

[Shakeel Ahmad]

Sent from Mail for Windows 10

-- 
https://mail.python.org/mailman/listinfo/python-list

Multiprocessing and memory management

[Israel Brewster]

I have a script that benefits greatly from multiprocessing (it’s generating a 
bunch of images from data). Of course, as expected each process uses a chunk of 
memory, and the more processes there are, the more memory used. The amount used 
per process can vary from around 3 GB (yes, gigabytes) to over 40 or 50 GB, 
depending on the amount of data being processed (usually closer to 10GB, the 
40/50 is fairly rare). This puts me in a position of needing to balance the 
number of processes with memory usage, such that I maximize resource 
utilization (running one process at a time would simply take WAY to long) while 
not overloading RAM (which at best would slow things down due to swap). 

Obviously this process will be run on a machine with lots of RAM, but as I 
don’t know how large the datasets that will be fed to it are, I wanted to see 
if I could build some intelligence into the program such that it doesn’t 
overload the memory. A couple of approaches I thought of:

1) Determine the total amount of RAM in the machine (how?), assume an average 
of 10GB per process, and only launch as many processes as calculated to fit. 
Easy, but would run the risk of under-utilizing the processing capabilities and 
taking longer to run if most of the processes were using significantly less 
than 10GB

2) Somehow monitor the memory usage of the various processes, and if one 
process needs a lot, pause the others until that one is complete. Of course, 
I’m not sure if this is even possible.

3) Other approaches?


---
Israel Brewster
Software Engineer
Alaska Volcano Observatory 
Geophysical Institute - UAF 
2156 Koyukuk Drive 
Fairbanks AK 99775-7320
Work: 907-474-5172
cell:  907-328-9145

-- 
https://mail.python.org/mailman/listinfo/python-list

EuroPython 2019: Find a new job at the conference

[M.-A. Lemburg]

We’d like to draw your attention to our job board, with plenty of job
ads from our sponsors:

  EuroPython 2019 Job Board

 * https://ep2019.europython.eu/sponsor/job-board/ *

Our sponsors would love to get in touch with you, so please have a look
and visit them at their booth or contact them via the links and email
addresses given on the page.

Job ad emails
-

We will also send out job ad emails to attendees who have agreed to
receiving these emails. If you are interested, please log in, go to your
profile and enable the recruiting email option in the privacy section:

https://ep2019.europython.eu/user-panel/privacy-settings/

Note that we will not give your email addresses to sponsors, but only
send out these emails on behalf of them.


Dates and Venues


EuroPython will be held from July 8-14 2019 in Basel, Switzerland, at
the Congress Center Basel (CCB) for the main conference days (Wed-Fri)
and the FHNW Muttenz for the workshops/trainings/sprints days
(Mon-Tue, Sat-Sun).

The schedule is available at:

https://ep2019.europython.eu/events/schedule/

Tickets can be purchased on our registration page:

https://ep2019.europython.eu/registration/buy-tickets/

For more details, please have a look at our website and the FAQ:

https://ep2019.europython.eu/faq


Help spread the word


Please help us spread this message by sharing it on your social
networks as widely as possible. Thank you !

Link to the blog post:

https://blog.europython.eu/post/18602032/europython-2019-find-a-new-job-at-the-conference

Tweet:

https://twitter.com/europython/status/1146342161354543106


Enjoy,
--
EuroPython 2019 Team
https://ep2019.europython.eu/
https://www.europython-society.org/


-- 
https://mail.python.org/mailman/listinfo/python-list