Re: [Tutor] code review
Post it somewhere on github and I'll try to take a look at it. Lukas On 06/10/2014 05:51 PM, Adam Gold wrote: Hi there. I've been writing a script that is now finished and working (thanks, in part, to some helpful input from this board). What I'd really like to do now is go through it with an 'expert' who can point out ways I may have been able to code more efficiently/effectively. I don't think it would be appropriate to post the whole script here and ask how could I do this better (!) so I was wondering if anyone knows of ways for python noobs to connect with python experts for this sort of exercise. I understand people can be really busy so I'm happy to pay for someone's time if necessary. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] code review
On 11/06/14 00:04, Steven D'Aprano wrote: On Tue, Jun 10, 2014 at 04:51:20PM +0100, Adam Gold wrote: Hi there. I've been writing a script that is now finished and working (thanks, in part, to some helpful input from this board). What I'd really like to do now is go through it with an 'expert' who can point out ways I may have been able to code more efficiently/effectively. I don't think it would be appropriate to post the whole script here and ask how could I do this better (!) so I was wondering if anyone knows of ways for python noobs to connect with python experts for this sort of exercise. I understand people can be really busy so I'm happy to pay for someone's time if necessary. How big is the script? A single file, or hundreds of files? Fifty lines of code? A thousand? Thanks for the reply Steven. It's no more than 100 lines at a guess (I'm quite a noob although I think/hope that the brevity of the script may be partially attributable to some choices I made). In simple English, what does it do? Does it require specialized knowledge to understand? It's a backup script, more specifically: - create lv snapshots of pre-selected logical volumes on my server which are running xen VMs - dd and bzip2 (using a pipe) the snapshots to .img.bz2 files for storage on the same server - gpg encrypt the same files and uploads them to s3 - removes the lv snapshots and the .gpg files - deletes files in the S3 directory which are older than X days Is it available somewhere on the Internet? E.g. on Google code, github, sourceforge, your own personal website? Are there confidentiality restrictions on it? The are no restrictions (license or otherwise), no confidentiality issues. The file is not currently available on the internet but I can make it available easily enough on pastebin, as a downloadable file etc. The answer to these questions will influence the type of code review you get. I look forward to hearing further thoughts. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python sockets
Ok, so after making the changes the code does bind the startingPort variable but that is the only port that gets bound. Also when connecting to the startingPort I receive the following error: Please enter starting port: 65520 listening... ...connected! Traceback (most recent call last): File response.py, line 31, in module thread.start_new_thread(setup(port)) TypeError: start_new_thread expected at least 2 arguments, got 1 On Tue, Jun 10, 2014 at 4:23 PM, Marc Tompkins marc.tompk...@gmail.com wrote: On Tue, Jun 10, 2014 at 9:28 AM, Jon Engle jon.en...@gmail.com wrote: for port in range (startingPort, 65535): thread.start_new_thread(setup, (port,)) startingPort=startingPort+1 #print startingPort I think you just need this: for port in range (startingPort, 65535): thread.start_new_thread(setup(port)) #print port and inside of setup, get rid of this line: PORT = startingPort#arbitrary port not currently in use -- Cheers, Jon S. Engle jon.en...@gmail.com ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python sockets
Ok, so when I run the code it immediately terminates and never 'listens' to
the ports in the loop. I have verified by running netstat -an | grep 65530
and the startingPort is not binding.
***Server***
Jons-Mac:Desktop Jon$ python response.py
Please enter starting port: 65530
Jons-Mac:Desktop Jon$
Jons-Mac:Desktop Jon$ netstat -an | grep 65530
Jons-MacDesktop Jon$
***Code***
#!/usr/bin/python # This is server.py file
from socket import * #import the socket library
import thread #import the thread library
startingPort=input(\nPlease enter starting port: )
startingPort=int(startingPort)
def setup(PORT):
##let's set up some constants
HOST = ''#we are the host
#PORT = startingPort#arbitrary port not currently in use
ADDR = (HOST,PORT)#we need a tuple for the address
BUFSIZE = 4096#reasonably sized buffer for data
## now we create a new socket object (serv)
## see the python docs for more information on the socket types/flags
serv = socket( AF_INET,SOCK_STREAM)
##bind our socket to the address
serv.bind((ADDR))#the double parens are to create a tuple with one
element
serv.listen(5)#5 is the maximum number of queued connections we'll allow
print 'listening...'
conn,addr = serv.accept() #accept the connection
print '...connected!'
conn.send('TEST')
conn.close()
for port in range (startingPort, 65535):
thread.start_new_thread(setup, (port,))
startingPort=startingPort+1
#print startingPort
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] code review
On 11/06/14 00:30, Adam Gold wrote: Thanks for the reply Steven. It's no more than 100 lines at a guess In that case just copy and paste it into a message and send it to the group. Anyone with time available can then take a peek. hth -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python sockets
On 11/06/14 00:08, Jon Engle wrote: Ok, so when I run the code it immediately terminates and never 'listens' This has nothing to do with your immediate problem but... ***Code*** #!/usr/bin/python # This is server.py file from socket import * #import the socket library import thread#import the thread library startingPort=input(\nPlease enter starting port: ) startingPort=int(startingPort) You said you were using Python 2.7. Do not use input() in Python v2 use raw_input() instead. Especially since you are using int() to convert the data anyway. v2 input() evaluates whatever the user types which could be damaging code (whether deliberate or inadvertent). v2 input() is so dangerous it was removed from v3 and raw_input renamed as input. (Which has just created lots of confusion IMHO!) def setup(PORT): ##let's set up some constants HOST = ''#we are the host BTW I'm still losing nearly all indentation on your posts. Are you posting in plain text? Nobody else is complaining so it might just be my reader that's barfing on it... -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python sockets
On Tue, Jun 10, 2014 at 4:08 PM, Jon Engle jon.en...@gmail.com wrote:
Ok, so when I run the code it immediately terminates and never 'listens' to
the ports in the loop. I have verified by running netstat -an | grep 65530
and the startingPort is not binding.
The problem is that all threads started by a program terminate when
the program terminates - and you haven't told your program to stick
around when it's done setting up.So it's setting up and then
immediately exiting - and by the time you run netstat a few seconds
later you find nothing. (Also, by leaving HOST = '', you're listening
at address 0.0.0.0, so good luck catching any traffic...)
Try something like this:
#!/usr/bin/python # This is server.py file
from socket import * #import the socket library
import thread #import the thread library
def setup(PORT):
HOST = '127.0.0.1'#we are the host
ADDR = (HOST,PORT)#we need a tuple for the address
BUFSIZE = 4096#reasonably sized buffer for data
serv = socket( AF_INET,SOCK_STREAM)
serv.bind((ADDR))#the double parens are to create a tuple with
one element
serv.listen(5)#5 is the maximum number of queued connections we'll allow
print '\nlistening on port %i...' % PORT
conn,addr = serv.accept() #accept the connection
print '\n...port %i connected!' % PORT
conn.send('TEST')
conn.close()
def main():
startingPort=int(raw_input(\nPlease enter starting port: ))
for port in range (startingPort, 65535):
thread.start_new_thread(setup, (port,))
quitNow = ''
while quitNow not in ('Q', 'q'):
quitNow = raw_input('Enter Q to quit.')
if __name__ == '__main__':
main()
This will stick around until you enter 'Q', and if you run netstat in
another window you'll see that it's LISTENING on all the ports you
asked for. (All of those print statements will show up in a
surprising order!)
I'm not running the other side of this experiment, so I haven't tested
a successful connection... good luck.
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] code review
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 11/06/14 08:11, Alan Gauld wrote: On 11/06/14 00:30, Adam Gold wrote: Thanks for the reply Steven. It's no more than 100 lines at a guess In that case just copy and paste it into a message and send it to the group. Anyone with time available can then take a peek. hth One way noobs anywhere can learn is by listening in to other people's conversations - it's called lurking, I believe. So I would say, please do this on the list, and many more people than Adam may benefit. Others can ignore the thread if they wish. Bob - -- Bob Williams System: Linux 3.11.10-11-desktop Distro: openSUSE 13.1 (x86_64) with KDE Development Platform: 4.13.1 Uptime: 06:00am up 3 days 11:36, 3 users, load average: 0.05, 0.03, 0.05 -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlOYFfkACgkQ0Sr7eZJrmU5EUwCgkvjIWEdp1AzodJj6j5fY5yAL wtsAoKFSwk2U4kq5HW5KsmeErH+9fcXI =lJCF -END PGP SIGNATURE- ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python sockets
Jon Engle wrote:
Ok, so when I run the code it immediately terminates and never 'listens'
to the ports in the loop. I have verified by running netstat -an | grep
65530 and the startingPort is not binding.
As I've already hinted the easiest way to keep your listening threads alive
is to use the threading instead of the thread module:
$ cat bind_ports.py
#!/usr/bin/python
import socket
import threading
import sys
HOST = ''
STARTPORT = int(sys.argv[1])
ENDPORT = int(sys.argv[2])
def setup(port):
print 'setting up port', port
addr = (HOST, port)
serv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
serv.bind(addr)
serv.listen(1)
conn, addr = serv.accept()
print 'port', port, '...connected!'
conn.sendall('TEST')
conn.close()
print 'port', port, '...CLOSED!'
if __name__ == __main__:
for port in range(STARTPORT, ENDPORT):
threading.Thread(target=setup, args=(port,)).start()
$ python bind_ports.py 0 3
[1] 9214
$ setting up port 0
setting up port 1
setting up port 2
netstat -an | grep
tcp0 0 0.0.0.0:0 0.0.0.0:* LISTEN
tcp0 0 0.0.0.0:1 0.0.0.0:* LISTEN
tcp0 0 0.0.0.0:2 0.0.0.0:* LISTEN
To test it I've applied a minor modification to the client of the echoserver
example in https://docs.python.org/2/library/socket.html#example
$ cat read_socket.py
import socket
import sys
HOST = 'localhost'
PORT = int(sys.argv[1])
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
s.sendall('Hello, world')
data = s.recv(1024)
s.close()
print 'Received', repr(data)
$ python read_socket.py 0
port 0 ...connected!
port 0 ...CLOSED!
Received 'TEST'
$ python read_socket.py 2
port 2 ...connected!
port 2 ...CLOSED!
Received 'TEST'
$ python read_socket.py 1
port 1 ...connected!
port 1 ...CLOSED!
Received 'TEST'
$ fg
bash: fg: Programm ist beendet.
[1]+ Fertig python bind_ports.py 0 3
$
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] [OT] Long delay until my posts appear
Alan Gauld wrote: On 10/06/14 09:43, Peter Otten wrote: I'm posting via gmane. Since last month there is a delay (usually a few hours I think) until my posts appear and I seem to be getting a Your message to Tutor awaits moderator approval, would you like to cancel... Something has changed on the list server which means that any time someone edits their settings it automatically sets up full moderation. Presumably you changed a setting somewhere... Anyway I have now gone in and switched off moderation for your account. Normal service should have resumed. It has! Thank you. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python sockets
Alan Gauld wrote: On 11/06/14 00:08, Jon Engle wrote: Ok, so when I run the code it immediately terminates and never 'listens' This has nothing to do with your immediate problem but... ***Code*** #!/usr/bin/python # This is server.py file from socket import * #import the socket library import thread#import the thread library startingPort=input(\nPlease enter starting port: ) startingPort=int(startingPort) You said you were using Python 2.7. Do not use input() in Python v2 use raw_input() instead. Especially since you are using int() to convert the data anyway. v2 input() evaluates whatever the user types which could be damaging code (whether deliberate or inadvertent). v2 input() is so dangerous it was removed from v3 and raw_input renamed as input. (Which has just created lots of confusion IMHO!) def setup(PORT): ##let's set up some constants HOST = ''#we are the host BTW I'm still losing nearly all indentation on your posts. Are you posting in plain text? Nobody else is complaining so it might just be my reader that's barfing on it... No, it's not just you, I don't see any indentation either. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] [OT] Long delay until my posts appear
On 11/06/14 09:49, Peter Otten wrote: Something has changed on the list server which means that any time someone edits their settings it automatically sets up full moderation. Normal service should have resumed. It has! Thank you. Good, now if only I could find a way to stop the server switching on moderation for anything except new members joining... -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python sockets
Thank you for your help, this definitely gets me going in the right
direction!
On Wed, Jun 11, 2014 at 4:16 AM, Marc Tompkins marc.tompk...@gmail.com
wrote:
On Tue, Jun 10, 2014 at 4:08 PM, Jon Engle jon.en...@gmail.com wrote:
Ok, so when I run the code it immediately terminates and never 'listens'
to
the ports in the loop. I have verified by running netstat -an | grep
65530
and the startingPort is not binding.
The problem is that all threads started by a program terminate when
the program terminates - and you haven't told your program to stick
around when it's done setting up.So it's setting up and then
immediately exiting - and by the time you run netstat a few seconds
later you find nothing. (Also, by leaving HOST = '', you're listening
at address 0.0.0.0, so good luck catching any traffic...)
Try something like this:
#!/usr/bin/python # This is server.py file
from socket import * #import the socket library
import thread #import the thread library
def setup(PORT):
HOST = '127.0.0.1'#we are the host
ADDR = (HOST,PORT)#we need a tuple for the address
BUFSIZE = 4096#reasonably sized buffer for data
serv = socket( AF_INET,SOCK_STREAM)
serv.bind((ADDR))#the double parens are to create a tuple with
one element
serv.listen(5)#5 is the maximum number of queued connections we'll
allow
print '\nlistening on port %i...' % PORT
conn,addr = serv.accept() #accept the connection
print '\n...port %i connected!' % PORT
conn.send('TEST')
conn.close()
def main():
startingPort=int(raw_input(\nPlease enter starting port: ))
for port in range (startingPort, 65535):
thread.start_new_thread(setup, (port,))
quitNow = ''
while quitNow not in ('Q', 'q'):
quitNow = raw_input('Enter Q to quit.')
if __name__ == '__main__':
main()
This will stick around until you enter 'Q', and if you run netstat in
another window you'll see that it's LISTENING on all the ports you
asked for. (All of those print statements will show up in a
surprising order!)
I'm not running the other side of this experiment, so I haven't tested
a successful connection... good luck.
--
Cheers,
Jon S. Engle
jon.en...@gmail.com
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] code review
Thanks for the reply Steven. It's no more than 100 lines at a guess In that case just copy and paste it into a message and send it to the group. Anyone with time available can then take a peek. One way noobs anywhere can learn is by listening in to other people's conversations - it's called lurking, I believe. So I would say, please do this on the list, and many more people than Adam may benefit. Others can ignore the thread if they wish. Bob Oke doke, here it is below. Just for convenience's sake, I'm going to repeat what the basic steps are. It's a backup script for certain xen virtual machines (VM) running on my server. Each VM runs on its own logical volume (as opposed to a file-based loop device). From my own (bitter) experience, the absolutely best way to back up a VM running on a logical volume is to clone it to an image file using dd. I'm aware that a separate discussion could be had around this (on a different mailing list) but, unless someone thinks this is a horribly flawed approach, it may be best to assume this approach is 'fine' so as not to distract from the code review!! Here are the steps: 1) create snapshots of the xen logical volumes using the built in snapshot feature of LVM2 (this way I can backup each logical volume without having to shut down the VM) 2) dd and bzip2 (using a pipe) the snapshots to .img.bz2 files for storage on the same server 3) gpg encrypt the same files and upload them to Amazon s3 4) remove the logical volume snapshots (because they accumulate disk space and I'm doing this daily) and the .gpg files 5) deletes files in the s3 directory which are older than X days As I've mentioned, I'm a real noob, so I'm still mastering some basic stuff. The script works fine for my purposes, I'm keen to understand where it could be improved from a python pov. Finally, yes I could have written this in bash but I prefer python! P.S. I think some of the comments have been wrapped onto more than one line by my email client, I hope this doesn't cause too much inconvenience. #!/usr/bin/python3 ## XEN VIRTUAL MACHINE BACKUP SCRIPT ## ## Copyright (C) 2014 Adam Gold ## ## This program is free software: you can redistribute it and/or modify ## it under the terms of the GNU General Public License as published by ## the Free Software Foundation, either version 3 of the License, or (at ## your option) any later version. ## ## This program is distributed in the hope that it will be useful, but ## WITHOUT ANY WARRANTY; without even the implied warranty of ## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See ## the GNU General Public License for more details. ## ## You should have received a copy of the GNU General Public License ## along with this program. If not see http://gnu.org/licenses/ ## ## Version: 0.4 ## 2014-06-10 import datetime, time, subprocess, shlex, os, gnupg, glob, shutil # logical volumes exist in two different volume groups, vgxen and vg_data # hence two lists of vms vgxenList = ['vm1', 'vm2', 'vm3', 'vm4', 'vm5', 'vm6' ] vg_dataList = ['vm1', 'vm2'] backupList = [ ] snapNameList = [ ] # create snapshot names like the following: 2014-06-10T01-00-01.vm1.img.bz2 for i in vgxenList: DATE = datetime.datetime.now().strftime(%Y-%m-%d + T + %H-%M-%S) vgxenName = /dev/vgxen/ lvName = i origName = vgxenName + lvName snapName= DATE + . + lvName snapNameList.append(snapName) backupList.append(vgxenName + snapName) subprocess.call(['lvcreate', '-s', '-L1G', origName, '-n', snapName]) for h in vg_dataList: DATE = datetime.datetime.now().strftime(%Y-%m-%d + T + %H-%M-%S) vg_dataName = /dev/vg_data/ lvName = h origName = vg_dataName + lvName snapName = DATE + . + lvName snapNameList.append(snapName) backupList.append(vg_dataName + snapName) subprocess.call(['lvcreate', '-s', '-L1G', origName, '-n', snapName]) # backupPath is list of full paths of each snapshot # the string is extacted from backupList using 'join' backupPath = ' '.join(backupList) for j, k in zip(backupList, snapNameList): backupPath = j backupSnapshot = k # run dd and pipe to bz2 file using subprocess module ddIf = shlex.split(dd if=%s bs=4k conv=noerror,notrunc,sync % (backupPath)) compress = pbzip2 filename = /home/files/temp/%s.img.bz2 % (backupSnapshot) p1 = subprocess.Popen(ddIf, stdout=subprocess.PIPE) with p1.stdout as fin, open(filename, w) as fout: p2 = subprocess.Popen(compress, stdin=fin, stdout=fout) ret1 = p1.wait() ret2 = p2.wait() # create list of files to be encrypted with full path names # start with list of unencrypted files cryptDir = '/home/files/temp/' unencrypted = [u for u in os.listdir(cryptDir)] # join absolute path to file names to create new list (list comprehension) cryptDir_unencrypted = [
Re: [Tutor] code review
Ok, not so bad, hoewer there are some parts of the code that could be
done a bit cleaner.
I'll write them below in the response.
Thanks for the reply Steven. It's no more than 100 lines at a
guess
In that case just copy and paste it into a message and send it to
the group. Anyone with time available can then take a peek.
One way noobs anywhere can learn is by listening in to other people's
conversations - it's called lurking, I believe.
So I would say, please do this on the list, and many more people than
Adam may benefit. Others can ignore the thread if they wish.
Bob
Oke doke, here it is below. Just for convenience's sake, I'm going to
repeat what the basic steps are. It's a backup script for certain xen
virtual machines (VM) running on my server. Each VM runs on its own
logical volume (as opposed to a file-based loop device). From my own
(bitter) experience, the absolutely best way to back up a VM running on
a logical volume is to clone it to an image file using dd. I'm aware
that a separate discussion could be had around this (on a different
mailing list) but, unless someone thinks this is a horribly flawed
approach, it may be best to assume this approach is 'fine' so as not to
distract from the code review!!
Here are the steps:
1) create snapshots of the xen logical volumes using the built in
snapshot feature of LVM2 (this way I can backup each logical volume
without having to shut down the VM)
2) dd and bzip2 (using a pipe) the snapshots to .img.bz2 files for
storage on the same server
3) gpg encrypt the same files and upload them to Amazon s3
4) remove the logical volume snapshots (because they accumulate disk
space and I'm doing this daily) and the .gpg files
5) deletes files in the s3 directory which are older than X days
As I've mentioned, I'm a real noob, so I'm still mastering some basic
stuff. The script works fine for my purposes, I'm keen to understand
where it could be improved from a python pov. Finally, yes I could have
written this in bash but I prefer python!
P.S. I think some of the comments have been wrapped onto more than one
line by my email client, I hope this doesn't cause too much inconvenience.
#!/usr/bin/python3
## XEN VIRTUAL MACHINE BACKUP SCRIPT
##
## Copyright (C) 2014 Adam Gold
##
## This program is free software: you can redistribute it and/or modify
## it under the terms of the GNU General Public License as published by
## the Free Software Foundation, either version 3 of the License, or (at
## your option) any later version.
##
## This program is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See
## the GNU General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with this program. If not see http://gnu.org/licenses/
##
## Version: 0.4
## 2014-06-10
import datetime, time, subprocess, shlex, os, gnupg, glob, shutil
imports should be one module per line and alphabetically- more readable:
import datetime
import glob
import gnupg
import os
import shlex
import shutil
import subprocess
import time
# logical volumes exist in two different volume groups, vgxen and vg_data
# hence two lists of vms
vgxenList = ['vm1', 'vm2', 'vm3', 'vm4', 'vm5', 'vm6' ]
vg_dataList = ['vm1', 'vm2']
backupList = [ ]
snapNameList = [ ]
# create snapshot names like the following: 2014-06-10T01-00-01.vm1.img.bz2
for i in vgxenList:
DATE = datetime.datetime.now().strftime(%Y-%m-%d + T + %H-%M-%S)
vgxenName = /dev/vgxen/
lvName = i
origName = vgxenName + lvName
snapName= DATE + . + lvName
snapNameList.append(snapName)
backupList.append(vgxenName + snapName)
subprocess.call(['lvcreate', '-s', '-L1G', origName, '-n', snapName])
for h in vg_dataList:
DATE = datetime.datetime.now().strftime(%Y-%m-%d + T + %H-%M-%S)
vg_dataName = /dev/vg_data/
lvName = h
origName = vg_dataName + lvName
snapName = DATE + . + lvName
snapNameList.append(snapName)
backupList.append(vg_dataName + snapName)
subprocess.call(['lvcreate', '-s', '-L1G', origName, '-n', snapName])
Here in the two loops, you can notice they're almost the same ...
NOTE: I thing you're dealing with file paths, you should use os.path and
os.path.join instead of string conactenation
and you should also check if the file exists first before
calling subprocess ... you can figure out how yourself
Basic DRY (don't repeat yourself) principle - distill their essence into
a function or more:
def snapshot_name(item, name):
:param item: string (one of 'vm1', 'vm2' ..)
:param name: string ('/dev/vgxen/')
isodate = datetime.datetime.now().isoformat() # notice datetime
already has this time format
[Tutor] A couple newbie questions about Python
Hi. Everywhere I have read, the 'standard practice' for indentation is 4 spaces, but I am running into 2 space indentation in a lot of tutorials and such. Should I keep with the 4 spaces, or does it even matter, as long as it is consistent? I just recently became aware of the inaccuracy of calculations using floats and I am concerned about that. I am using Python 3, just fyi. Thanks for any enlightenment on these questions. Deb in WA, USA FREE 3D MARINE AQUARIUM SCREENSAVER - Watch dolphins, sharks orcas on your desktop! Check it out at http://www.inbox.com/marineaquarium ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] A couple newbie questions about Python
On 11/06/14 21:46, Deb Wyatt wrote: Hi. Everywhere I have read, the 'standard practice' for indentation is 4 spaces, That's a style recommendation. Python doesn't care. But your readers will. 2 spaces is the absolute minimum, 8 spaces is about the absolute maximum. Outside that it gets hard to read the code. 3, 4 or 5 spaces is pretty good and you are unlikely to get complaints except from style pedants or if you are submitting code for the standard library where they like to stick with the official guidance. I just recently became aware of the inaccuracy of calculations using floats and I am concerned about that. The inaccuracies are an inevitable result of the way your computer processes floating point numbers. In what way are you concerned? Its not a Python issue - the exact same issues occur in any computer - or even your pocket calculator. If it concerns you in Excel or in Visual Basic/Java/C++ etc too then the good news is that Python has alternative renderings that can often reduce/eliminate them - but at the cost of speed and complexity. HTH -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] code review
On 11/06/14 11:43, Adam Gold wrote: # create snapshot names like the following: 2014-06-10T01-00-01.vm1.img.bz2 for i in vgxenList: DATE = datetime.datetime.now().strftime(%Y-%m-%d + T + %H-%M-%S) Why use addition? You could just insett the literal T... DATE = datetime.datetime.now().strftime(%Y-%m-%dT%H-%M-%S) Also using all caps for the name suggests a constant by convention. vgxenName = /dev/vgxen/ lvName = i why not just say for lvname in vxgenlist and avoid the extra assignment? origName = vgxenName + lvName snapName= DATE + . + lvName Again you could have done that in the original strftime() call. snapNameList.append(snapName) backupList.append(vgxenName + snapName) subprocess.call(['lvcreate', '-s', '-L1G', origName, '-n', snapName]) for h in vg_dataList: DATE = datetime.datetime.now().strftime(%Y-%m-%d + T + %H-%M-%S) see above vg_dataName = /dev/vg_data/ lvName = h again, why not just use the meaningful name in the for loop? origName = vg_dataName + lvName snapName = DATE + . + lvName snapNameList.append(snapName) backupList.append(vg_dataName + snapName) subprocess.call(['lvcreate', '-s', '-L1G', origName, '-n', snapName]) In fact these loops are so similar you should probably make a function with a couple of parameters and call it from both loops. The only things different are the list you loop over and the path. Make those params and you get something like this (untested): def create_volumes(volList, path): for name in volList: dt = datetime.datetime.now().strftime(%Y-%m-%dT%H-%M-%S.+name) dataName = path origName = dataName + name snapName = DATE + . + name snapNameList.append(snapName) backupList.append(dataName + snapName) subprocess.call(['lvcreate', '-s', '-L1G', origName, '-n', snapName]) # backupPath is list of full paths of each snapshot # the string is extacted from backupList using 'join' backupPath = ' '.join(backupList) for j, k in zip(backupList, snapNameList): backupPath = j backupSnapshot = k # run dd and pipe to bz2 file using subprocess module ddIf = shlex.split(dd if=%s bs=4k conv=noerror,notrunc,sync % (backupPath)) compress = pbzip2 filename = /home/files/temp/%s.img.bz2 % (backupSnapshot) p1 = subprocess.Popen(ddIf, stdout=subprocess.PIPE) with p1.stdout as fin, open(filename, w) as fout: p2 = subprocess.Popen(compress, stdin=fin, stdout=fout) Take note of the warning in the subprocess documentation about using stdin/stdout directly. You should ideally access them via Popen.communicate(). ret1 = p1.wait() ret2 = p2.wait() I may just be missing it but I don't see you do anything with these ret1,ret2 variables? # create list of files to be encrypted with full path names # start with list of unencrypted files cryptDir = '/home/files/temp/' unencrypted = [u for u in os.listdir(cryptDir)] This is a redundant comprehension. listdir() returns the list for you. Any time you see foo = [item for item in a_list] it probably should just be: foo = a_list Or if a_list is really an iterator it could be foo = list(an_iter) Comprehensions are good when you have a filter condition on the end or if you are returning a function/expression involving item. # join absolute path to file names to create new list (list comprehension) cryptDir_unencrypted = [ os.path.join(cryptDir, s) for s in unencrypted ] Like that one for example... # encrypt files for G in cryptDir_unencrypted: You seem to like single letter loop variables. Its better to use something more meaningful. Say: for filename in cryptDir_unencrypted: gpg = gnupg.GPG(gnupghome='/root/.gnupg') phrase = passphrase # HORRIBLE SECURITY, I KNOW! The script is running as a cronjob so I can't interactively enter the passphrase. Suggestions are welcome. Use a config file that you can edit when needed. Read the passwords from that. It could even be encrypted... cipher = AES256 with open(G, 'rb') as f: status = gpg.encrypt_file(f, None, armor=False, passphrase=phrase, symmetric=cipher.upper(), output=G + '.gpg') # move unencypted files out of temp directory for data in glob.glob(cryptDir + '*.bz2'): You should really use os.join() instead of string addition... shutil.move(data,'/home/files/') # delete snapshots for r in snapNameList: removeSnapshots1 = 'lvremove -f ' + vgxenName + r subprocess.call(shlex.split(removeSnapshots1)) removeSnapshots2 = 'lvremove -f ' + vg_dataName + r subprocess.call(shlex.split(removeSnapshots2)) # create list of file names to be uploaded (list comprehension) uploads = [y for y in os.listdir(cryptDir)] see previous comment # join absolute path to file names to create new list (list comprehension) cryptDir_uploads = [ os.path.join(cryptDir, t) for t in uploads ] #
Re: [Tutor] A couple newbie questions about Python
Deb Wyatt codemon...@inbox.com Wrote in message: Hi. Everywhere I have read, the 'standard practice' for indentation is 4 spaces, but I am running into 2 space indentation in a lot of tutorials and such. Should I keep with the 4 spaces, or does it even matter, as long as it is consistent? 4 spaces is an excellent choice in my opinion, and many other people's. We just tell our editor to turn the tab key into 4 column alignment and pretty much forget about it. The python interpreter doesn't care. But other people's opinions will matter as soon as you share your code, or work on a multi person project. Note that when you're looking at other people's code, you may be seeing it differently than they, chief reason being html (thank you for remembering to post here in text). The other reason you may think they're using 2 is proportional spacing. It should be off for code. I just recently became aware of the inaccuracy of calculations using floats and I am concerned about that. I learned programming in 1967 with Fortran, and McCracken spent a chapter warning about that same thing. Probably everything he warned about still applies to Python and modern computers. It is impossible to do serious computing for long without running into these issues. But Python has as many ways of avoiding them as any mainstream language. You can use decimal to avoid some types of problems, and fractions to avoid others. And common sense for others. You will need to understand your tools. BTW, calculators and spreadsheets frequently use decimal rather than binary, and I wrote a decimal floating package for a computer in the mid 70's. It didn't even offer binary. -- DaveA ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] A couple newbie questions about Python
Hi Deb, My responses below, interleaved with your questions. On Wed, Jun 11, 2014 at 12:46:11PM -0800, Deb Wyatt wrote: Hi. Everywhere I have read, the 'standard practice' for indentation is 4 spaces, but I am running into 2 space indentation in a lot of tutorials and such. Should I keep with the 4 spaces, or does it even matter, as long as it is consistent? Four spaces is common, and recommended by PEP 8, which describes the coding styles for the Python standard library: https://www.python.org/dev/peps/pep-0008 But it's not compulsory (except for contributions to the standard library). There are good reasons for sticking to four spaces, but eight spaces or a single tab are also common. In my opinion, two spaces is too little, and one space is right out. But as far as the Python interpreter is concerned, it doesn't care, so long as you're consistent. (However, other people reading your code may care.) I just recently became aware of the inaccuracy of calculations using floats and I am concerned about that. Floating point mathematics is tricky. What you're seeing is not a bug in Python, but a general problem with floating point calculations in general. Regardless of whether you use Python, or C, or Java, or some other programming language, you have to face the same issues that calculations with floats are not always accurate compared to what you expect from pure mathematics. (Think about your calculator: how often do you get a number like 3.001 instead of 3, or 0.499 instead of 0.5?) You can read up on some of the issues here: http://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html although it's quite technical. Some less technical discussions can be found here: http://randomascii.wordpress.com/2012/04/05/floating-point-complexities/ http://support.microsoft.com/kb/42980 but however you look at it, it's complicated, and inherent to the problem, not the fault of Python. -- Steven ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] A couple newbie questions about Python
On 2014-06-11 20:08, Dave Angel wrote: I learned programming in 1967 with Fortran, and McCracken spent a chapter warning about that same thing. Probably everything he warned about still applies to Python and modern computers. A google search for McCracken did not yield anyone that seems to fit the person you reference. Can you clarify, please? ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] A couple newbie questions about Python
Hi, responses below...
Dne 11. 6. 2014 22:46, Deb Wyatt napsal(a):
Hi. Everywhere I have read, the 'standard practice' for indentation is 4
spaces, but I am running into 2 space indentation in a lot of tutorials and
such. Should I keep with the 4 spaces, or does it even matter, as long as it
is consistent?
I recommend to use ammount of spaces that is consistent with already
existing project.
If it is a new project, use whatever you like - 4 spaces is recommended
(looks good :)
I just recently became aware of the inaccuracy of calculations using floats and
I am concerned about that.
If there are numerical operations where precision is necessary
(accounting or banking) use decimal type
from decimal import Decimal
a = Decimal('0.1')
b = Decimal('0.9')
print a + b
Note these numbers are passed to Decimal as a string, that is for a
reason - if you pass float to Decimal, it will already loose precision
and the whole thing doesn't make sense - and aslo will throw an exception.
I am using Python 3, just fyi.
Thanks for any enlightenment on these questions.
Deb in WA, USA
FREE 3D MARINE AQUARIUM SCREENSAVER - Watch dolphins, sharks orcas on your
desktop!
Check it out at http://www.inbox.com/marineaquarium
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Lukas
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
