[Tutor] a par2 creator and verification program
Hello everyone. I'm thinking through a short program I want to write that will 'par2'/generate ECCs for all of my work files which branch out from a single directory and number approximately 15,000. Specifically: 1) day one: - create a mirror copy of the directory tree empty of all files (there are a bunch of ways in bash of doing this). - recurse down the directory tree which has the files and run a par2 create calculation on each file which generates approximately 10 *.par2 fileblocks. I will then copy the *.par2 fileblocks to the mirror directory tree into the same position as the 'principal file. Therefore assuming 10 *.par2 fileblocks for every actual file, the mirror tree will have around 150,000 *.par2 fileblocks (space and CPU time are a non-issue). 2) day two: - for each file in the primary directory, par2 verify it with respect to its corresponding *.par2 fileblocks in the mirror tree. If it's ok, move on to the next file, if not, repair it, generate a new set of *.par2 fileblocks and copy them over to the mirror. 3) day three: - same as day two, ongoing. I'm aware that most par2 programs need the file and *.par blocks to be in the same location but let's assume I find a way around this. Also, I believe it would be possible to par2 the top directory (which will give me work1.par2 - work10.par2) but the problem is performed this way, the blocks treat all files as a single whole so if I detect corruption, I have no way of locating which file. I'm considering two ways of doing this: Option A: - This seems the most obvious if somewhat inelegant: define a few functions, and incorporate them into a for loop which will be applied to each file as described in 1) - 3) above. Option B: - I'm afraid my thinking is not entirely clear regards this option but somehow I import metadata for every (primary) file into a list (I think all that's needed is file name and location), perhaps even a nested list although I'm not sure if that provides an advantage. Then I apply the operations for 1) - 3) above sequentially per list item, the assumption being the list data and my home made functions will be sufficient. I've found various par2 programs on PyPi and possibly pyFileFixity could be used but in this instance I'd rather give it a go myself. For various reasons I can't use ZFS which would, of course, negate the need for doing any of this. It seems this would be my consolation prize :) ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] pexpect interactive password entry for luks device
I'm trying to do something really simply (I think) with pexpect but must be missing an obvious step. I want to ssh from host to guest, establish a CIFS connection on the guest to a share somewhere else on the local network, open a luks loop device on the share by entering the password (I don't want this to be non-interactive, either with a key file or by including the password in the pexpect script) and then mounting the device. I'm using the following code: password = getpass.getpass('Enter passphrase for /opt/luks: ') child = pexpect.spawn('ssh root@192.168.56.101') child.interact() child.sendline('mount.cifs -o credentials=/root/cred_f //192.168.56.1/f /opt') child.sendline('losetup /dev/loop0 /opt/luks') child.sendline('cryptsetup luksOpen /dev/loop0 l') child.expect('Enter passphrase for /opt/luks: ') child.sendline(password) child.sendline('mount /dev/mapper/l /opt') child.sendline('exit') The best that happens is the CIFS share gets mounted and the program exits while leaving me in the child terminal. I've tried varying the placement of child.interact() but that makes no difference or replacing the first line which uses getpass with a 'child.expect' line when the password is required. Could anyone help me? ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] code review
On 12/06/14 00:38, Alan Gauld wrote: > > HTH Thanks Alan and Lukáš for your very helpful comments. I will attempt to revise the script in light of them and will revert if I hit any brick walls :) ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] code review
>>> Thanks for the reply Steven. It's no more than 100 lines at a >>> guess > >> In that case just copy and paste it into a message and send it to >> the group. Anyone with time available can then take a peek. > > One way noobs anywhere can learn is by listening in to other people's > conversations - it's called lurking, I believe. > > So I would say, please do this on the list, and many more people than > Adam may benefit. Others can ignore the thread if they wish. > > Bob Oke doke, here it is below. Just for convenience's sake, I'm going to repeat what the basic steps are. It's a backup script for certain xen virtual machines ("VM") running on my server. Each VM runs on its own logical volume (as opposed to a file-based loop device). From my own (bitter) experience, the absolutely best way to back up a VM running on a logical volume is to clone it to an image file using dd. I'm aware that a separate discussion could be had around this (on a different mailing list) but, unless someone thinks this is a horribly flawed approach, it may be best to assume this approach is 'fine' so as not to distract from the code review!! Here are the steps: 1) create snapshots of the xen logical volumes using the built in snapshot feature of LVM2 (this way I can backup each logical volume without having to shut down the VM) 2) dd and bzip2 (using a pipe) the snapshots to .img.bz2 files for storage on the same server 3) gpg encrypt the same files and upload them to Amazon s3 4) remove the logical volume snapshots (because they accumulate disk space and I'm doing this daily) and the .gpg files 5) deletes files in the s3 directory which are older than X days As I've mentioned, I'm a real noob, so I'm still mastering some basic stuff. The script works fine for my purposes, I'm keen to understand where it could be improved from a python pov. Finally, yes I could have written this in bash but I prefer python! P.S. I think some of the comments have been wrapped onto more than one line by my email client, I hope this doesn't cause too much inconvenience. ==== #!/usr/bin/python3 ## XEN VIRTUAL MACHINE BACKUP SCRIPT ## ## Copyright (C) 2014 Adam Gold ## ## This program is free software: you can redistribute it and/or modify ## it under the terms of the GNU General Public License as published by ## the Free Software Foundation, either version 3 of the License, or (at ## your option) any later version. ## ## This program is distributed in the hope that it will be useful, but ## WITHOUT ANY WARRANTY; without even the implied warranty of ## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See ## the GNU General Public License for more details. ## ## You should have received a copy of the GNU General Public License ## along with this program. If not see <http://gnu.org/licenses/> ## ## Version: 0.4 ## 2014-06-10 import datetime, time, subprocess, shlex, os, gnupg, glob, shutil # logical volumes exist in two different volume groups, vgxen and vg_data # hence two lists of vms vgxenList = ['vm1', 'vm2', 'vm3', 'vm4', 'vm5', 'vm6' ] vg_dataList = ['vm1', 'vm2'] backupList = [ ] snapNameList = [ ] # create snapshot names like the following: 2014-06-10T01-00-01.vm1.img.bz2 for i in vgxenList: DATE = datetime.datetime.now().strftime("%Y-%m-%d" + "T" + "%H-%M-%S") vgxenName = "/dev/vgxen/" lvName = i origName = vgxenName + lvName snapName= DATE + "." + lvName snapNameList.append(snapName) backupList.append(vgxenName + snapName) subprocess.call(['lvcreate', '-s', '-L1G', origName, '-n', snapName]) for h in vg_dataList: DATE = datetime.datetime.now().strftime("%Y-%m-%d" + "T" + "%H-%M-%S") vg_dataName = "/dev/vg_data/" lvName = h origName = vg_dataName + lvName snapName = DATE + "." + lvName snapNameList.append(snapName) backupList.append(vg_dataName + snapName) subprocess.call(['lvcreate', '-s', '-L1G', origName, '-n', snapName]) # backupPath is list of full paths of each snapshot # the string is extacted from backupList using 'join' backupPath = ' '.join(backupList) for j, k in zip(backupList, snapNameList): backupPath = j backupSnapshot = k # run dd and pipe to bz2 file using subprocess module ddIf = shlex.split("dd if=%s bs=4k conv=noerror,notrunc,sync" % (backupPath)) compress = "pbzip2" filename = "/home/files/temp/%s.img.bz2" % (backupSnapshot)
Re: [Tutor] code review
On 11/06/14 00:04, Steven D'Aprano wrote: > On Tue, Jun 10, 2014 at 04:51:20PM +0100, Adam Gold wrote: >> Hi there. I've been writing a script that is now finished and working >> (thanks, in part, to some helpful input from this board). What I'd >> really like to do now is go through it with an 'expert' who can point >> out ways I may have been able to code more efficiently/effectively. I >> don't think it would be appropriate to post the whole script here and >> ask "how could I do this better" (!) so I was wondering if anyone knows >> of ways for python noobs to connect with python experts for this sort of >> exercise. I understand people can be really busy so I'm happy to pay >> for someone's time if necessary. > > > How big is the script? A single file, or hundreds of files? Fifty lines > of code? A thousand? Thanks for the reply Steven. It's no more than 100 lines at a guess (I'm quite a noob although I think/hope that the brevity of the script may be partially attributable to some choices I made). > In simple English, what does it do? Does it require specialized > knowledge to understand? It's a backup script, more specifically: - create lv snapshots of pre-selected logical volumes on my server which are running xen VMs - dd and bzip2 (using a pipe) the snapshots to .img.bz2 files for storage on the same server - gpg encrypt the same files and uploads them to s3 - removes the lv snapshots and the .gpg files - deletes files in the S3 directory which are older than X days > > Is it available somewhere on the Internet? E.g. on Google code, github, > sourceforge, your own personal website? Are there confidentiality > restrictions on it? The are no restrictions (license or otherwise), no confidentiality issues. The file is not currently available on the internet but I can make it available easily enough on pastebin, as a downloadable file etc. > The answer to these questions will influence the type of code review > you get. I look forward to hearing further thoughts. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] code review
Hi there. I've been writing a script that is now finished and working (thanks, in part, to some helpful input from this board). What I'd really like to do now is go through it with an 'expert' who can point out ways I may have been able to code more efficiently/effectively. I don't think it would be appropriate to post the whole script here and ask "how could I do this better" (!) so I was wondering if anyone knows of ways for python noobs to connect with python experts for this sort of exercise. I understand people can be really busy so I'm happy to pay for someone's time if necessary. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] gnupg within a for loop
On 02/06/14 01:35, Danny Yoo wrote: >> Thanks Danny, that was spot on. I actually used os.chdir to change to >> the base directory (which I assigned to a variable) just before the open >> statement. I don't know if that's 'pythonically' correct but it seemed >> like a simple way to do it. Again, thank you for helping me fix this. > > I am glad! I would recommend using os.path.join instead to build absolute > paths, as it's less fragile to external factors. I think of current > working directory the same way I think of global variables: use it > judiciously. Yup, makes sense, thanks for the advice, ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] gnupg within a for loop
On 01/06/14 18:28, Danny Yoo wrote: > Hi Adam, > > Ah; I've seen this before. Make sure the file name is either relative > to current working directory, or make the file name absolute. What's > happening is that os.listdir() is giving you file names that are > relative to the base directory you've passed it, but open() doesn't > know about the base directory. > > You can use os.path.join() to create such paths: > > https://docs.python.org/2/library/os.path.html#os.path.join > Thanks Danny, that was spot on. I actually used os.chdir to change to the base directory (which I assigned to a variable) just before the open statement. I don't know if that's 'pythonically' correct but it seemed like a simple way to do it. Again, thank you for helping me fix this. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] gnupg within a for loop
Hi there. I'm trying to do the following using python 3: create a list from all files in a particular directory, then loop over that list symmetrically encrypting each file using the gnupg module (note, for the moment I'm embedding the passphrase in the code while obviously in practice it would have to be inputted by the user). I start with the following which can be used to encrypt a single file (assume I have the indentations correct in the actual code, I can't seem to modify the wrapping with my email client): phrase = '12345' cipher = 'AES256' gpg = gnupg.GPG(gnupghome='/home/adam/.gnupg') with open('/home/adam/file1', 'rb') as f: status = gpg.encrypt_file(f, None, passphrase=phrase, symmetric=cipher.upper(), output='/home/adam/file1.gpg') This produces a single encrypted file, 'file1.gpg'. I then try to embed this in a for loop: unencrypted = [u for u in os.listdir('/home/adam/temp')] for G in unencrypted: gpg = gnupg.GPG(gnupghome='/home/adam/.gnupg') phrase = '12345' cipher = 'AES256' with open (G, 'rb') as f: status = gpg.encrypt_file(f, None, passphrase=phrase, symmetric=cipher.upper(), output=G + '.gpg') This produces the following error: Traceback (most recent call last): File "", line 5, in with open (G, 'rb') as f: FileNotFoundError: [Errno 2] No such file or directory: 'file1' It seems 'G', which I'm intending to represent each successive element of the list as the for loop iterates, does not find its way into the gnupg code lines. I have a feeling I'm missing something really basic here :) ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] pipes and redirecting
> On 27/05/14 21:01, Adam Gold wrote: > >> "dd if=/home/adam/1 bs=4k conv=noerror,notrunc,sync | pbzip2 > 1.img.bz2" >> >> The first thing I do is break it into two assignments > > And that's the start of the problem because it should be three: > The first command, the second command and the output file. > >> ddIf = shlex.split("dd if=/home/adam/1 bs=4k conv=noerror,notrunc,sync") >> compress = shlex.split("pbzip2 > /home/adam/1.img.bz2") > > compress = "pbzip2" > outfile = open('/home/adam/1.img.bz2','w') > > The redirection symbol is not something subprocess can > use as an argument. > >> p1 = subprocess.Popen(ddIf, stdout=subprocess.PIPE) >> p2 = subprocess.Popen(compress, stdin=p1.stdout, stdout=subprocess.PIPE) > > Use the output file here. > > p2 = subprocess.Popen(compress, stdin=p1.stdout, stdout=outfile) > > >> I think that the '>' redirect needs to be dealt with using the >> subprocess module as well but I can't quite put the pieces together. >> I'd appreciate any guidance. Thanks. > > Alternatively read the output into a variable using communicate but then > write it out to the file manually at the end. > > [You might also be able to use shell=TRUE but that introduces other > issues. But I don't know whether using shell includes redirection.] > Thanks Alan, yes, I realise now I needed a third assignment to make this work. I actually had an exchange with subscriber 'eryksun' yesterday who did a great job of pointing me in the right direction. As a bit of a noob, I think I replied to the individual rather than the list, hence it doesn't seem to be in the thread. For the benefit of the archives I append below eryksun's initial (there was a bit of follow up but nothing too important) reply to me. = Send p2's stdout to a file: import subprocess import shlex ddIf = shlex.split("dd if=/home/adam/1 bs=4k " "conv=noerror,notrunc,sync") compress = "pbzip2" filename = "/home/adam/1.img.bz2" p1 = subprocess.Popen(ddIf, stdout=subprocess.PIPE) with p1.stdout as fin, open(filename, "w") as fout: p2 = subprocess.Popen(compress, stdin=fin, stdout=fout) ret1 = p1.wait() ret2 = p2.wait() Does this work? ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] pipes and redirecting
I'm trying to run the following unix command from within Python as opposed to calling an external Bash script (the reason being I'm doing it multiple times within a for loop which is running through a list): "dd if=/home/adam/1 bs=4k conv=noerror,notrunc,sync | pbzip2 > 1.img.bz2" The first thing I do is break it into two assignments (I know this isn't strictly necessary but it makes the code easier to deal with): ddIf = shlex.split("dd if=/home/adam/1 bs=4k conv=noerror,notrunc,sync") compress = shlex.split("pbzip2 > /home/adam/1.img.bz2") I have looked at the docs here (and the equivalent for Python 3) https://docs.python.org/2/library/subprocess.html. I can get a 'simple' pipe like the following to work: p1 = subprocess.Popen(["ps"], stdout=PIPE) p2 = subprocess.Popen(["grep", "ssh"], stdin=p1.stdout, stdout=subprocess.PIPE) p1.stdout.close() output = p2.communicate()[0] I then try to adapt it to my example: p1 = subprocess.Popen(ddIf, stdout=subprocess.PIPE) p2 = subprocess.Popen(compress, stdin=p1.stdout, stdout=subprocess.PIPE) p1.stdout.close() output = p2.communicate()[0] I get the following error: pbzip2: *ERROR: File [>] NOT found! Skipping... --- pbzip2: *ERROR: Input file [/home/adam/1.img.bz2] already has a .bz2 extension! Skipping I think that the '>' redirect needs to be dealt with using the subprocess module as well but I can't quite put the pieces together. I'd appreciate any guidance. Thanks. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] ssh socks proxy
Many thanks Kushal (it's always the silly things one has to look for!). The answer to your question is I don't have shell access to the proxy server, I can only access it for the purpose of setting up a socks proxy. Btw, I'm probably pushing my luck here but any idea what the same script would look like if one used paramiko? On 17 May 2012, at 04:01, kushal.kumaran+pyt...@gmail.com wrote: kushal.kumaran+pyt...@gmail.com wrote: > Adam Gold wrote: > >> I'm trying to write a 'simple' script that will set up a socks proxy >> over ssh and maintain the connection until manually terminated. It's >> not possible to use key-based authentication so a password will need > to >> be supplied. Also, note, the user is presented with a list of servers >> to choose from at the beginning. The actual ssh command is: 'ssh > -vNCD >> 2 user@host'. >> >> I've been tinkering with both pexpect and paramiko but fear I'm making >> a mountain out of a mole hill. I'm aware both have example scripts > for >> ssh forwarding but, to be honest, they are both too complicated (aka I >> don't know how to customise them). Here is one script I've attempted >> to get working and the associated error listing: >> >> http://pastebin.com/jj8Fgvwm - script >> http://pastebin.com/jRA8zpzi - error >> >> Could anyone help me either correct the script I've started or suggest >> an ockham's-razor-adherent alternative! Many thanks. > > The error message indicates that there hostname HOST could not be > resolved. You need to replace that with the value of the variable HOST. > > COMMAND = "ssh -vNDR 2 {}@{}".format(USER, HOST) > > I'm curious to know why you can't use keys. They make things much > simpler. Excuse the incorrect ssh command. New mail client tripped me up. -- regards, kushal Use the link below to report this message as spam. https://lavabit.com/apps/teacher?sig=3087369&key=478586400 ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
[Tutor] ssh socks proxy
I'm trying to write a 'simple' script that will set up a socks proxy over ssh and maintain the connection until manually terminated. It's not possible to use key-based authentication so a password will need to be supplied. Also, note, the user is presented with a list of servers to choose from at the beginning. The actual ssh command is: 'ssh -vNCD 2 user@host'. I've been tinkering with both pexpect and paramiko but fear I'm making a mountain out of a mole hill. I'm aware both have example scripts for ssh forwarding but, to be honest, they are both too complicated (aka I don't know how to customise them). Here is one script I've attempted to get working and the associated error listing: http://pastebin.com/jj8Fgvwm - script http://pastebin.com/jRA8zpzi - error Could anyone help me either correct the script I've started or suggest an ockham's-razor-adherent alternative! Many thanks. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] different behaviour in Idle shell vs Mac terminal
> Date: Sun, 8 Jan 2012 23:34:15 + > From: Adam Gold > To: > Subject: [Tutor] different behaviour in Idle shell vs Mac terminal > Message-ID: > Content-Type: text/plain; charset="iso-8859-1" > > > I have short piece of code I'm using to print a string to the terminal one > letter at a time.? It works fine when I invoke the script from within Idle; > each letter appears after the preceding one according to the designated time > interval.? However if I run it in the Mac terminal ('python3 ./script.py'), > there's a pause and then the whole string prints in one go.? Here's the > relevant code: > > import sys > import time > > text = "this text is printing one letter at a time..." > for char in text: > ??? sys.stdout.write(char) > ??? time.sleep(0.03) > > I'm thinking this may be a tty issue (is stdout going to the right terminal?) > but I'm still finding my way and would therefore appreciate any guidance.? Of > course if there's a better way of printing out one letter at a time, I'm also > interested to know that.? Thanks. > > P.S. if it's relevant, this is part of a simple financial maths program and > it's used to display the results after certain inputs have been gathered. > > > -- > > Message: 3 > Date: Mon, 09 Jan 2012 10:56:29 +1100 > From: Steven D'Aprano > To: tutor@python.org > Subject: Re: [Tutor] different behaviour in Idle shell vs Mac terminal > Message-ID: <4f0a2d2d.9000...@pearwood.info> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Adam Gold wrote: > > I have short piece of code I'm using to print a string to the terminal one > > letter at a time. It works fine when I invoke the script from within Idle; > > each letter appears after the preceding one according to the designated > > time interval. However if I run it in the Mac terminal ('python3 > > ./script.py'), there's a pause and then the whole string prints in one go. > > Here's the relevant code: > > > > import sys > > import time > > > > text = "this text is printing one letter at a time..." > > for char in text: > > sys.stdout.write(char) > > time.sleep(0.03) > > > > I'm thinking this may be a tty issue (is stdout going to the right > > terminal?) > > It's a buffering issue. > > [...] > > P.S. if it's relevant, this is part of a simple financial maths program and > > it's used to display the results after certain inputs have been gathered. > > To annoy your users? I'm not sure why you think it's a good idea to pretend > that the computer has to type the letters one at a time. This isn't some > stupid imaginary program in a Hollywood movie, I assume it is meant to > actually be useful and usable, and trust me on this, waiting while the program > pretends to type gets old *really* fast. > > (What are you trying to emulate? A stock ticker or something? Do those things > still even exist? I haven't seen one since the last time I watched the Addams > Family TV series. The old one, in black & white.) > > But if you must, after writing each character, call sys.stdout.flush() to > flush the buffer. > > > > -- > Steven > Thanks Steven that worked. In terms of why I'm using this: I shouldn't overstate what I'm doing when I say financial maths. One of the elements is a mortgage calculator for my mother who's, shall we say, not a "power user". After taking the basic inputs, it prints out a few stats (monthly payment etc.). Without some literal "brake" in how the info gets written on the screen, it all appears in one go and having tested it on said power user, it was too much at once. Hence I want to slow things down. I appreciate, I'm not pushing the boundaries here but one step at a time ('excuse the pun!). Anyway, again, thanks for your help. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
[Tutor] different behaviour in Idle shell vs Mac terminal
I have short piece of code I'm using to print a string to the terminal one letter at a time. It works fine when I invoke the script from within Idle; each letter appears after the preceding one according to the designated time interval. However if I run it in the Mac terminal ('python3 ./script.py'), there's a pause and then the whole string prints in one go. Here's the relevant code: import sys import time text = "this text is printing one letter at a time..." for char in text: sys.stdout.write(char) time.sleep(0.03) I'm thinking this may be a tty issue (is stdout going to the right terminal?) but I'm still finding my way and would therefore appreciate any guidance. Of course if there's a better way of printing out one letter at a time, I'm also interested to know that. Thanks. P.S. if it's relevant, this is part of a simple financial maths program and it's used to display the results after certain inputs have been gathered. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor