Re: Addendum to Strategy/ Advice for How to Best Attack this Problem?
Thank you for the feedback - I have only been programming for 8 months now and
am fairly new to best practice and what is considered acceptable in public
forums. I appreciate the feedback on how to best address this problem.
On Sunday, March 29, 2015 at 8:33:43 AM UTC-4, Peter Otten wrote:
> Saran Ahluwalia wrote:
>
> > On Sunday, March 29, 2015 at 7:33:04 AM UTC-4, Saran Ahluwalia wrote:
> >> Below are the function's requirements. I am torn between using the OS
> >> module or some other quick and dirty module. In addition, my ideal
> >> assumption that this could be cross-platform. "Records" refers to
> >> contents in a file. What are some suggestions from the Pythonistas?
> >>
> >> * Monitors a folder for files that are dropped throughout the day
> >>
> >> * When a file is dropped in the folder the program should scan the file
> >>
> >> o IF all the records in the file have the same length
> >>
> >> o THEN the file should be moved to a "success" folder and a text file
> >> written indicating the total number of records processed
> >>
> >> o IF the file is empty OR the records are not all of the same length
> >>
> >> o THEN the file should be moved to a "failure" folder and a text file
> >> written indicating the cause for failure (for example: Empty file or line
> >> 100 was not the same length as the rest).
> >
> > Below are some functions that I have been playing around with. I am not
> > sure how to create a functional program from each of these constituent
> > parts. I could use decorators or simply pass a function within another
> > function.
>
> Throwing arbitrary code at a task in the hope that something sticks is not a
> good approach. You already have given a clear description of the problem, so
> start with that and try to "pythonize" it. Example:
>
> def main():
> while True:
> files_to_check = get_files_in_monitored_folder()
> for file in files_to_check:
> if is_good(file):
> move_to_success_folder(file)
> else:
> move_to_failure_folder(file)
> wait_a_minute()
>
> if __name__ == "__main__":
> main()
>
> Then write bogus implementations for the building blocks:
>
> def get_files_in_monitored_folder():
> return ["/foo/bar/ham", "/foo/bar/spam"]
>
> def is_good(file):
> return file.endswith("/ham")
>
> def move_to_failure_folder(file):
> print("failure", file)
>
> def move_to_success_folder(file):
> print("success", file)
>
> def wait_a_minute():
> raise SystemExit("bye") # we don't want to enter the loop while
> developing
>
> Now successively replace the dummy function with functions that do the right
> thing. Test them individually (preferrably using unit tests) so that when
> you have completed them all and your program does not work like it should
> you can be sure that there is a flaw in the main function.
>
> > [code]
> > import time
> > import fnmatch
> > import os
> > import shutil
> >
> >
> > #If you want to write to a file, and if it doesn't exist, do this:
>
> Hm, are these your personal notes or is it an actual script?
>
> > if not os.path.exists(filepath):
> > f = open(filepath, 'w')
> > #If you want to read a file, and if it exists, do the following:
> >
> > try:
> > f = open(filepath)
> > except IOError:
> > print 'I will be moving this to the '
> >
> >
> > #Changing a directory to "/home/newdir"
> > os.chdir("/home/newdir")
>
> Never using os.chdir() is a good habit to get into.
>
> > def move(src, dest):
> > shutil.move(src, dest)
> >
> > def fileinfo(file):
> > filename = os.path.basename(file)
> > rootdir = os.path.dirname(file)
> > lastmod = time.ctime(os.path.getmtime(file))
> > creation = time.ctime(os.path.getctime(file))
> > filesize = os.path.getsize(file)
> >
> > print "%s**\t%s\t%s\t%s\t%s" % (rootdir, filename, lastmod, creation,
> > filesize)
> >
> > searchdir = r'D:\Your\Directory\Root'
> > matches = []
> >
> > def search
>
> Everytime you post code that doesn't even compile you lose some goodwill.
> In a few lines of code meant to demonstrate a problem a typo may be
> acceptable, but for something that you probably composed in an editor you
> should take the time to run it and fix at least the syntax errors.
>
> > for root, dirnames, filenames in os.walk(searchdir):
> > ## for filename in fnmatch.filter(filenames, '*.c'):
> > for filename in filenames:
> > ## matches.append(os.path.join(root, filename))
> > ##print matches
> > fileinfo(os.path.join(root, filename))
> >
> >
> > def get_files(src_dir):
> > # traverse root directory, and list directories as dirs and files as files
> > for root, dirs, files in os.walk(src_dir):
> > path = root.split('/')
> > for file in files:
> > process(os.path.join(root, file))
> > os.remove(os.path.join(root, file))
> >
> > def del_dirs(src_dir
Re: Strategy/ Advice for How to Best Attack this Problem?
On Sunday, March 29, 2015 at 10:04:45 PM UTC-4, Chris Angelico wrote: > On Mon, Mar 30, 2015 at 12:08 PM, Paul Rubin wrote: > > Saran Ahluwalia writes: > >> cross-platform... > >> * Monitors a folder for files that are dropped throughout the day > > > > I don't see a cross-platform way to do that other than by waking up and > > scanning the folder every so often (once a minute, say). The Linux way > > is with inotify and there's a Python module for it (search terms: python > > inotify). There might be comparable but non-identical interfaces for > > other platforms. > > All too often, "cross-platform" means probing for one option, then > another, then another, and using whichever one you can. On Windows, > there's FindFirstChangeNotification and ReadDirectoryChanges, which > Tim Golden wrote about, and which I coded up into a teleporter for > getting files out of a VM automatically: > > http://timgolden.me.uk/python/win32_how_do_i/watch_directory_for_changes.html > https://github.com/Rosuav/shed/blob/master/senddir.py > > ChrisA @Dave, Chris, Paul and Dennis: Thank you for resources and the notes regarding what I should keep in mind. I have an initial commit: https://github.com/ahlusar1989/IntroToPython/blob/master/Project1WG_with_assumptions_and_comments.py I welcome your thoughts on this -- https://mail.python.org/mailman/listinfo/python-list
Re: Strategy/ Advice for How to Best Attack this Problem?
On Monday, March 30, 2015 at 2:36:02 PM UTC-4, Dave Angel wrote: > On 03/30/2015 12:45 PM, Saran A wrote: > > On Sunday, March 29, 2015 at 10:04:45 PM UTC-4, Chris Angelico wrote: > >> On Mon, Mar 30, 2015 at 12:08 PM, Paul Rubin > >> wrote: > >>> Saran Ahluwalia writes: > >>>> cross-platform... > >>>> * Monitors a folder for files that are dropped throughout the day > >>> > >>> I don't see a cross-platform way to do that other than by waking up and > >>> scanning the folder every so often (once a minute, say). The Linux way > >>> is with inotify and there's a Python module for it (search terms: python > >>> inotify). There might be comparable but non-identical interfaces for > >>> other platforms. > >> > >> All too often, "cross-platform" means probing for one option, then > >> another, then another, and using whichever one you can. On Windows, > >> there's FindFirstChangeNotification and ReadDirectoryChanges, which > >> Tim Golden wrote about, and which I coded up into a teleporter for > >> getting files out of a VM automatically: > >> > >> http://timgolden.me.uk/python/win32_how_do_i/watch_directory_for_changes.html > >> https://github.com/Rosuav/shed/blob/master/senddir.py > >> > >> ChrisA > > > > @Dave, Chris, Paul and Dennis: Thank you for resources and the notes > > regarding what I should keep in mind. I have an initial commit: > > https://github.com/ahlusar1989/IntroToPython/blob/master/Project1WG_with_assumptions_and_comments.py > > > > I welcome your thoughts on this > > > > It's missing a number of your requirements. But it's a start. > > If it were my file, I'd have a TODO comment at the bottom stating known > changes that are needed. In it, I'd mention: > > 1) your present code is assuming all filenames come directly from the > commandline. No searching of a directory. > > 2) your present code does not move any files to success or failure > directories > > 3) your present code doesn't calculate or write to a text file any > statistics. > > 4) your present code runs once through the names, and terminates. It > doesn't "monitor" anything. > > 5) your present code doesn't check for zero-length files > > I'd also wonder why you bother checking whether the > os.path.getsize(file) function returns the same value as the os.SEEK_END > and ftell() code does. Is it that you don't trust the library? Or that > you have to run on Windows, where the line-ending logic can change the > apparent file size? > > I notice you're not specifying a file mode on the open. So in Python 3, > your sizes are going to be specified in unicode characters after > decoding. Is that what the spec says? It's probably safer to > explicitly specify the mode (and the file encoding if you're in text). > > I see you call strip() before comparing the length. Could there ever be > leading or trailing whitespace that's significant? Is that the actual > specification of line size? > > -- > DaveA @ Dave A On Monday, March 30, 2015 at 2:36:02 PM UTC-4, Dave Angel wrote: > On 03/30/2015 12:45 PM, Saran A wrote: > > On Sunday, March 29, 2015 at 10:04:45 PM UTC-4, Chris Angelico wrote: > >> On Mon, Mar 30, 2015 at 12:08 PM, Paul Rubin > >> wrote: > >>> Saran Ahluwalia writes: > >>>> cross-platform... > >>>> * Monitors a folder for files that are dropped throughout the day > >>> > >>> I don't see a cross-platform way to do that other than by waking up and > >>> scanning the folder every so often (once a minute, say). The Linux way > >>> is with inotify and there's a Python module for it (search terms: python > >>> inotify). There might be comparable but non-identical interfaces for > >>> other platforms. > >> > >> All too often, "cross-platform" means probing for one option, then > >> another, then another, and using whichever one you can. On Windows, > >> there's FindFirstChangeNotification and ReadDirectoryChanges, which > >> Tim Golden wrote about, and which I coded up into a teleporter for > >> getting files out of a VM automatically: > >> > >> http://timgolden.me.uk/python/win32_how_do_i/watch_directory_for_changes.html > >> https://github.com/Rosuav/shed/blob/master/senddir.py > >> > >> ChrisA > > > > @Dave, Chris, Paul and Denn
Re: Strategy/ Advice for How to Best Attack this Problem?
On Tuesday, March 31, 2015 at 9:19:37 AM UTC-4, Dave Angel wrote:
> On 03/31/2015 07:00 AM, Saran A wrote:
>
> > @DaveA: This is a homework assignment. Is it possible that you
> could provide me with some snippets or guidance on where to place your
> suggestions (for your TO DOs 2,3,4,5)?
> >
>
>
> > On Monday, March 30, 2015 at 2:36:02 PM UTC-4, Dave Angel wrote:
>
> >>
> >> It's missing a number of your requirements. But it's a start.
> >>
> >> If it were my file, I'd have a TODO comment at the bottom stating known
> >> changes that are needed. In it, I'd mention:
> >>
> >> 1) your present code is assuming all filenames come directly from the
> >> commandline. No searching of a directory.
> >>
> >> 2) your present code does not move any files to success or failure
> >> directories
> >>
>
> In function validate_files()
> Just after the line
> print('success with %s on %d reco...
> you could move the file, using shutil. Likewise after the failure print.
>
> >> 3) your present code doesn't calculate or write to a text file any
> >> statistics.
>
> You successfully print to sys.stderr. So you could print to some other
> file in the exact same way.
>
> >>
> >> 4) your present code runs once through the names, and terminates. It
> >> doesn't "monitor" anything.
>
> Make a new function, perhaps called main(), with a loop that calls
> validate_files(), with a sleep after each pass. Of course, unless you
> fix TODO#1, that'll keep looking for the same files. No harm in that if
> that's the spec, since you moved the earlier versions of the files.
>
> But if you want to "monitor" the directory, let the directory name be
> the argument to main, and let main do a dirlist each time through the
> loop, and pass the corresponding list to validate_files.
>
> >>
> >> 5) your present code doesn't check for zero-length files
> >>
>
> In validate_and_process_data(), instead of checking filesize against
> ftell, check it against zero.
>
> >> I'd also wonder why you bother checking whether the
> >> os.path.getsize(file) function returns the same value as the os.SEEK_END
> >> and ftell() code does. Is it that you don't trust the library? Or that
> >> you have to run on Windows, where the line-ending logic can change the
> >> apparent file size?
> >>
> >> I notice you're not specifying a file mode on the open. So in Python 3,
> >> your sizes are going to be specified in unicode characters after
> >> decoding. Is that what the spec says? It's probably safer to
> >> explicitly specify the mode (and the file encoding if you're in text).
> >>
> >> I see you call strip() before comparing the length. Could there ever be
> >> leading or trailing whitespace that's significant? Is that the actual
> >> specification of line size?
> >>
> >> --
> >> DaveA
> >
> >
>
> > I ask this because I have been searching fruitlessly through for some time
> > and there are so many permutations that I am bamboozled by which is
> > considered best practice.
> >
> > Moreover, as to the other comments, those are too specific. The scope of
> > the assignment is very limited, but I am learning what I need to look out
> > or ask questions regarding specs - in the future.
> >
>
>
> --
> DaveA
@DaveA
My most recent commit
(https://github.com/ahlusar1989/WGProjects/blob/master/P1version2.0withassumptions_mods.py)
has more annotations and comments for each file.
I have attempted to address the functional requirements that you brought up:
1) Before, my present code was assuming all filenames come directly from the
commandline. No searching of a directory. I think that I have addressed this.
2) My present code does not move any files to success or failure directories
(requirements for this assignment1). I am still wondering if and how I should
use shututil() like you advised me to. I keep receiving a syntax error when
declaring this below the print statement.
3) You correctly reminded me that my present code doesn't calculate or write to
a text file any statistics or errors for the cause of the error. (Should I use
the copy or copy2 method in order provide metadata? If so, should I wrap it
into a try and except logic?)
4) Before, my present code runs once through the names, and terminates. It
doesn't "monitor" anything. I think I have addressed this with the main
function - correct?
5) Before, my present code doesn't check for zero-length files - I have added
a comment there in case that is needed)
I realize appreciate your invaluable feedback. I have grown a lot with this
assignment!
Sincerely,
Saran
--
https://mail.python.org/mailman/listinfo/python-list
Re: Addendum to Strategy/ Advice for How to Best Attack this Problem?
On Sunday, March 29, 2015 at 8:33:43 AM UTC-4, Peter Otten wrote:
> Saran Ahluwalia wrote:
>
> > On Sunday, March 29, 2015 at 7:33:04 AM UTC-4, Saran Ahluwalia wrote:
> >> Below are the function's requirements. I am torn between using the OS
> >> module or some other quick and dirty module. In addition, my ideal
> >> assumption that this could be cross-platform. "Records" refers to
> >> contents in a file. What are some suggestions from the Pythonistas?
> >>
> >> * Monitors a folder for files that are dropped throughout the day
> >>
> >> * When a file is dropped in the folder the program should scan the file
> >>
> >> o IF all the records in the file have the same length
> >>
> >> o THEN the file should be moved to a "success" folder and a text file
> >> written indicating the total number of records processed
> >>
> >> o IF the file is empty OR the records are not all of the same length
> >>
> >> o THEN the file should be moved to a "failure" folder and a text file
> >> written indicating the cause for failure (for example: Empty file or line
> >> 100 was not the same length as the rest).
> >
> > Below are some functions that I have been playing around with. I am not
> > sure how to create a functional program from each of these constituent
> > parts. I could use decorators or simply pass a function within another
> > function.
>
> Throwing arbitrary code at a task in the hope that something sticks is not a
> good approach. You already have given a clear description of the problem, so
> start with that and try to "pythonize" it. Example:
>
> def main():
> while True:
> files_to_check = get_files_in_monitored_folder()
> for file in files_to_check:
> if is_good(file):
> move_to_success_folder(file)
> else:
> move_to_failure_folder(file)
> wait_a_minute()
>
> if __name__ == "__main__":
> main()
>
> Then write bogus implementations for the building blocks:
>
> def get_files_in_monitored_folder():
> return ["/foo/bar/ham", "/foo/bar/spam"]
>
> def is_good(file):
> return file.endswith("/ham")
>
> def move_to_failure_folder(file):
> print("failure", file)
>
> def move_to_success_folder(file):
> print("success", file)
>
> def wait_a_minute():
> raise SystemExit("bye") # we don't want to enter the loop while
> developing
>
> Now successively replace the dummy function with functions that do the right
> thing. Test them individually (preferrably using unit tests) so that when
> you have completed them all and your program does not work like it should
> you can be sure that there is a flaw in the main function.
>
> > [code]
> > import time
> > import fnmatch
> > import os
> > import shutil
> >
> >
> > #If you want to write to a file, and if it doesn't exist, do this:
>
> Hm, are these your personal notes or is it an actual script?
>
> > if not os.path.exists(filepath):
> > f = open(filepath, 'w')
> > #If you want to read a file, and if it exists, do the following:
> >
> > try:
> > f = open(filepath)
> > except IOError:
> > print 'I will be moving this to the '
> >
> >
> > #Changing a directory to "/home/newdir"
> > os.chdir("/home/newdir")
>
> Never using os.chdir() is a good habit to get into.
>
> > def move(src, dest):
> > shutil.move(src, dest)
> >
> > def fileinfo(file):
> > filename = os.path.basename(file)
> > rootdir = os.path.dirname(file)
> > lastmod = time.ctime(os.path.getmtime(file))
> > creation = time.ctime(os.path.getctime(file))
> > filesize = os.path.getsize(file)
> >
> > print "%s**\t%s\t%s\t%s\t%s" % (rootdir, filename, lastmod, creation,
> > filesize)
> >
> > searchdir = r'D:\Your\Directory\Root'
> > matches = []
> >
> > def search
>
> Everytime you post code that doesn't even compile you lose some goodwill.
> In a few lines of code meant to demonstrate a problem a typo may be
> acceptable, but for something that you probably composed in an editor you
> should take the time to run it and fix at least the syntax errors.
>
> > for root, dirnames, filenames in os.walk(searchdir):
> > ## for filename in fnmatch.filter(filenames, '*.c'):
> > for filename in filenames:
> > ## matches.append(os.path.join(root, filename))
> > ##print matches
> > fileinfo(os.path.join(root, filename))
> >
> >
> > def get_files(src_dir):
> > # traverse root directory, and list directories as dirs and files as files
> > for root, dirs, files in os.walk(src_dir):
> > path = root.split('/')
> > for file in files:
> > process(os.path.join(root, file))
> > os.remove(os.path.join(root, file))
> >
> > def del_dirs(src_dir):
> > for dirpath, _, _ in os.walk(src_dir, topdown=False): # Listing the
> > files
> > if dirpath == src_dir:
> > break
> > try:
> > os.rmdir(dirpath)
> > except OSError
New to Programming: TypeError: coercing to Unicode: need string or buffer, list found
Good Morning:
I understand this error message when I run this code. However, I am curious to
know what the most pythonic way is to convert the list to a string? I use
Python 2.7.
"Traceback (most recent call last):
before = dict([(f, None) for f in os.listdir(dirlist)])
TypeError: coercing to Unicode: need string or buffer, list found"
The sample code that I am trying to run is:
path = "/Users/Desktop/Projects/"
dirlist = os.listdir(path)
before = dict([(f, None) for f in os.listdir(dirlist)])
def main(dirlist):
while True:
time.sleep(10) #time between update check
after = dict([(f, None) for f in os.listdir(dirlist)])
added = [f for f in after if not f in before]
if added:
print('Successfully added new file - ready to validate')
if __name__ == "__main__":
main()
--
https://mail.python.org/mailman/listinfo/python-list
Re: New to Programming: TypeError: coercing to Unicode: need string or buffer, list found
On Thursday, April 2, 2015 at 8:26:01 AM UTC-4, Chris Angelico wrote:
> On Thu, Apr 2, 2015 at 11:02 PM, Saran A wrote:
> > I understand this error message when I run this code. However, I am curious
> > to know what the most pythonic way is to convert the list to a string? I
> > use Python 2.7.
> >
>
> I don't think you actually want to convert a list into a string, here.
> Tell me if I'm understanding your code's intention correctly:
>
> > The sample code that I am trying to run is:
> >
> > path = "/Users/Desktop/Projects/"
> > dirlist = os.listdir(path)
> > before = dict([(f, None) for f in os.listdir(dirlist)])
>
> Start up and get a full list of pre-existing files.
>
> > def main(dirlist):
> > while True:
> > time.sleep(10) #time between update check
>
> Then, every ten seconds...
>
> > after = dict([(f, None) for f in os.listdir(dirlist)])
> > added = [f for f in after if not f in before]
>
> ... get a list of files, and if there are new ones...
>
> > if added:
> > print('Successfully added new file - ready to validate')
> > if __name__ == "__main__":
> > main()
>
> ... print out a message.
>
> If that's what you're trying to do, I would suggest using a directory
> notification system instead. Here's one that I use on Linux:
>
> https://github.com/Rosuav/shed/blob/master/dirwatch.py
>
> Here's another one, this time built for Windows:
>
> https://github.com/Rosuav/shed/blob/master/senddir.py
>
> But even if you absolutely have to poll, like that, you'll need to
> make a few code changes. The exception you're getting is symptomatic
> of just one problem with the code as published. My suspicion is that
> you just want to use listdir(path) rather than listdir(dirlist) - but
> if you want subdirectories, then you'll need to do things a bit
> differently (probably using os.walk instead).
>
> Also: You say you're using Python 2.7. If you have no particular
> reason to use 2.7, you'll do better to jump to Python 3. Your code
> will probably run identically, when it's this simple.
>
> ChrisA
@ChrisA - this is a smaller function that will take the most updated file. My
intention is the following:
* Monitor a folder for files that are dropped throughout the day
* When a file is dropped in the folder the program should scan the file
o IF all the contents in the file have the same length (let's assume line
length)
o THEN the file should be moved to a "success" folder and a text file written
indicating the total number of records/lines/words processed
o IF the file is empty OR the contents are not all of the same length
o THEN the file should be moved to a "failure" folder and a text file written
indicating the cause for failure (for example: Empty file or line 100 was not
the same length as the rest).
Here is the code I have written:
import os
import time
import glob
import sys
def initialize_logger(output_dir):
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
# create console handler and set level to info
handler = logging.StreamHandler()
handler.setLevel(logging.INFO)
formatter = logging.Formatter("%(levelname)s - %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)
# create error file handler and set level to error
handler = logging.FileHandler(os.path.join(output_dir, "error.log"),"w",
encoding=None, delay="true")
handler.setLevel(logging.ERROR)
formatter = logging.Formatter("%(levelname)s - %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)
# create debug file handler and set level to debug
handler = logging.FileHandler(os.path.join(output_dir, "all.log"),"w")
handler.setLevel(logging.DEBUG)
formatter = logging.Formatter("%(levelname)s - %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)
#Helper Functions for the Success and Failure Folder Outcomes, respectively
#checks the length of the file
def file_len(filename
with open(filename) as f:
for i, l in enumerate(f):
pass
return i + 1
#copies file to new destination
def copyFile(src, dest):
try:
shutil.copy(src, dest)
# eg. src and dest are the same file
except shutil.Error as e:
print('Error: %s' % e)
# eg. source or destination doesn't exist
except IOError as e:
print('Error: %s' % e.strerror)
#Failure Folder
def move_to_failure_folder_and_r
Re: New to Programming: TypeError: coercing to Unicode: need string or buffer, list found
On Thursday, April 2, 2015 at 8:26:51 AM UTC-4, Peter Otten wrote:
> Saran A wrote:
>
> > Good Morning:
> >
> > I understand this error message when I run this code. However, I am
> > curious to know what the most pythonic way is to convert the list to a
> > string? I use Python 2.7.
> >
> > "Traceback (most recent call last):
> > before = dict([(f, None) for f in os.listdir(dirlist)])
> > TypeError: coercing to Unicode: need string or buffer, list found"
> >
> >
> > The sample code that I am trying to run is:
> >
> > path = "/Users/Desktop/Projects/"
> > dirlist = os.listdir(path)
>
> At this point dirlist is a list of names of the files and directories in
>
> "/Users/Desktop/Projects/"
>
> Assuming that the Projects folder contains the subfolders or files
> /Users/Desktop/Projects/foo, /Users/Desktop/Projects/bar and
> /Users/Desktop/Projects/baz dirlist looks like this:
>
> ["foo", "bar", "baz"]
>
> It makes no sense to pass this list to os.listdir() as you do below:
>
> > before = dict([(f, None) for f in os.listdir(dirlist)])
>
> Forget about the other details in the error message; the actual problem is
> the "list found" part.
>
> Now what would be a possible fix? Sorry, I have no idea what your intention
> is. Again, you don't need to convert your list to string, you need to decide
> what directory you want to pass to listdir(). If you have multiple such
> directories you need to invoke listdir() multiple times with a single
> directory, typically in a loop.
>
> Bonus info:
>
> > while True:
> > time.sleep(10) #time between update check
>
> This loop will never terminate.
@Peter I understand that the intention of this program is to not terminate.
Here is what I have written so far:
I thought I would run this by you, since you offer such valuable feedback, in
the past. Just a quick rundown on what I want my program to do:
* Monitor a folder for files that are dropped throughout the day
* When a file is dropped in the folder the program should scan the file
o IF all the contents in the file have the same length (let's assume line
length)
o THEN the file should be moved to a "success" folder and a text file written
indicating the total number of records/lines/words processed
o IF the file is empty OR the contents are not all of the same length
o THEN the file should be moved to a "failure" folder and a text file written
indicating the cause for failure (for example: Empty file or line 100 was not
the same length as the rest).
Here is the code I have written:
import os
import time
import glob
import sys
def initialize_logger(output_dir):
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
# create console handler and set level to info
handler = logging.StreamHandler()
handler.setLevel(logging.INFO)
formatter = logging.Formatter("%(levelname)s - %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)
# create error file handler and set level to error
handler = logging.FileHandler(os.path.join(output_dir, "error.log"),"w",
encoding=None, delay="true")
handler.setLevel(logging.ERROR)
formatter = logging.Formatter("%(levelname)s - %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)
# create debug file handler and set level to debug
handler = logging.FileHandler(os.path.join(output_dir, "all.log"),"w")
handler.setLevel(logging.DEBUG)
formatter = logging.Formatter("%(levelname)s - %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)
def main(dirslist):
while True:
for file in os.listdir(dirslist) :
return validate_files(file)
time.sleep(5)
if __name__ == "__main__":
main()
#Helper Functions for the Success and Failure Folder Outcomes, respectively
#checks the length of the file
def file_len(filename
with open(filename) as f:
for i, l in enumerate(f):
pass
return i + 1
#copies file to new destination
def copyFile(src, dest):
try:
shutil.copy(src, dest)
# eg. src and dest are the same file
except shutil.Error as e:
print('Error: %s' % e)
# eg. source or destination doesn't exist
except IOError as e:
print('Error: %s' % e.strerror)
#Failure Folder
def move_to_failure_folder_and_return_error_file():
os.mkdir('Failure')
copyFile(filename, 'Failure')
initialize_logger(
Re: Strategy/ Advice for How to Best Attack this Problem?
On Wednesday, April 1, 2015 at 7:52:27 PM UTC-4, Dave Angel wrote:
> On 04/01/2015 09:43 AM, Saran A wrote:
> > On Tuesday, March 31, 2015 at 9:19:37 AM UTC-4, Dave Angel wrote:
> >> On 03/31/2015 07:00 AM, Saran A wrote:
> >>
> >> > @DaveA: This is a homework assignment. Is it possible that you
> >> could provide me with some snippets or guidance on where to place your
> >> suggestions (for your TO DOs 2,3,4,5)?
> >> >
> >>
> >>
> >>> On Monday, March 30, 2015 at 2:36:02 PM UTC-4, Dave Angel wrote:
> >>
> >>>>
> >>>> It's missing a number of your requirements. But it's a start.
> >>>>
> >>>> If it were my file, I'd have a TODO comment at the bottom stating known
> >>>> changes that are needed. In it, I'd mention:
> >>>>
> >>>> 1) your present code is assuming all filenames come directly from the
> >>>> commandline. No searching of a directory.
> >>>>
> >>>> 2) your present code does not move any files to success or failure
> >>>> directories
> >>>>
> >>
> >> In function validate_files()
> >> Just after the line
> >> print('success with %s on %d reco...
> >> you could move the file, using shutil. Likewise after the failure print.
> >>
> >>>> 3) your present code doesn't calculate or write to a text file any
> >>>> statistics.
> >>
> >> You successfully print to sys.stderr. So you could print to some other
> >> file in the exact same way.
> >>
> >>>>
> >>>> 4) your present code runs once through the names, and terminates. It
> >>>> doesn't "monitor" anything.
> >>
> >> Make a new function, perhaps called main(), with a loop that calls
> >> validate_files(), with a sleep after each pass. Of course, unless you
> >> fix TODO#1, that'll keep looking for the same files. No harm in that if
> >> that's the spec, since you moved the earlier versions of the files.
> >>
> >> But if you want to "monitor" the directory, let the directory name be
> >> the argument to main, and let main do a dirlist each time through the
> >> loop, and pass the corresponding list to validate_files.
> >>
> >>>>
> >>>> 5) your present code doesn't check for zero-length files
> >>>>
> >>
> >> In validate_and_process_data(), instead of checking filesize against
> >> ftell, check it against zero.
> >>
> >>>> I'd also wonder why you bother checking whether the
> >>>> os.path.getsize(file) function returns the same value as the os.SEEK_END
> >>>> and ftell() code does. Is it that you don't trust the library? Or that
> >>>> you have to run on Windows, where the line-ending logic can change the
> >>>> apparent file size?
> >>>>
> >>>> I notice you're not specifying a file mode on the open. So in Python 3,
> >>>> your sizes are going to be specified in unicode characters after
> >>>> decoding. Is that what the spec says? It's probably safer to
> >>>> explicitly specify the mode (and the file encoding if you're in text).
> >>>>
> >>>> I see you call strip() before comparing the length. Could there ever be
> >>>> leading or trailing whitespace that's significant? Is that the actual
> >>>> specification of line size?
> >>>>
> >>>> --
> >>>> DaveA
> >>>
> >>>
> >>
> >>> I ask this because I have been searching fruitlessly through for some
> >>> time and there are so many permutations that I am bamboozled by which is
> >>> considered best practice.
> >>>
> >>> Moreover, as to the other comments, those are too specific. The scope of
> >>> the assignment is very limited, but I am learning what I need to look out
> >>> or ask questions regarding specs - in the future.
> >>>
> >>
> >>
> >> --
> >> DaveA
> >
> > @DaveA
> >
> > My most recent commit
> > (https://github.com/ahlusar1989/WGProjects/blob/master/P1version2.0withassumptions_mods.py)
> > has more annotations and comments for each file.
>
> Perhaps you don't realize how github w
Re: New to Programming: TypeError: coercing to Unicode: need string or buffer, list found
On Thursday, April 2, 2015 at 9:06:49 AM UTC-4, Chris Angelico wrote:
> On Thu, Apr 2, 2015 at 11:46 PM, Saran A wrote:
> > @ChrisA - this is a smaller function that will take the most updated file.
> > My intention is the following:
> >
> > * Monitor a folder for files that are dropped throughout the day
> >
> > * When a file is dropped in the folder the program should scan the file
> >
> > o IF all the contents in the file have the same length (let's assume line
> > length)
> >
> > o THEN the file should be moved to a "success" folder and a text file
> > written indicating the total number of records/lines/words processed
> >
> > o IF the file is empty OR the contents are not all of the same length
> >
> > o THEN the file should be moved to a "failure" folder and a text file
> > written indicating the cause for failure (for example: Empty file or line
> > 100 was not the same length as the rest).
> >
>
> Sounds like a perfect job for inotify, then. Your function will be
> called whenever there's a new file.
>
> > Here is the code I have written:
> >
> > def initialize_logger(output_dir):
> > logger = logging.getLogger()
> > ...
> > def file_len(filename
> > with open(filename) as f:
> > for i, l in enumerate(f):
> > pass
> > return i + 1
>
> These functions are all getting defined inside your
> initialize_logger() function. I suspect you want them to be flush left
> instead.
>
> > def copyFile(src, dest):
> > try:
> > shutil.copy(src, dest)
> > # eg. src and dest are the same file
> > except shutil.Error as e:
> > print('Error: %s' % e)
> > # eg. source or destination doesn't exist
> > except IOError as e:
> > print('Error: %s' % e.strerror)
>
> Recommendation: Skip the try/except, and just let exceptions bubble
> up. Don't just print out messages and keep going.
>
> > def move_to_failure_folder_and_return_error_file():
> > os.mkdir('Failure')
> > copyFile(filename, 'Failure')
> > initialize_logger('rootdir/Failure')
> > logging.error("Either this file is empty or the lines")
>
> This doesn't move the file, it copies it. Is that your intention?
>
> Moving a file is pretty easy. Just use os.rename().
>
> > if __name__ == '__main__':
> >import sys
> >validate_files(sys.argv[1:])
>
> I've no idea what validate_files() does, as you haven't included that.
>
> I think you could code this fairly efficiently as a simple callback
> off pyinotify, or if you're not on Linux, with one of the equivalent
> services. What you're doing here (watching for files, looking inside
> them, and moving them when done) is pretty common around the world.
>
> ChrisA
@ChrisA
validate_files will:
#double check for record time and record length - logic to be written to either
pass to Failure or Success folder respectively. I welcome your thoughts on
this.
def validate_files():
creation = time.ctime(os.path.getctime(added))
lastmod = time.ctime(os.path.getmtime(added))
Does this address the issue. I particularly like writing my own exceptions as
they provide me with more information on what could be the root cause. I know
that in other circumstances, try and except are not the best practice. I
appreciate the reminder though.
Does this modification to copyFile do the job of moving the file? I haven't
written a test yet.
Thanks for catching the indentation for the helper functions.
def copyFile(src, dest):
> > try:
> > shutil.rename(src, dest)
> > # eg. src and dest are the same file
> > except shutil.Error as e:
> > print('Error: %s' % e)
> > # eg. source or destination doesn't exist
> > except IOError as e:
> > print('Error: %s' % e.strerror)
--
https://mail.python.org/mailman/listinfo/python-list
Re: Strategy/ Advice for How to Best Attack this Problem?
On Thursday, April 2, 2015 at 5:11:20 PM UTC-4, Dave Angel wrote:
> On 04/02/2015 09:06 AM, Saran A wrote:
>
> >
> > Thanks for your help on this homework assignment. I started from scratch
> > last night. I have added some comments that will perhaps help clarify my
> > intentions and my thought process. Thanks again.
> >
> > from __future__ import print_function
>
> I'll just randomly comment on some things I see here. You've started
> several threads, on two different forums, so it's impractical to figure
> out what's really up.
>
>
>
> >
> > #Helper Functions for the Success and Failure Folder Outcomes, respectively
> >
> > def file_len(filename):
>
> This is an indentation error, as you forgot to start at the left margin
>
> > with open(filename) as f:
> > for i, l in enumerate(f):
> > pass
> > return i + 1
> >
> >
> > def copy_and_move_File(src, dest):
>
> ditto
>
> > try:
> > shutil.rename(src, dest)
>
> Is there a reason you don't use the move function? rename won't work if
> the two directories aren't on the same file system.
>
> > # eg. src and dest are the same file
> > except shutil.Error as e:
> > print('Error: %s' % e)
> > # eg. source or destination doesn't exist
> > except IOError as e:
> > print('Error: %s' % e.strerror)
> >
> >
> > # Call main(), with a loop that calls # validate_files(), with a sleep
> > after each pass. Before, my present #code was assuming all filenames come
> > directly from the commandline. There was no actual searching #of a
> > directory.
> >
> > # I am assuming that this is appropriate since I moved the earlier versions
> > of the files.
> > # I let the directory name be the argument to main, and let main do a
> > dirlist each time through the loop,
> > # and pass the corresponding list to validate_files.
> >
> >
> > path = "/some/sample/path/"
> > dirlist = os.listdir(path)
> > before = dict([(f, None) for f in dirlist)
> >
> > #Syntax Error? before = dict([(f, None) for f in dirlist)
> > ^
> > SyntaxError: invalid syntax
>
> Look at the line in question. There's an unmatched set of brackets. Not
> that it matters, since you don't need these 2 lines for anything. See
> my comments on some other forum.
>
> >
> > def main(dirlist):
>
> bad name for a directory path variable.
>
> > while True:
> > time.sleep(10) #time between update check
>
> Somewhere inside this loop, you want to obtain a list of files in the
> specified directory. And you want to do something with that list. You
> don't have to worry about what the files were last time, because
> presumably those are gone. Unless in an unwritten part of the spec,
> you're supposed to abort if any filename is repeated over time.
>
>
> > after = dict([(f, None) for f in dirlist)
> > added = [f for f in after if not f in before]
> > if added:
> > print('Sucessfully added new file - ready to validate')
> >add return statement here to pass to validate_files
> > if __name__ == "__main__":
> > main()
>
> You'll need an argument to call main()
>
> >
> >
> > #check for record time and record length - logic to be written to either
> > pass to Failure or Success folder respectively
> >
> > def validate_files():
>
> Where are all the parameters to this function?
>
> > creation = time.ctime(os.path.getctime(added))
> > lastmod = time.ctime(os.path.getmtime(added))
> >
> >
> >
> > #Potential Additions/Substitutions - what are the
> > implications/consequences for this
> >
> > def move_to_failure_folder_and_return_error_file():
> > os.mkdir('Failure')
> > copy_and_move_File(filename, 'Failure')
> > initialize_logger('rootdir/Failure')
> > logging.error("Either this file is empty or there are no lines")
> >
> >
> > def move_to_success_folder_and_read(f):
> > os.mkdir('Success')
> > copy_and_move_File(filename, 'Success')
> > print("Success", f)
> > return file_len()
> >
>
Re: New to Programming: TypeError: coercing to Unicode: need string or buffer, list found
On Thursday, April 2, 2015 at 8:03:53 PM UTC-4, Dennis Lee Bieber wrote:
> On Thu, 2 Apr 2015 05:46:57 -0700 (PDT), Saran A
> declaimed the following:
>
> >
> >@ChrisA - this is a smaller function that will take the most updated file.
> >My intention is the following:
> >
> >* Monitor a folder for files that are dropped throughout the day
> >
> I would suggest that your first prototype is to be a program that
> contains a function whose only purpose is to report on the files it finds
> -- forget about all the processing/moving of the files until you can
> successfully loop around the work of fetching the directory and handling
> the file names found (by maybe printing the names of the ones determined to
> be new since last fetch).
>
> >* When a file is dropped in the folder the program should scan the file
> >
> >o IF all the contents in the file have the same length (let's assume line
> >length)
> >
> >o THEN the file should be moved to a "success" folder and a text file
> >written indicating the total number of records/lines/words processed
> >
> >o IF the file is empty OR the contents are not all of the same length
> >
> >o THEN the file should be moved to a "failure" folder and a text file
> >written indicating the cause for failure (for example: Empty file or line
> >100 was not the same length as the rest).
> >
> You still haven't defined how you determine the "correct length" of the
> record. What if the first line is 79 characters, and all the others are 80
> characters? Do you report ALL lines EXCEPT the first as being the wrong
> length, when really it is the first line that is wrong?
>
> Also, if the files are Unicode (UTF-8, in particular) -- the byte
> length of a line could differ but the character length could be the same.
>
> >Here is the code I have written:
> >
> >import os
> >import time
> >import glob
> >import sys
> >
> >def initialize_logger(output_dir):
> >logger = logging.getLogger()
> >logger.setLevel(logging.DEBUG)
> >
> ># create console handler and set level to info
> >handler = logging.StreamHandler()
> >handler.setLevel(logging.INFO)
> >formatter = logging.Formatter("%(levelname)s - %(message)s")
> >handler.setFormatter(formatter)
> >logger.addHandler(handler)
> >
> ># create error file handler and set level to error
> >handler = logging.FileHandler(os.path.join(output_dir, "error.log"),"w",
> > encoding=None, delay="true")
> >handler.setLevel(logging.ERROR)
> >formatter = logging.Formatter("%(levelname)s - %(message)s")
> >handler.setFormatter(formatter)
> >logger.addHandler(handler)
> >
> ># create debug file handler and set level to debug
> >handler = logging.FileHandler(os.path.join(output_dir, "all.log"),"w")
> >handler.setLevel(logging.DEBUG)
> >formatter = logging.Formatter("%(levelname)s - %(message)s")
> >handler.setFormatter(formatter)
> >logger.addHandler(handler)
> >
> >#Helper Functions for the Success and Failure Folder Outcomes, respectively
> >
> >#checks the length of the file
> >def file_len(filename
> >with open(filename) as f:
> >for i, l in enumerate(f):
> >pass
> >return i + 1
> >
> >#copies file to new destination
> >
> >def copyFile(src, dest):
> >try:
> >shutil.copy(src, dest)
> ># eg. src and dest are the same file
> >except shutil.Error as e:
> >print('Error: %s' % e)
> ># eg. source or destination doesn't exist
> >except IOError as e:
> >print('Error: %s' % e.strerror)
> >
> >#Failure Folder
> >
> >def move_to_failure_folder_and_return_error_file():
> >os.mkdir('Failure')
> >copyFile(filename, 'Failure')
> >initialize_logger('rootdir/Failure')
> >logging.error("Either this file is empty or the lines")
> >
> ># Success Folder Requirement
> >
> >def move_to_success_folder_and_read(file):
> >os.mkdir('Success')
> >copyFile(filename, 'Success')
> >print("Success", file)
> >return file_len()
> >
> >
> >#This simply checks the file information by name
> &
New to Programming: Adding custom functions with ipynotify classes
Hello All:
Here is the program that I am trying to write (with specs):
* Monitors a folder for files that are dropped throughout the day
* When a file is dropped in the folder the program should scan the file
o IF all the records in the file have the same length (line length)
o THEN the file should be moved to a "success" folder and a text file written
indicating the total number of records processed
o IF the file is empty OR the records are not all of the same length
o THEN the file should be moved to a "failure" folder and a text file written
indicating the cause for failure (for example: Empty file or line 100 was not
the same length as the rest).
Many on forums suggest using ipynotify. I am wondering how to combine my
current script and add it to the ipynotify.
Below is my original script (the ipynotify script is provided after this)
[code]
# # # Without data to examine here, I can only guess based on this
requirement's language that
# # fixed records are in the input.
##I made the assumption that the directories are in the same filesystem
# # Takes the function fileinfo as a starting point and demonstrates calling a
function from within a function.
# I tested this little sample on a small set of files created with MD5
checksums. I wrote the Python in such a way as it
# would work with Python 2.x or 3.x (note the __future__ at the top).
# # # There are so many wonderful ways of failure, so, from a development
standpoint, I would probably spend a bit
# # more time trying to determine which failure(s) I would want to report to
the user, and how (perhaps creating my own Exceptions)
# # # The only other comments I would make are about safe-file handling.
# # # #1: Question: After a user has created a file that has failed (in
# # #processing),can the user create a file with the same name?
# # #If so, then you will probably want to look at some sort
# # #of file-naming strategy to avoid overwriting evidence of
# # #earlier failures.
# # # File naming is a tricky thing. I referenced the tempfile module [1] and
the Maildir naming scheme to see two different
# # types of solutions to the problem of choosing a unique filename.
## I am assuming that all of my files are going to be specified in unicode
## Utilized Spyder's Scientific Computing IDE to debug, check for indentation
errors and test function suite
from __future__ import print_function
import os.path
import time
import difflib
import logging
def initialize_logger(output_dir):
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
# create console handler and set level to info
handler = logging.StreamHandler()
handler.setLevel(logging.INFO)
formatter = logging.Formatter("%(levelname)s - %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)
# create error file handler and set level to error
handler = logging.FileHandler(os.path.join(output_dir, "error.log"),"w",
encoding=None, delay="true")
handler.setLevel(logging.ERROR)
formatter = logging.Formatter("%(levelname)s - %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)
# create debug file handler and set level to debug
handler = logging.FileHandler(os.path.join(output_dir, "all.log"),"w")
handler.setLevel(logging.DEBUG)
formatter = logging.Formatter("%(levelname)s - %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)
#This function's purpose is to obtain the filename, rootdir and filesize
def fileinfo(f):
filename = os.path.basename(f)
rootdir = os.path.dirname(f)
filesize = os.path.getsize(f)
return filename, rootdir, filesize
#This helper function returns the length of the file
def file_len(f):
with open(f) as f:
for i, l in enumerate(f):
pass
return i + 1
#This helper function attempts to copy file and move file to the respective
directory
#I am assuming that the directories are in the same filesystem
# If directories ARE in different file systems, I would use the following
helper function:
# def move(src, dest):
# shutil.move(src, dest)
def copy_and_move_file(src, dest):
try:
os.rename(src, dest)
# eg. src and dest are the same file
except IOError as e:
print('Error: %s' % e.strerror)
path = "."
dirlist = os.listdir(path)
# Caveats of the "main" function is that it does not scale well
#(although it is appropriate if one assumes that there will be few changes)
# It does not account for updated files existing in the directory - only new
files "dropped" in
# (If this was included in the requirements, os.stat would be appropriate here)
def main(dirlist):
before = dict([(f, 0) for f in dirlist])
while True:
time.sleep(1) #time between update check
after = dict([(f, None) for f in dirlist])
added = [f for f in after if not f in before]
Re: Strategy/ Advice for How to Best Attack this Problem?
On Friday, April 3, 2015 at 8:05:14 AM UTC-4, Dave Angel wrote:
> On 04/02/2015 07:43 PM, Saran A wrote:
>
> >
> > I debugged and rewrote everything. Here is the full version. Feel free to
> > tear this apart. The homework assignment is not due until tomorrow, so I am
> > currently also experimenting with pyinotify as well. I do have questions
> > regarding how to make this function compatible with the ProcessEvent Class.
> > I will create another post for this.
> >
> > What would you advise in regards to renaming the inaptly named dirlist?
>
> Asked and answered. You called it path at one point. dir_name would
> also be good. The point is that if you have a good name, you're less
> likely to have unreasonable code processing that name. For example,
>
> >
> > # # # Without data to examine here, I can only guess based on this
> > requirement's language that
> > # # fixed records are in the input. If so, here's a slight revision to the
> > helper functions that I wrote earlier which
> > # # takes the function fileinfo as a starting point and demonstrates
> > calling a function from within a function.
> > # I tested this little sample on a small set of files created with MD5
> > checksums. I wrote the Python in such a way as it
> > # would work with Python 2.x or 3.x (note the __future__ at the top).
> >
> > # # # There are so many wonderful ways of failure, so, from a development
> > standpoint, I would probably spend a bit
> > # # more time trying to determine which failure(s) I would want to report
> > to the user, and how (perhaps creating my own Exceptions)
> >
> > # # # The only other comments I would make are about safe-file handling.
> >
> > # # # #1: Question: After a user has created a file that has failed (in
> > # # #processing),can the user create a file with the same name?
> > # # #If so, then you will probably want to look at some sort
> > # # #of file-naming strategy to avoid overwriting evidence of
> > # # #earlier failures.
> >
> > # # # File naming is a tricky thing. I referenced the tempfile module [1]
> > and the Maildir naming scheme to see two different
> > # # types of solutions to the problem of choosing a unique filename.
> >> ## I am assuming that all of my files are going to be specified in unicode
> >
> > ## Utilized Spyder's Scientific Computing IDE to debug, check for
> > indentation errors and test function suite
> >
> > from __future__ import print_function
> >
> > import os.path
> > import time
> > import logging
> >
> >
> > def initialize_logger(output_dir):
>
>
> I didn't ever bother to read the body of this function, since a simple
>
>print(mydata, file=mylog_file)
>
> will suffice to add data to the chosen text file. Why is it constantly
> getting more complex, to solve a problem that was simple in the beginning?
>
> >
> >
> > #Returns filename, rootdir and filesize
> >
> > def fileinfo(f):
> > filename = os.path.basename(f)
> > rootdir = os.path.dirname(f)
> > filesize = os.path.getsize(f)
> > return filename, rootdir, filesize
> >
> > #returns length of file
> > def file_len(f):
> > with open(f) as f:
> > for i, l in enumerate(f):
> > pass
> > return i + 1
>
> Always returns 1 or None. Check the indentation, and test to see what
> it does for empty file, for a file with one line, and for a file with
> more than one line.
>
>
> > #attempts to copy file and move file to it's directory
> > def copy_and_move_file(src, dest):
>
> Which is it, are you trying to copy it, or move it? Pick one and make a
> good function name that shows your choice.
>
> > try:
> > os.rename(src, dest)
>
> Why are you using rename, when you're trying to move the file? Take a
> closer look at shutil, and see if it has a function that does it safer
> than rename. The function you need uses rename, when it'll work, and
> does it other ways when rename will not.
>
> > # eg. src and dest are the same file
> > except IOError as e:
> > print('Error: %s' % e.strerror)
>
> A try/except that doesn't try to correct the problem is not generally
> useful. Figure out what could be triggering the exception, and how
> you're going to handle it. If it cannot be handled, terminate the program.
>
> For example, what
Re: Strategy/ Advice for How to Best Attack this Problem?
On Friday, April 3, 2015 at 6:46:21 AM UTC-4, Peter Otten wrote:
> Saran A wrote:
>
> > I debugged and rewrote everything. Here is the full version. Feel free to
> > tear this apart. The homework assignment is not due until tomorrow, so I
> > am currently also experimenting with pyinotify as well.
>
> Saran, try to make a realistic assessment of your capability. Your
> "debugged" code still has this gem
>
> >while True:
> >time.sleep(1) #time between update check
>
> and you are likely to miss your deadline anyway.
>
> While the inotify approach may be the "right way" to attack this problem
> from the point of view of an experienced developer like Chris as a newbie
> you should stick to the simplest thing that can possibly work.
>
> In a previous post I mentioned unit tests, but even an ad-hoc test in the
> interactive interpreter will show that your file_len() function doesn't work
>
> > #returns length of file
> > def file_len(f):
> > with open(f) as f:
> > for i, l in enumerate(f):
> > pass
> > return i + 1
> >
>
> $ python
> Python 2.7.6 (default, Mar 22 2014, 22:59:56)
> [GCC 4.8.2] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> #returns length of file
> ... def file_len(f):
> ... with open(f) as f:
> ... for i, l in enumerate(f):
> ... pass
> ... return i + 1
> ...
> >>> with open("tmp.txt", "w") as f: f.write("abcde")
> ...
> >>> file_len("tmp.txt")
> 1
>
> It reports size 1 for every file.
>
> A simple unit test would look like this (assuming watchscript.py is the name
> of the script to be tested):
>
> import unittest
>
> from watchscript import file_len
>
> class FileLen(unittest.TestCase):
> def test_five(self):
> LEN = 5
> FILENAME = "tmp.txt"
> with open(FILENAME, "w") as f:
> f.write("*" * LEN)
>
> self.assertEqual(file_len(FILENAME), 5)
>
> if __name__ == "__main__":
> unittest.main()
>
> Typically for every tested function you add a new TestCase subclass and for
> every test you perform for that function you add a test_...() method to that
> class. The unittest.main() will collect these methods, run them and generate
> a report. For the above example it will complain:
>
> $ python test_watchscript.py
> F
> ==
> FAIL: test_five (__main__.FileLen)
> --
> Traceback (most recent call last):
> File "test_watchscript.py", line 12, in test_five
> self.assertEqual(file_len(FILENAME), 5)
> AssertionError: 1 != 5
>
> --
> Ran 1 test in 0.001s
>
> FAILED (failures=1)
>
>
> The file_len() function is but an example of the many problems with your
> code. There is no shortcut, you have to decide for every function what
> exactly it should do and then verify that it actually does what it is meant
> to do. Try to negotiate an extra week or two with your supervisor and then
> start small:
>
> On day one make a script that moves all files from one directory to another.
> Make sure that the source and destination directory are only stored in one
> place in your script so that they can easily be replaced. Run the script
> after every change and devise tests that demonstrate that it works
> correctly.
>
> On day two make a script that checks all files in a directory. Put the
> checks into a function that takes a file path as its only parameter.
> Write a few correct and a few broken files and test that only the correct
> ones are recognized as correct, and only the broken ones are flagged as
> broken, and that all files in the directory are checked.
>
> On day three make a script that combines the above and only moves the
> correct files into another directory. Devise tests to verify that both
> directories contain the expected files before and after your script is
> executed.
>
> On day four make a script that also moves the bad files into another
> directory. Modify your tests from the previous day to check the contents of
> all three directories.
>
> On day five wrap your previous efforts in a loop that runs forever.
> I seems to work? You are not done. Write tests that demonstrate that it does
> work. T
Re: New to Programming: Adding custom functions with ipynotify classes
On Friday, April 3, 2015 at 12:40:11 PM UTC-4, Denis McMahon wrote: > On Thu, 02 Apr 2015 18:30:42 -0700, Saran A wrote: > > > Here is the program that I am trying to write (with specs): > > Saran, please stop prefacing every subject with "New to programming:" - > it does not give an clue whatsoever as to what your post is about. > > -- > Denis McMahon, [email protected] @Denis Thank you for your feedback -- https://mail.python.org/mailman/listinfo/python-list
JSON Object to CSV file
I would like to have this JSON object written out to a CSV file so that the
keys are header fields (for each of the columns) and the values are values that
are associated with each header field. Is there a best practice for working
with this? Ideally I would like to recursively iterate through the key value
pairs. Thank you in advance. I am using Python 3.4 on Windows. My editor is
Sublime 2.
{
"CF": {
"A": "5",
"FEC": "1/1/0001 12:00:00 AM",
"TE": null,
"Locator": null,
"Message": "Transfer Fee",
"AT": null,
"FT": null,
"FR": "True",
"FY": null,
"FR": null,
"FG": "0",
"Comment": null,
"FUD": null,
"cID": null,
"GEO": null,
"ISO": null,
"TRID": null,
"XTY": "931083",
"ANM": null,
"NM": null
},
"CF": "Fee",
"ID": "2"
}
--
https://mail.python.org/mailman/listinfo/python-list
