[Tutor] Error terminal

[Richard Rizk]

Hello

I wanted to send you this email to ask if you would have someone that can solve 
the problem I'm having with python. 

I'm having issues with terminal on mac, is there someone that can help with 
this?

Best regards,
Richard
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Name for this type of class?

[Richard Damon]

On 8/3/19 1:40 PM, Alex Kleider wrote:
> On 2019-08-03 10:16, Mats Wichmann wrote:
>
>> .  It also comes up here:
>>
>>>> file_count
>>>> line_count
>>>> byte_count
>>>> row_count
>>>> batch_count
>>>> job_count
>>>> error_count
>>>> warning_count
>>
>> why not "files", "lines", "bytes"... the plural form already tells you
>> it's a counter.
>
> To me "files", "lines", "bytes" implies collections of files, lines
> bytes. (i.e. arrays or tuples...)
> "___count" implies an integer.
> If the latter, I'd use "n_files", "n_lines" ... (possibly without the
> underscores if you find typing them a bother.)
> Comments? 

I agree, plural nouns imply collections, and collections should
generally be plural nouns.

file_count is fine to me, or it could be numfiles or nfiles (possibly
adding the underscore)

-- 
Richard Damon

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Output reason

[Richard Damon]

If I remember how that works right, there is a single empty list that is 
created and used for all the calls that use the default argument, and then your 
function modifies that empty list so it is no longer empty, and that modified 
list is used on future calls. (Not good to use a mutable as a default 
parameter).

A better solution would be to make the default something like None, and test if 
at the beginning of the function li is None, and if so set it to an empty list, 
and that empty list will be in function scope so it goes away and a new one is 
created on a new call.

> On Jul 12, 2019, at 10:24 AM, Gursimran Maken  
> wrote:
> 
> Hi,
> 
> Can someone please explain me the reason for below output.
> 
> Program:
> def fun(n,li = []):
>a = list(range(5))
>li.append(a)
>print(li)
> 
> fun(4)
> fun(5,[7,8,9])
> fun(4,[7,8,9])
> fun(5) # reason for output (why am I getting to values in this output.)
> 
> Output:
> [[0, 1, 2, 3, 4]]
> [7, 8, 9, [0, 1, 2, 3, 4]]
> [7, 8, 9, [0, 1, 2, 3, 4]]
> [[0, 1, 2, 3, 4], [0, 1, 2, 3, 4]]
> 
> Thank you,
> Gursimran
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Make a linked list subscriptable?

[Richard Damon]

On 7/11/19 10:55 AM, Mats Wichmann wrote:
> On 7/10/19 6:30 PM, Sarah Hembree wrote:
>> How might I best make a linked list subscriptable? Below is skeleton code
>> for a linked list (my
>> actual is much more). I've done __iter__ and __next__ but I would like to
>> be able to do start:stop:stride I just can't figure out how. Suggestions or
>> just hints please?
> As a learning exercise this can be interesting, but as to practical
> applications, one would like to ask "why"?  If index into the list is
> important, then choose a regular list; the "interesting" part of a
> linked list, which is "next node" is then available as index + 1.
>
To expand on the question, the primary use of something like a linked
list is that you want cheap insertions/deletions (of O(1)) and in
exchange for that indexing becomes O(n), verse an array based list which
has O(1) indexing but O(N) insertions/deletions (since you need to
compact the array). Both can be iterated in O(1).

You can add an index operator that takes O(N) time to a linked list.
obj[n] will call obj.__getitem__ (and you will also want to implement
__setitem__, __delitem__), and check if the argument is a slice to
handle slices.

-- 
Richard Damon

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] Setting Command Line Arguments in IDLE

[Richard Damon]

I am working on a python script that will be provided arguments when run
from the system command line. Is there any place in IDLE to provide
equivalent arguments for testing while developing in IDLE?

Is there any way to define the working directory for the program, or
will it always be the directory the script is in (it will be typically
run using the PATH, so not the same directory as the script)?

If not, is there an easy way to detect that I am running in IDLE so I
can fake the command line arguments when testing?

-- 
Richard Damon

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] [SPAM?] Re: Case Insensitive Globing

[Richard Damon]

On 5/18/19 6:52 AM, Alan Gauld via Tutor wrote:
> On 18/05/2019 03:14, Richard Damon wrote:
>
>> The same directory, running the same program under Mac OS X, which also
>> is a case insensitive file system, 
> That is your mistake. Darwin, the core of the MacOS X system
> is a version of BSD Unix and like all Unix OS is very much
> case sensitive.
>
> Some of the GUI tools in MacOS X may work as if they were case
> insensitive (presumably for backwards compatibility to MacOS 9)
> but the underlying OS is case sensitive and all the Terminal
> type tools and commands, including Python, follow suit.
Ok, I guess I have gotten used to the GUI tools and their behavior, that
would explain why glob works the way it does.
>> Is there an easy was to make glob match files as a case insensitive manner?
> Depends on what you mean by easy :-)
> You could include both upper and lower case letters in the search:
>
> glob.glob("[aA][mM][iI][xX][eE][dD][nN][aA][mM][eE]")
>
> And you could write a helper function to generate the search strings.
>
> But personally I'd probably just use listdir and compare with
> my search string expressed as a regex.
>
> for fname in os.listdir('.'):
> if re.match("aregex", fname)
>...
>
> Another option might be to use fnmatch against the uppercase
> version of the name
>
> for fname in os.listdir('.'):
>if fnmatch(pattern, fname.upper())
>  ...
>
> HTH

I was hoping for something simpler, but I guess an explicit directory
search isn't that bad.

Thanks.

-- 
Richard Damon

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] Case Insensitive Globing

[Richard Damon]

I am working on a program to process some files created by an old
windows program that created it files with varying case with a python
program.

Using glob.glob on Windows seems to ignore the case, and find all the
matching files.

The same directory, running the same program under Mac OS X, which also
is a case insensitive file system, is only files that match the case of
the glob, and is missing many of the files that were found under windows.

Is there an easy was to make glob match files as a case insensitive manner?

Or a simple way to do this with something else.

I am trying to do something like:

  for file in glob.glob(pattern): processfile(file)

-- 
Richard Damon

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] TESTING

[richard mwenya via Tutor]

Hello.
Everyone is quiet or is it my email thats not working?
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] program code for Python Programming for the Absolute Beginner, 3rd ed.?

[Richard Grose]

https://www.google.co.uk/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0ahUKEwiw1tD3qurUAhULKlAKHWMOAPMQFggkMAA&url=https%3A%2F%2Fgithub.com%2FCWade3051%2FPy%2Ftree%2Fmaster%2FAbsolute%2520Book%2Fpy3e_source&usg=AFQjCNG36WhZfh5ftqWncjtgZy3z6xgh6g&cad=rjt

Py/Absolute Book/py3e_source at master · CWade3051/Py · GitHub 
<https://github.com/CWade3051/Py/tree/master/Absolute%20Book/py3e_source>

Richard G
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] Help with scoping

[Richard Mcewan]

Hi

Thanks Alan and all. 

I've incorporated your points. I've explored how values are returned and used 
by functions. I switched to Python to check a Swift3 bug and I think I can see 
a number of issues more clearly now. 

I put the count and call for next user guess back a tab at the end of the while 
loop instead of duplicating at the end of if conditions. 

Appreciate the help. Will come back to the list with new topics in the future 
no doubt and look forward to mailings. 

Thanks Richard

Code from 2.7 Python running on Pythonista iOS app. 

# coding: utf-8
import random

#guess number game

#number to guess
def generateNumber():
someValue = int(random.randrange(0,101))
return someValue #sic


#get user guess
def getUser():
userGuess = int(raw_input('Guess a number: '))
return userGuess

#loop to check guess and report
def main():
count = 0

#call for hidden number and user 1st guess
hiddenNumber = generateNumber()
userGuess = getUser()

#main loop to report on guess
while userGuess != hiddenNumber:
if userGuess < hiddenNumber:
print('Too low\n')
elif userGuess > hiddenNumber:
print('Too high\n')
#note new position
count +=1
userGuess = getUser()

print('Correct! ' , count+1, 'guesses')

main()

#future: deal with bad input. 
#invite user to set size of game.
#Range of fun messages reported. 
#Record best scores. 


___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] Help with function scoping

[Richard Mcewan]

Hi

Thanks Mats, Joel and Alan for helpful advice. 

This code (below) behaves as I wanted now. And think I see where I was going 
wrong with functions. 

Thank you very much. 

Richard
Ps I'll reflect on appropriate variable names also. 

# coding: utf-8
import random

#guess number game

#computer generates random number
def generateNumber():
computerGuess = int(random.randrange(0,101))
return computerGuess

#get user
def getUser():
userGuess = int(raw_input('Guess a number'))
return userGuess

#call for computer and user
computerGuess = generateNumber()
userGuess = getUser()


#loop to check guess and report
while userGuess != computerGuess:
if userGuess < computerGuess:
print('Too low')
userGuess = getUser()
elif userGuess > computerGuess:
print('Too high')
userGuess = getUser()

print('Correct!')

#Add counter of guess later. 




Sent from my iPhone
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] Help with function scoping

[Richard Mcewan]

Hi

I wonder if you can help. 

I'm confused about how functions should work. Below is some code I write to 
check my understanding. 

I'm expecting two functions to be defined. Then called. One returns a random 
number. The other user input (guessing the number). 

I expect the return values to act like variables that I can use in a while 
loop. 

The programme asks for the user input then crashes. The error report says 
'NameError:The name 'userGuess' is not defined. 

My understanding is wrong somewhere. I thought functions would allow me to 
return values I could use elsewhere without having to define them initially. 
I.e. Encapsulation. 

I've looked for other examples but get the same issue. I expect a function to 
return a variable I can use elsewhere. Not sure how to do. In the case below 
I'm not sure how I can use the work of the function elsewhere. 

Advice welcome. 

Richard 

Using Pythonista App on iOS with Python 2.7 on iPhone 5s

# coding: utf-8
import random

#guess number game

#computer generates random number
def generateNumber():
computerGuess = int(random.randrange(0,101))
return computerGuess

#get user
def getUser():
userGuess = int(input('Guess a number'))
return userGuess

#call for computer and user
generateNumber()
getUser()

#loop to check guess and report
while userGuess != computerGuess:
if userGuess < computerGuess:
print('Too low')
getUser()
elif userGuess > computerGuess:
print('Too high')
getUser()
elif userGuess > computerGuess:
print('Correct!')

#Add counter of guess later


Sent from my iPhone
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] formatting xml (again)

[richard kappler]

It occurred to me last night on the drive home that I should just run this
through an xml parser, then lo and behold this email was sitting in my
inbox when I got  home. Having tried that, my data is not as clean as I
first thought. It seems like a fairly simple fix, but durned if I can
figure out how to do it. One of the problems is data such as this (viewed
in the text editor, this is a log, not a stream):

1\x02 data data data \x03\x02 more data more data more data \x03\x02 even
more data even
2more data even more data\x03\x02 Mary had a little\x03\x02 lamb whose
fleece was white as
3snow\x03\x02

and so on. The 1,2,3 at the beginning of each above are just line numbers
in the text editor, they do not actually exist.

How do I read in the file, either in it's entirety or line by line, then
output the text with as \x02 the event data \x03 on each line, and when
python sees the \x03 it goes to a new line and continues to output?

On Tue, Dec 27, 2016 at 7:46 PM, David Rock  wrote:

> * Alan Gauld via Tutor  [2016-12-28 00:40]:
> > On 27/12/16 19:44, richard kappler wrote:
> > > Using python 2.7 - I have a large log file we recorded of streamed xml
> data
> > > that I now need to feed into another app for stress testing. The
> problem is
> > > the data comes in 2 formats.
> > >
> > > 1. each 'event' is a full set of xml data with opening and closing
> tags +
> > > x02 and x03 (stx and etx)
> > >
> > > 2. some events have all the xml data on one 'line' in the log, others
> are
> > > in typical nested xml format with lots of white space and multiple
> 'lines'
> > > in the log for each event, the first line of th e 'event' starting
> with an
> > > stx and the last line of the 'event' ending in an etx.
> >
> > It sounds as if an xml parser should work for both. After all
> > xml doesn't care about layout and whitespace etc.
> >
> > Which xml parser are you using - I assume you are not trying
> > to parse it manually using regex or string methjods - that's
> > rarely a good idea for xml.
>
> Yeah, since everything appears to be .., the "event" flags
> of [\x02] [\x03] may not even matter if you use an actual parser.
>
> --
> David Rock
> da...@graniteweb.com
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] formatting xml (again)

[richard kappler]

The input is consistent in that it all has stx at the beginning of each
'event.' I'm leaning towards regex. When you say:

" find stx, stuff lines until I see the next stx, then dump and continue"

Might I trouble you for an example of how you do that? I can find stx, I
can find etx using something along the lines of :

a = [m.start() for m in re.finditer(r"", line)]

but then I get a little lost, mostly because I have some lines that have
"data data [\x03][\x02] data" and then to the next line. More succinctly,
the stx aren't always at the beginning of the line, etx not always at the
end. No problem, I can find them, but then I'm guessing I would have to
write to a buffer starting with stx, keep writing to the buffer until I get
to etx, write the buffer to file (or send it over the socket, either way is
fine) then continue on. The fact that 'events' span multiple lines is
challenging me.

On Tue, Dec 27, 2016 at 3:55 PM, David Rock  wrote:

> * richard kappler  [2016-12-27 15:39]:
> > I was actually working somewhat in that direction while I waited. I had
> in
> > mind to use something along the lines of:
> >
> >
> > stx = '\x02'
> > etx = '\x03'
> > line1 = ""
> >
> > with open('original.log', 'r') as f1:
> >with open('new.log', 'w') as f2:
> > for line in f1:
> > if stx in line:
> > line1 = line1 + line
> > if not stx in line:
> > if not etx in line:
> > line1 = line1 + line
> > if etx in line:
> > line1 = line1 + line + '\n'
> > f2.write(line1)
> > line1 = ""
> >
> >
> > but that didn't work. It neither broke each line on etx (multiple events
> > with stx and etx on one line) nor did it concatenate the multi-line
> events.
>
> A big part of the challenge sounds like it's inconsistent data
> formatting.  You are going to have to identify some way to reliably
> check for the beginning/end of your data for it to work.  Do you know if
> you will always have \x02 at the start of a section of input, for example?
>
> The way I usually do log parsing in that case is use the stx as a flag
> to start doing other things (ie, if I find stx, stuff lines until I see
> the next stx, then dump and continue).  If you have intermediary data
> that is not between your stx and etx (comment lines, other data that you
> don't want), then it gets a lot harder.
>
> If you don't have at least a marginally consistent input, your only real
> option is probably going to be scanning by character and looking for the
> \x02 and \x03 to get a glob of data, then parse that glob with some kind
> of xml parser, since the data between those two is likely safe-ish.
>
> --
> David Rock
> da...@graniteweb.com
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] formatting xml (again)

[richard kappler]

I was actually working somewhat in that direction while I waited. I had in
mind to use something along the lines of:


stx = '\x02'
etx = '\x03'
line1 = ""

with open('original.log', 'r') as f1:
   with open('new.log', 'w') as f2:
for line in f1:
if stx in line:
line1 = line1 + line
if not stx in line:
if not etx in line:
line1 = line1 + line
if etx in line:
line1 = line1 + line + '\n'
f2.write(line1)
line1 = ""


but that didn't work. It neither broke each line on etx (multiple events
with stx and etx on one line) nor did it concatenate the multi-line events.

On Tue, Dec 27, 2016 at 3:25 PM, David Rock  wrote:

> * richard kappler  [2016-12-27 14:44]:
> >
> > I have tried to feed this raw into our other app (Splunk) and the app
> reads
> > each line (gedit numbered line) as an event. I want everything in between
> > each stx and etx to be one event.
> >
> > I have tried:
> >
> > #
> > with open("original.log", 'r') as f1:
> > with open("new.log", 'a') as f2:
> > for line in f1:
> > line2 = line.replace("\n", "")
> > f2.write(line2)
> > ##
> >
> > Now this obviously doesn't work because, as stated above, each tag and
> > datum in the example above from lines 2 to 7 is on a different line, so
> > python is doing exactly as I tell it, it's stripping the \n and then
> > printing the line, but not concatenating everything between stx and etx
> on
> > one line, which is what I want it to do.
> >
> > What I'm trying to do is collapse the 'expanded lines' between stx and
> etx
> > to one line, but I just can't wrap my head around how to do it. Or to
> put,
> > and do, it another way, how do I read each line from the original file,
> but
> > write it to another file so that everything from stx to etx, including
> stx
> > and etx, are on one line in the file?
>
> Concatinate all your lines into a single output variable first, then
> write that to your log
>
> Pseudocode (ie, this won't run), but this should give you the idea
> (defining the parts to loop will be the challenge you will have to
> define for yourself).
>
> with open("original.log", 'r') as f1:
> with open("new.log", 'a') as f2:
> output = ""
> for line in f1:
> if between [x02] and [x03]:
> output =+ line.strip()
> else:
> f2.write(output)
> output = ""
>
> Basically, you need to loop over everything between your markers and put
> them in a single entry, then send that one entry all at once instead of
> piecemeal.
>
> I can come up with something a little more complete if you still need
> more help.
>
> --
> David Rock
> da...@graniteweb.com
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] formatting xml (again)

[richard kappler]

Using python 2.7 - I have a large log file we recorded of streamed xml data
that I now need to feed into another app for stress testing. The problem is
the data comes in 2 formats.

1. each 'event' is a full set of xml data with opening and closing tags +
x02 and x03 (stx and etx)

2. some events have all the xml data on one 'line' in the log, others are
in typical nested xml format with lots of white space and multiple 'lines'
in the log for each event, the first line of th e 'event' starting with an
stx and the last line of the 'event' ending in an etx.

Examples of how the data looks from an editor (gedit specifically in this
case):
1[x02]some stuffsome more
stuff>[x03]
2[x02]
3 
4   some stuff
5   somestuff
6
7[x03]

I have tried to feed this raw into our other app (Splunk) and the app reads
each line (gedit numbered line) as an event. I want everything in between
each stx and etx to be one event.

I have tried:

#
with open("original.log", 'r') as f1:
with open("new.log", 'a') as f2:
for line in f1:
line2 = line.replace("\n", "")
f2.write(line2)
##

Now this obviously doesn't work because, as stated above, each tag and
datum in the example above from lines 2 to 7 is on a different line, so
python is doing exactly as I tell it, it's stripping the \n and then
printing the line, but not concatenating everything between stx and etx on
one line, which is what I want it to do.

What I'm trying to do is collapse the 'expanded lines' between stx and etx
to one line, but I just can't wrap my head around how to do it. Or to put,
and do, it another way, how do I read each line from the original file, but
write it to another file so that everything from stx to etx, including stx
and etx, are on one line in the file?

regards Richard
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] please help me modify this code so that I can utilize raw_input

[Richard Koeman]

I would like to modify this code so that instead of me calling the function
with the list as shown [1,2,3,4], the user inputs the list with raw_input.

Thanks in advance


"""Define a function sum() and a function multiply() that sums and
multiplies (respectively) all the numbers in a list of numbers. For
example, sum([1, 2, 3, 4]) should return 10, and multiply([1, 2, 3, 4])
should return 24."""

def sum(mylist = [], *args):
  print mylist
  sum_of_number = 0
  for i in mylist:
sum_of_number += i
  print "The sum of the digits in your number is %s" %(sum_of_number)
def multiply(mylist = [], *args):
  product_of_number = 1
  for i in mylist:
product_of_number = product_of_number * i
  print "The product of the digits in your number is %s"
%(product_of_number)

sum([1,2,3,4])
multiply([1,2,3,4])

-- 
This is a staff email account managed by Simcoe Muskoka Catholic District 
School Board.  This email and any files transmitted with it are 
confidential and intended solely for the use of the individual or entity to 
whom they are addressed. If you have received this email in error please 
notify the sender.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] python coding problem

[Richard Koeman]

This is my first time using this so I hope it works.
I am trying to find out why this code doesnt work.
Its simple.  I want to find out which number is bigger.

I hope you can help and that I am using this python feature properly.
Thanks.
The function prints the first two print statements then nothing else
happens.

def maximum(n1, n2):
  print "the first number is" ,n1
  print "the second number is", n2
  if n1 > n2:
return
print n1 , "is the bigger number"
  elif n1 < n2:
return n2
print n2
  elif n1 == n2:
print "they are equal"
  else:
print "Sorry, No response found"
maximum(1,2)

-- 
This is a staff email account managed by Simcoe Muskoka Catholic District 
School Board.  This email and any files transmitted with it are 
confidential and intended solely for the use of the individual or entity to 
whom they are addressed. If you have received this email in error please 
notify the sender.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] Buffer code review

[richard kappler]

, i)
else:
brokenLine = 1
buf += line
logging.info('broken line sending to buf line ' +
str(i))
f3.write(buf + '\n')
elif '' in line:
logging.info('sending heartbeatdata line ' + str(i))
brokenLine = 0
sendLine(line, i)
else:
brokenLine = 1
buf += line
logging.info('broken line sending to buf line ' +
str(i))
f3.write(buf + '\n')
else:
logging.error('tag match fail line ' + str(i))
buf = ''

# update log.txt file size
eof = neweof
logging.info('eof set to neweof, sleeping 2, eof =' + str(eof))
time.sleep(2)

 elif neweof < eof:
# this resets eof at night when old log file zipped and new log
file started
eof = 0
logging.info('neweof < eof, set eof = 0, sleeping 2')
time.sleep(2)

 elif neweof == eof:
# file hasn't changed, do nothing
logging.info('neweof = eof, file has not changed, sleeping 2')
time.sleep(2)
#

regards, Richard

-- 

*Java is like Alzheimers; it starts slow and eventually, it takes away all
of your memory.*
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] declare a variable inside a class

[richard kappler]

On Fri, Feb 12, 2016 at 2:23 AM, Ian Winstanley 
wrote:

> To refer to a class variable you need to put the class name first for
> example MyClass.xslt
>

That indeed did the trick, thanks Ian.

regards, Richard

-- 

*Java is like Alzheimers; it starts slow and eventually, it takes away all
of your memory.*
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] declare a variable inside a class

[richard kappler]

Thanks for the reply Martin, and in this instance I cannot post the actual
code (company rules). What I can do is say that with the xslt variable
defined within the formatter method, everything works, but when I pull it
out and put it in the upper level of the class, it gives me a traceback
that says the global variable xslt is not defined. Does that help?

regards, Richard

On Thu, Feb 11, 2016 at 3:43 PM, Martin A. Brown 
wrote:

>
> Hi there again Richard,
>
> [snipped a bit of stuff]
>
> >The method that actually formats the messages opens the xslt file,
> >then uses it to format the message. Clever boy that I am, I figured
> >it would be better to open the xslt file once, rather than once per
> >message, so I moved the line
> >
> >xslt = ('path/to/xslt.file')
>
> Yes, this is a good idea.
>
> >out of the method it was in and up to the top level of the class, to wit:
> >
> >class MyClass():
> >
> >xslt = ('path/to/xslt.file')
> >
> >   def a_bunch-of-methods():
>
> An impossible method name.  That would be a syntax error.
>
> >and so on.
> >
> >Obviously this didn't work,
>
> Why is it obvious?  What was obvious to me is the syntax error, but
> that does not appear to be what you are asking.
>
> What was the error message?
>
> >when the formatting method was called, it through an exception that
> >the variable xslt wasn't defined. This is my first not-a-toy try at
> >OOP, I could use a clue.
>
> So, before addressing your question, could I gently recommend that
> you post your actual code (clean up any variables or data that you
> don't want the public to see), whether it is running or not, along
> with any error messages (pasted, please).
>
> This is just a reminder, that this reduces the guesswork on the part
> of the members of the list.
>
> I think your problem is simply not knowing the name of the variable
> you called 'xslt'.  Perhaps the below example helps?
>
> -Martin
>
> #! /usr/bin/python3
>
> class MyClass(object):
>
> xslt = '/usr/share/nmap/nmap.xsl'
>
> def find_xslt(self):
> print(xslt)
>
> def oh_there_it_is(self):
> print(self.xslt)
>
> def redefine_in_instance(self):
> self.xslt = '/usr/share/sgml/X11/xorg.xsl'
>
> if __name__ == '__main__':
>
> # -- instantiate the class
> c = MyClass()
>
> # -- this is what I think you were doing
> try:
> c.where_is_xslt()
> except AttributeError:
> pass
>
> # -- but, try locating the class attribute through self
> c.oh_there_it_is()
>
> # -- class attribute can be overridden in instances
> c.redefine_in_instance()
> c.oh_there_it_is()
>
> # -- newly defined instances will get the default class attribute
> d = MyClass()
> d.oh_there_it_is()
>
> --
> Martin A. Brown
> http://linux-ip.net/
>



-- 

*Java is like Alzheimers; it starts slow and eventually, it takes away all
of your memory.*
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] declare a variable inside a class

[richard kappler]

Trying to optimize the modular input we wrote for Splunk, the basic
structure (dictated by Splunk) is the entire script is a class with
numerous methods within. The modular input reads a tcp stream, starts a new
thread for each source, reads the messages from that source into a buffer,
checks for stx and etx, formats it using an xslt and sends the result out
to splunk.

The method that actually formats the messages opens the xslt file, then
uses it to format the message. Clever boy that I am, I figured it would be
better to open the xslt file once, rather than once per message, so I moved
the line

xslt = ('path/to/xslt.file')

out of the method it was in and up to the top level of the class, to wit:

class MyClass():

xslt = ('path/to/xslt.file')

   def a_bunch-of-methods():

and so on.

Obviously this didn't work, when the formatting method was called, it
through an exception that the variable xslt wasn't defined. This is my
first not-a-toy try at OOP, I could use a clue.

regards, Richard

-- 

*Java is like Alzheimers; it starts slow and eventually, it takes away all
of your memory.*
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] script follows a file into zip?

[richard kappler]

Ah, I see, makes perfect sense now that it's right under my nose. Thanks
Peter!

regards, Richard

On Fri, Feb 5, 2016 at 12:04 PM, Peter Otten <__pete...@web.de> wrote:

> richard kappler wrote:
>
> > I have a script that checks a file and if there are new additions to the
> > file, parses and sends the new additions. This log rolls over and zips at
> > midnight creating a new log of the same name. The problem is, when the
> > file rolls over, the new file seems to follow the old file the way the
> > script is currently written.
> >
> > Current script (edited for brevity):
> > ###
> > from time import sleep
> >
> > f1 = open('theFile.log', 'r')
> >
> > f1.seek(0,2)
> > eof = f1.tell()
> >
> > While True:
> > try:
> > #check file size to see if grown, set neweof
> > f1.seek(0,2)
> > neweof = f1.tell()
> > except ValueError:
> > f1 = open(rd1, 'r')
> > f1.seek(0,2)
> > neweof = f1.tell()
> >
> > #if the file is larger...
> > if neweof > eof:
> > do a bunch of stuff
> >
> > # update log.txt file size
> > eof = neweof
> > time.sleep(10)
> >
> > elif neweof < eof:
> > # this resets eof at night when old log file zipped and new log
> > file started
> > eof = 0
> > time.sleep(10)
> >
> > elif neweof == eof:
> >  # file hasn't changed, do nothing
> >  time.sleep(10)
> > #
> >
> > The script works great all day until the logrotate at midnight. I would
> > expect to get the elif neweof < eof bit then, but my log shows I'm
> getting
> > elif neweof == eof for the rest of the night, which is what leads me to
> > believe that the script, as written, is following the zipped file, not
> > looking at the new file.
> >
> > I have two thoughts on this, not sure which to use, or if something else
> > completely might be appropriate.
> > 1. Instead of
> >
> > f1 = open('theFile.log', 'r')
> >
> > while True:
> > 
> >
> > Would it make a difference if I used
> >
> > with open('theFile.log', 'r') as f1:
> > while True:
> > and so on
>
> No. The problem is that you keep the file open. When the file is
> overwritten
> by another application only the entry in the file system is replaced while
> the underlying file (identified by its inode) is kept.
>
> The file you see has now no entry in the file system and therefore could
> only be changed by you; as you only read it no more changes will happen.
>
> To see the file that is  is currently registered under the name
> "theFile.log" you have to reopen:
>
> while True:
> with open("theFile.log") as f1:
> ...
>
>
> > 2. Get rid of the elif neweof == eof statement and just make it
> > else:
> >pass
> >
> > to send it back to the beginning of the loop (doesn't make sense to me,
> > but wanted to throw it out there).
> >
> > regards, Richard
>
>
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>



-- 

No matter where you go, there you are. - Buckaroo Banzai
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] script follows a file into zip?

[richard kappler]

I have a script that checks a file and if there are new additions to the
file, parses and sends the new additions. This log rolls over and zips at
midnight creating a new log of the same name. The problem is, when the file
rolls over, the new file seems to follow the old file the way the script is
currently written.

Current script (edited for brevity):
###
from time import sleep

f1 = open('theFile.log', 'r')

f1.seek(0,2)
eof = f1.tell()

While True:
try:
#check file size to see if grown, set neweof
f1.seek(0,2)
neweof = f1.tell()
except ValueError:
f1 = open(rd1, 'r')
f1.seek(0,2)
neweof = f1.tell()

#if the file is larger...
if neweof > eof:
do a bunch of stuff

# update log.txt file size
eof = neweof
time.sleep(10)

elif neweof < eof:
# this resets eof at night when old log file zipped and new log
file started
eof = 0
time.sleep(10)

elif neweof == eof:
 # file hasn't changed, do nothing
 time.sleep(10)
#

The script works great all day until the logrotate at midnight. I would
expect to get the elif neweof < eof bit then, but my log shows I'm getting
elif neweof == eof for the rest of the night, which is what leads me to
believe that the script, as written, is following the zipped file, not
looking at the new file.

I have two thoughts on this, not sure which to use, or if something else
completely might be appropriate.
1. Instead of

f1 = open('theFile.log', 'r')

while True:


Would it make a difference if I used

with open('theFile.log', 'r') as f1:
while True:
and so on

2. Get rid of the elif neweof == eof statement and just make it
else:
   pass

to send it back to the beginning of the loop (doesn't make sense to me, but
wanted to throw it out there).

regards, Richard
-- 

No matter where you go, there you are. - Buckaroo Banzai
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] reading an input stream

[richard kappler]

On Thu, Jan 7, 2016 at 5:07 PM, Cameron Simpson  wrote:

>
> Just a few followup remarks:
>
> This is all Python 3, where bytes and strings are cleanly separated.
> You've got a binary stream with binary delimiters, so we're reading binary
> data and returning the binary XML in between. We separately decode this
> into a string for handing to your XML parser. Just avoid Python 2
> altogether; this can all be done in Python 2 but it is not as clean, and
> more confusing.
>

Love to, can't. Splunk uses 2.7 so that's what we have to work with. That
will not change in the forseeable future. Doing other homework right now,
but will more closely review this and the other posts that have come in
since I left work later tonight or first thing in the morning.

regards, Richard
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Syntax error

[richard kappler]

Hi Sarah!

instead of 'python hello.py', try

>>>import hello.py

using python hello.py works from the Linux command line (presumably Windows
as well) and it starts python then runs the hello.py script. From within
the python interpreter, you import the file and it executes.

HTH, Richard

On Thu, Jan 7, 2016 at 1:01 PM, Joel Goldstick 
wrote:

> On Thu, Jan 7, 2016 at 8:40 AM, Sarah Rasco 
> wrote:
>
> > Hello,
> >
> > I'm new to programming and was told that Python would be a good language
> to
> > start with. I downloaded version 3.5.1, and I have Windows 10.
> >
> > In IDLE, I typed print ("Hello, world!") and hit enter, and it returned
> the
> > message. I saved the file as hello.py in C:\python. Then, when I tried to
> > run it in IDLE, I got a syntax error and it highlighted the '5' in the
> > prompt 'python 3.5.1'.
> >
> > I also tried to run it in my windows command prompt. I put in cd
> C:\python
> > and it gave me the python prompt. Then, when I tried to open the file by
> > typing python hello.py, I was given a syntax error again. Does anyone
> have
> > any suggestions as to what the problem could be?
> >
> > Thank you!
> > ___
> > Tutor maillist  -  Tutor@python.org
> > To unsubscribe or change subscription options:
> > https://mail.python.org/mailman/listinfo/tutor
> >
>
> Welcome Sarah.
>
> Can you copy and paste the traceback (error message) that you get when you
> run your code.  Also, copy and paste your complete code text.
>
> --
> Joel Goldstick
> http://joelgoldstick.com/stats/birthdays
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>



-- 

All internal models of the world are approximate. ~ Sebastian Thrun
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] multipart socket streaming problem: threading??

[richard kappler]

This is a continuation of the thread 'reading an input stream' I had to
walk away from for a few days due to the holidays and then other work
considerations, and I figured it best to break my confusion into separate
chunks, I hope that's appropriate. In short, my script needs to read a
stream of xml data from a socket (port 2008), the data coming in from as
many as 30 different machines, but usually 4 or less, as many as 3 messages
per second from each machine at times, messages block format delimited by
stx(\x02) and etx (\x03), send the data in those blocks to a parser
(already built using lxml and an xslt file) and send it out to splunk using
a native 'event writer'.

And finally, for now, I suspect I'll need to run each 'connection', that is
the data from each sending machine on a separate thread. I'm not well
versed in threading, but have read some of the documentation and it seems
pretty straight forward, if I have questions I will of course ask, but the
preliminary question is: Do I in fact need to do this? Is threading the way
to go or is just running all the data through the same 'pipe'
feasible/Pythonic/efficient?

regards, Richard
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] multipart socket streaming problem: the delimiters

[richard kappler]

This is a continuation of the thread 'reading an input stream' I had to
walk away from for a few days due to the holidays and then other work
considerations, and I figured it best to break my confusion into separate
chunks, I hope that's appropriate. In short, my script needs to read a
stream of xml data from a socket (port 2008), the data coming in from as
many as 30 different machines, but usually 4 or less, as many as 3 messages
per second from each machine at times, messages block format delimited by
stx(\x02) and etx (\x03), send the data in those blocks to a parser
(already built using lxml and an xslt file) and send it out to splunk using
a native 'event writer'.

Once I figure out how to get the messages in and read in general (my
current test file has only one message) I'll need to figure out how to pull
them, one at a time (one message from stx to etx) and send to the parser. I
am totally lost here. Once I can get the data read from stream rather from
file, how do I write the part of the script that recognizes block format?
I've looked for libraries but am apparently not writing a good search, as I
always get sites that discuss blocking and non-blocking sockets, not block
format messages. Is there a library or tutorial out there anyone knows of
that might help with this?

regards, Richard
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] multipart socket streaming problem: reading the stream

[richard kappler]

This is a continuation of the thread 'reading an input stream' I had to
walk away from for a few days due to the holidays and then other work
considerations, and I figured it best to break my confusion into separate
chunks, I hope that's appropriate. In short, my script needs to read a
stream of xml data from a socket (port 2008), the data coming in from as
many as 30 different machines, but usually 4 or less, as many as 3 messages
per second from each machine at times, messages block format delimited by
stx(\x02) and etx (\x03), send the data in those blocks to a parser
(already built using lxml and an xslt file) and send it out to splunk using
a native 'event writer'.

I am a bit lost in the woods on 'reading' the stream. My first attempt
tried to read the stream directly (from the buffer 'I think'):

#!/usr/bin/env python


import socket

import lxml.etree as ET


# receive socket

sockrx = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

sockrx_address = ('', 2008)

print 'opening sockrx on %s port %s' % sockrx_address

try:

sockrx.bind(sockrx_address)

except socket.error as msg:

print 'Bind failed. Error code: ' + str(msg)

sys.exit()


sockrx.listen(5)

print 'listening'


while True:

# wait for a connection

connection, client_address = sockrx.accept()

data = connection.recv(8192)

if data:

print 'receiving data'

f1 = open('parser.out', 'a')

xslt = ET.parse('stack13.xsl')

dom = ET.parse(data)

transform = ET.XSLT(xslt)

newdom = transform(dom)

f1.write(str(newdom))



# close sockrx

connection.close()


f1.close()


This did not work:

Traceback (most recent call last):

  File "streamer-01.py", line 27, in 

dom = ET.parse(data)

  File "lxml.etree.pyx", line 3239, in lxml.etree.parse
(src/lxml/lxml.etree.c:69955)

  File "parser.pxi", line 1748, in lxml.etree._parseDocument
(src/lxml/lxml.etree.c:102066)

  File "parser.pxi", line 1774, in lxml.etree._parseDocumentFromURL
(src/lxml/lxml.etree.c:102330)

  File "parser.pxi", line 1678, in lxml.etree._parseDocFromFile
(src/lxml/lxml.etree.c:101365)

  File "parser.pxi", line 1110, in lxml.etree._BaseParser._parseDocFromFile
(src/lxml/lxml.etree.c:96817)

  File "parser.pxi", line 582, in
lxml.etree._ParserContext._handleParseResultDoc
(src/lxml/lxml.etree.c:91275)

  File "parser.pxi", line 683, in lxml.etree._handleParseResult
(src/lxml/lxml.etree.c:92461)

  File "parser.pxi", line 620, in lxml.etree._raiseParseError
(src/lxml/lxml.etree.c:91722)

IOError: Error reading file

and then the error message printed out the entire file to screen, which I
won't here  for brevity.


I was, however, able to receive the the stream as above, write it to a
file, then read the file to the parser and it worked. I tired using
makefile() but that did not work either. I'm looking for some general
direction here, more specific questions to follow.

regards, Richard
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] multipart socket streaming problem: the socket

[richard kappler]

This is a continuation of the thread 'reading an input stream' I had to
walk away from for a few days due to the holidays and then other work
considerations, and I figured it best to break my confusion into separate
chunks, I hope that's appropriate. In short, my script needs to read a
stream of xml data from a socket (port 2008), the data coming in from as
many as 30 different machines, but usually 4 or less, as many as 3 messages
per second from each machine at times, messages block format delimited by
stx(\x02) and etx (\x03), send the data in those blocks to a parser
(already built using lxml and an xslt file) and send it out to splunk using
a native 'event writer'.

Here's what I have thus far for the socket, it works though has not been
tested for multiple connections yet, how should it be improved to be more
Pythonic, robust and efficient?:

#!/usr/bin/env python

import socket

# receive socket
sockrx = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sockrx_address = ('', 2008)
print 'opening sockrx on %s port %s' % sockrx_address
try:
sockrx.bind(sockrx_address)
except socket.error as msg:
print 'Bind failed. Error code: ' + str(msg)
sys.exit()

sockrx.listen(5)
print 'listening'

# wait for a connection
connection, client_address = sockrx.accept()
data = connection.recv(8192)

I have questions including:
- Should I increase the 5 connection limit to 30+ as it may be listening to
that many machines?
- Buffer size is set at 8192, yet the messages may be much larger, can I
safely increase that? Should I?
- I'm presuming that the OS handles assembly of a message that comes in
more than one packet using the TCP/IP protocols, but is that true?

regards, Richard
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] reading an input stream

[richard kappler]

Sorry it took so long to respond, just getting back from the holidays. You
all have given me much to think about. I've read all the messages through
once, now I need to go trough them again and try to apply the ideas. I'll
be posting other questions as I run into problems. BTW, Danny, best
explanation of generators I've heard, well done and thank you.

regards, Richard

On Thu, Dec 24, 2015 at 4:54 PM, Danny Yoo  wrote:

> > I think what I need to do would be analogous to (pardon if I'm using the
> > wrong terminology, at this poing in the discussion I am officially out of
> > my depth) sending the input stream to a buffer(s) until  the ETX for that
> > message comes in, shoot the buffer contents to the parser while accepting
> > the next STX + message fragment into the buffer, or something analogous.
>
> Yes, I agree.  It sounds like you have one process read the socket and
> collect chunks of bytes delimited by the STX markers.  It can then
> send those chunks to the XML parser.
>
>
> We can imagine one process that reads the socket and spits out a list
> of byte chunks:
>
> chunks = readDelimitedChunks(socket)
>
> and another process that parses those chunks and does something with them:
>
> for chunk in chunks:
> 
>
>
> It would be nice if we could organize the program like this.  But one
> problem is that chunks might not be finite!  The socket might keep on
> returning bytes.  If it keeps returning bytes, we can't possibly
> return a finite list of the chunked bytes.
>
>
> What we really want is something like:
>
> chunkStream = readDelimitedChunks(socket)
> for chunk in chunkStream:
> 
>
> where chunkStream is itself like a socket: it should be something that
> we can repeatedly read from as if it were potentially infinite.
>
>
> We can actually do this, and it isn't too bad.  There's a mechanism in
> Python called a generator that allows us to write function-like things
> that consume streams of input and produce streams of output.  Here's a
> brief introduction to them.
>
> For example, here's a generator that knows how to produce an infinite
> stream of numbers:
>
> ##
> def nums():
> n = 0
> while True:
> yield n
> n += 1
> ##
>
> What distinguishes a generator from a regular function?  The use of
> "yield".  A "yield" is like a return, but rather than completely
> escape out of the function with the return value, this generator will
> remember what it was doing  at that time.  Why?  Because it can
> *resume* itself when we try to get another value out of the generator.
>
> Let's try it out:
>
> #
>
> >>> numStream = nums()
> >>> numStream.next()
> 0
> >>> numStream.next()
> 1
> >>> numStream.next()
> 2
> >>> numStream.next()
> 3
> >>> numStream.next()
> 4
> #
>
> Every next() we call on a generator will restart it from where it left
> off, until it reaches its next "yield".  That's how we get this
> generator to return an infinite sequence of things.
>
>
> That's how we produce infinite sequences.  And we can write another
> generator that knows how to take a stream of numbers, and square each
> one.
>
> 
> def squaring(stream):
> for n in stream:
> yield n
> 
>
>
> Let's try it.
>
>
> 
>
> >>> numStream = nums()
> >>> squaredNums = squaring(numStream)
> >>> squaredNums.next()
> 0
> >>> squaredNums.next()
> 1
> >>> squaredNums.next()
> 4
> >>> squaredNums.next()
> 9
> >>> squaredNums.next()
> 16
> 
>
>
> If you have experience with other programming languages, you may have
> heard of the term "co-routine".  What we're doing with this should be
> reminiscent of coroutine-style programming.  We have one generator
> feeding input into the other, with program control bouncing back and
> forth between the generators as necessary.
>
>
> So that's a basic idea of generators.  It lets us write processes that
> can deal with and produce streams of data.  In the context of sockets,
> this is particularly helpful, because sockets can be considered a
> stream of bytes.
>
>
> Here's another toy example that's closer to the problem you're trying
> to solve.  Let's say that we're working on a program to alphabetize
> th

[Tutor] reading an input stream

[richard kappler]

I have to create a script that reads  xml data over a tcp socket, parses it
and outputs it to console. Not so bad, most of which I already know how to
do. I know how to set up the socket, though I am using a file for
development and testing, am using lxml and have created an xslt that does
what I want with the xml, and it outputs it to console.

What I'm not really sure of, each xml 'message' is preceeded by an STX
(\x02) and ends with an ETX (\x03). These 'messages' (Danny, are you noting
I don't say -lines- anymore? :-)  ) need to be parsed and output whole as
opposed to partial.

My concern is, there will actually be numerous machines sending data to the
tcp socket, so it's entirely likely the messages will come in fragmented
and the fragments will need to be held until complete so they can be sent
on whole to the parser. While this is the job of tcp, my script needs to

I think what I need to do would be analogous to (pardon if I'm using the
wrong terminology, at this poing in the discussion I am officially out of
my depth) sending the input stream to a buffer(s) until  the ETX for that
message comes in, shoot the buffer contents to the parser while accepting
the next STX + message fragment into the buffer, or something analogous.

Any guidance here?

And Merry Christmas / Happy Holidays / Festive whatever you celebrate!!!

regards, Richard

-- 

"I want to makes shoes!" -> elf fleeing the fire engulfed Keebler Tree
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] Modules and variable usage in functions from other files

[Richard Bekenstein]

   Dear All,

   I have a main .py script which contains code that occasionally calls some
   functions. This script calls in these defined functions from other .py
   files in the same directory as the main .py. However, even after importing
   the functions from the different .py files, I get error messages, such as
   "global name 'np' is not defined" even though I define "import numpy as
   np" initially in my main .py script.
   To show some detail, I import some modules at the beggining of my main .py
   script as such:

   from scipy.constants import m_p,G,k
   import numpy as np
   from math import cos, pi, floor
   import matplotlib.pyplot as plt
   from findall import findall
   from pylab import *
   import sys
   import timeit
   from copy import copy
   from array import array
   from bol_heatFTCS_nonadaptive import heatFTCS #this is my first
   user-defined function that calls the defined function 'heatFTCS' from file
   'bol_heatFTCS_nonadaptive.py'
   from bol_runge_kutta_functions import dy1_dt,dy2_dt,dy3_dt,dy5_dt
   #this is my second user-defined function that calls the defined functions
   'dyi_dt's from file 'bol_runge_kutta_functions.py'
   from bol_runge_kutta_evl_nonadaptive import runge_kutta_evl #this is
   my first user-defined function that calls the defined function
   'runge_kutta_evl' from file 'bol_runge_kutta_evl_nonadaptive.py'
   m_range,p_range,T_range,r_range,rho_range = np.loadtxt('(there are
   some other directories here that don't matter for this
   discussion)/planetary_structure_params.txt', unpack=True,
   usecols=[0,1,2,3,4])  #imports some data to 5 arrays


   When I run this main.py, I get an error message from the imported
   'runge_kutta_evl' function that numpy is not defined. Why is it that even
   though I have it defined in my script it does not recognize it in my
   function called to the script? It turns out that even if I then state
   'import numpy as np' in my 'runge_kutta_evl' function, I then another
   error that 'r_range' is not defined. But aren't 'r_range' and 'np'
   supposed to be accesible to all called functions in the main .py?

   Thank you,
   R.B.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] responding to command line

[richard kappler]

I need to write a script that runs ssh-copy-id to over 500 clients from one
master (setting up keys to use ansible).

I understand how to get my script to read each IP address in a list, use it
as input and then move on to the next, but not sure how to handle the
password feed, particularly as there will be a considerable delay in the
passwords request. Specifically:

$ssh-copy-id user@IPofClient

 delay of as much as 90 or more seconds.

user@IPofClient's password:


and I want the script to detect that password request and provide the
password (Same for all 500 + machines, so no issue there).



-- 

All internal models of the world are approximate. ~ Sebastian Thrun
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] parsing xml as lines

[richard kappler]

I have an xml file that get's written to as events occur. Each event writes
a new 'line' of xml to the file, in a specific format, eg: sometthing like
this:

http://www.w3.org/2001/XMLSchema-instance";
xsi:noNamespaceSchemaLocation="Logging.xsd" version="1.0">somestuff

and each 'line' has that same structure or format.

I've written a script that parses out the needed data and forwards it on
using regex's, but think it might be better to use an xml parser. I can
parse out what I need to if I have just one line in the file, but when
there are number of lines as there actually are, I can't figure out how to
get it to work.

In other words, with a one line file, this works fine and I understand it:

import xml.etree.cElementTree as ET
tree = ET.ElementTree(file='1lineTest.log'
grandchild = tree.find('grandchild')
print grandchild.tag, grandchild.text

and I get the output I desire:

grandchild Sally

But if I have several lines in the file try to run a loop:

import xml.etree.cElementTree as ET
f1 = open('5lineTest.log', 'r')
lineList = f1.readlines()
Imax = len(lineList)

i = 0
while i <= Imax:
tree = ET.ElementTree(lineList[i])
grandchild = tree.find('grandchild')
print grandchild.tag, grandchild.txt
i += 1

Traceback (most recent call last):
  File "", line 4, in 
AttributeError: 'int' object has no attribute 'tag'

and yet I can do:

print lineList[0] and it will print out the first line.

I get why (I think), I just can't figure out a way around it.

Guidance please?
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] socket communications and threading

[richard kappler]

Sorry, thought it was clear. Each of the three different data generating
machines (in the test env, the python script that sends the data with 3
different device names) goes over a different thread so the developers tell
me. In production, those three machines are microcontrollers, not full
blown computers. VM2 is a computer, so that must be where the threads
'occur' (again, I don't understand this well) but in the examples I read,
it was about which was server and which was client, hence the connection
between client/server and threading. With your explanation I am off to
re-read the tutorials and examples, thank you.

regards, Richard

On Tue, Oct 27, 2015 at 1:44 PM, Alan Gauld 
wrote:

> On 27/10/15 14:52, richard kappler wrote:
>
> In our test environment we have simulated this by building three vm's. VM1
>> has a python script that sends raw data over tcp to VM2 which parses the
>> data and sends it over tcp to VM3 upon which we are developing our
>> analytics apps.
>> ...
>>
>
> 1. The data from the three different machines each gets it's own thread in
>> production, so that would have to happen on 'VM2' as the 'VM1' are
>> actually
>> just microcontrollers out  in production. From a socket and threading
>> perspective, which would be considered the client  and which the server,
>> VM1 (the sender) or VM2 (the receiver)?
>>
>
> Client and server are about roles. The question is therefore
> which machine is requesting a service and which is providing
> it? Sounds like for the first transaction VM1 is asking VM2 to
> store the data, so VM1 is client, VM2 is server.
>
> However, for the analytics part, VM2 is asking for analysis and
> VM3 doing the work so VM2 is client in that transaction and VM3
> the server.
>
> 2. The process has worked mediocre at best thus far. When I developed the
>> two python scripts (tunnelSim to send over socket and parser to rx and tx
>> over socket) I started by just reading and writing to files so I could
>> concentrate on the parsing bit. Once that was done and worked very well I
>> added in sockets for data flow and commented out the read from and to
>> files
>> bits, and everything seemed to work fine, VM1 sent a number of 'lines',
>> VM2
>> received the same number of 'lines', parsed them and, seemed to send them
>> on. Some days analytics (VM3) got them all, some days it did not. Not sure
>> where to look, and any thoughts on troubleshooting this would be helpful,
>> but the main point of the entire email is question 1, threading.
>>
>
> Where is the threading question in #1? I only saw a question about
> client/server - which has nothing at all to do with threading?
>
> Slightly confused.
>
> --
> Alan G
> Author of the Learn to Program web site
> http://www.alan-g.me.uk/
> http://www.amazon.com/author/alan_gauld
> Follow my photo-blog on Flickr at:
> http://www.flickr.com/photos/alangauldphotos
>
>
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>



-- 

All internal models of the world are approximate. ~ Sebastian Thrun
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] socket communications and threading

[richard kappler]

I'm having difficulty wrapping my arms around sockets and threading, not so
much from a 10,000 foot/ network perspective, but from a low level
perspective.

In our production environment we have three machines that generate data and
forward it over tcp to a computer that stores the data, parses it and sends
it on to another computer over tcp for analytics.

In our test environment we have simulated this by building three vm's. VM1
has a python script that sends raw data over tcp to VM2 which parses the
data and sends it over tcp to VM3 upon which we are developing our
analytics apps.

Note that VM1 script reads from a .log file which is the actual output from
three real machines, differentiated by a unique deviceID, and each line or
'event' in the .log file is read and sent over tcp with a 0.5 second delay
between each read.

VM2 has a python script that receives through a socket the raw data over
tcp, parses it and sends it on through a socket over tcp to VM3.

My lack of confusion arises for a couple reasons.

1. The data from the three different machines each gets it's own thread in
production, so that would have to happen on 'VM2' as the 'VM1' are actually
just microcontrollers out  in production. From a socket and threading
perspective, which would be considered the client  and which the server,
VM1 (the sender) or VM2 (the receiver)?

2. The process has worked mediocre at best thus far. When I developed the
two python scripts (tunnelSim to send over socket and parser to rx and tx
over socket) I started by just reading and writing to files so I could
concentrate on the parsing bit. Once that was done and worked very well I
added in sockets for data flow and commented out the read from and to files
bits, and everything seemed to work fine, VM1 sent a number of 'lines', VM2
received the same number of 'lines', parsed them and, seemed to send them
on. Some days analytics (VM3) got them all, some days it did not. Not sure
where to look, and any thoughts on troubleshooting this would be helpful,
but the main point of the entire email is question 1, threading.

regards, Richard
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] general exception questions

[richard kappler]

>
>
>
> how else do you skip the current line if the 'try' can't be done, and go on
>> to the next line exiting the program with a trace error?
>>
>
> That last sentence confused me. If you use pass (or continue)
> you will NOT get any trace error. If you want to store the
> error to report it at the end then it's quite a bit more work,
> you would need something like:
>

Yup, confused myself when I reread it, sorry Alan. It should have been ..if
the 'try' can't be done, and go on to the next line INSTEAD OF exiting the
program with a trace error.

>
> errors = []
> for line in file:
>try:
>  process(line)
>except SomeError as e:
>   errors.append((line,e))
>
> if errors:
>for e in errors:
>   display_error(e)
>
> where display_error() is an exercise for the student :-)
>

Interesting. Sounds like something else I need to take a round turn on.

regards, Richard
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] general exception questions

[richard kappler]

I'm reading up on exception handling, and am a little confused. If you have
an exception that just has 'pass' in it, for example in a 'for line in
file:' iteration, what happens? Does the program just go to the next line?

EX:

for line in file:
try:
do something
except:
pass

I know (so many of you have told me :-) that using pass is a bad idea, but
how else do you skip the current line if the 'try' can't be done, and go on
to the next line exiting the program with a trace error?

regards, Richard
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] stx, etx (\x02, \x03)

[richard kappler]

>
> Hi Richard,
>
> Just to check: what operating system are you running your program in?
> Also, what version of Python?
>

Hi Danny, using Linux and Python 2.7

>
>
>
> ##
> with open('input/test.xml', 'rU') as f1: ...
> ##
>
>
> Question: can you explain why the program is opening 'mod1.xml' in 'a'
> append mode?  Why not in 'w' write mode?
>
>
> This may be important because multiple runs of the program will append
> to the end of the file, so if you inspect the output file, you may be
> confusing the output of prior runs of your program.  Also, it's likely
> that the output file will be malformed, since there should just be one
> XML document per file.  In summary: opening the output file in append
> mode looks a bit dubious here.
>
>
>
> To your other question:
>
> > What am I missing?
>
> Likely, the lines being returned from f1 still have a line terminator
> at the end.  You'll want to interpose the '\x03' right before the line
> terminator.  Mark Laurence's suggestion to use:
>
> s = '\x02' + line[:-1] + '\x03\n'
>
> looks ok to me.
>

Actually, you solved it, but it required both

with open('input/test.xml', 'rU') as f1:...

and

s = 'x02' + line[:-1] + 'x03\n'

Either of the above alone do not resolve the issue, but both together do.

As far as the write vs append, I understand your concerns but that is for
testing during scripting only. Ultimately the script outputs to a socket on
another machine, not to a file, so that gets commented out. The append is
because I am using a 100 line test file with a

for line in f1:

statement in it, and I want the results for each line appended to the
output file. I delete the results file before each run.

Thanks for all the help!

regards, Richard



-- 

All internal models of the world are approximate. ~ Sebastian Thrun
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] stx, etx (\x02, \x03)

[richard kappler]

>
>>
> Reread my original post and you'll see that your "s" isn't set the same
> way mine was


No.  My implementation of your code:

#!/usr/bin/env python

with open('input/PS06Test_100Packages.xml', 'r') as f1:
with open('mod1.xml', 'a') as f2:
for line in f1:
s = '\x02' + line[:-1] + '\x03\n'
f2.write(s)

gives me:
line 1 starts with the \x02 hex then the line
line 2 is the \xo3 hex alone
line 3 starts with the \x02 hex then the line
line 4 is the \x03 hex alone
lather rinse repeat.

So I mispoke, please accept my apology, it wasn't exactly the same result
as my original code, it put the \x03 on it's own line.

Oddly enough, I'm wondering if it's the xml that's making things screwy. I
ran a little experiment, creating a test text file:

line one
line two
line three
line four

and ran the above code (changing the file names of course) and it came out
perfect. Each line begins with the \x02 hex and ends with the \x03 hex.

regards, Richard
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] stx, etx (\x02, \x03)

[richard kappler]

>
>
>
> > That is *very* unlikely.
>

Perhaps, but I assure you it is what is happening.


>
> > Verify that you deleted the existing mod1.xml. As you open the file in
> > append mode existing faulty lines persist.
>

I did, hence the above assurance.

Here's the modified code:

#!/usr/bin/env python

stx = '\x02'
etx = '\x03'

with open('input/PS06Test_100Packages.xml', 'r') as f1:
with open('mod1.xml', 'a') as f2:
for line in f1:
s = stx + line[:-1] + etx
f2.write(s)


mod1.xml has the \x02 hex at the beginning of line 1, then line 2 and all
further lines begin with the \x03 hex then the \x02 hex and then the line.

regards, Richard
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] stx, etx (\x02, \x03)

[richard kappler]

Thanks for the reply Mark, tried that, same result.

On Tue, Sep 22, 2015 at 8:58 AM, Mark Lawrence 
wrote:

> On 22/09/2015 13:37, richard kappler wrote:
>
>> I have a file with several lines. I need to prepend each line with \x02
>> and
>> append each line with \x03 for reading into Splunk. I can get the \x02 at
>> the beginning of each line, no problem, but can't get the \x03 to go on
>> the
>> end of the line. Instead it goes to the beginning of the next line.
>>
>> I have tried:
>>
>> #!/usr/bin/env python
>>
>> with open('input/test.xml', 'r') as f1:
>>  with open('mod1.xml', 'a') as f2:
>>  for line in f1:
>>  s = ('\x02' + line + '\x03')
>>  f2.write(s)
>>
>> as well as the same script but using .join, to no avail. What am I
>> missing?
>>
>> regards, Richard
>>
>>
> Your line has EOL on the end.  Getting rid of the unneeded brackets try:-
>
> s = '\x02' + line[:-1] + '\x03\n'
>
> --
> My fellow Pythonistas, ask not what our language can do for you, ask
> what you can do for our language.
>
> Mark Lawrence
>
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>



-- 

All internal models of the world are approximate. ~ Sebastian Thrun
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] stx, etx (\x02, \x03)

[richard kappler]

I have a file with several lines. I need to prepend each line with \x02 and
append each line with \x03 for reading into Splunk. I can get the \x02 at
the beginning of each line, no problem, but can't get the \x03 to go on the
end of the line. Instead it goes to the beginning of the next line.

I have tried:

#!/usr/bin/env python

with open('input/test.xml', 'r') as f1:
with open('mod1.xml', 'a') as f2:
for line in f1:
s = ('\x02' + line + '\x03')
f2.write(s)

as well as the same script but using .join, to no avail. What am I missing?

regards, Richard

-- 

All internal models of the world are approximate. ~ Sebastian Thrun
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] changing a variable with raw_input

[richard kappler]

Nevermind, figured it out. it needed to be delay = 0 and delay = 0.5, not ==

regards, Richard the virus ridden

On Wed, Sep 16, 2015 at 1:03 PM, richard kappler 
wrote:

> Missing something obvious here, but down with the flu so I'm foggy and
> just can't see it. I need to set a variable 'delay' to be used in
> time.sleep(delay) later on in a script, based on user input. Something odd
> is going on in setting the variable though.  Here's the snippet where I'm
> trying to set the variable:
>
> #!/usr/bin/env python
>
> delay = 7
>
> print("1 - Batch")
> print("2 - 2 per second")
> print("3 - Real Time")
> choice = raw_input("Enter feed speed choice (then press enter): ")
> choice = int(choice)
>
> if choice == 1:
> print "running as batch"
> delay == 0
>
> elif choice == 2:
> print "feeding 2 events per second"
> delay == 0.5
>
> elif choice == 3:
> print "feeding real time"
> delay = 'real time'
>
> else:
> print "choice not available"
> os._exit(0)
>
> print 'delay = ' + str(delay)
>
>
> If I enter 1 or 2, I get delay = 7, if I enter 3, I get delay = real time.
>
> HUH???
>
> regards, Richard
>
> --
>
> All internal models of the world are approximate. ~ Sebastian Thrun
>



-- 

All internal models of the world are approximate. ~ Sebastian Thrun
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] changing a variable with raw_input

[richard kappler]

Missing something obvious here, but down with the flu so I'm foggy and just
can't see it. I need to set a variable 'delay' to be used in
time.sleep(delay) later on in a script, based on user input. Something odd
is going on in setting the variable though.  Here's the snippet where I'm
trying to set the variable:

#!/usr/bin/env python

delay = 7

print("1 - Batch")
print("2 - 2 per second")
print("3 - Real Time")
choice = raw_input("Enter feed speed choice (then press enter): ")
choice = int(choice)

if choice == 1:
print "running as batch"
delay == 0

elif choice == 2:
print "feeding 2 events per second"
delay == 0.5

elif choice == 3:
print "feeding real time"
delay = 'real time'

else:
print "choice not available"
os._exit(0)

print 'delay = ' + str(delay)


If I enter 1 or 2, I get delay = 7, if I enter 3, I get delay = real time.

HUH???

regards, Richard

-- 

All internal models of the world are approximate. ~ Sebastian Thrun
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] ftp socket.error

[richard kappler]

Thanks for all the assistance, turned out it was a problem with the
iptables not having an accept for eth1. Onward and upward!

regards, Richard

On Sat, Sep 12, 2015 at 12:24 PM, Martin A. Brown 
wrote:

>
> Hello and good morning
>
> I may be mistaken, but it looks like you are trying to open the socket on
>> port 2021. Standard ftp uses 21. Is the server listening on 2021?
>>
>
> Ooof!  And, in fact, that is a great point!  I overlooked that in the
> original snippet!
>
> Everything I wrote still stands, except that you need to tell the
> ip_conntrack_ftp (or nf_conntrack_ftp) kernel module to watch for a command
> channel on TCP/2021.
>
>   modprobe ip_conntrack_ftp ports=21,2021
>
> That means that the ip_conntrack_ftp module will watch flows on both ports.
>
> I'm glad you observed that important detail, Robert!
>
> -Martin
>
>
> Strictly speaking, it's no Python question, but... good ol' FTP.
>>>
>>> socket.error: [Errno 113] No route to host
>>>
>>>>
>>>>>>
>>>>> Your program is receiving an EHOSTUNREACH.
>>>
>>>  >>> import errno
>>>  >>> errno.errorcode[113]
>>>   'EHOSTUNREACH'
>>>
>>> This occurs at precisely the moment that your program is trying to
>>> initiate a data transfer.  Every firewall administrator in the world
>>> loves
>>> FTP for precisely this reason.  (And, when I say "love", you can replace
>>> this with a verb or  of your choice.)
>>>
>>> Without packet captures, I will merely guess (based on experience).
>>>
>>>   1. The receiving machine is running the Python program, builds a
>>>  connection on port 21 (this is called the FTP command
>>>  channel), you log in and all is well.
>>>   2. The moment you try to transfer any data, the FTP client (your
>>>  receiving machine) and the FTP server negotiate either FTP
>>>  passive mode or FTP active (retronym) mode.  I'm guessing
>>>  that your FTP client is choosing passive mode.  (Your FTP
>>>  client might produce logging of this negotiation.)
>>>   3. Using the connection information, the client attempts to build
>>>  an FTP data channel.  So, your machine running the Python
>>>  program initiates a connection to the FTP server.
>>>   4. The FTP server is (probably) configured to allow connections
>>>  inbound to TCP/21 (FTP Command Channel), but doesn't know to
>>>  allow the connections to the ephemeral port(s) negotiated
>>>  during step 2 (above).  So, the firewall on the FTP Server
>>>  sends an ICMP Type 3, Code 1 [0].
>>>
>>> Figured it out. On the receiving machine  I had to
>>>
>>>>
>>>>> # modprobe ip_conntrack_ftp
>>>>>
>>>>>
>>>> Right instinct!  Try this same command on the FTP server side. Unless
>>> your
>>> Python FTP client is negotiating active mode, the server will be the
>>> "problem" in this case.
>>>
>>> No, apparently I didn't figure it out. I thought I had as after the
>>>
>>>> modprobe I was getting a an EOFError, but now I'm getting the no route
>>>> to
>>>> host error again. I can ping it, and as you can see from the original
>>>> post,
>>>> I am able to establish a connection and log in, it's just when I try to
>>>> send a file it goes bollocks up. Still need ideas.
>>>>
>>>>
>>> Hopefully, my idea #1 helps.  (If not, you'll need to do some packet
>>> captures and probably crank up the logging on the FTP server, too.)
>>>
>>> I do have another idea, though.  Have you ever wondered about the slow
>>> demise of FTP?  All of this command-channel, data-channel, PORT or PASV
>>> nonsense goes away when you use a protocol that runs over a single TCP
>>> port.  Worked fine in the early days of the Internet before firewalls and
>>> NAT.
>>>
>>> Anyway, short idea #2:
>>>
>>>   If it's anonymous access, use HTTP.
>>>   If authenticated access, use ssh/scp/sftp.
>>>
>>> Good luck,
>>>
>>> -Martin
>>>
>>>  [0] http://www.networksorcery.com/enp/protocol/icmp/msg3.htm
>>>
>>> --
>>> Martin A. Brown
>>> http://linux-ip.net/
>>> ___
>>> Tutor maillist  -  Tutor@python.org
>>> To unsubscribe or change subscription options:
>>> https://mail.python.org/mailman/listinfo/tutor
>>>
>>>
>>
> --
> Martin A. Brown
> http://linux-ip.net/
>



-- 

All internal models of the world are approximate. ~ Sebastian Thrun
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] Type Error

[richard kappler]

Still working on my data feed script, if you'll recall from previous
emails, it reads incoming data and creates a name for image files based on
the incoming data in a test environment. below is a snippet of that code
that copies the next image in the pool renaming it as it copies, sends to
another machine via ftp, the closes the file and *** should then delete the
file *** but does not.

img = str(i) + '.jpg'
ID = device + '_' + timestamp + '_' + counter + '.jpg'
src = '/home/test/DataFeed/input/BOT/' + img
dst = '/home/test/DataFeed/output/BOT' + ID
shututil.copyfile(src, dst)
file = open(dst, 'rb')
sendBOT01.storbinary('STOR ' + ID, file)
file.close()
os.remove(file)


everything works except the os.remove(file) which gives the following error:

Traceback (most recent call last):
  File "DataFeedBatch.py", line 104, in 
 os.remove(file)
TypeError: coercing to Unicode: need string or buffer, file found

I don't understand the error
-- 

All internal models of the world are approximate. ~ Sebastian Thrun
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] ftp socket.error

[richard kappler]

No, apparently I didn't figure it out. I thought I had as after the
modprobe I was getting a an EOFError, but now I'm getting the no route to
host error again. I can ping it, and as you can see from the original post,
I am able to establish a connection and log in, it's just when I try to
send a file it goes bollocks up. Still need ideas.

regards, Richard

On Fri, Sep 11, 2015 at 12:17 PM, richard kappler 
wrote:

> Figured it out. On the receiving machine  I had to
>
> # modprobe ip_conntrack_ftp
>
>
> On Fri, Sep 11, 2015 at 12:00 PM, richard kappler 
> wrote:
>
>> I can connect via ftp, but when I try to send a file, I get a no route to
>> host error, I don't understand.
>>
>> code:
>>
>> >>> import ftplib
>> >>> from ftplib import FTP
>> >>> fBOT = FTP()
>> >>> oldfile = '/home/test/DataFeed/input/images/BOT/1.jpg'
>> >>> newfile = 'new.jpg'
>> >>> oldfile = open('/home/test/DataFeed/input/images/BOT/1.jpg', 'rb')
>> >>> fBOT.connect('192.168.2.23', 2021)
>> '220 Service ready for new user.'
>> >>> fBOT.login('BOT', 'sick')
>> '230 User logged in, proceed.'
>> >>> fBOT.storbinary('STOR  newfile', oldfile)
>> Traceback (most recent call last):
>>   File "", line 1, in 
>>   File "/usr/lib64/python2.6/ftplib.py", line 452, in storbinary
>> conn = self.transfercmd(cmd)
>>   File "/usr/lib64/python2.6/ftplib.py", line 360, in transfercmd
>> return self.ntransfercmd(cmd, rest)[0]
>>   File "/usr/lib64/python2.6/ftplib.py", line 326, in ntransfercmd
>> conn = socket.create_connection((host, port), self.timeout)
>>   File "/usr/lib64/python2.6/socket.py", line 567, in create_connection
>> raise error, msg
>> socket.error: [Errno 113] No route to host
>>
>> Any ideas?
>>
>> regards, Richard
>>
>> --
>>
>> All internal models of the world are approximate. ~ Sebastian Thrun
>>
>
>
>
> --
>
> All internal models of the world are approximate. ~ Sebastian Thrun
>



-- 

All internal models of the world are approximate. ~ Sebastian Thrun
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] ftp socket.error

[richard kappler]

Figured it out. On the receiving machine  I had to

# modprobe ip_conntrack_ftp


On Fri, Sep 11, 2015 at 12:00 PM, richard kappler 
wrote:

> I can connect via ftp, but when I try to send a file, I get a no route to
> host error, I don't understand.
>
> code:
>
> >>> import ftplib
> >>> from ftplib import FTP
> >>> fBOT = FTP()
> >>> oldfile = '/home/test/DataFeed/input/images/BOT/1.jpg'
> >>> newfile = 'new.jpg'
> >>> oldfile = open('/home/test/DataFeed/input/images/BOT/1.jpg', 'rb')
> >>> fBOT.connect('192.168.2.23', 2021)
> '220 Service ready for new user.'
> >>> fBOT.login('BOT', 'sick')
> '230 User logged in, proceed.'
> >>> fBOT.storbinary('STOR  newfile', oldfile)
> Traceback (most recent call last):
>   File "", line 1, in 
>   File "/usr/lib64/python2.6/ftplib.py", line 452, in storbinary
> conn = self.transfercmd(cmd)
>   File "/usr/lib64/python2.6/ftplib.py", line 360, in transfercmd
> return self.ntransfercmd(cmd, rest)[0]
>   File "/usr/lib64/python2.6/ftplib.py", line 326, in ntransfercmd
> conn = socket.create_connection((host, port), self.timeout)
>   File "/usr/lib64/python2.6/socket.py", line 567, in create_connection
> raise error, msg
> socket.error: [Errno 113] No route to host
>
> Any ideas?
>
> regards, Richard
>
> --
>
> All internal models of the world are approximate. ~ Sebastian Thrun
>



-- 

All internal models of the world are approximate. ~ Sebastian Thrun
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] ftp socket.error

[richard kappler]

I can connect via ftp, but when I try to send a file, I get a no route to
host error, I don't understand.

code:

>>> import ftplib
>>> from ftplib import FTP
>>> fBOT = FTP()
>>> oldfile = '/home/test/DataFeed/input/images/BOT/1.jpg'
>>> newfile = 'new.jpg'
>>> oldfile = open('/home/test/DataFeed/input/images/BOT/1.jpg', 'rb')
>>> fBOT.connect('192.168.2.23', 2021)
'220 Service ready for new user.'
>>> fBOT.login('BOT', 'sick')
'230 User logged in, proceed.'
>>> fBOT.storbinary('STOR  newfile', oldfile)
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib64/python2.6/ftplib.py", line 452, in storbinary
conn = self.transfercmd(cmd)
  File "/usr/lib64/python2.6/ftplib.py", line 360, in transfercmd
return self.ntransfercmd(cmd, rest)[0]
  File "/usr/lib64/python2.6/ftplib.py", line 326, in ntransfercmd
    conn = socket.create_connection((host, port), self.timeout)
  File "/usr/lib64/python2.6/socket.py", line 567, in create_connection
raise error, msg
socket.error: [Errno 113] No route to host

Any ideas?

regards, Richard

-- 

All internal models of the world are approximate. ~ Sebastian Thrun
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] indent error on if/elif/else

[richard kappler]

Here's my code, no tabs were used, all whitespace verified made with
spacebar:

print("Please enter a number for feed speed...")
print("1 - Batch")
print("2 - 2 per second")
print("3 - Real Time")
print("4 - Exit")

if x == ord('1'):
delay = 0
elif x == ord('2'):
delay = 0.5
elif x == ord('3'):
# set delay real time
# read timestamp from line, create an offset
# send lin iaw server time with offset
else:
print("bad choice, exiting")
os.exit()

#

Here's the Traceback:

  File "dataFeedTest.py", line 44
else:
   ^
IndentationError: expected an indented block


Not a clue why it's doing this. Any help?

regards, Richard

-- 

All internal models of the world are approximate. ~ Sebastian Thrun
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] A further question about opening and closing files

[richard kappler]

Thanks, tried them both, both work great on Linux. Now I understand better.

regards, Richard

On Wed, Sep 9, 2015 at 11:28 AM, Laura Creighton  wrote:

> >I did a little experiment:
> >
> >>>> f1 = open("output/test.log", 'a')
> >>>> f1.write("this is a test")
> >>>> f1.write("this is a test")
> >>>> f1.write('why isn\'t this writing')
> >>>> f1.close()
>
> If you want the thing written out, use f1.flush() whenever you want to
> make sure this happens.
>
> If you want completely unbuffered writing, then you can open your file
> this way, with f1 = open("output/test.log", 'a', 0) I think if you are
> on windows you can only get unbuffered writing if you open your file
> in binary mode.
>
> Laura
>
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>



-- 

All internal models of the world are approximate. ~ Sebastian Thrun
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] iterating through a directory

[richard kappler]

Albert-Jan, thanks for the response. shutil.copyfile does seem to be one of
the tools I need to make the copying, renaming the copy and saving it
elsewhere in one line instead of three or more.

Still not sure how to efficiently get the script to keep moving to the next
file in the directory though, in other words, for each iteration in the
loop, I want it to fetch, rename and send/save the next image in line. Hope
that brings better understanding.

Thanks for the tip!

regards, Richard

On Wed, Sep 9, 2015 at 9:46 AM, Albert-Jan Roskam 
wrote:

> > Date: Wed, 9 Sep 2015 09:32:34 -0400
> > From: richkapp...@gmail.com
> > To: tutor@python.org
> > Subject: [Tutor] iterating through a directory
>
> >
> > Yes, many questions today. I'm working on a data feed script that feeds
> > 'events' into our test environment. In production, we monitor a camera
> that
> > captures an image as product passes by, gathers information such as
> > barcodes and package ID from the image, and then sends out the data as a
> > line of xml to one place for further processing and sends the image to
> > another place for storage. Here is where our software takes over,
> receiving
> > the xml data and images from vendor equipment. Our software then
> processes
> > the xml data and allows retrieval of specific images associated with each
> > line of xml data. Our test environment must simulate the feed from the
> > vendor equipment, so I'm writing a script that feeds the xml from an
> actual
> > log to one place for processing, and pulls images from a pool of 20,
> which
> > we recycle, to associate with each event and sends them to another dir
> for
> > saving and searching.
> >
> > As my script iterates through each line of xml data (discussed yesterday)
> > to simulate the feed from the vendor camera equipment, it parses ID
> > information about the event then sends the line of data on. As it does
> so,
> > it needs to pull the next image in line from the image pool directory,
> > rename it, send it to a different directory for saving. I'm pretty solid
> on
> > all of this except iterating through the image pool. My idea is to just
> > keep looping through the image pool, as each line of xmldata is parsed,
> the
> > next image in line gets pulled out, renamed with the identifying
> > information from the xml data, and both are sent on to different places.
> >
> > I only have one idea for doing this iterating through the image pool, and
> > frankly I think the idea is pretty weak. The only thing I can come up
> with
> > is to change the name of each image in the pool from something
> > like 0219PS01CT1_2029_04_00044979.jpg to 1.jpg, 2.jpg, 3.jpg
> etc.,
> > then doing something like:
> >
> > i = 0
> >
> > here begins my for line in file loop:
> > if i == 20:
> > i = 1
> > else:
> > i += 1
> > do stuff with the xml file including get ID info
> > rename i.jpg to IDinfo.jpg
> > send it all on
> >
> > That seems pretty barbaric, any thoughts on where to look for better
> ideas?
> > I'm presuming there are modules out there that I am unaware of or
> > capabilities I am unaware of in modules I do know a little about that I
> am
> > missing.
>
> I do not really understand what you intend to do, but the following
> modules might come in handy
> -os  (os.rename, os.listdir)
> -glob (glob.iglob or glob.glob)
> -shutil (shutil.copy)
>
>
>
>


-- 

All internal models of the world are approximate. ~ Sebastian Thrun
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] A further question about opening and closing files

[richard kappler]

Under a different subject line (More Pythonic?) Steven D'Aprano commented:

> And this will repeatedly open the file, append one line, then close it
> again. Almost certainly not what you want -- it's wasteful and
> potentially expensive.

And I get that. It does bring up another question though. When using

with open(somefile, 'r') as f:
with open(filename, 'a') as f1:
for line in f:

the file being appended is opened and stays open while the loop iterates,
then the file closes when exiting the loop, yes? Does this not have the
potential to be expensive as well if you are writing a lot of data to the
file?

I did a little experiment:

>>> f1 = open("output/test.log", 'a')
>>> f1.write("this is a test")
>>> f1.write("this is a test")
>>> f1.write('why isn\'t this writing')
>>> f1.close()

monitoring test.log as I went. Nothing was written to the file until I
closed it, or at least that's the way it appeared to the text editor in
which I had test.log open (gedit). In gedit, when a file changes it tells
you and gives you the option to reload the file. This didn't happen until I
closed the file. So I'm presuming all the writes sat in a buffer in memory
until the file was closed, at which time they were written to the file.

Is that actually how it happens, and if so does that not also have the
potential to cause problems if memory is a concern?

regards, Richard
-- 

All internal models of the world are approximate. ~ Sebastian Thrun
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] More Pythonic?

[richard kappler]

> It's not clear why you need the try...except: pass. Please provide some
more background information.

I don't need the try, this was more of a "are there different ways to do
this, which is better and why?" experiment. I am learning, so tend to write
script that is more brute force than elegant and pythonic, wish to write
better code. I do okay, but there are many nuances to Python that I just
haven't run across. For example:

> with open(sourcefile) as instream:
>with open(destfile, "a") as outstream:
>outstream.writelines(process_lines(instream))

I had no idea I could nest with statements like that. It seems obvious now,
but I didn't know.

For the record, I have made a couple other posts this morning that explain
the script constraints far better than I did here. For the sake of brevity
I shant repeat the info here other than to say it's not reading from stdin,
but from a log file to simulate stdin in a test environment.

regards, Richard

On Wed, Sep 9, 2015 at 9:37 AM, Peter Otten <__pete...@web.de> wrote:

> richard kappler wrote:
>
> > Would either or both of these work, if both, which is the better or more
> > Pythonic way to do it, and why?
> >
> > ###
> >
> > import whatIsNeeded
> >
> > writefile = open("writefile", 'a')
> >
> > with open(readfile, 'r') as f:
> > for line in f:
> > if keyword in line:
> > do stuff
> > f1.write(line)
> > else:
> > f1.write(line)
>
> Why do you invoke f1.write() twice?
>
> > writefile.close()
> >
> > ##
> >
> > import whatIsNeeded
> >
> > with open(readfile, 'r') as f:
> > for line in f:
> > try:
> > if keyword in line:
> > do stuff
> > except:
>
> What exceptions are you expecting here? Be explicit. You probably don't
> want
> to swallow a KeyboardInterrupt. And if something unexpected goes wrong a
> noisy complaint gives you the chance to either fix an underlying bug or
> explicitly handle the exception in future runs of the script.
>
> > do nothing
>
> That's spelt
>   pass
>
> > with open(writefile, 'a') as f1:
> > f1.write(line)
>
> Opening the file once per line written seems over-the-top to me.
>
> > ##
> >
> > or something else altogether?
>
> I tend to put the processing into into a generator. That makes it easy to
> replace the source or the consumer:
>
> def process_lines(instream):
> for line in instream:
> if keyword in line:
> do stuff
> yield line
>
> with open(sourcefile) as instream:
> with open(destfile, "a") as outstream:
> outstream.writelines(process_lines(instream))
>
> Now if you want to read from stdin and print to stdout:
>
> sys.stdout.writelines(process_lines(sys.stdin))
>
> > I'm thinking the first way is better as it only opens the files once
> > whereas it seems to me the second script would open and close the
> > writefile once per iteration, and the do nothing in the except seems just
> > wrong to me.
>
> It's not clear why you need the try...except: pass.
> Please provide some more background information.
>
>
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>



-- 

All internal models of the world are approximate. ~ Sebastian Thrun
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] Fwd: Fwd: find second occurance of string in line

[richard kappler]

> It looks likes I was not clear enough: XML doesn't have the concept of lines.
When you process XML "by line" you have buggy code.

No Peter, I'm pretty sure it was I who was less than clear. The xml data is
generated by events, one line in a log for each event, so while xml doesn't
have the concept of lines, each event creates a new line of xml data in the
log from which I will be reading. Sorry about the confusion. I should have
phrased it better, perhaps calling them events instead of lines?

> Just for fun take the five minutes to install lxml and compare the output
of the two scripts. If it's the same now there's no harm switching to lxml,
and you are making future failures less likely.

I'll take a look at it, thanks.

regards, Richard
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] iterating through a directory

[richard kappler]

Yes, many questions today. I'm working on a data feed script that feeds
'events' into our test environment. In production, we monitor a camera that
captures an image as product passes by, gathers information such as
barcodes and package ID from the image, and then sends out the data as a
line of xml to one place for further processing and sends the image to
another place for storage. Here is where our software takes over, receiving
the xml data and images from vendor equipment. Our software then processes
the xml data and allows retrieval of specific images associated with each
line of xml data. Our test environment must simulate the feed from the
vendor equipment, so I'm writing a script that feeds the xml from an actual
log to one place for processing, and pulls images from a pool of 20, which
we recycle, to associate with each event and sends them to another dir for
saving and searching.

 As my script iterates through each line of xml data (discussed yesterday)
to simulate the feed from the vendor camera equipment, it parses ID
information about the event then sends the line of data on. As it does so,
it needs to pull the next image in line from the image pool directory,
rename it, send it to a different directory for saving. I'm pretty solid on
all of this except iterating through the image pool. My idea is to just
keep looping through the image pool, as each line of xmldata is parsed, the
next image in line gets pulled out, renamed with the identifying
information from the xml data, and both are sent on to different places.

I only have one idea for doing this iterating through the image pool, and
frankly I think the idea is pretty weak. The only thing I can come up with
is to change the name of each image in the pool from something
like 0219PS01CT1_2029_04_00044979.jpg to 1.jpg, 2.jpg, 3.jpg etc.,
then doing something like:

i = 0

here begins my for line in file loop:
if i == 20:
i = 1
else:
i += 1
do stuff with the xml file including get ID info
rename i.jpg to IDinfo.jpg
send it all on

That seems pretty barbaric, any thoughts on where to look for better ideas?
I'm presuming there are modules out there that I am unaware of or
capabilities I am unaware of in modules I do know a little about that I am
missing.

-- 

All internal models of the world are approximate. ~ Sebastian Thrun
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] More Pythonic?

[richard kappler]

Would either or both of these work, if both, which is the better or more
Pythonic way to do it, and why?

###

import whatIsNeeded

writefile = open("writefile", 'a')

with open(readfile, 'r') as f:
for line in f:
if keyword in line:
do stuff
f1.write(line)
else:
f1.write(line)

writefile.close()

##

import whatIsNeeded

with open(readfile, 'r') as f:
for line in f:
try:
if keyword in line:
do stuff
except:
do nothing
with open(writefile, 'a') as f1:
f1.write(line)

##

or something else altogether?

I'm thinking the first way is better as it only opens the files once
whereas it seems to me the second script would open and close the writefile
once per iteration, and the do nothing in the except seems just wrong to
me. Is my thinking on target here?

regards, Richard




-- 

All internal models of the world are approximate. ~ Sebastian Thrun
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] Fwd: find second occurance of string in line

[richard kappler]

> Do you want to find just the second occurence in the *file* or the second
occurence within a given tag in the file (and there could be multiple such
tags)?

There are multiple objectdata lines in the file and I wish to find the
second occurence of timestamp in each of those lines.

> Is objectdata within a specific tag? Usually when parsing XML its the
tags you look for first since "lines" can be broken over multiple lines and
multiple tags can exist on one literal line.

objectdata is within a tag as is timestamp. Here's an example:

http://www.w3.org/2001/XMLSchema-instance";
xsi:noNamespaceSchemaLocation="Logging.xsd"
version="1.0">0381UDI1322015-06-18T14:28:06.570531630381UDI12015-06-18T14:27:5037913062015-06-18T14:27:50.81151.450-10.66F001SE0381icdim01322142C0050.00017.20015.2IN10
 
0044]C0037962200190175168000643903637408AcSuccess,AC,LFT,Cubic,ValidDim004951834962200190175168000643903637408FxgEpic,RefBarcode,ScannersTop,Ref,ValidRead,SortableBarcode,EpicBarcode700660001019911281372936-501307200084.06]C0037962200190175168000643903637408

> import re

I know I don't use this in the snippet of code I sent, it is used elsewhere
in the script and I missed that when trimming.

> You should assign this once above the loops, it saves a lot of duplicated
work.

Yes, I understand, again, it is a snippet of a much larger piece of code.

> second = line.index(x,first+1)

I think that is what I was looking for, off to test now. Thanks Alan!

regards, Richard
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] find second occurance of string in line

[richard kappler]

I need to find the index of the second occurance of a string in an xml file
for parsing. I understand re well enough to do what I want to do on the
first instance, but despite many Google searches have yet to find something
to get the index of the second instance, because split won't really work on
my xml file (if I understand split properly) as there are no spaces.
Specifically I'm looking for the second  in an objectdata line.
Not all lines are objectdata lines, though all objectdata lines do have
more than one .

What I have so far:

import re

with open("example.xml", 'r') as f:
for line in f:
if "objectdata" in line:
if "" in line:
x = ""
first = x.index(line)
second = x[first+1:].index(line)
print first, second
else:
print "no timestamp"
else:
print "no objectdata"

my traceback:

Traceback (most recent call last):
  File "2iter.py", line 10, in 
first = x.index(line)
ValueError: substring not found



-- 

All internal models of the world are approximate. ~ Sebastian Thrun
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] iteration help

[richard kappler]

Running python 2.7 on Linux

While for and if loops always seem to give me trouble. They seem obvious
but I often don't get the result I expect and I struggle to figure out why.
Appended below is a partial script. Ultimately, this script will read a
log, parse out two times from each line of the log, a time the line was
written to the lg (called serverTime in the script) and an action time from
elsewhere in the line, then get the difference between the two. I don't
want every difference, but rather the average per hour, so I have a line
count. The script will output the average time difference for each hour.
I've got most of the pieces working in test scripts, but I'm stymied with
the single output bit.

The idea is that the script takes the hour from the server time of the
first line of the log and sets that as the initial serverHr. That works,
has been tested. Next the script is supposed to iterate through each line
of the log (for line in f1) and then check that there is a time in the line
(try), and if not skip to the next line. That works, has been tested.

As each line is iterated over, my intent was that the variable newServerHr
(read from the current line) is compared to serverHr and if they are the
same, the script will increase the count by one and add the difference to a
cummulative total then go to the next line. If the newServerHr and serverHr
are not the same, then we have entered a new clock hour, and the script
should calculate averages and output those, zero all counts and cummulative
totals, then carry on. The idea being that out of 117,000 ish lines of log
(the test file) that have inputs from 0200 to 0700, I would get 6 lines of
output.

I've got everything working properly in a different script except I get 25
lines of output instead of 6, writing something like 16 different hours
instead of 02 - 07.

In trying to chase down my bug, I wrote the appended script, but it outputs
117,000 ish lines (times 02-07, so that bit is better), not 6. Can someone
tell me what I'm misunderstanding?

#!/usr/bin/env python

import re

f1 = open('ATLA_PS4_red5.log', 'r')
f2 = open('recurseOut.log', 'a')

# read server time of first line to get hour
first_line = f1.readline()
q = re.search(r'\d\d:\d\d:\d\d', first_line)
q2 = q.start()
serverHr = (first_line[q2:q2+2])


for line in f1:
try:
s = line
#read server time
a = re.search(r'\d\d:\d\d:\d\d', s)  # find server time in line
b = a.start()# find 1st position of srvTime
newServerHr = (s[b:b+2])  # what hour is it now?
if newServerHr != serverHr:
f2.write('hour ' + newServerHr + '\n')
else:
serverHr == newServerHr

except:
pass

-- 

All internal models of the world are approximate. ~ Sebastian Thrun
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] line iteration in a file

[richard kappler]

SOLVED: Sometimes one just has to be an idiot. One must remember that
computers count from zero, not from one. Changes my list indexes to reflect
that small but crucial fundamental point, and all worked fine.

regards, Richard

On Wed, Jun 3, 2015 at 10:37 PM, richard kappler 
wrote:

> Figured out the string delimiters problem, thanks for all the help. Now
> I've run into another.
>
> I've used the re.finditer that I think it was Peter suggested. So I have:
>
> for line in file:
> s = line
> t = [m.start() for m in re.finditer(r"]", s)]
> q = len(t)
>
> which works fine, in testing it finds the number and position of the ]'s
> in any line I throw at it. I then wrote a series of if/elif statements
> based on q, in other words
>
> if q == 1:
> do something
> elif q == 2:
> do something else
> elif q == 3:
> do a third thing
> else:
> pass
>
> as I looked through enough example to figure out that the most ]'s I can
> have is 3, but the pass is there just in case.
>
> I keep getting a list index out of range error, and my best guess is that
> it's because t and q are set on the first line read, not each line read, is
> that right? If not, what might be the problem and either way, how do I fix
> it?
>
> regards, Richard
> who is proving to his Linux box that he is an idiot pretty regularly
> --
>
> Windows assumes you are an idiot…Linux demands proof.
>



-- 

Windows assumes you are an idiot…Linux demands proof.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] line iteration in a file

[richard kappler]

Figured out the string delimiters problem, thanks for all the help. Now
I've run into another.

I've used the re.finditer that I think it was Peter suggested. So I have:

for line in file:
s = line
t = [m.start() for m in re.finditer(r"]", s)]
q = len(t)

which works fine, in testing it finds the number and position of the ]'s in
any line I throw at it. I then wrote a series of if/elif statements based
on q, in other words

if q == 1:
do something
elif q == 2:
do something else
elif q == 3:
do a third thing
else:
pass

as I looked through enough example to figure out that the most ]'s I can
have is 3, but the pass is there just in case.

I keep getting a list index out of range error, and my best guess is that
it's because t and q are set on the first line read, not each line read, is
that right? If not, what might be the problem and either way, how do I fix
it?

regards, Richard
who is proving to his Linux box that he is an idiot pretty regularly
-- 

Windows assumes you are an idiot…Linux demands proof.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string delimiters

[richard kappler]

Perhaps the better way for me to have asked this question would have been:

How can I find the location within a string of every instance of a
character such as ']'?

regards, Richard

On Wed, Jun 3, 2015 at 5:16 PM, Alex Kleider  wrote:

> On 2015-06-03 12:53, Alan Gauld wrote:
> ...
>
>> If this is really about parsing dates and times have
>> you looked at the datetime module and its parsing/formatting
>> functions (ie strptime/strftime)?
>>
>
> I asssume strftime gets its name from 'string from time.'
> What about strptime? How did that get its name?
>
>
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>



-- 

Windows assumes you are an idiot…Linux demands proof.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string delimiters

[richard kappler]

figured that out from your last post, and thank you, now I understand how
that works. I thought I was looking for the entire string, not each
character. That bit all makes sense now.

A descriptor is, for example, for the following part of a string '0032.4'
the descriptor would be weight, so the formatted output would be
weight:0032.4, and so on. each bit of the strings in the post where I
provided the two examples has specific meaning, and I have to parse the
lines so that I add a descriptor (okay, bad word, what should I use?) to
each bit of data from the line.

At the moment I'm doing it by position, which is, I'm sure, a really bad
way to do it, but I need this quickly and don't know enough to know if
there is a better way. I have to parse and output the entire line, but
there are, as I said, two 'types' of string and some are variable in
length. I'm eager for direction. What other information would better help
explain?

regards, Richard

On Wed, Jun 3, 2015 at 4:31 PM, Alan Gauld 
wrote:

> On 03/06/15 21:23, richard kappler wrote:
>
>> hold the phone
>>
>> I have no idea why it worked, would love an explanation, but I changed my
>> previous test script by eliminating
>>
>> for tag in ("icdm"):
>>
>
> This loops over the string assigning the characters i,c,d and m to tag
>
>  if 'icdm' in line:
>>
>
> This checks if the 4 character string 'icdm' is in the line.
> Completely different.
>
>
> --
> Alan G
> Author of the Learn to Program web site
> http://www.alan-g.me.uk/
> http://www.amazon.com/author/alan_gauld
> Follow my photo-blog on Flickr at:
> http://www.flickr.com/photos/alangauldphotos
>
>


-- 

Windows assumes you are an idiot…Linux demands proof.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string delimiters

[richard kappler]

hold the phone

I have no idea why it worked, would love an explanation, but I changed my
previous test script by eliminating

for tag in ("icdm"):

and changing

if tag in line

to

if 'icdm' in line:

and it works perfectly! It only iterates over the file once, and the else
executes so both types of lines format correctly except for the multiple
identifiers in the non-icdm lines

I could still use some help with that bit, please.

regards, Richard

On Wed, Jun 3, 2015 at 4:13 PM, richard kappler 
wrote:

> I was trying to keep it simple, you'd think by now I'd know better. My
> fault and my apology.
>
> It's definitely  not all dates and times, the data and character types
> vary. This is the output from my log parser script which you helped on the
> other day. there are essentially two types of line:
>
> Tue Jun  2 10:22:42 2015 name="SE">SE201506012200310389PS01CT1407166S0011.40009.7.6IN
>  0018.1LB000258]C10259612019466862270088094]L0223PDF
> Tue Jun  2 10:22:43 2015 name="SE">SE0389icdim01307755C0038.20033.20012.0IN10
>  0032]C10259612804038813568089577
>
> I have to do several things:
> the first type can be of variable length, everything after the ] is an
> identifier that I have to separate, some lines have one, some have more
> than one, variable length, always delimited by a ]
> the second type (line 2) doesn't have the internal datetime stamp, so I
> just need to add 14 x's to fill in the space where that date time stamp
> would be.
>
> and finally, I have to break these apart and put a descriptor with each.
>
> While I was waiting for a response to this, I put together a script to
> start figuring things out (what could possibly go wrong?!?!?! :-) )
>
> and I can't post the exact script but the following is the guts of it:
>
> f1 = open('unformatted.log', 'r')
> f2 = open('formatted.log', 'a')
>
> for line in f1:
> for tag in ("icdm"):
> if tag in line:
> newline = 'log datestamp:' + line[0:24] # + and so on to
> format the lines with icdm in them including adding 14 x's for the missing
> timestamp
> f2.write(newline) #write the formatted output to the new log
> else:
> newline = 'log datestamp:' + line[0:24] # + and so on to
> format the non-icdm lines
> f2.write(newline)
>
> The problems are:
> 1. for some reason this iterates over the 24 line file 5 times, and it
> writes the 14 x's to every file, so my non-icdm code (the else:) isn't
> getting executed. I'm missing something basic and obvious but have no idea
> what.
> 2. I still don't know how to handle the differences in the end of the
> non-icdm files (potentially more than identifier ] delimited as described
> above).
>
> regards, Richard
>
> On Wed, Jun 3, 2015 at 3:53 PM, Alan Gauld 
> wrote:
>
>> On 03/06/15 20:10, richard kappler wrote:
>>
>>> for formatting a string and adding descriptors:
>>>
>>> test = 'datetimepart1part2part3the_rest'
>>>
>>
>> If this is really about parsing dates and times have
>> you looked at the datetime module and its parsing/formatting
>> functions (ie strptime/strftime)?
>>
>>  Can I stop using position numbers and start looking for specific
>>> characters
>>> (the delimiter)  and proceed to the end (which is always a constant
>>> string
>>> btw).
>>>
>>
>> The general answer is probably to look at regular expressions.
>> But they get messy fast so usually I'd suggest trying regular
>> string searches/replaces and  splits first.
>>
>> But if your pattern is genuinely complex and variable then
>> regex may be the right solution.
>>
>> But if its dates check the strptime() functions first.
>>
>>
>> --
>> Alan G
>> Author of the Learn to Program web site
>> http://www.alan-g.me.uk/
>> http://www.amazon.com/author/alan_gauld
>> Follow my photo-blog on Flickr at:
>> http://www.flickr.com/photos/alangauldphotos
>>
>>
>> ___
>> Tutor maillist  -  Tutor@python.org
>> To unsubscribe or change subscription options:
>> https://mail.python.org/mailman/listinfo/tutor
>>
>
>
>
> --
>
> Windows assumes you are an idiot…Linux demands proof.
>



-- 

Windows assumes you are an idiot…Linux demands proof.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string delimiters

[richard kappler]

I was trying to keep it simple, you'd think by now I'd know better. My
fault and my apology.

It's definitely  not all dates and times, the data and character types
vary. This is the output from my log parser script which you helped on the
other day. there are essentially two types of line:

Tue Jun  2 10:22:42 2015SE201506012200310389PS01CT1407166S0011.40009.7.6IN
 0018.1LB000258]C10259612019466862270088094]L0223PDF
Tue Jun  2 10:22:43 2015SE0389icdim01307755C0038.20033.20012.0IN10
 0032]C10259612804038813568089577

I have to do several things:
the first type can be of variable length, everything after the ] is an
identifier that I have to separate, some lines have one, some have more
than one, variable length, always delimited by a ]
the second type (line 2) doesn't have the internal datetime stamp, so I
just need to add 14 x's to fill in the space where that date time stamp
would be.

and finally, I have to break these apart and put a descriptor with each.

While I was waiting for a response to this, I put together a script to
start figuring things out (what could possibly go wrong?!?!?! :-) )

and I can't post the exact script but the following is the guts of it:

f1 = open('unformatted.log', 'r')
f2 = open('formatted.log', 'a')

for line in f1:
for tag in ("icdm"):
if tag in line:
newline = 'log datestamp:' + line[0:24] # + and so on to format
the lines with icdm in them including adding 14 x's for the missing
timestamp
f2.write(newline) #write the formatted output to the new log
else:
newline = 'log datestamp:' + line[0:24] # + and so on to format
the non-icdm lines
f2.write(newline)

The problems are:
1. for some reason this iterates over the 24 line file 5 times, and it
writes the 14 x's to every file, so my non-icdm code (the else:) isn't
getting executed. I'm missing something basic and obvious but have no idea
what.
2. I still don't know how to handle the differences in the end of the
non-icdm files (potentially more than identifier ] delimited as described
above).

regards, Richard

On Wed, Jun 3, 2015 at 3:53 PM, Alan Gauld 
wrote:

> On 03/06/15 20:10, richard kappler wrote:
>
>> for formatting a string and adding descriptors:
>>
>> test = 'datetimepart1part2part3the_rest'
>>
>
> If this is really about parsing dates and times have
> you looked at the datetime module and its parsing/formatting
> functions (ie strptime/strftime)?
>
>  Can I stop using position numbers and start looking for specific
>> characters
>> (the delimiter)  and proceed to the end (which is always a constant string
>> btw).
>>
>
> The general answer is probably to look at regular expressions.
> But they get messy fast so usually I'd suggest trying regular
> string searches/replaces and  splits first.
>
> But if your pattern is genuinely complex and variable then
> regex may be the right solution.
>
> But if its dates check the strptime() functions first.
>
>
> --
> Alan G
> Author of the Learn to Program web site
> http://www.alan-g.me.uk/
> http://www.amazon.com/author/alan_gauld
> Follow my photo-blog on Flickr at:
> http://www.flickr.com/photos/alangauldphotos
>
>
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>



-- 

Windows assumes you are an idiot…Linux demands proof.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] string delimiters

[richard kappler]

for formatting a string and adding descriptors:

test = 'datetimepart1part2part3the_rest'
newtest = 'date='  + test[0:4] + ' time=' + test[4:8] + ' part1=' +
test[8:13] + ' part2=' + test[13:18] + ' part3=' + test[18:23] + ' the
rest=' + test[23:]

and while this may be ugly, it does what I want it to do.

The question is, if instead of 'the_rest' I have ']the_rest' and sometimes
there's not just one. how do I handle that?

In other words, this script will iterate over numerous lines in a file, and
each one is identical up to the delimiter before the rest, and sometimes
there is only one, sometimes there is two, they vary in length.

Can I stop using position numbers and start looking for specific characters
(the delimiter)  and proceed to the end (which is always a constant string
btw).

regards, Richard

-- 

Windows assumes you are an idiot…Linux demands proof.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] inserting path to open a file from a variable

[richard kappler]

I found the problem, but the answer confuses me.

If I run my script to open a file in Documents/MyScripts/fileMonitor which
is where I'm doing my building and testing, with the variable rd1 (however
created, my way and ConfigParser way both work) from within
Documents/MyScripts/fileMonitor, the script fails with a

Traceback (most recent call last):
  File "21FileMonitor.py", line 28, in 
file = open(rd1, 'r')
IOError: [Errno 2] No such file or directory:
'Documents/MyScripts/fileMonitor/log.txt'

but if I run the exact same script from the Home directory, it works fine,
does exactly what I expected it to do (ie. opens the files).

I thought it might be because because I used Doc... instead of ~/Doc... for
my path, but I got the same traceback.

curiouser and curiouser was, Richard

On Thu, May 28, 2015 at 3:01 PM, Felix Dietrich <
felix.dietr...@sperrhaken.name> wrote:

> richard kappler  writes:
>
> > Now I've been tasked to change the script so that the script doesn't need
> > to be in the same directory as the log file, which makes perfect sense.
> > Furthermore, the path can't be hard coded into the script, but rather
> > should read the installer should be able to edit a text file to specify
> the
> > paths to the read file (log from which we're extracting data) and the
> write
> > file (file to which we're send the extracted data). I thought this would
> be
> > a trivial exercise, but I'm stuck.
>
> An alternative way to configurate execution parameters is to pass the
> filenames as arguments to your program which you can access via
> /sys.argv/ (modules to consider: /getopt/, /optparser/):
>
> import sys
> print sys.argv[0] # the first element is the script's name
> print sys.argv[1] # second element first argument (e.g. rd)
> print sys.argv[2] # third element second argument (e.g. wd)
>
> If you have these lines in a file "argv_test.py" try:
>
> python argv_test.py read_filename write_filename
>
>
> Another good alternative for filters is to simply read from stdin and
> output the results to stdout; then use the shell to redirect those
> from/to the respective files:
>
> python filter.py wdfile
> # < redirects stdin
> # > redirects stdout
> # or with a pipe and cat
> cat rdfile | python filter.py > wdfile
>
> Within python stdin can be read via sys.stdin:
>
> import sys
> for l in sys.stdin:
> print l
>
>
> The /ConfigParser/-Module provides a way to read and write configuration
> files.
>
> > # read the config file to get file locations for a script
> > conf = open('fileMonitor.conf', 'r')
> > read_it = conf.read()
> >
> > for line in read_it.splitlines():
> > if line.startswith('rdfile:'):
> > rd = line
> > elif line.startswith('wrtfile:'):
> > wrt = line
>
> Instead of reading all the content of a file into memory one can simply
> iterate it and retrieve the contents line my line (I believe this way is
> also considered more "pythonic"):
>
> conf = open('fileMonitor.conf', 'r')
> for line in conf:
> ...
>
> One more "pythonic" thing to do is to wrap the interaction with a file
> in a /with/-block: that way file closing is ensured after one is done
> with the file:
>
> with open('fileMonitor.conf', 'r') as conf:
> for line in conf:
> ...
>
>
> > At the moment, I am, for example, opening the file to be read from with a
> > simple
> >
> > file = open('log.txt', 'r')
> >
> > but I need to replace 'log.txt' with rd1 (path and file name to log.txt).
> >
> > And I'm stumped. rd1 and wrt1 exist, I can print them, and I get what I
> > expect (path/filename) for example print rd1 gives
> > me Documents/MyScripts/fileMonitor/log.txt
> >
> > But how in the heck do I get that into the open() statement?
> >
> > What I've tried (none worked):
> >
> > file = open(rd1, 'r')
> > file = open('rd1', 'r')
>
> What do you mean by "none worked"?  Did python respond with an error?
> How did you figure that the calls to /open/ failed?
>
> Also: The first line opens a file having the path of the string the
> variable /rd1/ currently holds (presumably
> "Documents/MyScripts/fileMonitor/log.txt").  The second calls /open/
> with the string "rd1" causing /open/ to try and open a file with the
> name rd1.
>
> 'r' will fail when the file does not exist.
>
> --
> Felix Dietrich
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>



-- 

Windows assumes you are an idiot…Linux demands proof.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] inserting path to open a file from a variable

[richard kappler]

Sorry Alan, I think I might have accidentally replied to only you. Love
your Python Projects book btw! Working through it now.

I've looked at, and played with a bit, ConfigParser and yes, it seems that
may be a better idea, thank you.

I still have the issue of how to get the path into my code, regardless of
whether I use ConfigParser or

What I've tried (none worked):
> file = open(rd1, 'r')
>

This should have worked.

What happened when you tried it? "Did not work" is a tad vague!

Traceback (most recent call last):
  File "21FileMonitor.py", line 28, in 
file = open(rd1, 'r')
IOError: [Errno 2] No such file or directory:
'Documents/MyScripts/fileMonitor/log.txt'

Should I not be using os.path or somesuch? If my path to the file, as
provided via ConfigParser, is rd1, would I do:

file = os.path.open(%s, 'r') % (rd1)
or
file = os.path.open(rd1, 'r')

or do I not need os.path...

Kind of lost in the woods here.

regards, Richard

On Thu, May 28, 2015 at 1:56 PM, Alan Gauld 
wrote:

> On 28/05/15 18:39, richard kappler wrote:
>
>  I've created a config file which the user would edit named
>> fileMonitor.conf:
>>
>> # line two is the absolute path to the log you are parsing data from
>> # keep 'rdfile:' as is, path starts after it, no spaces
>> rdfile:Documents/MyScripts/fileMonitor/log.txt
>> # line 4 is the absolute path to the log you are appending the parsed data
>> too
>> # keep 'wrtfile:' as is, path starts after it, no spaces
>> wrtfile:Documents/MyScripts/fileMonitor/newlog.txt
>>
>
> You should maybe look at the config parser module and
> use a standard file format. You are likely to be adding
> more entries into this file...
>
>  and I have written up a script that reads the paths and strips off the
>> unusable bit named readConfig.py:
>>
>> # read the config file to get file locations for a script
>> conf = open('fileMonitor.conf', 'r')
>> read_it = conf.read()
>>
>> for line in read_it.splitlines():
>>
>
> You should consider using something like
>
> with open('fileMonitor.conf', 'r') as conf:
>for file in conf:
>
> Rather rthan reading/splitting the file in memory.
>
>  rd1 = rd.replace('rdfile:',"",1)
>> wrt1 = wrt.replace('wrtfile:',"",1)
>>
>> This worked fine, if I print rd1 and wrt1 I get just the paths that are
>> entered into the conf file.
>>
>> file = open('log.txt', 'r')
>>
>> but I need to replace 'log.txt' with rd1 (path and file name to log.txt).
>>
>> What I've tried (none worked):
>> file = open(rd1, 'r')
>>
>
> This should have worked.
>
> What happened when you tried it? "Did not work" is a tad vague!
>
>
> --
> Alan G
> Author of the Learn to Program web site
> http://www.alan-g.me.uk/
> http://www.amazon.com/author/alan_gauld
> Follow my photo-blog on Flickr at:
> http://www.flickr.com/photos/alangauldphotos
>
>
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>



-- 

Windows assumes you are an idiot…Linux demands proof.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] inserting path to open a file from a variable

[richard kappler]

This is a continuation of the read data script I asked for help on
yesterday, which works very well thanks to all the help from the list.

My script opens and reads in new lines from an in service log file,
extracts specific data, writes it to another file for analysis. All of that
works fine, tested out great.

Now I've been tasked to change the script so that the script doesn't need
to be in the same directory as the log file, which makes perfect sense.
Furthermore, the path can't be hard coded into the script, but rather
should read the installer should be able to edit a text file to specify the
paths to the read file (log from which we're extracting data) and the write
file (file to which we're send the extracted data). I thought this would be
a trivial exercise, but I'm stuck.

This is python 2.6.6 running on a Linux machine.

I've created a config file which the user would edit named fileMonitor.conf:

# line two is the absolute path to the log you are parsing data from
# keep 'rdfile:' as is, path starts after it, no spaces
rdfile:Documents/MyScripts/fileMonitor/log.txt
# line 4 is the absolute path to the log you are appending the parsed data
too
# keep 'wrtfile:' as is, path starts after it, no spaces
wrtfile:Documents/MyScripts/fileMonitor/newlog.txt

and I have written up a script that reads the paths and strips off the
unusable bit named readConfig.py:

# read the config file to get file locations for a script
conf = open('fileMonitor.conf', 'r')
read_it = conf.read()

for line in read_it.splitlines():
if line.startswith('rdfile:'):
rd = line
elif line.startswith('wrtfile:'):
wrt = line

rd1 = rd.replace('rdfile:',"",1)
wrt1 = wrt.replace('wrtfile:',"",1)

This worked fine, if I print rd1 and wrt1 I get just the paths that are
entered into the conf file.

Now I need to add that into my fileMonitor.py (parses log for data
extraction) so it goes to the two files.

At the moment, I am, for example, opening the file to be read from with a
simple

file = open('log.txt', 'r')

but I need to replace 'log.txt' with rd1 (path and file name to log.txt).

And I'm stumped. rd1 and wrt1 exist, I can print them, and I get what I
expect (path/filename) for example print rd1 gives
me Documents/MyScripts/fileMonitor/log.txt

But how in the heck do I get that into the open() statement?

What I've tried (none worked):
file = open(rd1, 'r')
file = open('rd1', 'r')

and I ran out of ideas.

Help?

regards, Richard




-- 

Windows assumes you are an idiot…Linux demands proof.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Extracting data between strings

[richard kappler]

>Hi Richard,

I'm not sure how advanced you are, whether you have any experience or if
you're a total beginner. If anything I say below doesn't make sense,
please ask! Keep your replies on the list, and somebody will be happy to
answer.

In between. I've been learning Python off and on as a hobby for about 4-5
years, learning what I need to know to do what I want to do, but I just
took a new job and part of it requires ramping up my Python knowledge as
I'm the only Python (and Linux) guy here.

On Wed, May 27, 2015 at 09:26:15AM -0400, richard kappler wrote:

> I'm writing a script that reads from an in-service log file in xml format
> that can grow to a couple gigs in 24 hours, then gets zipped out and
> restarts at zero.

Please be more specific -- does the log file already get zipped up each
day, and you have to read from it, or is your script responsible for
reading AND zipping it up?

No, all I have to do is read it and recognize when it gets emptied
(zipped). The Linux system its on already rotates and zips. My script just
needs to extract the specific data we're looking for.


Now, on to the script that extracts data from the logs...

> My script must check to see if new entries have been
> made, find specific lines based on 2 different start tags, and from those
> lines extract data between the start and end tags (hopefully including the
> tags) and write it to a file. I've got the script to read the file, see if
> it's grown, find the appropriate lines and write them to a file.

It's probably better to use a directory listener rather than keep
scanning the file over and over again.

Perhaps you can adapt this?

http://code.activestate.com/recipes/577968-log-watcher-tail-f-log/

I'll check it out, thanks.


More comments below:


> I  still
> need to strip out just the data I need (between the open and close tags)
> instead of writing the entire line, and also to reset eof when the nightly
> zip / new log file creation occurs. I could use some guidance on stripping
> out the data, at the moment I'm pretty lost, and I've got an idea about
the
> nightly reset but any comments about that would be welcome as well. Oh,
and
> the painful bit is that I can't use any modules that aren't included in
the
> initial Python install. My code is appended below.
>
> regards, Richard
>
>
> import time
>
> while True:
> #open the log file containing the data
> file = open('log.txt', 'r')
> #find inital End Of File offset
> file.seek(0,2)
> eof = file.tell()
> #set the file size again
> file.seek(0,2)
> neweof = file.tell()
> #if the file is larger...


You can tell how big the file is without opening it:

import os
filesize = os.path.getsize('path/to/file')

Brilliant! We were just discussing the potential interference and overhead
of opening and closing the file more frequently than 5 minutes, this solves
that problem.


> if neweof > eof:
> #go back to last position...
> file.seek(eof)
> # open file to which the lines will be appended
> f1 = open('newlog.txt', 'a')
> # read new lines in log.txt
> for line in file.readlines():

For huge files, it is MUCH more efficient to write:

for line in file: ...

than to use file.readlines().

Ok.


> #check if line contains needed data
> if "usertag1" in line or "SeMsg" in line:
>
> 
>  this should extract the data between usertag1 and  
>  and /usertag1, and between SeMsg and /SeMsg,   
>  writing just that data to the new file for 
>  analysis. For now, write entire line to file   
> 

>You say "between usertag1 ... AND between SeMsg..." -- what if only one
>set of tags are available? I'm going to assume that it's possible that
>both tags could exist.

Right, I should have been more specific, sorry. Each line will contain one
or the other or neither, never both.

for line in file:
for tag in ("usertag1", "SeMsg"):
if tag in line:
start = line.find(tag)
end = line.find("/" + tag, start)
    if end == -1:
 print("warning: /%d found!" % tag)
else:
 data = line[start:end+len(tag)+1]
 f1.write(data)



Does this help?

I think it does, I'll go give it a try and post back this afternoon. Thanks
Steven!

On Wed, May 27, 2015 at 10:38 AM, Steven D'Aprano 
wrote:

> Hi Richard,
>
> I'm not sure how advanced you are, whether y

[Tutor] Extracting data between strings

[richard kappler]

I'm writing a script that reads from an in-service log file in xml format
that can grow to a couple gigs in 24 hours, then gets zipped out and
restarts at zero. My script must check to see if new entries have been
made, find specific lines based on 2 different start tags, and from those
lines extract data between the start and end tags (hopefully including the
tags) and write it to a file. I've got the script to read the file, see if
it's grown, find the appropriate lines and write them to a file. I  still
need to strip out just the data I need (between the open and close tags)
instead of writing the entire line, and also to reset eof when the nightly
zip / new log file creation occurs. I could use some guidance on stripping
out the data, at the moment I'm pretty lost, and I've got an idea about the
nightly reset but any comments about that would be welcome as well. Oh, and
the painful bit is that I can't use any modules that aren't included in the
initial Python install. My code is appended below.

regards, Richard


import time

while True:
#open the log file containing the data
file = open('log.txt', 'r')
#find inital End Of File offset
file.seek(0,2)
eof = file.tell()
#set the file size again
file.seek(0,2)
neweof = file.tell()
#if the file is larger...
if neweof > eof:
#go back to last position...
file.seek(eof)
# open file to which the lines will be appended
f1 = open('newlog.txt', 'a')
# read new lines in log.txt
for line in file.readlines():
#check if line contains needed data
if "usertag1" in line or "SeMsg" in line:


 this should extract the data between usertag1 and  
 and /usertag1, and between SeMsg and /SeMsg,   
 writing just that data to the new file for 
 analysis. For now, write entire line to file   


# if yes, send line to new file
f1.write(line)
# update log.txt file size
eof = neweof

###
 need an elif for when neweof < eof (nightly   
 reset of log file) that continues loop with reset 
 eof and neweof
###
file.close()
f1.close()
# wait x number of seconds until back to begining of loop
# set at ten for dev and test, set to 300 for production
time.sleep(10)
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] How to paste text from from clipboard to command line?

[Richard D. Moores]

On Wed, Jan 21, 2015 at 8:10 AM, Steven D'Aprano  wrote:
> https://duckduckgo.com/?q=windows+command+line+paste

Ah, check QuickEdit Mode in Properties. Thanks very much.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] How to paste text from from clipboard to command line?

[Richard D. Moores]

On Wed, Jan 21, 2015 at 8:04 AM, Steven D'Aprano  wrote:
> Huh? Thanks for what?

Please note the subject line.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] How to paste text from from clipboard to command line?

[Richard D. Moores]

Thanks,

Dick Moores
Python 3.4.1
Win 7
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] format command help

[Richard Dillon]

I create column headings using \t

 print('base1\tbase2\theight\tarea')

and I would like the numbers to align with the headings. I think that I need to 
use format instead of doing this:

print(A,' ',B,' ',C,' ',int(area1))
print(D,' ',E,' ',F,' ',int(area2))

but I don't know how.
I've looked at code examples and I can't figure out how.

Thanks 


___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] This is driving my crazy! My file hasn't been corrupted

[Richard Dillon]

My text file has five numbers, 1-5
I don't what the problem is.
I've written the file using Word (saved as .txt ) as well as TextEdit
Python 3.4.1 on a Mac

Here's the code:

#   Richard Dillon

#   This program reads data from a .txt file and calculates a total
#   data in text file: 1,2,3,4 and 5 for a total of 15
#   error message: Non-numeric data found in the file

def main():


total = 0.0

try:

infile = open('/Users/richarddillon/Desktop/numbers.txt', 'r')

for line in infile:
amount = float(line)
total += amount

infile.close()

print(format(total, ',.2f'))

except IOError:
print('An error occured trying to read the file.')

except ValueError:
print('Non-numeric data found in the file.')

except:
print('An error occured.')

main()
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] reading strings and calculating totals

[Richard Dillon]

When I tried

total = 0
with open('/Users/richarddillon/Desktop/numbers.txt', 'r') as infile:
   for line in infile:
   total += float(line)
print(total)

Python returned "ValueError: could not convert string to float: "

Richard

On Aug 30, 2014, at 1:13 PM, Alan Gauld  wrote:

> total = 0
> with open('/Users/richarddillon/Desktop/numbers.txt', 'r') as infile:
>for line in infile:
>total += float(line)
> print(total)

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] reading strings and calculating totals

[Richard Dillon]

I apologize in advance - This is my third week using Python (3.4.1 on a Mac)

I need to read a text file, convert the values into numbers and calculate a 
total.
The total I get doesn't match the values entered in the file.

def main():
total = 0
infile = open('/Users/richarddillon/Desktop/numbers.txt', 'r') 
# read first record
line = infile.readline()
a = float(line)
# read rest of records
while line != '':
total = total + a
line = infile.readline()
infile.close()
print(total)

main()
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] Python 3.4.1 question for Mac users

[Richard Dillon]

I’m teaching myself Python 3.4.1 on a Mac and the book I’m using is written for 
Windows users.

I’m trying to open a file on the desktop and I created a path using the  
example in the book.

 

Any Mac users out there with a solution? My main drive is named “OS”.

Here’s my code:


  

def main():

my_file = input('Enter file to open: ')

infile = open(r'\OS\Users\richarddillon\Desktop\my_file','r')

file_contents = infile.read()

infile.close()

print(file.contents)

 

main()___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Error message

[RICHARD KENTISH]

Thanks for your response. I have tried your suggestion and get the same error.  
Here's the text from terminal.

Richards-MacBook-Pro:Python resources richardkentish$ pwd
/Users/richardkentish/desktop/Python resources
Richards-MacBook-Pro:Python resources richardkentish$ python blankgame.py
Traceback (most recent call last):
  File "blankgame.py", line 1, in 
    import pygame, sys
  File 
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pygame/__init__.py",
 line 95, in 
    from pygame.base import *
ImportError: 
dlopen(/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pygame/base.so,
 2): no suitable image found.  Did find:
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pygame/base.so:
 no matching architecture in universal wrapper
Richards-MacBook-Pro:Python resources richardkentish$ 

I installed the python files from the python site, latest version for mac.

Best wishes,

Richard



 From: Steven D'Aprano 
To: tutor@python.org 
Sent: Sunday, 10 August 2014, 12:30
Subject: Re: [Tutor] Error message
 

On Sun, Aug 10, 2014 at 10:32:52AM +0100, RICHARD KENTISH wrote:
> Hi!
> 
> Ive installed Python 2.7.8 and pygame 1.9.1 onto a macbook pro 10.9.4 
> Mavericks.
> 
> The code is taken direct from the 'making games' book. Here it is pasted from 
> idle:

Whenever you have a mysterious error in Python that doesn't appear to be 
an error on your part -- and I admit that as a beginner, it's hard to 
tell which errors are yours and which aren't -- it's always worth trying 
again running Python directly. IDEs like IDLE sometimes do funny things 
to the environment in order to provide an Integrated Development 
Environment, and occasionally they can interfere with the clean running 
of Python code. So, just to be sure, I suggest you try running your code 
again outside of IDLE:

* Save your code to a file (which I see you've already done).

* Open up a terminal so you have a command line.

* Change into the directory where your file is saved. Type this command:

  cd "/Users/richardkentish/Desktop/Python resources"

  then press ENTER.

* Run the file directly using Python by typing this command:

  python blankgame.py

  then press ENTER.

If the error goes away, and you either get no error at all, or a 
different error, then it may be a problem with IDLE. Copy and paste the 
error here, and we can advise further.

Having said all that, looking at the exception you get:


> Traceback (most recent call last):
>   File "/Users/richardkentish/Desktop/Python resources/blankgame.py", line 1, 
> in 
>     import pygame, sys
>   File 
> "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pygame/__init__.py",
>  line 95, in 
>     from pygame.base import *
> ImportError: 
> dlopen(/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pygame/base.so,
>  
> 2): no suitable image found.  Did find: 
> /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pygame/base.so:
>  
> no matching architecture in universal wrapper

it looks to me like perhaps you have a version of Pygame which is not 
compatible with the Mac. How did you install it? I'm not a Mac expert 
(it's been about, oh, 17 years since I've used a Mac) but somebody else 
may be able to advise.


-- 
Steven
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Error message

[RICHARD KENTISH]

Hi All,

I have found a work around - not entirely sure what I did but followed this 
website 
http://www.reddit.com/r/pygame/comments/21tp7n/how_to_install_pygame_on_osx_mavericks/

Still can't run through idle but dragging the saved .py file to the python 
launcher works!

Thanks for your help.

Best wishes,

Richard



 From: RICHARD KENTISH 
To: "tutor@python.org"  
Sent: Sunday, 10 August 2014, 10:32
Subject: [Tutor] Error message
 


Hi!

Ive installed Python 2.7.8 and pygame 1.9.1 onto a macbook pro 10.9.4 Mavericks.

The code is taken direct from the 'making games' book. Here it is pasted from 
idle:

import pygame, sys
from pygame.locals import *

pygame.init()
displaysurf = pygame.dispaly.set_mode((400, 300))
pygame.display.set_caption('Hello World!')

while True: # main game loop
    for event in pygame.event.get():
        if event.type == quit:
            pygame.quit()
            sys.exit()
    pygame.display.update()

Here is the error I am getting.

Python 2.7.8 (v2.7.8:ee879c0ffa11, Jun 29 2014, 21:07:35) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "copyright", "credits" or "license()" for more information.
>>>  RESTART 
>>> 

Traceback (most recent call last):
  File "/Users/richardkentish/Desktop/Python resources/blankgame.py", line 1, 
in 
    import pygame, sys
  File 
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pygame/__init__.py",
 line 95, in 
    from pygame.base import *
ImportError: 
dlopen(/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pygame/base.so,
 2): no suitable image found.  Did find:
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pygame/base.so:
 no matching architecture in universal wrapper


Thanks for your help.

Best wishes,

Richard
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] Error message

[RICHARD KENTISH]

Hi!

Ive installed Python 2.7.8 and pygame 1.9.1 onto a macbook pro 10.9.4 Mavericks.

The code is taken direct from the 'making games' book. Here it is pasted from 
idle:

import pygame, sys
from pygame.locals import *

pygame.init()
displaysurf = pygame.dispaly.set_mode((400, 300))
pygame.display.set_caption('Hello World!')

while True: # main game loop
    for event in pygame.event.get():
        if event.type == quit:
            pygame.quit()
            sys.exit()
    pygame.display.update()

Here is the error I am getting.

Python 2.7.8 (v2.7.8:ee879c0ffa11, Jun 29 2014, 21:07:35) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "copyright", "credits" or "license()" for more information.
>>>  RESTART 
>>> 

Traceback (most recent call last):
  File "/Users/richardkentish/Desktop/Python resources/blankgame.py", line 1, 
in 
    import pygame, sys
  File 
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pygame/__init__.py",
 line 95, in 
    from pygame.base import *
ImportError: 
dlopen(/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pygame/base.so,
 2): no suitable image found.  Did find:
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pygame/base.so:
 no matching architecture in universal wrapper


Thanks for your help.

Best wishes,

Richard___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] trace / profile function calls with inputs

[Richard Cziva]

Hi All,

I am trying to print out every function that is being called while my 
Python program is running (own functions and library calls too). I can 
not modify the Python programs I am trying to profile.


Let me give an example. A program contains a function like this:

def foo(x):
  y = math.cos(x)
  z = 1 + 1
  time.sleep(y+1)
  return x * 50

And it calls the function:

print foo(100)

I would like to retrieve an execution trace that shows me each function 
called with the value or hash of its arguments. According to the 
example, I am looking for a technique to extract something similar:


foo(100)
math.cos(100)
time.sleep(0.87)

Things I have tried with only partial success:
- trace module
- profile module / cProfile

Could you suggest me a way of doing this?

Thanks,
Richard
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] trace / profile every function with inputs

[Richard Cziva]

Hi All,

I am struggling with a "simple" problem: I would like to print out every 
function that is being executed while my Python program is running. I can not 
modify the Python programs that I would like to profile.

Let me give an example. A program contains a function and a call like this:

def foo(x):
  y = math.cos(x)
  time.sleep(y+1)
  return x * 50

print foo(100)

I would like to retrieve an execution trace that shows me each function called 
(with the value or hash of the function arguments). According to the example, I 
am looking for a way to extract something similar:

foo(100)
math.cos(100)
time.sleep(0.87)

Things I have tried with only partial success:
- trace: couldn't get the function names in some cases
- profile / cProfile: no information about the arguments

Could you suggest me a way of doing this?

Thanks,
Richard
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] truncated dictionary return

[richard kappler]

Would something like

if len(dict) = 8
return d
else
continue

work?


On Sun, Dec 1, 2013 at 8:54 PM, richard kappler wrote:

> Now I'm completely lost. While opening the serial port outside the
> function sounds like a good idea, I'm thinking that might not work unless I
> am mistaken. The sensorRead function once it's called would then basically
> own the serial port barring other traffic, yes? That won't work as the same
> serial port that receives sensor data from the arduino sends propulsion and
> nav signals to the arduino which, along with controlling/reading the
> sensors, also controls the motors used for propulsion, hence only opening
> the port when the data is called for. The sensorRead function works, heck
> it's not even mine, it was written by one of the gurus here in response to
> a question I posed months ago (either Alan or Eryksun IIRC) and does
> exactly what it's supposed to do, except for the timing bit.
>
> Perhaps I'm looking for a simple solution where none exists but I rather
> doubt it. I was thinking something along the lines of (psuedo code here)
> check incoming dict for length or number of elements
> if 8, keep
> else retry
>
> While I appreciate the above comments and any help that is offered, I
> neither understand them as presented nor think they will fix the problem
> with the limited understanding I do have. Again, it could be my lack of
> knowledge is preventing me from seeing the light here, but it feels like
> we're reinventing the wheel.
>
> I hope that didn't come across as rude, it truly was not intended to be
> such.
>
> regards, Richard
>
>
> On Sun, Dec 1, 2013 at 3:06 PM, spir  wrote:
>
>> On 12/01/2013 08:28 PM, richard kappler wrote:
>>
>>> I have a script that reads sensor values gathered by an Arduino board
>>> from
>>> serial as a dictionary, said values to later be used in the AI for Nav &
>>> Control. Here's the script:
>>>
>>> #!/usr/bin/python
>>>
>>> def sensorRead():
>>>  import serial
>>>  from time import sleep
>>>
>>>  sensors = {}
>>>  sensors = dict.fromkeys('Sonar1 Sonar2 Sonar3 Sonar4 Dewpoint
>>> Temperature Humidity Light'.split())
>>>
>>>  arduino = serial.Serial('/dev/ttyACM0', 9600)
>>>  sleep(1)
>>>  line = arduino.readline().strip()
>>>  line = line.lstrip('{').rstrip('}').strip()
>>>
>>>  d = {}
>>>  for item in line.split(','):
>>>  item = item.strip()
>>>  key, value = item.split(':')
>>>  key = key.strip()
>>>  value = value.strip()
>>>  d[key]=int(value)
>>>  return d
>>>
>>> I hope that comes through okay, I copied it from the text file so
>>> indentation and such should be fine, if not let me know.
>>>
>>> The script works great with one exception. I understand the problem, I'm
>>> just not sure how to address it. The problem is:
>>>
>>> The Arduino runs on a constant loop, it reads each sensor, sends the key
>>> and the value to the serial bus in format for python to read it as a
>>> dictionary, lather, rinse, repeat.
>>>
>>> Python querries the bus when told. Usually the python script gets the
>>> full
>>> dictionary (all 8 values with keys, brackets etc) but sometimes it
>>> doesn't.
>>> Sometimes it only gets the last few values, sometimes it gets nothing or
>>> misses a bracket and throws an error. This makes sense. They are not in
>>> sync.
>>>
>>> What I need to figure out how to do is have the python script wait until
>>> the next round of values as signified by the opening bracket "{" or check
>>> that it has all 8 values and if not retry or something.
>>>
>>> Would this be an if/else? try? exception?
>>>
>>> I've not yet delved into any of these in my quest to learn python except
>>> if/else and that doesn't feel right for this, so I'm at a loss as to how
>>> to
>>> proceed.
>>>
>>
>> * What is the point of the 'sensors' dict? (also, you don't need to
>> initialise it as an enmpty dict)
>> * If you know about regexps or another matching utility, use that to
>> decode the input line into (key,value) pairs. This is even easier if input
>> has a strict format. Would tell us?
>> * About wrong input (in

Re: [Tutor] truncated dictionary return

[richard kappler]

Now I'm completely lost. While opening the serial port outside the function
sounds like a good idea, I'm thinking that might not work unless I am
mistaken. The sensorRead function once it's called would then basically own
the serial port barring other traffic, yes? That won't work as the same
serial port that receives sensor data from the arduino sends propulsion and
nav signals to the arduino which, along with controlling/reading the
sensors, also controls the motors used for propulsion, hence only opening
the port when the data is called for. The sensorRead function works, heck
it's not even mine, it was written by one of the gurus here in response to
a question I posed months ago (either Alan or Eryksun IIRC) and does
exactly what it's supposed to do, except for the timing bit.

Perhaps I'm looking for a simple solution where none exists but I rather
doubt it. I was thinking something along the lines of (psuedo code here)
check incoming dict for length or number of elements
if 8, keep
else retry

While I appreciate the above comments and any help that is offered, I
neither understand them as presented nor think they will fix the problem
with the limited understanding I do have. Again, it could be my lack of
knowledge is preventing me from seeing the light here, but it feels like
we're reinventing the wheel.

I hope that didn't come across as rude, it truly was not intended to be
such.

regards, Richard


On Sun, Dec 1, 2013 at 3:06 PM, spir  wrote:

> On 12/01/2013 08:28 PM, richard kappler wrote:
>
>> I have a script that reads sensor values gathered by an Arduino board from
>> serial as a dictionary, said values to later be used in the AI for Nav &
>> Control. Here's the script:
>>
>> #!/usr/bin/python
>>
>> def sensorRead():
>>  import serial
>>  from time import sleep
>>
>>  sensors = {}
>>  sensors = dict.fromkeys('Sonar1 Sonar2 Sonar3 Sonar4 Dewpoint
>> Temperature Humidity Light'.split())
>>
>>  arduino = serial.Serial('/dev/ttyACM0', 9600)
>>  sleep(1)
>>  line = arduino.readline().strip()
>>  line = line.lstrip('{').rstrip('}').strip()
>>
>>  d = {}
>>  for item in line.split(','):
>>  item = item.strip()
>>  key, value = item.split(':')
>>  key = key.strip()
>>  value = value.strip()
>>  d[key]=int(value)
>>  return d
>>
>> I hope that comes through okay, I copied it from the text file so
>> indentation and such should be fine, if not let me know.
>>
>> The script works great with one exception. I understand the problem, I'm
>> just not sure how to address it. The problem is:
>>
>> The Arduino runs on a constant loop, it reads each sensor, sends the key
>> and the value to the serial bus in format for python to read it as a
>> dictionary, lather, rinse, repeat.
>>
>> Python querries the bus when told. Usually the python script gets the full
>> dictionary (all 8 values with keys, brackets etc) but sometimes it
>> doesn't.
>> Sometimes it only gets the last few values, sometimes it gets nothing or
>> misses a bracket and throws an error. This makes sense. They are not in
>> sync.
>>
>> What I need to figure out how to do is have the python script wait until
>> the next round of values as signified by the opening bracket "{" or check
>> that it has all 8 values and if not retry or something.
>>
>> Would this be an if/else? try? exception?
>>
>> I've not yet delved into any of these in my quest to learn python except
>> if/else and that doesn't feel right for this, so I'm at a loss as to how
>> to
>> proceed.
>>
>
> * What is the point of the 'sensors' dict? (also, you don't need to
> initialise it as an enmpty dict)
> * If you know about regexps or another matching utility, use that to
> decode the input line into (key,value) pairs. This is even easier if input
> has a strict format. Would tell us?
> * About wrong input (incomplete data), avoid try/except, except if ever
> most cases are ok (it is very costly in case of exception). Anyway, you
> need to decode input, so use that to check for errors/missing stuff.
>
> If you used a matching tool, its absence of result would directly tell
> about wrong input (provided your pattern is correct! ;-) As of now, just
> place checks in your decoding sequence. Possible checks places, at first
> sight, marked below:
>
>
> line = arduino.readline().strip()
> line = line.lstrip('{').rstrip('}')

[Tutor] truncated dictionary return

[richard kappler]

I have a script that reads sensor values gathered by an Arduino board from
serial as a dictionary, said values to later be used in the AI for Nav &
Control. Here's the script:

#!/usr/bin/python

def sensorRead():
import serial
from time import sleep

sensors = {}
sensors = dict.fromkeys('Sonar1 Sonar2 Sonar3 Sonar4 Dewpoint
Temperature Humidity Light'.split())

arduino = serial.Serial('/dev/ttyACM0', 9600)
sleep(1)
line = arduino.readline().strip()
line = line.lstrip('{').rstrip('}').strip()

d = {}
for item in line.split(','):
item = item.strip()
key, value = item.split(':')
key = key.strip()
value = value.strip()
d[key]=int(value)
return d

I hope that comes through okay, I copied it from the text file so
indentation and such should be fine, if not let me know.

The script works great with one exception. I understand the problem, I'm
just not sure how to address it. The problem is:

The Arduino runs on a constant loop, it reads each sensor, sends the key
and the value to the serial bus in format for python to read it as a
dictionary, lather, rinse, repeat.

Python querries the bus when told. Usually the python script gets the full
dictionary (all 8 values with keys, brackets etc) but sometimes it doesn't.
Sometimes it only gets the last few values, sometimes it gets nothing or
misses a bracket and throws an error. This makes sense. They are not in
sync.

What I need to figure out how to do is have the python script wait until
the next round of values as signified by the opening bracket "{" or check
that it has all 8 values and if not retry or something.

Would this be an if/else? try? exception?

I've not yet delved into any of these in my quest to learn python except
if/else and that doesn't feel right for this, so I'm at a loss as to how to
proceed.

regards, Richard

-- 

*Mater tua criceta fuit, et pater tuo redoluit bacarum sambucus*
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] strip and split?

[richard kappler]

I'm using psutil to generate some somatic data with the following script:

import psutil as ps

cpu = ps.cpu_percent()
mem = ps.virtual_memory()
disk = ps.disk_usage('/')

All works well, but except for cpu I am struggling to learn how to strip
out what I don't need.

For example, once I do the above, if I then enter "disk" I get:

disk usage(total=302264549376, used=73844322304, free=213066088448,
percent=24.4)

So if I want only the number following percent, I get that I need to
convert this to a built in type and do some split and strip, but that's
where I'm floundering. Might I get a little guidance on this please?

regards, Richard

-- 

*Mater tua criceta fuit, et pater tuo redoluit bacarum sambucus*
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] IndexError: list index out of range

[richard kappler]

izing a face

# return from a function
#def get_face(arg):
#some_variable = 10 * arg
#return some_variable

#result = get_face(5)


## end facerec #



## sphinx speech rec will go here ##



### Google speech rec ##


import shlex

print " say something"
os.system('sox -r 16000 -t alsa default recording.flac silence 1 0.1 1% 1
1.5 1%')
cmd='wget -q -U "Mozilla/5.0" --post-file recording.flac
--header="Content-Type: audio/x-flac; rate=16000" -O - "
http://www.google.com/speech-api/v1/recognize?lang=en-us&client=chromium";'


args = shlex.split(cmd)
output,error = subprocess.Popen(args,stdout = subprocess.PIPE, stderr=
subprocess.PIPE).communicate()

if not error:
a = eval(output)

#a = eval(open("data.txt").read())
confidence= a['hypotheses'][0]['confidence']
speech=a['hypotheses'][0]['utterance']
 print "you said:   ", speech, "  with  ",confidence,"confidence"


# end Google speech rec code ###


Any help would be greatly appreciated.

regards, Richard

-- 

*Mater tua criceta fuit, et pater tuo redoluit bacarum sambucus*
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Having trouble figuring out bug

[richard kappler]

On Wed, Feb 13, 2013 at 6:56 PM, Steven D'Aprano wrote:

> On 14/02/13 10:14, richard kappler wrote:
>
>> I have tried to run the Google speech recognition code found here:
>> https://github.com/jeysonmc/**python-google-speech-scripts/**
>> blob/master/stt_google.py<https://github.com/jeysonmc/python-google-speech-scripts/blob/master/stt_google.py>
>>
>> I am getting the following traceback:
>> Traceback (most recent call last):
>>File "", line 1, in
>>File "GoogSTT.py", line 43, in listen_for_speech
>>  data = stream.read(chunk)
>>File "/usr/lib/python2.7/dist-**packages/pyaudio.py", line 605, in
>> read
>>  return pa.read_stream(self._stream, num_frames)
>>
>
>
> Awesome!
>
> Richard, if you hang around this mailing list for a while, you will soon
> see that we tutors have a lot of trouble convincing newbies to post the
> entire traceback, not just the error message. But this is the first time
> that somebody has posted the traceback EXCEPT for the error message.
>
> Well done, you win the Internet :-)
>
> Please try again and post the entire traceback, including the error
> message at the end.
>
> Thank you.
>
>
> And now you see my dilemma. That's the error message, the whole bloody
thing. But for ye unbelievers out there, here's my entire screen, from the
call to the crash:

python GoogSTT.py
ALSA lib pcm.c:2217:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.rear
ALSA lib pcm.c:2217:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.center_lfe
ALSA lib pcm.c:2217:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.side
ALSA lib audio/pcm_bluetooth.c:1614:(audioservice_expect)
BT_GET_CAPABILITIES failed : Input/output error(5)
ALSA lib audio/pcm_bluetooth.c:1614:(audioservice_expect)
BT_GET_CAPABILITIES failed : Input/output error(5)
ALSA lib audio/pcm_bluetooth.c:1614:(audioservice_expect)
BT_GET_CAPABILITIES failed : Input/output error(5)
ALSA lib audio/pcm_bluetooth.c:1614:(audioservice_expect)
BT_GET_CAPABILITIES failed : Input/output error(5)
ALSA lib pcm_dmix.c:957:(snd_pcm_dmix_open) The dmix plugin supports only
playback stream
Cannot connect to server socket err = No such file or directory
Cannot connect to server request channel
jack server is not running or cannot be started
* listening. CTRL+C to finish.
starting record
^CTraceback (most recent call last):
  File "GoogSTT.py", line 108, in 
listen_for_speech()
  File "GoogSTT.py", line 43, in listen_for_speech
data = stream.read(chunk)
  File "/usr/lib/python2.7/dist-packages/pyaudio.py", line 605, in read
return pa.read_stream(self._stream, num_frames)
KeyboardInterrupt

So I call the script, when it says *listening I speak into the microphone,
then hit CTRL+C and everything after starting to record spits out. When I
run the script that fails not, it provides EXACTLY the same ALSA and
jackserver output, but records the wav.

regards, Richard

-- 

quando omni flunkus moritati
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Having trouble figuring out bug

[richard kappler]

> See that KeyboardInterrupt line? That was missing from your first post.
>
> You get that because you have hit Ctrl-C, which is Python's way of halting
> the running code (if possible). If you don't want this exception, then
> don't hit Ctrl-C.
>
> Which of course brings you to a dilemma -- the software tells you to use
> Ctrl-C to stop recording, but it apparently lies. I suggest that it is
> buggy. Looking at the source code to the library, here:
>
> https://github.com/jeysonmc/**python-google-speech-scripts/**
> blob/master/stt_google.py<https://github.com/jeysonmc/python-google-speech-scripts/blob/master/stt_google.py>
>
> I'm pretty sure it is buggy. There is no attempt to catch the Ctrl-C and
> continue processing. I could be wrong, because I haven't actually tested it
> or studied it in detail, but I can't see how this could possibly work as
> advertised.
>
> Also the library includes something which gives me the absolute
> heebie-jeebies: it downloads a website from the Internet, then *executes it
> as code* without making any attempt to see what it is. If you run this
> library, you are giving Google, or anyone that manages to intercept your
> connection to Google, carte blanche to run ANY CODE THEY LIKE on your
> computer. I wouldn't touch that library with a fifty-foot pole until that
> is fixed.


Uh huh... That actually makes perfect sense and I'm a bit annoyed I didn't
catch it. So... methinks that if I code in for the recording to end on say
a second or two of silence, and get rid of the CTRL+C, then the code should
proceed as necessary. Then I neeed to figure out how to make it more secure
per your heebie-jeebies. Okay, once more into the breach dear tutors!

thanks for the help, Richard
-- 

quando omni flunkus moritati
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

[Tutor] Having trouble figuring out bug

[richard kappler]

I have tried to run the Google speech recognition code found here:
https://github.com/jeysonmc/python-google-speech-scripts/blob/master/stt_google.py

I am getting the following traceback:
Traceback (most recent call last):
  File "", line 1, in 
  File "GoogSTT.py", line 43, in listen_for_speech
data = stream.read(chunk)
  File "/usr/lib/python2.7/dist-packages/pyaudio.py", line 605, in read
return pa.read_stream(self._stream, num_frames)

which refers to this part of the code:

while (True):
data = stream.read(chunk)

slid_win.append (abs(audioop.avg(data, 2)))

if(True in [ x>THRESHOLD for x in slid_win]):
if(not started):
print "starting record"
started = True
all_m.append(data)
elif (started==True):
print "finished"
#the limit was reached, finish capture and deliver
filename = save_speech(all_m,p)
stt_google_wav(filename)
#reset all
started = False
slid_win = deque(maxlen=SILENCE_LIMIT*rel)
all_m= []
print "listening ..."

I have run another bit of code is very similar to the above, to wit:

while 1:
data = stream.read(CHUNK_SIZE)
L = unpack('<' + ('h'*(len(data)/2)), data) # little endian, signed
short
L = array('h', L)
LRtn.extend(L)


and get no errors, the code runs fine and records. Please note that the
line in question is virtually identical in both snippets of code, one
works, one does not. To the best of my ability I have verified what I know
how to, for example chunk is set with a value previous, etc. Not sure where
I'm running into problems, could use some guidance. NOTE: I did not post
the entire code here for brevity, but it is available at the listed website
or, if it is preferred, I can post it in a follow-on email.

regards, Richard


-- 

quando omni flunkus moritati
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

[Tutor] sending a wav file to a website?

[richard kappler]

Maybe way off scope here, I hope not. I just can't get the accuracy I need
with pocketsphinx at the moment(though I continue to work on it). Google's
webkit-speech actually works pretty durned well, here's an example:
http://slides.html5rocks.com/#speech-input  But that doesn't really solve
any problems either.

So the question is, how do I get a wav file to a website like the one
listed above and retrieve the text result using python (or don't I)? I've
looked at and played around with urllib, I get how it works, I'm pretty
comfy with it, but it seems to be for sending text only? Or is it?  I can
use rec in the terminal (command line) to record speech into a wav file,
I'm pretty sure I can figure out how to make it start and stop recording as
desired, (yes, I've looked at pyaudio, my system crashed when I used it
this morning, come to find out that may be a different problem so will look
again), but sending the wav file to the url and retrieving the text for
further processing has me a bit baffled.

Any ideas?

regards, Richard

-- 

quando omni flunkus moritati
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

[Tutor] sockets and networking

[richard kappler]

I want to be able to communicate between a raspberry pi and a laptop
running ubuntu, transferring text from the pi to the laptop. Specifically
the Pi would be running poscketsphinx speech rec and would send the
generated text through the crossover between the two ethernet ports to a
python program on the laptop. Someone suggest using sockets, with which I
am completely unfamiliar. I've read the socket docs and admit I find them
quite confusing. Can someone point me to a/some good beginner tutorials on
sockets/networking for me to look at?

regards, Richard

-- 

quando omni flunkus moritati
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

[Tutor] a Pygtk question sort of

[richard kappler]

Before anybody jumps me about this question being inappropriate for this
list, yes, I know it probably is BUT, the two places where it might be
appropriate are down pretty hard, so this is my only option (I think).

The question is in regards to pygtk I think, and pocketsphinx obliquely.
Appended below is code from the CMU Sphinx website that I wish to ever so
slightly modify, but am not quite sure how to proceed. This code creates a
gui with a button that stops and starts the pocketsphinx speech recognition
engine. When the button reads "speak" and you push it, the button goes
blank iirc, pocketsphinx listens, performs speech recognition and sends the
resulting text to the gui, changing the button text back to "speak" and
putting the pocketsphinx engine ostensibly (though not in actuality) on
pause.

I understand how the below code does all of this, including how
pocketsphinx works. Here's the issue: I want to use this code or code like
it (pocketsphinx can be imported and run directly in python but apparently
only to decode wav files, not as a real-time decoder unless you run it
through gst as shown in the appended code as I understand it) in my bot
program, so I don't need the gui, button or any of that. I need
pocketsphinx to work exactly as below, but send the text output back to the
main program or to a different program (chatbot) instead of the gui. Make
sense? I've been pouring over this code off and on for months as I've been
learning, and it's not quite as simple as dump the gui method and the
button. The problem is the button controls the vader (determines begin and
end of utterances) as well. Detailed explanation here:
http://cmusphinx.sourceforge.net/wiki/gstreamer

So can anyone give me some guidance here or point me towards a place to
discuss this? The forums at Python.org are under construction, the
CMUSphinx forums at Sourceforge are down (404) so I'm not quite sure where
to go for help.

regards, Richard

#!/usr/bin/env python

# Copyright (c) 2008 Carnegie Mellon University.
#
# You may modify and redistribute this file under the same terms as
# the CMU Sphinx system.  See
# http://cmusphinx.sourceforge.net/html/LICENSE for more information.

import pygtk
pygtk.require('2.0')
import gtk

import gobject
import pygst
pygst.require('0.10')
gobject.threads_init()
import gst

class DemoApp(object):
"""GStreamer/PocketSphinx Demo Application"""
def __init__(self):
"""Initialize a DemoApp object"""
self.init_gui()
self.init_gst()

def init_gui(self):
"""Initialize the GUI components"""
self.window = gtk.Window()
self.window.connect("delete-event", gtk.main_quit)
self.window.set_default_size(400,200)
self.window.set_border_width(10)
vbox = gtk.VBox()
self.textbuf = gtk.TextBuffer()
self.text = gtk.TextView(self.textbuf)
self.text.set_wrap_mode(gtk.WRAP_WORD)
vbox.pack_start(self.text)
self.button = gtk.ToggleButton("Speak")
self.button.connect('clicked', self.button_clicked)
vbox.pack_start(self.button, False, False, 5)
self.window.add(vbox)
self.window.show_all()

def init_gst(self):
"""Initialize the speech components"""
self.pipeline = gst.parse_launch('gconfaudiosrc ! audioconvert
! audioresample '
 + '! vader name=vad
auto-threshold=true '
 + '! pocketsphinx name=asr ! fakesink')
asr = self.pipeline.get_by_name('asr')
asr.connect('partial_result', self.asr_partial_result)
asr.connect('result', self.asr_result)
asr.set_property('configured', True)

bus = self.pipeline.get_bus()
bus.add_signal_watch()
bus.connect('message::application', self.application_message)

self.pipeline.set_state(gst.STATE_PAUSED)

def asr_partial_result(self, asr, text, uttid):
"""Forward partial result signals on the bus to the main thread."""
struct = gst.Structure('partial_result')
struct.set_value('hyp', text)
struct.set_value('uttid', uttid)
asr.post_message(gst.message_new_application(asr, struct))

def asr_result(self, asr, text, uttid):
"""Forward result signals on the bus to the main thread."""
struct = gst.Structure('result')
struct.set_value('hyp', text)
struct.set_value('uttid', uttid)
asr.post_message(gst.message_new_application(asr, struct))

def application_message(self, bus, msg):
  

[Tutor] garbage collection/class question

[richard kappler]

Class is still something I struggle with. I think I'm finally starting to
get my head wrapped around it, but the discussion in a different thread has
sparked a question. First, please check my understanding:
A class creates objects, it's like a template that allows me to create as
many copies as I want of the object but allows me to have slightly
different versions of the object by having different values for the
variables within the object, which I can set with arguments?
By using  __init__ (self) I instantiate a new copy of the object?

Whether the above is correct or not (and do please correct me/ tutor me),
my question is, if I create objects in a while True loop, do the objects
get garbage collected, ie. disappear when the loop returns to the beginning
and creates new versions of the objects?

Psuedo code example:

(presuming I have a class that creates a lemon, a lime and has a function
that operates on them called juice)

while True:
lemon = yellow() # create a yellow object called lemon
lime = green() # create a green object called lime
drink = juice(lemon, lime) # operate on objects lemon and lime in a
function called juice resident in the class

So, using the above example, when the loop reaches the end and returns to
the beginning, new lemons and limes are created, yes? What happens to the
old ones? Or have I still got this completed boggled?

regards, RIchard

-- 

quando omni flunkus moritati
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

[Tutor] .strip question

[richard kappler]

I have a sort of a dictionary resulting from psutil.disk_usage('/') that
tells me info about my hard drive, specifically:

usage(total=147491323904, used=62555189248, free=77443956736, percent=42.4)

I'm having a bit of a brain fudge here and can't remember how to strip out
what I want. All I want to end up with is the number percent (in this case
42.4)  I started playing with .strip but the usage term before the parens
gets in the way. If it weren't for that I could convert this into a dict
and just pull the number by the key, right? So how do I strip out the
'usage' string? Once I do that, I think I know what I'm doing but here's my
proposed code to look at if you would.

import psutil as ps

disk = ps.disk_usage('/')

# whatever I need to do to strip usage out

d = {}
for item in disk.split(','):
item = item.strip()
key, value = item.split(':')
key = key.strip()
value = value.strip()
d[key] = float(value)
return d

Mind you, this is as of yet untested code, so before you ask for
tracebacks, I can't give any until I figure out how to get rid of the
preceding 'usage'.

regards, Richard


-- 

quando omni flunkus moritati
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor