Re: [Tutor] reading lines from a list of files

[Albert-Jan Roskam via Tutor]

--
On Tue, May 12, 2015 9:54 PM CEST Peter Otten wrote:

>Albert-Jan Roskam wrote:
>
>> It was not that long ago that I found out about the fileinput module, so I
>> sometimes forget to use it. It is not specify the encoding of the files,
>> is it? It'd be nice if one could specify a tuple of encodings, e.g.
>> ('utf-8-sig', 'latin1', 'cp1252').
>> 
>> input(files=None, inplace=0, backup='', bufsize=0, mode='r',
>> openhook=None)
>> input([files[, inplace[, backup[, mode[, openhook])
>
>Whatever you plan to do with these encodings, it should be possible with a 
>custom openhook.

Hi Peter,

Sorry, I should have read the documentation better. I did not know about 
openhooks. There's one useful hook already:

fileinput.hook_encoded(encoding)
Returns a hook which opens each file with codecs.open(), using the given 
encoding to read the file.
Usage example: 
fi = fileinput.FileInput(openhook=fileinput.hook_encoded("iso-8859-1"))
Note With this hook, FileInput might return Unicode strings depending on the 
specified encoding.
New in version 2.5.

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Open a Url in new tab and tab switching in IE using python and selenium

[shweta kaushik]

Hi Laura,

I have posted same question in stackoverflow.com also but no body replied
till now, For timebeing I am using another approach to do my work.

Thank you all for your suggestions.

Regards,
Shweta

On Tue, May 12, 2015 at 3:06 PM, Laura Creighton  wrote:

> In a message of Wed, 06 May 2015 00:15:39 +0100, Alan Gauld writes:
> >On 05/05/15 19:22, shweta kaushik wrote:
> >> Thanks Steve for the information.
> >> I searched but was not able to find suitable forum to shoot this
> question.
> >> So posted here if anyone can help out.
> >>
> >
> >As Steve suggested the main python list is probably your best bet.
> >
>
> Or stackoverflow.com
>
> There are people whom I know know selenium well who read that and
> don't read python-list (any more).
>
> Laura
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] reading lines from a list of files

[Peter Otten]

Albert-Jan Roskam wrote:

> It was not that long ago that I found out about the fileinput module, so I
> sometimes forget to use it. It is not specify the encoding of the files,
> is it? It'd be nice if one could specify a tuple of encodings, e.g.
> ('utf-8-sig', 'latin1', 'cp1252').
> 
> input(files=None, inplace=0, backup='', bufsize=0, mode='r',
> openhook=None)
> input([files[, inplace[, backup[, mode[, openhook])

Whatever you plan to do with these encodings, it should be possible with a 
custom openhook.

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] my membership and access to the Tutor list

[Alan Gauld]

Forwarding to the list for comment.
Always use Reply All9Or Reply List if your mailer supports it)
when including the list members.

Alan G

On 12/05/15 11:24, Stewart Lawton wrote:

Hi Alan
 I have worked though the file permissions cogniscent of  your 
comments to see if  I can find what
is failing in apache access to a python created unix socket. Points 1) 
,..., 12) give the results.
In particular I do not understand how to set the user of uds_socket to 
apache or set the write permission of
uds_socket group to rwx. I think that either change should enable 
successful operation, comments please!

In answer to your other questions:
I chose Unix Sockets since I had very similar access problems with IP 
sockets.
I would like to remote control an embedded device from a laptop. The 
target will be Raspberrypi that in turn communicates to ARM Cortex M3 
devices that are capable of  correct Nyquist sampling, that Unix based 
devices cannot guarantee. I chose Python since it is so widely used 
and I need to learn that language processor. I appreciate there are 
many ways other ways of achieving this end but I think this one ought 
to work!

Many Thanks for your help,
Stewart Lawton

1) /etc/httpd/conf/httpd.conf species the apache server user and group 
as:-


# User/Group: The name (or #number) of the user/group to run httpd as.
# It is usually good practice to create a dedicated user and group for
# running httpd, as with most system services.
#
User apache
Group apache
2)apache is started with command sudo ./startapache that contains:-
systemctl start httpd.service

3) the process status of apache is found by command ps -el
the following is taken from the status report:-
F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY TIME CMD
4 S 0  2226 1  0  80   0 -  7493 poll_s ? 00:00:00 httpd
5 S48  2227  2226  0  80   0 -  7493 inet_c ? 00:00:00 httpd
5 S48  2228  2226  0  80   0 -  7493 inet_c ? 00:00:00 httpd
5 S48  2229  2226  0  80   0 -  7493 inet_c ? 00:00:00 httpd
5 S48  2230  2226  0  80   0 -  7493 inet_c ? 00:00:00 httpd
5 S48  2233  2226  0  80   0 -  7493 inet_c ? 00:00:00 httpd
4) The user identity UID ,48,is used to find the user and group given 
in /etc/passwd :-

apache:x:48:48:Apache:/usr/share/httpd:/sbin/nologin
The user and group identies are given as 48:48 so the user and group 
are apache and apache.
5)The above hopefully establishes that the server has permissions on 
user and groups named as apache.
6)I created a test directory at /test to be used to contain the file 
node uds_socket.

drwxrwxr-x.   2 apache apache  4096 May 11 20:15 test
7) on starting the Socket server that listens for an incoming connection
the new uds_socket is created with user and group permissions as shown:-
srwxr-xr-x. 1 johnlawton apache 0 May 12 10:22 /test/uds_socket
8)I do not understand what function the s bit performs here.
  I note that group access cannot write the file.
9) When I execute the myUnix2.cgi script from /var/www/cgi_bin with 
johnlawton as user with primary group apache the script executes and 
the listening server responds correctly. I note johnlawton has rwx 
access but the group apache access is limited  to r-x.
10) When the apache server executes the myUnix2.cgi script failure 
results in failing to access the socket.
11) Summary. I think the server fails as it can only get group access 
and group access is limited to

r-x NO w permission.
12) How can I get UDS_Socket to be created with apache as user(hence 
allowing rwx) or enable apache group access with w permission?





*From:* Alan Gauld 
*To:* Stewart Lawton ; tutor 


*Sent:* Friday, 8 May 2015, 10:33
*Subject:* Re: my membership and access to the Tutor list

On 08/05/15 09:09, Stewart Lawton wrote:
> Hi Alan
> Thank you very much for your response to my Tutor@python.org 
 question.

> I thought my membership was complete and that I could log in to answer
> your comments.

The tutor list is a mailing list not a web forum. You don't login to 
answer

comments you  send an email reply. Use Reply to send to the individual
(as you've just done with me) or, more usually, use ReplyAll (or ReplyList
if your mail tool has that feature) to reply to everyone on the list.

Use plain text to preserve code layout and use interleaved posting
(as I'm doing here) rather than top-posting.

> I found I could not login again. PLEASE can you help to get my
> password reset?

Only you can change the password, its purely web based. I only
approve messages in the moderation queue, virtually nothing else.
But the password just gives you access to your admin settings.

> I think I am failing to understand what user and or group permissions
> are required between apache python, and the python myUnix2.cgi program
> I am using.

OK, I'm no expert here but several things about your program
have me puzzled.

First remember that the web s

Re: [Tutor] Open a Url in new tab and tab switching in IE using python and selenium

[Laura Creighton]

In a message of Wed, 06 May 2015 00:15:39 +0100, Alan Gauld writes:
>On 05/05/15 19:22, shweta kaushik wrote:
>> Thanks Steve for the information.
>> I searched but was not able to find suitable forum to shoot this question.
>> So posted here if anyone can help out.
>>
>
>As Steve suggested the main python list is probably your best bet.
>

Or stackoverflow.com

There are people whom I know know selenium well who read that and
don't read python-list (any more).

Laura
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] reading lines from a list of files

[Albert-Jan Roskam via Tutor]

--
On Tue, May 12, 2015 8:48 AM CEST Peter Otten wrote:

>Alex Kleider wrote:
>
>> Is there a better (more 'Pythonic') way to do the following?
>> 
>>  for f_name in f_names:
>>  with open(f_name, 'r') as f:
>>  for line in f:
>
>There's the fileinput module
>
>
>
>but personally I prefer the way you show above.

It was not that long ago that I found out about the fileinput module, so I 
sometimes forget to use it. It is not specify the encoding of the files, is it? 
It'd be nice if one could specify a tuple of encodings, e.g. ('utf-8-sig', 
'latin1', 'cp1252'). 

input(files=None, inplace=0, backup='', bufsize=0, mode='r', openhook=None)
input([files[, inplace[, backup[, mode[, openhook])
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Tutor Digest, Vol 135, Issue 12

[Laura Creighton]

In a message of Tue, 05 May 2015 09:55:32 +0100, Alan Gauld writes:
>On 05/05/15 08:17, Siya 360 wrote:
>
>> Twice i unsubscribed to this mailing list, and i still continue to get them, 
>> why?
>

>The web page is the only way to unsubscribe. Nobody else
>on the list can unsubscribe you.

This is late.  I've been away.  Putting on my mailman hat now. :)

Actually, the people who are the owners of the list _can_ do this.
That's you and wescpy.  But, of course, it's better to not cave in
to user requests to do this.  I just wanted to let people know that
this is not a limitation of mailman 2.x in case they were in the
market for a mailing list.

However, I believe I know what is going on here.  The original poster,
not cc'd on this list, was receiving his or her mail as a digest.  And,
after you unsubscribe, mailman has to figure out what to do with the
digest that was in the process of being assembled for you.  So digest
readers often get one, and in some odd circumstances 2 copies of
the digest they no longer want to receive.

However, it is easy to check if the account is still around if you
are a list admininstrator ... and that is often a good idea, because
people do get into trouble by unsubscribing a different account than
the one where they are receiving mail, with the perfectly reasonable
outcome that the mail keeps on coming.


Best,
Laura
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] reading lines from a list of files

[Peter Otten]

Oscar Benjamin wrote:

> A generator cannot guarantee that execution continues after a yield so
> any context manager used around a yield is dependent on __del__. I
> think a good rule of thumb is "don't yield from a with block".

Uh-oh, I am afraid I did this quite a few times. Most instances seem to be 
context managers though. Is something like

@contextmanager
def my_open(filename):
if filename == "-":
yield sys.stdin
else:
with open(filename) as f:
yield f


OK?


___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] reading lines from a list of files

[David Rock]

* Alex Kleider  [2015-05-12 02:33]:
> On 2015-05-11 23:48, Peter Otten wrote:
> > Alex Kleider wrote:
> > 
> >> Is there a better (more 'Pythonic') way to do the following?
> >> 
> >>  for f_name in f_names:
> >>  with open(f_name, 'r') as f:
> >>  for line in f:
> > 
> > There's the fileinput module
> > 
> > 
> > 
> > but personally I prefer the way you show above.
> 
> Then I'll stick with what you prefer and what I know.
> It seems silly to import yet another module for the sole
> purpose of saving one line of code although the reason
> for my inquiry was more to diminish levels of indentation
> than number of lines.
> Thanks,
> Alex

Personally, *I* prefer fileinput as my go-to file reader.

Don't knock fileinput for "saving one line." It does a lot more than
that.  It allows your script to manage the filelist as an input,
automatically handles stdin so your script can easily be both a filter
in a pipeline and a file reader, plus a host of other useful methods for
info about the file you are reading.

Part of what you really need to define is the context of your question
of "better."  What is your use case?  From where is your list of files
coming?  Is it truly just "read and forget"?  Your needs will dictate
what option is "best."  It may be what you've already done yourself, it
may be fileinput, or it may be something completely different.

-- 
David Rock
da...@graniteweb.com
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] reading lines from a list of files

[Oscar Benjamin]

On 12 May 2015 at 11:46, Peter Otten <__pete...@web.de> wrote:
>
> > although the reason
> > for my inquiry was more to diminish levels of indentation
> > than number of lines.
>
> You usually do that by factoring out the loops into a generator:
>
> def lines(files):
> for file in files:
> with open(files) as f:
> yield from f  # before python 3.3: for line in f: yield line
>
>
> for line in lines(files):
> ...
>
>
> Also possible, but sloppy as files are closed on garbage collection rather
> than explicitly:
>
> lines = (line for file in files for line in open(file))
> for line in lines:
>...

The lines generator function above relies on __del__ as well if the
loop exits from break, return or an exception in the loop body. This
happens whenever you yield or yield-from from inside a with block. The
chain of events that calls your context manager if I break from the
loop is:

1) Once the for loop discards the generator it has a zero reference count.
2) generator.__del__() called.
3) __del__() calls close()
4) close() throws GeneratorExit into the generator frame.
5) The GeneratorExit triggers the __exit__() methods of any context
managers active in the generator frame.

Try the following:

$ cat test_gen_cm.py
#!/usr/bin/env python3

class cleanup():
def __enter__(self):
pass
def __exit__(self, *args):
print("__exit__ called")__del__.

def generator_with_cm():
with cleanup():
yield 1
yield 2
yield 3

g = generator_with_cm()
for x in g:
break

print('Deleting g')
del g
print('g is now deleted')

$ ./test_gen_cm.py
Deleting g
__exit__ called
g is now deleted

A generator cannot guarantee that execution continues after a yield so
any context manager used around a yield is dependent on __del__. I
think a good rule of thumb is "don't yield from a with block".

Alex I apologise if what I've written here is confusing but really
what you started with is just fine. It is not important to fully
understand what I wrote above.


--
Oscar
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] reading lines from a list of files

[Peter Otten]

Alex Kleider wrote:

> On 2015-05-11 23:48, Peter Otten wrote:
>> Alex Kleider wrote:
>> 
>>> Is there a better (more 'Pythonic') way to do the following?
>>> 
>>>  for f_name in f_names:
>>>  with open(f_name, 'r') as f:
>>>  for line in f:
>> 
>> There's the fileinput module
>> 
>> 
>> 
>> but personally I prefer the way you show above.
> 
> Then I'll stick with what you prefer and what I know.
> It seems silly to import yet another module for the sole
> purpose of saving one line of code 

I think of importing a module as "cheap" unless it draws in a framework (e. 
g. numpy). And don't forget that even small pieces of code should be tested. 
So you aren't just saving the extra line, but also some of your tests.

> although the reason
> for my inquiry was more to diminish levels of indentation
> than number of lines.

You usually do that by factoring out the loops into a generator:

def lines(files):
for file in files:
with open(files) as f:
yield from f  # before python 3.3: for line in f: yield line


for line in lines(files):
...


Also possible, but sloppy as files are closed on garbage collection rather 
than explicitly:

lines = (line for file in files for line in open(file))
for line in lines:
   ...


___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] reading lines from a list of files

[Alex Kleider]

On 2015-05-11 23:48, Peter Otten wrote:

Alex Kleider wrote:


Is there a better (more 'Pythonic') way to do the following?

 for f_name in f_names:
 with open(f_name, 'r') as f:
 for line in f:


There's the fileinput module



but personally I prefer the way you show above.


Then I'll stick with what you prefer and what I know.
It seems silly to import yet another module for the sole
purpose of saving one line of code although the reason
for my inquiry was more to diminish levels of indentation
than number of lines.
Thanks,
Alex
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] TCP server/client application failing. Need some help with the basics!

[Anubhav Yadav]

But starting with a multi threaded server is not what I'd

> call starting small. Its quite advanced. I'd start getting
> non multi threaded servers running reliably first. Then
> I'd try adding some threading. As in maybe a dozen
> connections sharing some data source..
>
> Only then would I even consider the extra issues of
> scalability to thousands of connections...
>

I did implemented a small server which could only serve one client at a
time. In fact
after realizing that it is able to serve a single client at a moment, I
decided to go with multithreaded
approach. I just didn't thought of including that single threaded server
code here.


> It may be theoretically possible but when I was designing
> large client server/web based systems I used a rule of thumb
> that said not more than 100 connections per core. So for 5000 connections
> I'd need 50 cores. Allowing one core for the OS to
> run per box that leaves 7 cores per 8-core server, so I need
> 8 servers to handle 5000 connections. For resilience I'd
> add another one (so called N+1 architecture).
>
> Now those servers were all carrying significant processing
> requests not simple ping-like requests, so on that basis I'd
> guess that you don't need 50 cores for your example. But
> it does seem like a lot to ask of a single box. And if you
> intend to extend that to a network model think about the
> load on your network card too. My servers at work all
> had a couple of (gigabit) network cards connected.
>

I agree, but since I am trying to learn, I think I should limit my clients
to 100 rather than 1000 or 5000
per machine? Getting a server of that capacity is difficult for me.


> Too much code for me to read at this time of night, but I
> would try slimming down your testing, at least initially.
> Make sure it works on lower loads then ramp up slowly
> until it breaks.
>

I can relate, I wrote this mail at 2 am in the night and immediately hit
the bed. :)

>
> The other thing I'd consider is the timing of your requests.
> I didn't check but have you a decent gap between requests
> or are all 5000 arriving nearly simultaneously? It may
> be a network configuration issue - the socket/port queue
> simply getting choked to the point where packets time out.
>

I have a time.sleep(1) before sending each requests, from each client.

>

-- 
Regards,
Anubhav Yadav
KPIT Technologies,
Pune.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor