Re: using POE and ftp to build a win32 <-> *nix job system?
You also may want to take a look at POE::Component::DirWatch and POE::Component::DirWatch::Object as a way to trigger the start of the process. 1. FIle created by server --- Watcher on win32 discovers it 2. Watcher triggers the process action and generates result files 3. watcher on unix discovers results and does something with them Beware though that maybe im just eager for someone to use my module. (DirWatch::Object) groditi On 8/22/06, Jeff McCarrell <[EMAIL PROTECTED]> wrote: Hi folks. I need to create a system that - moves some files generated on a *nix system to a win32 system - does some processing on those files - moves the results back to the *nix system The win32 system is the server side of the connection, and it needs to be able to support multiple clients, be highly available, etc. All of the usual daemon reqs, except for security. This is running inside the firewall, so I don't have to worry about that class of problems. My thought is to use POE and some of the FTP modules to handle the file transfers for a couple of reasons: - I don't want to deal with \r\n <-> \n line ending conversions - the result files are not text files My first thought was that POE and, in particular, POE::Component::Client::FTP and POE::Component::Server::FTP, were perfect matches, but the comments at http://poe.perl.org/?POE_Components_List are not inspiring to say the least. I could fall back to using Net::FTP and Net::FTPServer, I think, but that would make my client and server pretty chunky, and not very cooperative-multitasking-friendly. I've been through most all of the POE tutorials and docs, and I have the control protocol that I want the client and server to talk worked out pretty well. But I'm unclear on how best to leverage the POE and FTP combination. So I seek your wisdom: if it were your decision, which would you choose? Thanks in advance... -- jeff
RE: using POE and ftp to build a win32 <-> *nix job system?
I don't know your reasons for FTP, but are you stuck on it? My preferred method for doing this has been to just mount a windows share via SMB/CIFS, and use it like just-another-unix-filesystem. IMHO There's no good reason to write networking code where file IO will suffice. Just another point of failure. You did mention that it's all inside the firewall, and your windows Server probably already has the directory in question shared. (C$ at the least!) All you need then is a mount statement, either manually in your script, or automounted in your /etc/fstab. A note though, If you use win 2k3 - you will need CIFS, not samba, to mount with proper permissions. This is easier on a 2.6 series kernel, as the 2.4 series, (on slackware at least,) has to be patched for cifs support. (Though it really is quite easy to do.) Regards, Gavin McDonald EVI Logistic Enterprises email: [EMAIL PROTECTED] phone: (604) 313-3845 > -Original Message- > From: Jeff McCarrell [mailto:[EMAIL PROTECTED] > Sent: Tuesday, August 22, 2006 4:01 PM > To: poe@perl.org > Subject: using POE and ftp to build a win32 <-> *nix job system? > > Hi folks. > > I need to create a system that > - moves some files generated on a *nix system to a win32 system > - does some processing on those files > - moves the results back to the *nix system > > The win32 system is the server side of the connection, > and it needs to be able to support multiple clients, > be highly available, etc. All of the usual daemon reqs, > except for security. This is running inside the firewall, > so I don't have to worry about that class of problems. > > My thought is to use POE and some of the FTP modules to handle > the file transfers for a couple of reasons: > - I don't want to deal with \r\n <-> \n line ending conversions > - the result files are not text files > > My first thought was that POE and, in particular, > POE::Component::Client::FTP and > POE::Component::Server::FTP, > were perfect matches, but the comments at > http://poe.perl.org/?POE_Components_List > are not inspiring to say the least. > > I could fall back to using Net::FTP and Net::FTPServer, I think, > but that would make my client and server pretty chunky, and not > very cooperative-multitasking-friendly. > I've been through most all of the POE tutorials and docs, and > I have the control protocol that I want the client and server > to talk worked out pretty well. > But I'm unclear on how best to leverage the POE and FTP combination. > > So I seek your wisdom: if it were your decision, which would you choose? > Thanks in advance... > > -- jeff
using POE and ftp to build a win32 <-> *nix job system?
Hi folks. I need to create a system that - moves some files generated on a *nix system to a win32 system - does some processing on those files - moves the results back to the *nix system The win32 system is the server side of the connection, and it needs to be able to support multiple clients, be highly available, etc. All of the usual daemon reqs, except for security. This is running inside the firewall, so I don't have to worry about that class of problems. My thought is to use POE and some of the FTP modules to handle the file transfers for a couple of reasons: - I don't want to deal with \r\n <-> \n line ending conversions - the result files are not text files My first thought was that POE and, in particular, POE::Component::Client::FTP and POE::Component::Server::FTP, were perfect matches, but the comments at http://poe.perl.org/?POE_Components_List are not inspiring to say the least. I could fall back to using Net::FTP and Net::FTPServer, I think, but that would make my client and server pretty chunky, and not very cooperative-multitasking-friendly. I've been through most all of the POE tutorials and docs, and I have the control protocol that I want the client and server to talk worked out pretty well. But I'm unclear on how best to leverage the POE and FTP combination. So I seek your wisdom: if it were your decision, which would you choose? Thanks in advance... -- jeff
Re: memory (session) leaks in perl
Its not impossible, its just more trouble to terminate a
session. POE suffers from poor huffman encoding as it is,
adding more obligatory code to do simple things makes it
worse. Now I have to put a line to register the signal, one
to deregister it, and a "stop" state, just to make sure I
deregister the signal, and I have to make sure all
terminating states yield to "stop" instead of just not
yielding when terminating.
Except for the UI_DESTROYED, how many signals do sessions
really wait for before terminating, as opposed to using
them informationaly?
Maybe have a call to stop the current session:
$KERNEL->stop_session()
The session stops, no question asked, lose all the aliases,
all the signals, all the postback/callback become invalid,
etc... and _stop gets called, not necessarily in that
order.
-Mathieu
--- Rocco Caputo <[EMAIL PROTECTED]> wrote:
> Do you have a use case where it's impossible to do
> something under
> the new behavior? I'm working under the assumption that
> a session
> can always find a way to call sig(YOUR_SIGNAL_HERE =>
> undef) when
> it's ready to be destroyed.
>
> To be fair regarding discussions on this list, Jonathan
> Steinert
> announced the intent to make sig() hold sessions alive in
> his 19
> October 2005 message titled "Nastiness, and wrapping up
> signal
> reforms". I replied that day with:
>
> > Big change. I don't mind this; the old semantics of
> not holding a
> > reference count were tied to _signal, which delivered
> signals
> > without sessions explicitly asking for them. _signal
> is gone now,
> > so we can tie the explicit interest of sig() into a
> reference count
> > to keep the session alive.
>
> Nobody else responded. 17 days later I replied with a
> public go-
> ahead to make the change.
>
> --
> Rocco Caputo - [EMAIL PROTECTED]
>
>
> On Aug 21, 2006, at 09:58, Nick Williams wrote:
>
> > There appears to be a lack of opinion on this issue.
> Would it be
> > therefore reasonable to backout the signal change in
> POE? Or can
> > anyone suggest a way around the problem?
> >
> > Nick.
> >
> > [EMAIL PROTECTED] wrote:
> >
> >> On Wed, 2 Aug 2006, Mathieu Longtin wrote:
> >>
> >>
> >>> It was my understanding that a session would stay
> alive as
> >>> long as it has child sessions. Did that behavior
> change?
> >>>
> >>
> >> No, this is not down to a child session - AFAIK, the
> behaviour
> >> there has
> >> not changed.
> >>
> >>
> >>> I'm in agreement with Nick here. If a session is
> stritly
> >>> waiting for a child session to finish, then have it
> call
> >>> waitpid.
> >>>
> >>> $_[KERNEL]->wait_for_child($session, "state", @args);
> >>>
> >>> wait_for_child would add a link back to the current
> >>> session, the same way set_delay does.
> >>>
> >>
> >> The issue is not wait_for_child, it's wait_for_signal
> (which doesn't
> >> exist). The original problem report was that people
> were confused by
> >> setting up a signal handler for UIDESTROY didn't make
> the session
> >> persistent. That is standard signal semantics and the
> persistence
> >> could've easily been changed by using aliases.
> >>
> >> The new behaviour in POE (as of .3202) is that when
> placing a signal
> >> handler on a session, e.g.
> >>$kernel->sig('uidestroy', 'do_something');
> >> that will now increment the reference count of the
> session and
> >> therefore
> >> make that session persistent until the signal handler
> is deleted via
> >> something (and note, not within the _stop event, since
> we don't
> >> get that
> >> far).
> >>
> >> Nick.
> >>
> >>
> >>> -Mathieu
> >>>
> >>>
> >>> --- Nick Williams <[EMAIL PROTECTED]>
> wrote:
> >>>
> >>>
> So, I've found the reason that recent releases of
> POE
> cause me to get
> memory leaks - I know others have had problems also,
> so I
> wanted to get
> an open discussion of what I'm being bitten by and
> possible ways to
> workaround this.
>
> It turns out for me that my sessions aren't being
> garbage
> collected. In
> general, my POE system doesn't *require* the _stop
> event
> to be processed
> and so I never noticed this. However, peeking into
> the
> system shows a
> large number of sessions and I can see that none of
> the
> _stop events
> have been called.
>
> This is because of the change:
>
> 2005-11-07 06:59:07 (r1852) by hachi
> poe/lib/POE/Resource/Signals.pm M;
> poe/lib/POE/Resource/Sessions.pm M
>
> Change signal watchers so they keep sessions
> alive.
>
> WARNING: This is a major semantics change in
> POE.
> It has the
> potential to make code 'hang' in places where
> it
> formerly did not.
>
> This change is necessary so sessions
> expressing
> an interest in SIG
> CH?LD do not die prematurely. (There is a
> planned
> mandatory warning
>
Re: memory (session) leaks in perl
Rocco Caputo wrote: Do you have a use case where it's impossible to do something under the new behavior? I'm working under the assumption that a session can always find a way to call sig(YOUR_SIGNAL_HERE => undef) when it's ready to be destroyed. "When it's ready to be destroyed" is the key. The new behaviour means that sessions need to track that themselves and (in effect) manage their own reference counting independent of POE's. Before, they could just rely on POE to let them know via _stop. Now, _stop is no longer called unless the session clears the sig first. This is just a catch-22. And it leaves a race condition whereby the session exists but has declared it no longer wants the signal. To be fair regarding discussions on this list, Jonathan Steinert announced the intent to make sig() hold sessions alive in his 19 October 2005 message titled "Nastiness, and wrapping up signal reforms". I replied that day with: Big change. I don't mind this; the old semantics of not holding a reference count were tied to _signal, which delivered signals without sessions explicitly asking for them. _signal is gone now, so we can tie the explicit interest of sig() into a reference count to keep the session alive. Nobody else responded. 17 days later I replied with a public go- ahead to make the change. Yes, I realise that there was this discussion previously. However, speaking purely for myself, I didn't understand the impact of this at the time, since I wasn't cognizant of the internals of session reference counting at the time. Now I've looked at this, and I can't see how the new implementation makes sense. My issue is that that the bugzilla that Jonathan was attempting to fix is just trivial to fix using existing POE mechanisms of aliases, since there's an easy point at which you know you want to start the persistence of the session, and there's a well-defined point at which you can release the persistence. However, by making the behaviour of persistence implicit within signals, there is simply no way to achieve the opposite effect (automatic garbage collection). The user (the application) must decide at which point it has no more work to do and at that point it can then clear the signal. And only then will POE do it's garbage collection and call back to the application. This just doesn't make sense. Especially when you compare signals in POE with signals in other dispatchers. Having a handler configured for a signal should not make that process persistent. Maybe I'm thinking about it wrong. To me, "explicit interest of sig()" does not mean to me that I (this session) want to stay around until that sig, it means to me that it's merely putting in place a handler IN CASE of that sig. It's a really really important distinction. With the huge differentiator that the latter behaviour of 'in-case-of-handlers' CANNOT be achieved in the new POE signal world without careful application coding and ignoring any of the benefits of POE's internal garbage collection. As a compromise, I've also proposed implicitly that maybe we should have a new function wait_for_sig() as well as just sig(), so that we can make the difference in semantics explicit to users. I don't mind which way around the functions and semantics are achieved, so long as there is a way of doing this. Nick
Re: memory (session) leaks in perl
Do you have a use case where it's impossible to do something under
the new behavior? I'm working under the assumption that a session
can always find a way to call sig(YOUR_SIGNAL_HERE => undef) when
it's ready to be destroyed.
To be fair regarding discussions on this list, Jonathan Steinert
announced the intent to make sig() hold sessions alive in his 19
October 2005 message titled "Nastiness, and wrapping up signal
reforms". I replied that day with:
Big change. I don't mind this; the old semantics of not holding a
reference count were tied to _signal, which delivered signals
without sessions explicitly asking for them. _signal is gone now,
so we can tie the explicit interest of sig() into a reference count
to keep the session alive.
Nobody else responded. 17 days later I replied with a public go-
ahead to make the change.
--
Rocco Caputo - [EMAIL PROTECTED]
On Aug 21, 2006, at 09:58, Nick Williams wrote:
There appears to be a lack of opinion on this issue. Would it be
therefore reasonable to backout the signal change in POE? Or can
anyone suggest a way around the problem?
Nick.
[EMAIL PROTECTED] wrote:
On Wed, 2 Aug 2006, Mathieu Longtin wrote:
It was my understanding that a session would stay alive as
long as it has child sessions. Did that behavior change?
No, this is not down to a child session - AFAIK, the behaviour
there has
not changed.
I'm in agreement with Nick here. If a session is stritly
waiting for a child session to finish, then have it call
waitpid.
$_[KERNEL]->wait_for_child($session, "state", @args);
wait_for_child would add a link back to the current
session, the same way set_delay does.
The issue is not wait_for_child, it's wait_for_signal (which doesn't
exist). The original problem report was that people were confused by
setting up a signal handler for UIDESTROY didn't make the session
persistent. That is standard signal semantics and the persistence
could've easily been changed by using aliases.
The new behaviour in POE (as of .3202) is that when placing a signal
handler on a session, e.g.
$kernel->sig('uidestroy', 'do_something');
that will now increment the reference count of the session and
therefore
make that session persistent until the signal handler is deleted via
something (and note, not within the _stop event, since we don't
get that
far).
Nick.
-Mathieu
--- Nick Williams <[EMAIL PROTECTED]> wrote:
So, I've found the reason that recent releases of POE
cause me to get
memory leaks - I know others have had problems also, so I
wanted to get
an open discussion of what I'm being bitten by and
possible ways to
workaround this.
It turns out for me that my sessions aren't being garbage
collected. In
general, my POE system doesn't *require* the _stop event
to be processed
and so I never noticed this. However, peeking into the
system shows a
large number of sessions and I can see that none of the
_stop events
have been called.
This is because of the change:
2005-11-07 06:59:07 (r1852) by hachi
poe/lib/POE/Resource/Signals.pm M;
poe/lib/POE/Resource/Sessions.pm M
Change signal watchers so they keep sessions
alive.
WARNING: This is a major semantics change in POE.
It has the
potential to make code 'hang' in places where it
formerly did not.
This change is necessary so sessions expressing
an interest in SIG
CH?LD do not die prematurely. (There is a planned
mandatory warning
for reaped children that were not being watched.)
This change fixes
RT 15215.
The problem with this change is that if I set up a signal
handler for
something innocuous (e.g. to handle DIE, or one of my
hand-rolled
signals), this means that the session will do everything
appropriately
except be garbage collected. I think the intention of
this change is
reasonable, but to make it the default behavior for all
possible signals
is a bit keen. If I put in place a handler for a signal,
it does NOT
mean that I want to WAIT for the signal, it usually means
that the
signal shouldn't even happen, but that I'm putting in
place a handler
*in-case* it happens. It certainly shouldn't make my
session persistent!
I'm not sure of "the best way" of fixing this. Possibly
enhancing the
API to have an explicity waitforsig() and maybesig()?
(That's half
tongue-in-cheek, but is actually one of the better
solutions). In terms
of "normal" API, a sig handler shouldn't block, so I
would expect the
sig() behaviour to follow that paradigm...
Thoughts, anyone?
Nick
__
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Announcement: POE 0.3601
POE 0.3601 is out there on CPAN. Right now. Get it. You won't be sorry. And if you are, send your reports and patches to bug- [EMAIL PROTECTED] === Significant features: Lots of new tests, thanks to Benjamin Smith and Google's Summer of Code. The final results will be available in a later release. Matt Sickler committed high- and low-watermark callbacks for flow control in POE::Component::Server::TCP (and Client). Changes to POE::Queue::Array allow it to be subclassed. Motivated by a request by Randal Schwartz. POE::Kernel stops resetting all signals at "use" time. Now it doesn't even DEFAULT/IGNORE signals unless it really must (as in SIGCHLD and SIGPIPE). Stuart Kendrick reported this bug. POE::Wheel::Run supports a new conduit mode: pty-pipe. STDIN and STDOUT are via pty in this mode, while STDERR arrives over a pipe. Thanks to Randal Schwartz for the patch. POE::Component::Server::TCP (and Client) return something meaningful now---the ID of the session just created. Matt Sickler committed that one. Nick Williams patched POE::Wheel::ReadLine to re-enable SIGINT when it's not actively waiting for input. You should set a sig(INT) in addition to checking for exceptions from ReadLine if you want to be sure to catch interrupts. === Significant fixes: Allen Donovan found a large memory leak in POE::Filter::Reference. It was only triggered when a serializer class was provided at constructor time, as newer versions of POE::Component::IKC did. Nick Williams patched POE::Wheel::ReadLine for better input history handling. Nick also patched POE::Kernel to maintain its integrity before rethrowing die(). Multiple documentation fixes in many areas, by a lot of people. To name a few: Guillermo Roditi, Mike Baas, Benjamin Smith, Chris Williams, and Tim Otten. Eben Rauhut improved POE::Filter::HTTPD's ability to be used with keep-alive connections. Randal Schwartz submitted a test case for exceptions in _stop causing an infinite loop in POE. The temporary fix is to not catch exceptions in _stop. A more permanent fix is forthcoming. Better Vanilla Perl installation support. The Vanilla Perl distribution doesn't include all the modules that ActivePerl does, and we had come to rely on some of them. Resolves a number of issues reported by Adam Kennedy. Applied Peter Farmer's fix for TRACE_FILENAME files gaining unneeded quotes. === And all the rest: What changed besides the highlights? A lot! Check the CHANGES for details. -- Rocco Caputo - [EMAIL PROTECTED]
Re: memory (session) leaks in perl
IRC would be the place to be, but like the list, there are a lot of
lurkers. Better chances of getting a hold of Rocco's (a.k.a. dngor)
ear though on IRC
On 8/21/06, Mathieu Longtin <[EMAIL PROTECTED]> wrote:
I'm with you on the wait_for_signal.
However, I note that there is very little traffic on this
mailing list, not even a announcement of the last two POE
release.
So, I wonder if POE discussions are all on some other
mailing list, or stritly on IRC at this point.
-Mathieu
--- Nick Williams <[EMAIL PROTECTED]> wrote:
> There appears to be a lack of opinion on this issue.
> Would it be
> therefore reasonable to backout the signal change in POE?
> Or can anyone
> suggest a way around the problem?
>
> Nick.
>
> [EMAIL PROTECTED] wrote:
>
> >On Wed, 2 Aug 2006, Mathieu Longtin wrote:
> >
> >
> >
> >>It was my understanding that a session would stay alive
> as
> >>long as it has child sessions. Did that behavior
> change?
> >>
> >>
> >
> >No, this is not down to a child session - AFAIK, the
> behaviour there has
> >not changed.
> >
> >
> >
> >>I'm in agreement with Nick here. If a session is
> stritly
> >>waiting for a child session to finish, then have it
> call
> >>waitpid.
> >>
> >>$_[KERNEL]->wait_for_child($session, "state", @args);
> >>
> >>wait_for_child would add a link back to the current
> >>session, the same way set_delay does.
> >>
> >>
> >
> >The issue is not wait_for_child, it's wait_for_signal
> (which doesn't
> >exist). The original problem report was that people were
> confused by
> >setting up a signal handler for UIDESTROY didn't make
> the session
> >persistent. That is standard signal semantics and the
> persistence
> >could've easily been changed by using aliases.
> >
> >The new behaviour in POE (as of .3202) is that when
> placing a signal
> >handler on a session, e.g.
> > $kernel->sig('uidestroy', 'do_something');
> >that will now increment the reference count of the
> session and therefore
> >make that session persistent until the signal handler is
> deleted via
> >something (and note, not within the _stop event, since
> we don't get that
> >far).
> >
> >Nick.
> >
> >
> >
> >>-Mathieu
> >>
> >>
> >>--- Nick Williams <[EMAIL PROTECTED]>
> wrote:
> >>
> >>
> >>
> >>>So, I've found the reason that recent releases of POE
> >>>cause me to get
> >>>memory leaks - I know others have had problems also,
> so I
> >>>wanted to get
> >>>an open discussion of what I'm being bitten by and
> >>>possible ways to
> >>>workaround this.
> >>>
> >>>It turns out for me that my sessions aren't being
> garbage
> >>>collected. In
> >>>general, my POE system doesn't *require* the _stop
> event
> >>>to be processed
> >>>and so I never noticed this. However, peeking into the
> >>>system shows a
> >>>large number of sessions and I can see that none of
> the
> >>>_stop events
> >>>have been called.
> >>>
> >>>This is because of the change:
> >>>
> >>> 2005-11-07 06:59:07 (r1852) by hachi
> >>> poe/lib/POE/Resource/Signals.pm M;
> >>>poe/lib/POE/Resource/Sessions.pm M
> >>>
> >>>Change signal watchers so they keep sessions
> >>>alive.
> >>>
> >>>WARNING: This is a major semantics change in
> POE.
> >>>It has the
> >>>potential to make code 'hang' in places where
> it
> >>>formerly did not.
> >>>
> >>>This change is necessary so sessions
> expressing
> >>>an interest in SIG
> >>>CH?LD do not die prematurely. (There is a
> planned
> >>>mandatory warning
> >>>for reaped children that were not being
> watched.)
> >>>This change fixes
> >>>RT 15215.
> >>>
> >>>
> >>>
> >>>The problem with this change is that if I set up a
> signal
> >>>handler for
> >>>something innocuous (e.g. to handle DIE, or one of my
> >>>hand-rolled
> >>>signals), this means that the session will do
> everything
> >>>appropriately
> >>>except be garbage collected. I think the intention of
> >>>this change is
> >>>reasonable, but to make it the default behavior for
> all
> >>>possible signals
> >>>is a bit keen. If I put in place a handler for a
> signal,
> >>>it does NOT
> >>>mean that I want to WAIT for the signal, it usually
> means
> >>>that the
> >>>signal shouldn't even happen, but that I'm putting in
> >>>place a handler
> >>>*in-case* it happens. It certainly shouldn't make my
> >>>session persistent!
> >>>
> >>>I'm not sure of "the best way" of fixing this.
> Possibly
> >>>enhancing the
> >>>API to have an explicity waitforsig() and maybesig()?
> >>>(That's half
> >>>tongue-in-cheek, but is actually one of the better
> >>>solutions). In terms
> >>>of "normal" API, a sig handler shouldn't block, so I
> >>>would expect the
> >>>sig() behaviour to follow that paradigm...
> >>>
> >>>Thoughts, anyone?
> >>>
> >>>Nick
> >>>
> >>>
> >>>
> >>>
> >>__
> >>Do You Yahoo!?
> >>Tired of spam? Yahoo! Mail has the best spam
> protection around
> >>http://mail.yahoo.com
> >>
> >>
> >>
>
>
_
