Re: Advice on POE and Daemons
ty wrote: > I was looking to turn some of the servers I have written in POE into > daemons that would be run as non-root users because I do not have root > permissions on the machine they will run on. Could anyone share their > experience of creating daemons with POE? How have you handled > ensuring the server is always running? How do you recover from a > crash? Have you added any logging to your daemon? > > Best Regards > > Ty > See Proc::Daemon::Init for daemonizing any long running Perl app. For our app we used Log::Log4perl for logging, worked incredibly well. We used external processes for checking pids and/or built in a way to talk to the server via a socket to confirm things were still running well. As for crash recovery, I'm a big fan of manual crash recovery cause I just don't think you can write a technically sophisticated process, otherwise you would have prevented the crash to begin with HTH, http://danconia.org
Re: What is postback
Abhishek Jain wrote: > Hi Friends, > I have searched a lot but was unable to figure out what is postback and how > is it different from yield or from post. > Can anyone here help me out. > Thanks in advance for your time and effort. > > -- > Regards > Abhishek jain > www.sms1.in > www.webbee.biz Did you read the two sections on postbacks in the POE::Session docs? http://search.cpan.org/~rcaputo/POE-0.3101/lib/POE/Session.pm There is a fairly well developed example in that doc in the postback method section. If you have read this then you probably need to be a bit more specific with your question. http://danconia.org
Re: need advice on using poe for implementation of regression framework
Manish Sapariya wrote: > Hi all, > I am testing a networking device and need to write a regression framework > for the device. I have evaluated STAF/STAX framework, but I think its > not addressing > some of our needs, that is resource scheduling. > I am not familar with STAF/STAX so can't speak to them. POE is definitely a great way to handle resource scheduling. > So we have a set of DUT (device under test) machines, and we have 1/2 > builds everyday. > We are looking for ( or a develop one) a regression framework, which has > scheduling > system which basically can, > > - Keep a database of DUT machines, a dut is classified as High end or > low end machine Is this a static DB, or one that is generated based on the DUT contacting the main system to let it know that it is alive? Either should work, but it might change the dynamics a bit. > - we should be able to queue up the regression task and the system > should take the >available machine when free and start the regression on it Assuming there is some way to notify the controlling system that the DUT is ready for work this should be pretty easy to do with a queue of sorts. I know there is at least one pre-existing component queue and I have described here at least once how I implemented a different sort (check the archives). One question I have is whether the controller is running the regression, or whether the DUT runs it when the controller tells it to. > - which can pause the regression on failure, or instructed by the test > operator Pausing a regression should be fairly trivial, just sending an event indicating a pause should work. The more difficult part to this is that you have to write two interfaces that generate that event, specifically the one caused by the failure *and* the human interactive one (which is the harder). > > Besides these we are looking for features like: > > - Centralized logging/reporting service, i.e. all the information about > the execution > of the regression to be routed to a central logging machine and > creating reports out > of those log/summary files about the health of a particular build > For this I would suggest Log::Log4perl. It sounds similar to what I have done in the past, only I had the logs on the local machine instead of using syslog or something on a separate machine. > - Health checking service for dut, being a big system, even if the part > of the system >is not functioning regression framework may carry on executing. So I > want to watch >for some parameters on the dut machine and declare the machine as > unusable and pause >the regression for analysis. > All of this is doable, just a matter of time. > So my question to the list is, can i develop such kind of services using > POE. Yes. > Any advantages/disadvantages? There are tons of advantages over doing this with other Perl systems, which are really just inherent to POE. The disadvantage is that POE does have a learning curve (though it is very useful once you are through it) and you will have a fair amount to write. Whether you could find another framework that is already closer to your goal I don't know, you might drop a line to the perl-qa mailing list to ask there about such frameworks. > Or is it overkill to use poe technology for this kind of purpose? If you are set on Perl it is definitely not overkill. > or are there any areas in my specs which cannot be addressed by perl/poe? Your spec seems fairly high level (as it would on a mailing list), I didn't see anything that couldn't be addressed. I would suggest you look into the Test:: suite to see if they can be used for writing your test cases if you haven't alreay done so. > Do you suggest any alternate existing framework? > > Thanks and Regards, > Manish > As an aside and my standard rant, I would start out with object based sessions rather than inline, I think it will serve you and save you time in the end. Good luck, sounds like an interesting project, http://danconia.org
Re: Program not work from crontab...why ?
Alex wrote: > it`s doing nothing...why so? > > thx Cron isn't running. Syntax errors. Perl isn't installed. Computer is turned off. Today is the 20th of June. Help us to help you... http://danconia.org
Re: POE::NFA
Tim Klein wrote: > In a different thread, Rocco wrote: > >> POE::NFA is not well documented. It seems to be the strange cousin of >> POE::Session that people rarely talk about, so time is focused >> elsewhere. > > > That's certainly true, but it's puzzling to me. POE::NFA is extremely > powerful! It's like POE::Session's big brother, rather than its strange > cousin. I don't know why more people aren't using it. A Session has > events, but an NFA has events and states! An event can be handled > differently depending on what state the NFA is in when the event > arrives. Totally cool. > > POE::Session is great for stateless applications, but it's been my > experience that most applications of any complexity do have states, even > if we don't always think of them that way. POE::NFA enables an > application to be explicit about what state it's currently in, and to > use event handlers specifically "tuned" for that state. And that, in > turn, enables me as an application writer to explicitly divide a complex > application into conceptual "chunks", where each chunk consists of the > event handlers for just one state. It's a powerful mental tool as well > as a powerful programming tool. I think of POE::NFA as a superset of > POE::Session. I'd hate to see it neglected! > > Tim Klein > Dallas, Texas > > I haven't messed with POE::NFA much, but from your description of states, I think the same (or even better) can be accomplished by using the object wrapped session rather than the inline sessions. By having every session wrapped by a regular object you can hide the implementation of the event handling, and essentially you have 100% control over the extent of a state, since it isn't in a single state, it is *just an object*, which already inherently knows its own state, etc. But I have ranted enough here, before, on the use of object sessions rather than inline ones :-)... http://danconia.org
Re: LaDBI multiple connections problem Re: SimpleDBI - the same.
Alex wrote:
>> I can only guess at your problem because you never explained the
>> nature of these "errors". Since the problem is identical with
>> different DBI components, I'm guessing that you've reached a shell or
>> system limit on the amount of resources you're allowed to consume.
>>
>> Talk to your system administrator about your system's resource limits.
>
>
>
> Yes, Postgres max_connections=200, I just need to make about 180 DBI
> sessions and use it.
>
> As I said before, I use Randal Schwartz`s script
> (http://www.stonehenge.com/merlyn/LinuxMag/col41.listing.txt). I`ve
> added DBI package to save all extracted links to DBI.
>
> Each time when I add a query to DBI I create new session and then there
> are more than 197 sessions there are errors...
>
You start out by creating 298 sessions? j=1 less than 3, is two
iterations, two times i=1 less than or equal to 150 is 298? This happens
before you even start the kernel...
Why the double loop anyways?
for my $i (0..195) {
...
}
>
> Code below:
>
>
> ...
> use constant LADBI => "ladbi";
> POE::Component::LaDBI->create( Alias => LADBI );
> my $r;
> for(my $j=1;$j<3;$j++)
> {
> for(my $i=1;$i<=150;$i++)
> {
> POE::Component::My::DBI->spawn(SQL => $SQL,EVENT => "insert");
> print "$j : $i\n";
> }
> sleep 1;
> }
> $poe_kernel->run;
> exit 0;
>
[snip]
http://danconia.org
Re: Sanity Check
Matt S Trout wrote: > On Sat, Apr 30, 2005 at 10:19:04AM -0500, Wiggins d'Anconia wrote: > >>I hate to say it but with all your various systems, protocols, etc. I >>would start from scratch with an XML based config system. I don't throw >>out XML lightly as I think it is overkill for a lot of things, but with >>all the different setups you are talking about it can be very effective. >> XML::Simple should do nicely. > > > At least a couple of people are using my Class::XML to automatically load > a tree of config objects from an XML file in a relatively clean format. I > find it much less of a pain in the neck, anyway :) > Looks interesting, more complex than I have ever needed. And I have never really used Class::DBI so it wouldn't be an easy jump for me. But I will certainly consider it in the future, especially for anything complex I get into. Been meaning to take Class::DBI for a test drive too. http://danconia.org
Re: Sanity Check
Rodney Rindels wrote: Hello Fellow POE Lovers. I had a project dropped on my desk, after extensively reading the requirements and reading the POE discussion groups and documentation, I'm convinced I finally have a POE worthy project. But. Since I Noticed Rocco mentions more than once the fact that some things are better left to cron.. I Thought I would get the more experience POE users opinion on this one. Sounds like a project for POE to me, certainly beyond cron's abilities, sure you could do it that way but it would be a nightmare. Basically I need to build an Event Based, Web Attached (for Management), FTP, SFTP,WWW::Mechanize, Expect, and Serial Device Controlled framework. wowzers, you forgot the kitchen sink. Make sure to use CPAN to its fullest with the above list, pretty sure everything is covered. A few features of the requirements are ability to start, stop, and restart from web console, as well as pull statistics and dynamic reconfiguration. Centralized logging to x,y, or z (Ie. Could be syslog, could be flat , could be database). Definitely check out Log4perl, http://log4perl.sf.net , it should help you accomplish the above in a very very nice way with minimal coding by you. I have used POE with Log4perl before and it was a dream. Persistant Database Connections And the ability to connect and transfer files to x number of system types based off either custom scripts or Centralized with Configuration files driving the changing pieces. I hate to say it but with all your various systems, protocols, etc. I would start from scratch with an XML based config system. I don't throw out XML lightly as I think it is overkill for a lot of things, but with all the different setups you are talking about it can be very effective. XML::Simple should do nicely. I started down the road of building at least (what I think) could be a core to keep the daemon running properly where I build _start event which kicks off kernel delay to call _start again every so often. Thoughts, Suggestions, Therapy? The only thing I would say specifically about POE, and I have said it before, is use the object wrapped sessions. The regular subroutine based sessions are nice for simple and small projects, or examples like those in the cookbook. But using the object based ones have made life so much easier for me in the past. No need to meddle in the HEAP and passing a bunch of values around. It allows you to restrict the events to things that should normally be firing events, and lets everything else that would normally just be a method call to an object remain as such, don't use a bunch of event calls just to pass values/objects around, etc. Just my $.02. Rodney Good luck. http://danconia.org
Re: using a separate session to process multiple client inputs from Component::Server::TCP
[EMAIL PROTECTED] wrote: First let me say that I think POE is awesome! It's been a mind-blowing couple of weeks trying to get my brain around it. I am rewriting a Network Monitoring system where the clients periodically connect to my server and send things. Right now I am using a simple accept and forking IO::Socket::INET server. The problem is that we are at the mercy of the clients and sometimes the clients hang or take a long time while sending data. Add to this that many clients may end up connecting at nearly the same time and you have a recipe for out of memory/cannot fork conditions. Some of this is due to other design problems. But I am doing a complete rewrite of the whole thing so I will fix those as well. So on the server I know I want to do something more robust and POE seems to encapsulate that already so that's what I am going to use. Basically my server would be getting an XML document from the client. I'll only process the data from the client after it disconnects. No need to answer back to it (at least not yet). I would also like to batch up the responses and process them with POE::Wheel::Run (because processing the queue involves a lot of slow running parsing of some XML and entering it into a database). So I am building up a queue and periodically processing it. The next step would be to take the entire queue, hand it of to POE::Wheel::Run, zero the queue, and keep moving. In my mind this is a Poe Man's pre-forking server. Here (below) is a skeleton of what I have so far to demonstrate this (but w/o the POE::Wheel::Run). Here are my questions: 1) Am I on the right track to have the Server and the queue processing in separate sessions where the Server "post"s to the queue's session? 2) Is there any way to high-water mark the queue so that it is processed immediately if reaching that mark? Or should I just be happy with periodically processing the queue as I have done below? 3) Am I using delay_set below in the way it was intended to be used? 4) Can anyone point me to examples of timing-out a client connection gracefully after a certain period of time. 5) Am I crazy? Thanks for reading, Lance [snip Lance's code] Sounds like you are on the right track to me, but I am no expert. This is where I begin to wonder if what I am about to say is better off in a separate discussion, that is, why does everyone go directly to the inline states sessions? I have been wondering this for about 4 months (aka since I found out about the object states), I found the object states setup much much much easier to control and work with than the inline states. Possibly because most of the components are done inline or because the cookbook (in order to keep the examples simple) generally uses them? Back to the point I currently have an app in production, that handles files in a similar type of queue to yours. Elements are added to a queue from a directory watcher rather than a socket but the principle is the same. In my scheme I also have a series of queues in order to throttle the processing, aka a single file is processed in 4 steps, each taking a varying amount of time and cpu usage. Yielding 4 queues per direction (inbound and outbound). Each of the queues manipulates an object using the same 'process' method but the object gets blessed from one type of "Processor" to another depending on which queue it is in, this keeps my queue code generic, allows the processors to all act in the same manner (which makes writing each of them simple), while allowing different processing to occur at each stage. Finally, the process method is called within a POE::Wheel::Run as you describe to allow for multi-tasking. Doing all of this with inline states gave me headaches, when I switched to the object states it was so much clearer, the queue itself is just a standard object, things pop in and out of the queue as if it was static, but the implementation that makes it "run" is based on the fact that the queue object wraps a POE::Session, this also allows me to completely de-couple the POE events from the rest of the application, which to me is simpler and keeps my head from falling off. Ok so that is a great story and all but how does it help you? Well I am not completely allowed to post full code, so I have stripped down my process queue, removed names to protect the innocent, removed the logging and other crap and then put it up online (at least temporarily). About the only other thing to know is that I use a home grown Exception object to throw errors, and normally the modules wouldn't have top level module names. I don't know if the code will run out of the box as I may have made some mistakes when cutting it down, but it may give you ideas. Anyone interested in it, or have suggestions of how it can be improved I would love to hear them. If the POE group is at all interested in what I have done, I will take the discussion to the next level wher
Re: Deprecate $_[KERNEL]?
Tim Wood wrote: At 06:52 PM 2/23/04, Rocco Caputo wrote: So someone unclear on the $_[KERNEL] concept needed some help with it on IRC. They wanted to call a function that set a delay in the current session, but they didn't understand that $_[KERNEL] is a reference to an object (a singleton) and didn't magically appear in called subroutines. Singleton is an implementation detail. Hide it behind a provider interface and no-one need know the cardinality. ($_[KERNEL] is a fine, perl-concise way to do this, BTW). I agree, programmers using POE likely need to understand a simple slice anyways. TW PS Wherezat [POE] tag on the subject line?? :> Not sure I understand this need? Filter on To:/Cc: adding 5 characters to the subject line seems silly... As for the earlier conversation about the dev stuff crowding the list, I actually like it, and agree with whomever posted that they wouldn't normally read it as much since it seems to happen on IRC. This list is very low traffic otherwise and it may help to make things seem less dead. Having said that, it doesn't much matter to me, I will subscribe to both/multiple anyways. Though I was pleasantly surprised to see that the traffic had picked up, just wished I could have found time to read it when it was hot. On the testing reform note (and this may be tangential) are there plans or would its development help users test hand written POE stuff? I am currently developing major test suites for an application that uses POE as its framework but haven't quite figured out (haven't researched much either) how to test objects that wrap a session, once I am finished with the non-POE parts I intend to look further into this issue. http://danconia.org
Re: Help with JobQueue
Georgios Pappas wrote:
Hello,
I'm pretty new to POE and I found it very useful to solve a simple
problem. Given an array of tasks I want to distribute them to an array
of hosts, but each host should deal with a single task. The list of
tasks is larger than the available hosts and as soon as the host
finishes its job a new task is assigned to it.
I tried to implement it using a JobQueue, but in my program instead
of distributing the tasks and running them in parallel, the task for
each host blocks the entire execution. The question is how can I
effectively distribute the tasks between hosts? How should I proceed if
two hosts finish exactly at the same time? Could this program be
implemented without JobQueue?
You should consider running your "long task" inside of a POE::Wheel::Run
that way it is forked as a non-blocking process so control would
return to your queue while the processing of a particular element of
that queue continues to happen. Then when there is output/error or exit
you can handle those events in the controller (queue) to start running a
new task. Because the controller is timesliced you don't have to worry
about all of the normal IPC locking, issues, etc.
I did something similar but never succeeded at getting PoCo::JobQueue to
do what I wanted effectively, but it was trivial to create my own queue
that handled this specific task. Because my queue(s) are part of a
larger app I created a queue object which had methods for adding
elements to the queue, popping off the next task to run, running a task,
getting the number of running tasks, number of waiting elements, etc.
Then I implemented the events, basically check queue, process element,
element output, element error, as object states so each session had
access to the object where the elements were stored (the queue),
avoiding the need to mess with the HEAP. To go one step further I also
made each of the elements of the queue an object, and then the 'process
element' event could handle any object that had the same 'process'
method, these are known as Processors. So essentially there is a
threshold, some other input source adds an element to a queue which
triggers a check queue event, that then checks to see if there are new
elements in the queue and if the threshold has not been hit starts a new
one. Then the element is processed inside of a wheel and the check
queue is called again. When the wheel is done (the process has
completed) then the exit event calls another check queue. This way
there is no need to poll and as soon as an element is added to the queue
it can be scheduled to process.
Sorry I can't post code (yet), I am working on the company to let me
reveal it. Someone may have a better method but I have found this to
work well for our application.
http://danconia.org
Thanks in advance,
Georgios Pappas
Here is the code:
use POE qw(Component::JobQueue);
my @HOSTS = qw/host1 host2 host3 host4/;
my $job_control = { count=>0 };
use vars qw (@HOSTS $job_control);
# They hold the time spent in the task
my @tasks = qw( 15 4 3 2 10 5 4 8 2 1 );
$job_control->{finished_host} = undef;
#
# Spawning the tasks
#
POE::Component::JobQueue->spawn
( Alias=> 'active',
WorkerLimit => scalar(@HOSTS),
Worker => \&worker_next,
Active =>
{ AckAlias => 'respondee',
AckState => 'response',
},
);
sub worker_next {
my $task;
my $host;
foreach my $host (@HOSTS) {
if ($job_control->{$_}->{busy}!=1) {
$task = shift @tasks;
if (defined $task) {
&spawn_worker( $host, $task );
}
}
}
}
sub worker_start {
my ($kernel, $heap, $host, $task ) =
@_[KERNEL, HEAP, ARG0..ARG1];
$job_control->{$host}->{busy} = 1;
$heap->{task} = $task;
$heap->{host} = $host;
# Runnig the task -> it may take a while !!!
warn "Task $task sent to host [$host].\n";
$kernel->yield("LongTask");
}
sub worker_stop {
my $heap = $_[HEAP];
print "finished on [$heap->{host}]\n";
$job_control->{finished_host} = $heap->{host};
$job_control->{count}++;
$job_control->{$heap->{host}}->{busy} = 0;
}
sub spawn_worker {
my ($outer_host, $outer_task) = @_;
POE::Session->create
( inline_states =>
{ _start => \&worker_start,
_stop => \&worker_stop,
LongTask => \&LongTask,
},
args => [ $outer_host, $outer_task ]
);
}
sub LongTask {
my ($heap) = @_[HEAP];
#
# WARNING: My task is actually a SOAP call to the server but
# I'm simulating this with sleep (and not $kernel->delay)
sleep($heap->{task});
}
Re: Feedback on POE::Wheel::Run changes
Rocco Caputo wrote: On Thu, Mar 06, 2003 at 11:00:26AM +1100, [EMAIL PROTECTED] wrote: I understand that there is a long-term goal to supercede POE::Wheel::Run with a component module with the tentative title of POE::Component::Process (http://poe.perl.org/?V1.0-TODOS) and there has been a discussion on the mailing list previously about how the POE::Wheel::Run module could be improved. In spite of this, or perhaps because of it, I have made some simple additions to the POE::Wheel::Run module which I have needed for some in-house projects which I would like to get some feedback on. The first of these changes is a simple addition to include a Directory parameter to specific the location from within which the program specified is executed. The diff for these changes against the CVS version of POE::Wheel::Run (version 1.44) is as follows: [patch] Note that this creates a locally scoped directory change for execution and should not affect execution of other components and wheels. I think it's an interesting feature, but I don't use Wheel::Run enough to know how generally useful it is. I'm looking for more discussion on it before making any decisions. I also have changes for username and group specification with name rather than UID/GID, bute these changes are very minor and depends upon the getpwnam and getgrnam functions which may affect portability of the change. Comments and feedback welcomed. I like the sounds of the proposed ideas and can certainly see instances where it would be desired, though unfortunately none of them pertain to my own app as we have a dedicated server with strict access rules really prevents us from needing a pseudo "chrooted" environment (as that is the best I can think to call it). Although it might make maintenance a nightmare, and lead to code duplication, etc. but what about implementing a POE::Wheel::RunSecure or SecureRun, etc. which works in essentially the same way but with the above specifications. Seems like backwards compatibility would be taken care of, and any future implementations of a completely different module could incorporate all of the functionality and merge the two modules back into a single all powerful RUN module (for lack of a true/better name) :-). Just some thoughts from a relative POE newbie. http://danconia.org
Re: POE::Wheel::Run ErrorEvent
Rocco Caputo wrote:
On Wed, Feb 12, 2003 at 08:38:25AM +1100, [EMAIL PROTECTED] wrote:
It's a valid interpretation of ErrorEvent, but it's not the way
POE::Wheel::Run was written.
StdoutEvent, StderrEvent, and ErrorEvent refer to the pipes that
attach the parent and child processes. The first two provide child
output that is received on those pipes, and the third lets you know if
there are errors reading from them.
A child program's return value comes to you differently: through
SIGCHLD. If it's significant, you'll need to set up a SIGCHLD handler
and match the returned process ID against the process ID of your
active wheel.
sub _start_handler {
my $kernel = $_[KERNEL];
$kernel->sig( CHLD => "sig_chld" );
my $heap = $_[HEAP];
my $wheel = POE::Wheel::Run->new( ... );
# Map PID to wheel reference here. Useful for managing multiple
# child processes in the same session.
$heap->{wheels}{$wheel->PID} = $wheel;
}
# ARG1 contains the PID of the ended child process.
# ARG2 contains the $? (perlvar) associated with this CHLD signal.
sub sig_chld_handler {
my ($pid, $return_value) = @_[ARG1, ARG2];
my $wheel = delete $_[HEAP]->{wheels}{$pid};
if (defined $wheel) {
print "Process $pid returned $return_value.\n";
}
}
Should the above be added to the POE::Wheel::Run docs or on the cookbook
page possibly? I think this is a good example of what should be done,
and was also confused by how the docs are worded. It also appeared to
me that the "CloseEvent" was *the* way to find out that the process was
complete, when clearly it is really just when STDOUT is closed, granted
this should happen when the process is over. I haven't had a chance to
work the above into my app yet but it definitely closes one question I
had lingering
http://danconia.org
Re: staying alive with an alias
Still in my POE early days, but see if I understand this correctly
Richard Liu wrote:
I need to be able to start a child POE process that stays alive even
when it has no events to keep it alive. The POE::Kernel man page
indicates that if I set an alias for the child, it will demonise itself
and stay alive. However when I try this with the following code ,
process with ID =5 still dies.
Why is this so?
You are only setting one session alias, so while that session *would*
stay open if it were possible further events could be directed at it,
there is nothing to direct events at it since no other session is still
alive, so it knows implicitly that it will never receive any work, and
can quit. How about something like, if session ID is 5 or 6...then see
if they continue, (though deadlock can seemingly occur).
http://danconia.org
if ($session->ID == 5 ) {
print "Settin alias for ", $session->ID, "\n";
$kernel->alias_set('five');
}
