Hi,
I'm facing an error under the client-server mode. As soon as I try to open a
file, paraview crashes with this error on the
server side :
ERROR: In /home/podallaire/ParaView/Servers/Common/vtkProcessModule.cxx, line
1065
vtkProcessModule (0x851da0): Cannot create object of type
"vtk1DTransferFunctionFilter".
while processing
Message 0 = New
Argument 0 = string_value {vtk1DTransferFunctionFilter}
Argument 1 = id_value {387}
ERROR: In /home/podallaire/ParaView/Servers/Common/vtkProcessModule.cxx, line
1066
vtkProcessModule (0x851da0): Aborting execution for debugging purposes.
....
Any idea what is going on ? I tried different files - same result.
Regards,
PO
On 2010-04-30, at 2:49 PM, burlen wrote:
> yeah, that's a typo in the bug report. Both last night and this morning I
> have been unable to access mantis or the the pv web site. Page never loads.
> I'll try later.
>
> pat marion wrote:
>> Hey Burlen, on the bug report page for 10283, I think you need to fix the
>> command line you are testing with :
>>
>> $ ssh remote cmd1 && cm2
>>
>> will execute cmd1 on remote and cmd2 locally. It should be:
>>
>> $ ssh remote "cmd1 && cmd2"
>>
>> Pat
>>
>> On Fri, Apr 30, 2010 at 9:12 AM, pat marion <pat.mar...@kitware.com
>> <mailto:pat.mar...@kitware.com>> wrote:
>>
>> I have applied your patch. I agree that paraview should explicity
>> close the child process. But... what I am pointing out is that
>> calling QProcess::close() does not help in this situation. What I
>> am saying is that, even when paraview does kill the process, any
>> commands run by ssh on the other side of the netpipe will be
>> orphaned by sshd. Are you sure you can't reproduce it?
>>
>>
>> $ ssh localhost sleep 1d
>> $ < press control-c >
>> $ pidof sleep
>> $ # sleep is still running
>>
>> Pat
>>
>>
>> On Fri, Apr 30, 2010 at 2:08 AM, burlen <burlen.lor...@gmail.com
>> <mailto:burlen.lor...@gmail.com>> wrote:
>>
>> Hi Pat,
>>
>> From my point of view the issue is philosophical, because
>> practically speaking I couldn't reproduce the orphans with out
>> doing something a little odd namely, ssh ... && sleep 1d.
>> Although the fact that a user reported suggests that it may
>> occur in the real world as well. The question is this: should
>> an application explicitly clean up resources it allocates? or
>> should an application rely on the user not only knowing that
>> there is the potential for a resource leak but also knowing
>> enough to do the right thing to avoid it (eg ssh -tt ...)? In
>> my opinion, as a matter of principle, if PV spawns a process
>> it should explicitly clean it up and there should be no way it
>> can become an orphan. In this case the fact that the orphan
>> can hold ports open is particularly insidious, because further
>> connection attempt on that port fails with no helpful error
>> information. Also it is not very difficult to clean up a
>> spawned process. What it comes down to is a little book
>> keeping to hang on to the qprocess handle and a few lines of
>> code called from pqCommandServerStartup destructor to make
>> certain it's cleaned up. This is from the patch I submitted
>> when I filed the bug report.
>>
>> + // close running process
>> + if (this->Process->state()==QProcess::Running)
>> + {
>> + this->Process->close();
>> + }
>> + // free the object
>> + delete this->Process;
>> + this->Process=NULL;
>>
>> I think if the cluster admins out there new which ssh options
>> (GatewayPorts etc) are important for ParView to work
>> seamlessly, then they might be willing to open them up. It's
>> my impression that the folks that build clusters want tools
>> like PV to be easy to use, but they don't necessarily know all
>> the in's and out's of confinguring and running PV.
>>
>> Thanks for looking at this again! The -tt option to ssh is
>> indeed a good find.
>>
>> Burlen
>>
>> pat marion wrote:
>>
>> Hi all!
>>
>> I'm bringing this thread back- I have learned a couple new
>> things...
>>
>> -----------------------
>> No more orphans:
>>
>> Here is an easy way to create an orphan:
>>
>> $ ssh localhost sleep 1d
>> $ <press control c>
>>
>> The ssh process is cleaned up, but sshd orphans the sleep
>> process. You can avoid this by adding '-t' to ssh:
>>
>> $ ssh -t localhost sleep 1d
>>
>> Works like a charm! But then there is another problem...
>> try this command from paraview (using QProcess) and it
>> still leaves an orphan, doh! Go back and re-read ssh's
>> man page and you have the solution, use '-t' twice: ssh -tt
>>
>> -------------------------
>> GatewayPorts and portfwd workaround:
>>
>> In this scenario we have 3 machines: workstation,
>> service-node, and compute-node. I want to ssh from
>> workstation to service-node and submit a job that will run
>> pvserver on compute-node. When pvserver starts on
>> compute-node I want it to reverse connect to service-node
>> and I want service-node to forward the connection to
>> workstation. So here I go:
>>
>> $ ssh -R11111:localhost:11111 service-node qsub
>> start_pvserver.sh
>>
>> Oops, the qsub command returns immediately and closes my
>> ssh tunnel. Let's pretend that the scheduler doesn't
>> provide an easy way to keep the command alive, so I have
>> resorted to using 'sleep 1d'. So here I go, using -tt to
>> prevent orphans:
>>
>> $ ssh -tt -R11111:localhost:11111 service-node "qsub
>> start_pvserver.sh && sleep 1d"
>>
>> Well, this will only work if GatewayPorts is enabled in
>> sshd_config on service-node. If GatewayPorts is not
>> enabled, the ssh tunnel will only accept connections from
>> localhost, it will not accept a connection from
>> compute-node. We can ask the sysadmin to enable
>> GatewayPorts, or we could use portfwd. You can run
>> portfwd on service-node to forward port 22222 to port
>> 11111, then have compute-node connect to
>> service-node:22222. So your job script would launch
>> pvserver like this:
>>
>> pvserver -rc -ch=service-node -sp=22222
>>
>> Problem solved! Also convenient, we can use portfwd to
>> replace 'sleep 1d'. So the final command, executed by
>> paraview client:
>>
>> ssh -tt -R 11111:localhost:11111 service-node "qsub
>> start_pvserver.sh && portfwd -g -c fwd.cfg"
>>
>> Where fwd.cfg contains:
>>
>> tcp { 22222 { => localhost:11111 } }
>>
>>
>> Hope this helps!
>>
>> Pat
>>
>> On Fri, Feb 12, 2010 at 7:06 PM, burlen
>> <burlen.lor...@gmail.com <mailto:burlen.lor...@gmail.com>
>> <mailto:burlen.lor...@gmail.com
>> <mailto:burlen.lor...@gmail.com>>> wrote:
>>
>>
>> Incidentally, this brings up an interesting point about
>> ParaView with client/server. It doesn't try to
>> clean up it's
>> child processes, AFAIK. For example, if you set up
>> this ssh
>> tunnel inside the ParaView GUI (e.g., using a
>> command instead
>> of a manual connection), and you cancel the
>> connection, it
>> will leave the ssh running. You have to track down
>> the ssh
>> process and kill it yourself. It's minor thing,
>> but it can
>> also prevent future connections if you don't
>> realize there's a
>> zombie ssh that kept your ports open.
>>
>> I attempted to reproduce on my kubuntu 9.10, qt 4.5.2
>> system, with
>> slightly different results, which may be qt/distro/os
>> specific.
>>
>> On my system as long as the process ParaView spawns
>> finishes on
>> its own there is no problem. That's usually how one
>> would expect
>> things to work out since when the client disconnects
>> the server
>> closes followed by ssh. But, you are right that PV never
>> explicitly kills or otherwise cleans up after the
>> process it
>> starts. So if the spawned process for some reason
>> doesn't finish
>> orphan processes are introduced.
>>
>> I was able to produce orphan ssh processes, giving the
>> PV client a
>> server start up command that doesn't finish. eg
>>
>> ssh ... pvserver ... && sleep 100d
>>
>> I get the situation you described which prevents further
>> connection on the same ports. Once PV tries and fails
>> to connect
>> on th eopen ports, there is crash soon after.
>>
>> I filed a bug report with a patch:
>> http://www.paraview.org/Bug/view.php?id=10283
>>
>>
>>
>> Sean Ziegeler wrote:
>>
>> Most batch systems have an option to wait until the
>> job is
>> finished before the submit command returns. I know
>> PBS uses
>> "-W block=true" and that SGE and LSF have similar
>> options (but
>> I don't recall the precise flags).
>>
>> If your batch system doesn't provide that, I'd
>> recommend
>> adding some shell scripting to loop through
>> checking the queue
>> for job completion and not return until it's done.
>> The sleep
>> thing would work, but wouldn't exit when the server
>> finishes,
>> leaving the ssh tunnels (and other things like
>> portfwd if you
>> put them in your scripts) lying around.
>>
>> Incidentally, this brings up an interesting point about
>> ParaView with client/server. It doesn't try to
>> clean up it's
>> child processes, AFAIK. For example, if you set up
>> this ssh
>> tunnel inside the ParaView GUI (e.g., using a
>> command instead
>> of a manual connection), and you cancel the
>> connection, it
>> will leave the ssh running. You have to track down
>> the ssh
>> process and kill it yourself. It's minor thing,
>> but it can
>> also prevent future connections if you don't
>> realize there's a
>> zombie ssh that kept your ports open.
>>
>>
>> On 02/08/10 21:03, burlen wrote:
>>
>> I am curious to hear what Sean has to say.
>>
>> But, say the batch system returns right away
>> after the job
>> is submitted,
>> I think we can doctor the command so that it
>> will live for
>> a while
>> longer, what about something like this:
>>
>> ssh -R XXXX:localhost:YYYY remote_machine
>> "submit_my_job.sh && sleep
>> 100d"
>>
>>
>> pat marion wrote:
>>
>> Hey just checked out the wiki page, nice! One
>> question, wouldn't this
>> command hang up and close the tunnel after
>> submitting
>> the job?
>> ssh -R XXXX:localhost:YYYY remote_machine
>> submit_my_job.sh
>> Pat
>>
>> On Mon, Feb 8, 2010 at 8:12 PM, pat marion
>> <pat.mar...@kitware.com
>> <mailto:pat.mar...@kitware.com>
>> <mailto:pat.mar...@kitware.com
>> <mailto:pat.mar...@kitware.com>>
>> <mailto:pat.mar...@kitware.com
>> <mailto:pat.mar...@kitware.com>
>>
>> <mailto:pat.mar...@kitware.com
>> <mailto:pat.mar...@kitware.com>>>> wrote:
>>
>> Actually I didn't write the notes at the
>> hpc.mil <http://hpc.mil>
>> <http://hpc.mil> <http://hpc.mil>
>>
>> link.
>>
>> Here is something- and maybe this is the
>> problem that
>> Sean refers
>> to- in some cases, when I have set up a
>> reverse ssh
>> tunnel from
>> login node to workstation (command executed
>> from
>> workstation) then
>> the forward does not work when the compute node
>> connects to the
>> login node. However, if I have the compute node
>> connect to the
>> login node on port 33333, then use portfwd
>> to forward
>> that to
>> localhost:11111, where the ssh tunnel is
>> listening on
>> port 11111,
>> it works like a charm. The portfwd tricks
>> it into
>> thinking the
>> connection is coming from localhost and
>> allow the ssh
>> tunnel to
>> work. Hope that made a little sense...
>>
>> Pat
>>
>>
>> On Mon, Feb 8, 2010 at 6:29 PM, burlen
>> <burlen.lor...@gmail.com
>> <mailto:burlen.lor...@gmail.com>
>> <mailto:burlen.lor...@gmail.com
>> <mailto:burlen.lor...@gmail.com>>
>> <mailto:burlen.lor...@gmail.com
>> <mailto:burlen.lor...@gmail.com>
>> <mailto:burlen.lor...@gmail.com
>> <mailto:burlen.lor...@gmail.com>>>> wrote:
>>
>> Nice, thanks for the clarification. I am
>> guessing that
>> your
>> example should probably be the recommended
>> approach rather
>> than the portfwd method suggested on the PV
>> wiki. :) I
>> took
>> the initiative to add it to the Wiki. KW
>> let me know
>> if this
>> is not the case!
>>
>>
>> http://paraview.org/Wiki/Reverse_connection_and_port_forwarding#Reverse_connection_over_an_ssh_tunnel
>>
>>
>>
>> Would you mind taking a look to be sure I
>> didn't miss
>> anything
>> or bollix it up?
>>
>> The sshd config options you mentioned may
>> be why your
>> method
>> doesn't work on the Pleiades system, either
>> that or
>> there is a
>> firewall between the front ends and compute
>> nodes. In
>> either
>> case I doubt the NAS sys admins are going to
>> reconfigure for
>> me :) So at least for now I'm stuck with
>> the two hop ssh
>> tunnels and interactive batch jobs. if
>> there were
>> someway to
>> script the ssh tunnel in my batch script I
>> would be
>> golden...
>>
>> By the way I put the details of the two hop
>> ssh tunnel
>> on the
>> wiki as well, and a link to Pat's hpc.mil
>> <http://hpc.mil>
>> <http://hpc.mil> <http://hpc.mil>
>>
>> notes. I don't dare try to summarize them
>> since I've never
>> used portfwd and it refuses to compile both
>> on my
>> workstation
>> and the cluster.
>>
>> Hopefully putting these notes on the Wiki
>> will save future
>> ParaView users some time and headaches.
>>
>>
>> Sean Ziegeler wrote:
>>
>> Not quite- the pvsc calls ssh with both the
>> tunnel options
>> and the commands to submit the batch job.
>> You don't even
>> need a pvsc; it just makes the interface
>> fancier. As long
>> as you or PV executes something like this
>> from your
>> machine:
>> ssh -R XXXX:localhost:YYYY remote_machine
>> submit_my_job.sh
>>
>> This means that port XXXX on remote_machine
>> will be the
>> port to which the server must connect. Port
>> YYYY (e.g.,
>> 11111) on your client machine is the one on
>> which PV
>> listens. You'd have to tell the server (in
>> the batch
>> submission script, for example) the name of
>> the node and
>> port XXXX to which to connect.
>>
>> One caveat that might be causing you
>> problems, port
>> forwarding (and "gateway ports" if the
>> server is running
>> on a different node than the login node)
>> must be enabled
>> in the remote_machine's sshd_config. If
>> not, no ssh
>> tunnels will work at all (see: man ssh and man
>> sshd_config). That's something that an
>> administrator
>> would need to set up for you.
>>
>> On 02/08/10 12:26, burlen wrote:
>>
>> So to be sure about what you're saying:
>> Your .pvsc
>> script ssh's to the
>> front end and submits a batch job which
>> when it's
>> scheduled , your batch
>> script creates a -R style tunnel and starts
>> pvserver
>> using PV reverse
>> connection. ? or are you using portfwd or a
>> second ssh
>> session to
>> establish the tunnel ?
>>
>> If you're doing this all from your .pvsc script
>> without a second ssh
>> session and/or portfwd that's awesome! I
>> haven't been
>> able to script
>> this, something about the batch system
>> prevents the
>> tunnel created
>> within the batch job's ssh session from
>> working. I
>> don't know if that's
>> particular to this system or a general fact
>> of life
>> about batch systems.
>>
>> Question: How are you creating the tunnel
>> in your
>> batch script?
>>
>> Sean Ziegeler wrote:
>>
>> Both ways will work for me in most cases,
>> i.e. a
>> "forward" connection
>> with ssh -L or a reverse connection with
>> ssh -R.
>>
>> However, I find that the reverse method is more
>> scriptable. You can
>> set up a .pvsc file that the client can
>> load and
>> will call ssh with
>> the appropriate options and commands for the
>> remote host, all from the
>> GUI. The client will simply wait for the
>> reverse
>> connection from the
>> server, whether it takes 5 seconds or 5
>> hours for
>> the server to get
>> through the batch queue.
>>
>> Using the forward connection method, if the
>> server
>> isn't started soon
>> enough, the client will attempt to connect and
>> then fail. I've always
>> had to log in separately, wait for the
>> server to
>> start running, then
>> tell my client to connect.
>>
>> -Sean
>>
>> On 02/06/10 12:58, burlen wrote:
>>
>> Hi Pat,
>>
>> My bad. I was looking at the PV wiki, and
>> thought you were talking about
>> doing this without an ssh tunnel and using
>> only port forward and
>> paraview's --reverse-connection option . Now
>> that I am reading your
>> hpc.mil <http://hpc.mil> <http://hpc.mil>
>> <http://hpc.mil> post I see
>>
>> what you
>> mean :)
>>
>> Burlen
>>
>>
>> pat marion wrote:
>>
>> Maybe I'm misunderstanding what you mean
>> by local firewall, but
>> usually as long as you can ssh from your
>> workstation to the login node
>> you can use a reverse ssh tunnel.
>>
>>
>> _______________________________________________
>> Powered by www.kitware.com
>> <http://www.kitware.com> <http://www.kitware.com>
>> <http://www.kitware.com>
>>
>> Visit other Kitware open-source projects at
>>
>> http://www.kitware.com/opensource/opensource.html
>>
>> Please keep messages on-topic and check the
>> ParaView Wiki at:
>> http://paraview.org/Wiki/ParaView
>>
>> Follow this link to subscribe/unsubscribe:
>>
>> http://www.paraview.org/mailman/listinfo/paraview
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the ParaView Wiki at:
> http://paraview.org/Wiki/ParaView
>
> Follow this link to subscribe/unsubscribe:
> http://www.paraview.org/mailman/listinfo/paraview
_______________________________________________
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Please keep messages on-topic and check the ParaView Wiki at:
http://paraview.org/Wiki/ParaView
Follow this link to subscribe/unsubscribe:
http://www.paraview.org/mailman/listinfo/paraview
