Burlen, I think I know what's causing the segfaults. I'll have a fix soon. Thanks for catching (and reporting) this :).
Utkarsh On Mon, Jun 3, 2013 at 8:05 PM, Burlen Loring <blor...@lbl.gov> wrote: > another data point on this, a similar crash occurs seemingly randomly (see > output below). I'm seeing this crash or a variant of it on pretty much all > the systems I'm using. Sometimes I can run for 5min and other a few hours, > but I'm crashing alot always with: Tag mismatch: got 1, expecting 22222. > > > ERROR: In > /home/bloring/builds/ParaView/VTK/Parallel/Core/vtkSocketCommunicator.cxx, > line 878 > vtkSocketCommunicator (0xf2d270): Tag mismatch: got 1, expecting 22222. > > pvserver: > /home/bloring/builds/ParaView/VTK/Parallel/Core/vtkMultiProcessStream.cxx:637: > vtkMultiProcessStream& vtkMultiProcessStream::operator>>(unsigned int&): > Assertion `this->Internals->Data.front() == vtkInternals::uint32_value' > failed. > Generic Warning: In > /home/bloring/builds/ParaView/VTK/Parallel/MPI/vtkMPICommunicator.cxx, line > 99 > > MPI had an error > ------------------------------------------------ > > A process has failed, error stack: > PMPI_Barrier(426)...........: MPI_Barrier(MPI_COMM_WORLD) failed > MPIR_Barrier_impl(294)......: > MPIR_Barrier_or_coll_fn(121): > MPIR_Barrier_intra(83)......: > dequeue_and_set_error(888)..: Communication error with rank 0 > MPIR_Barrier_impl(308)......: > MPIR_Bcast_impl(1369).......: > MPIR_Bcast_intra(1199)......: > MPIR_Bcast_binomial(220)....: Failure during collective > > Program Stack: > ------------------------------------------------ > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkParallelMPI-pv4.0.so.1 > [0x2b773f94aea7] > /home/bloring/installs/mpich/3.0.4/lib/libmpich.so.10(MPIR_Err_return_comm+0x1bd) > [0x2b77418465bd] > /home/bloring/installs/mpich/3.0.4/lib/libmpich.so.10(MPI_Barrier+0xda) > [0x2b77418bd7ca] > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkParallelMPI-pv4.0.so.1 > [0x2b773f94ac95] > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkParallelMPI-pv4.0.so.1(_ZN18vtkMPICommunicator18BroadcastVoidArrayEPvxii+0x2f) > [0x2b773f94e3ab] > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkParallelCore-pv4.0.so.1 > [0x2b773fb9b4bc] > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkParallelCore-pv4.0.so.1(_ZN15vtkCommunicator9BroadcastER21vtkMultiProcessStreami+0x110) > [0x2b773fb9a732] > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkPVClientServerCoreRendering-pv4.0.so.1 > [0x2b774d5c7901] > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkPVClientServerCoreRendering-pv4.0.so.1(_ZN30vtkPVSynchronizedRenderWindows20SatelliteStartRenderEP15vtkRenderWindow+0x46) > [0x2b774d60938a] > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkPVClientServerCoreRendering-pv4.0.so.1(_ZN30vtkPVSynchronizedRenderWindows17HandleStartRenderEP15vtkRenderWindow+0xc0) > [0x2b774d608f36] > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkPVClientServerCoreRendering-pv4.0.so.1 > [0x2b774d60db58] > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkCommonCore-pv4.0.so.1 > [0x2b77431ba183] > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkCommonCore-pv4.0.so.1(_ZN9vtkObject11InvokeEventEmPv+0x3d) > [0x2b77431ba693] > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkRenderingCore-pv4.0.so.1(_ZN15vtkRenderWindow6RenderEv+0x1e7) > [0x2b773b696c2d] > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkRenderingCore-pv4.0.so.1(_ZN19vtkHardwareSelector14CaptureBuffersEv+0x473) > [0x2b773b5fb60b] > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkPVVTKExtensionsRendering-pv4.0.so.1(_ZN21vtkPVHardwareSelector13PrepareSelectEv+0xef) > [0x2b774e61ce93] > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkPVVTKExtensionsRendering-pv4.0.so.1(_ZN21vtkPVHardwareSelector6SelectEPi+0x1c) > [0x2b774e61cee8] > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkPVClientServerCoreRendering-pv4.0.so.1(_ZN15vtkPVRenderView6SelectEiPi+0x9f) > [0x2b774d5f714b] > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkPVClientServerCoreRendering-pv4.0.so.1(_ZN15vtkPVRenderView11SelectCellsEPi+0x25) > [0x2b774d5f6e99] > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkPVServerManagerApplication-pv4.0.so.1(_ZN15vtkPVRenderView11SelectCellsEiiii+0x44) > [0x2b7739ab6736] > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkPVServerManagerApplication-pv4.0.so.1(_Z22vtkPVRenderViewCommandP26vtkClientServerInterpreterP13vtkObjectBasePKcRK21vtkClientServerStreamRS5_+0x22dc) > [0x2b7739ab214f] > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkClientServer-pv4.0.so.1(_ZN26vtkClientServerInterpreter20ProcessCommandInvokeERK21vtkClientServerStreami+0x18e) > [0x2b773ec67564] > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkClientServer-pv4.0.so.1(_ZN26vtkClientServerInterpreter17ProcessOneMessageERK21vtkClientServerStreami+0x135) > [0x2b773ec66c9d] > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkClientServer-pv4.0.so.1(_ZN26vtkClientServerInterpreter13ProcessStreamERK21vtkClientServerStream+0x2f) > [0x2b773ec66b37] > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkPVServerImplementationCore-pv4.0.so.1(_ZN16vtkPVSessionCore21ExecuteStreamInternalERK21vtkClientServerStreamb+0xdb) > [0x2b773a91b7bf] > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkPVServerImplementationCore-pv4.0.so.1(_ZN16vtkPVSessionCore30ExecuteStreamSatelliteCallbackEv+0xd5) > [0x2b773a91b6a3] > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkPVServerImplementationCore-pv4.0.so.1 > [0x2b773a91938f] > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkParallelCore-pv4.0.so.1(_ZN25vtkMultiProcessController10ProcessRMIEiPvii+0x3a6) > [0x2b773fba72f2] > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkParallelCore-pv4.0.so.1(_ZN25vtkMultiProcessController11ProcessRMIsEii+0x63c) > [0x2b773fba6e88] > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkParallelCore-pv4.0.so.1(_ZN25vtkMultiProcessController11ProcessRMIsEv+0x22) > [0x2b773fba684a] > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/pvserver [0x401d30] > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/pvserver(main+0x25) > [0x401dca] > /lib64/libc.so.6(__libc_start_main+0xf4) [0x3fab01d994] > /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/pvserver [0x401a09] > > > > > > On 05/31/2013 03:15 PM, Burlen Loring wrote: >> >> can also reproduce with the point source (open data, change to surface >> rep, add point source, pick) >> vtk dataset that can be used to reproduce: >> http://hpcvis.com/downloads/pr1-vti.tar.gz >> >> On 05/31/2013 01:54 PM, Burlen Loring wrote: >>> >>> Hi All, >>> >>> With git master (first noticed a few days ago, rebased this am) I'm >>> hitting crash when attempting to use the stream tracer interactive picking a >>> seed. >>> >>> 1) load the data, >>> 2) change to surface rep, >>> 3) add stream tracer filter >>> 4) hit the 'p' key to select a seed -> crash. >>> >>> I've only been able to reproduce with a real dataset (1024^3 image mesh). >>> the small data I've tried from ParaViewData doesn't lead to the crash, >>> However, there's plenty of ram (first encountered using 32 ranks 4 ranks per >>> node 48g per node on longhorn reduced to 4 ranks 24g on my workstation for >>> debugging). I can share this dataset if it helps to debug, or you may have >>> to use a large dataset to trigger. >>> >>> filed a bug report: http://paraview.org/Bug/view.php?id=14107 >>> >>> here's cerr output and a stack >>> >>> 3437 ERROR: In >>> /scratch/01237/bloring/ParaView/next/ParaView/VTK/Parallel/Core/vtkSocketCommunicator.cxx, >>> line 877 >>> 3438 vtkSocketCommunicator (0x2efd2d0): Tag mismatch: got 1, expecting >>> 22222. >>> 3439 >>> 3440 pvserver: >>> /scratch/01237/bloring/ParaView/next/ParaView/VTK/Parallel/Core/vtkMultiProcessStream.cxx:637: >>> vtkMultiProcessStream &vtkMultiProcessStream::operator>>(unsigned int &): >>> Assertion `this->Internals->Data.front() == vtkInter nals::uint32_value' >>> failed. >>> >>> #0 0x00000033aea30215 in raise () from /lib64/libc.so.6 >>> #1 0x00000033aea31cc0 in abort () from /lib64/libc.so.6 >>> #2 0x00000033aea29696 in __assert_fail () from /lib64/libc.so.6 >>> #3 0x00002b6ae159c3ed in vtkMultiProcessStream::operator>> >>> (this=0x7fffea94ed68, value=@0x7fffea94ed1c) >>> at >>> /scratch/01237/bloring/ParaView/next/ParaView/VTK/Parallel/Core/vtkMultiProcessStream.cxx:637 >>> #4 0x00002b6aef371cdd in >>> vtkPVSynchronizedRenderWindows::LoadWindowAndLayout (this=0x2835d50, >>> window=0x26e9650, stream=@0x7fffea94ed68) >>> at >>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ClientServerCore/Rendering/vtkPVSynchronizedRenderWindows.cxx:998 >>> #5 0x00002b6aef37155b in vtkPVSynchronizedRenderWindows::RootStartRender >>> (this=0x2835d50, renWin=0x26e9650) >>> at >>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ClientServerCore/Rendering/vtkPVSynchronizedRenderWindows.cxx:901 >>> #6 0x00002b6aef3711aa in >>> vtkPVSynchronizedRenderWindows::HandleStartRender (this=0x2835d50, >>> renWin=0x26e9650) >>> at >>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ClientServerCore/Rendering/vtkPVSynchronizedRenderWindows.cxx:814 >>> #7 0x00002b6aef36dc5a in >>> vtkPVSynchronizedRenderWindows::vtkObserver::Execute (this=0x282f660, >>> ocaller=0x26e9650, eventId=3) >>> at >>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ClientServerCore/Rendering/vtkPVSynchronizedRenderWindows.cxx:219 >>> #8 0x00002b6ae374c9ff in vtkSubjectHelper::InvokeEvent (this=0x282e970, >>> event=3, callData=0x0, self=0x26e9650) at >>> /scratch/01237/bloring/ParaView/next/ParaView/VTK/Common/Core/vtkObject.cxx:620 >>> #9 0x00002b6ae374d028 in vtkObject::InvokeEvent (this=0x26e9650, >>> event=3, callData=0x0) at >>> /scratch/01237/bloring/ParaView/next/ParaView/VTK/Common/Core/vtkObject.cxx:789 >>> #10 0x00002b6adc9ee77a in vtkRenderWindow::Render (this=0x26e9650) at >>> /scratch/01237/bloring/ParaView/next/ParaView/VTK/Rendering/Core/vtkRenderWindow.cxx:313 >>> #11 0x00002b6af42443f7 in vtkXOpenGLRenderWindow::Render (this=0x26e9650) >>> at >>> /scratch/01237/bloring/ParaView/next/ParaView/VTK/Rendering/OpenGL/vtkXOpenGLRenderWindow.cxx:1826 >>> #12 0x00002b6adc9183d6 in vtkHardwareSelector::CaptureBuffers >>> (this=0x26e9460) at >>> /scratch/01237/bloring/ParaView/next/ParaView/VTK/Rendering/Core/vtkHardwareSelector.cxx:288 >>> #13 0x00002b6af05119ef in vtkPVHardwareSelector::PrepareSelect >>> (this=0x26e9460) at >>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/VTKExtensions/Rendering/vtkPVHardwareSelector.cxx:62 >>> #14 0x00002b6af0511a54 in vtkPVHardwareSelector::Select (this=0x26e9460, >>> region=0x7fffea94f750) >>> at >>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/VTKExtensions/Rendering/vtkPVHardwareSelector.cxx:75 >>> #15 0x00002b6aef353e31 in vtkPVRenderView::Select (this=0x2833ce0, >>> fieldAssociation=1, region=0x7fffea94f750) >>> at >>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ClientServerCore/Rendering/vtkPVRenderView.cxx:597 >>> #16 0x00002b6aef353a19 in vtkPVRenderView::SelectCells (this=0x2833ce0, >>> region=0x7fffea94f750) >>> at >>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ClientServerCore/Rendering/vtkPVRenderView.cxx:556 >>> #17 0x00002b6ada8b1064 in vtkPVRenderView::SelectCells (this=0x2833ce0, >>> region0=218, region1=142, region2=218, region3=142) >>> at >>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ClientServerCore/Rendering/vtkPVRenderView.h:246 >>> #18 0x00002b6ada8ab300 in vtkPVRenderViewCommand (arlu=0x227b4f0, >>> ob=0x2833ce0, method=0x3dfbd19 "SelectCells", msg=@0x7fffea951538, >>> resultStream=@0x227bdf0) >>> at >>> /scratch/01237/bloring/ParaView/next/PV/Wrapping/ClientServer/vtkPVRenderViewClientServer.cxx:545 >>> #19 0x00002b6ae04b896e in >>> vtkClientServerInterpreter::ProcessCommandInvoke (this=0x227b4f0, >>> css=@0x7fffea951b90, midx=0) >>> at >>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ClientServerStream/vtkClientServerInterpreter.cxx:379 >>> #20 0x00002b6ae04b7d5b in vtkClientServerInterpreter::ProcessOneMessage >>> (this=0x227b4f0, css=@0x7fffea951b90, message=0) >>> at >>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ClientServerStream/vtkClientServerInterpreter.cxx:214 >>> #21 0x00002b6ae04b7c0e in vtkClientServerInterpreter::ProcessStream >>> (this=0x227b4f0, css=@0x7fffea951b90) >>> at >>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ClientServerStream/vtkClientServerInterpreter.cxx:183 >>> #22 0x00002b6adba173cd in vtkPVSessionCore::ExecuteStreamInternal >>> (this=0x227b990, stream=@0x7fffea951b90, ignore_errors=false) >>> at >>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ServerImplementation/Core/vtkPVSessionCore.cxx:665 >>> #23 0x00002b6adba1711e in vtkPVSessionCore::ExecuteStream >>> (this=0x227b990, location=21, stream=@0x7fffea951b90, ignore_errors=false) >>> at >>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ServerImplementation/Core/vtkPVSessionCore.cxx:635 >>> ---Type <return> to continue, or q <return> to quit--- >>> #24 0x00002b6adba11dde in vtkPVSessionBase::ExecuteStream >>> (this=0x1a6ae50, location=21, stream=@0x7fffea951b90, ignore_errors=false) >>> at >>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ServerImplementation/Core/vtkPVSessionBase.cxx:173 >>> #25 0x00002b6adba284b4 in vtkPVSessionServer::OnClientServerMessageRMI >>> (this=0x1a6ae50, message=0x3cd3d00, message_length=16) >>> at >>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ServerImplementation/Core/vtkPVSessionServer.cxx:510 >>> #26 0x00002b6adba25d2e in (anonymous namespace)::RMICallback >>> (localArg=0x1a6ae50, remoteArg=0x3cd3d00, remoteArgLength=16) >>> at >>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ServerImplementation/Core/vtkPVSessionServer.cxx:56 >>> #27 0x00002b6ae158abcc in vtkMultiProcessController::ProcessRMI >>> (this=0x2efd1a0, remoteProcessId=1, arg=0x3cd3d00, argLength=16, >>> rmiTag=55625) >>> at >>> /scratch/01237/bloring/ParaView/next/ParaView/VTK/Parallel/Core/vtkMultiProcessController.cxx:650 >>> #28 0x00002b6ae158a48f in vtkMultiProcessController::ProcessRMIs >>> (this=0x2efd1a0, reportErrors=0, dont_loop=1) >>> at >>> /scratch/01237/bloring/ParaView/next/ParaView/VTK/Parallel/Core/vtkMultiProcessController.cxx:596 >>> #29 0x00002b6adbec2d07 in >>> vtkTCPNetworkAccessManager::ProcessEventsInternal (this=0x174f090, >>> timeout_msecs=0, do_processing=true) >>> at >>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ClientServerCore/Core/vtkTCPNetworkAccessManager.cxx:268 >>> #30 0x00002b6adbec25d0 in vtkTCPNetworkAccessManager::ProcessEvents >>> (this=0x174f090, timeout_msecs=0) >>> at >>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ClientServerCore/Core/vtkTCPNetworkAccessManager.cxx:165 >>> #31 0x00000000004035e0 in RealMain (argc=5, argv=0x7fffea953418, >>> type=vtkProcessModule::PROCESS_SERVER) at >>> /scratch/01237/bloring/ParaView/next/ParaView/CommandLineExecutables/pvserver_common.h:90 >>> #32 0x000000000040305d in main (argc=5, argv=0x7fffea953418) at >>> /scratch/01237/bloring/ParaView/next/ParaView/CommandLineExecutables/pvserver.cxx:21 >>> >> > > _______________________________________________ > Powered by www.kitware.com > > Visit other Kitware open-source projects at > http://www.kitware.com/opensource/opensource.html > > Please keep messages on-topic and check the ParaView Wiki at: > http://paraview.org/Wiki/ParaView > > Follow this link to subscribe/unsubscribe: > http://www.paraview.org/mailman/listinfo/paraview _______________________________________________ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the ParaView Wiki at: http://paraview.org/Wiki/ParaView Follow this link to subscribe/unsubscribe: http://www.paraview.org/mailman/listinfo/paraview
