Re: [Paraview] Best file fomat for big data
I would recommend using xdmf (www.xdmf.org). It works well for large parallel datasets, and supports the same structures as vtk.ChrisOn 30 Nov 2017 10:13, a...@yandex.ru wrote:Hi! I just finish my own CFD simulation on 256 processors. The program made graphics output in 2000 time points. So large number of outputs I need to prepare smooth animation. I use XML VTK ascii format. So I have now 2000 pvtu files and 2000*256=512000 vtu files. I have problems with such big number of files. Now I understand that may be it was not the best idea to use VTK format. I have the one fixed unstructured grid with ~10^7 cells, about ten CFD variables on the cells and several thousands output points. My program is MPI parallel and output time points are irregular over time. So I'd like to keep value of time in the files (as I know it is impossible in the VTK format). And I'd like to use ParaView for the visualization. What is the better format for storing big amount of CFD data that I be able to write in parallel and load in the ParaView? Thank in advance! ___ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the ParaView Wiki at: http://paraview.org/Wiki/ParaView Search the list archives at: http://markmail.org/search/?q=ParaView Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/paraview
Re: [Paraview] Best file fomat for big data
A few months back there was a suggestion on the mailing list that you can apparently write a single vtk geometry and then use something like "map data" to load the new, updated fields (perhaps Ken posted it? - don't know any other details). I agree that these vtk format limitations can be quite frustrating and cause quite a bit of data bloat on the disk. For these situations, I would normally resort to using EnSight format. This gives you full separation of geometry and data, with moving or non-moving geometries etc. EnSight does support a server-of-servers data format, which makes it possible to a sub-case for each process and then load them together, but I've never used it. Instead we have the master process write a single file and stream the geometry and fields information across. The additional overhead of moving data across the network is offset by the fact that the filer only has a single process writing to it. If it helps, you can take a look at the foamToEnsight utility for a quick overview: https://develop.openfoam.com The harder bits are located under src/conversion/ensight/mesh/, src/conversion/ensight/output/, src/conversion/ensight/part/ If you find another possibility, please let us know since it will be of interest for other people too. Cheers, /mark -- Dr Mark OLESEN Principal Engineer, ESI-OpenCFD ESI GmbH | Einsteinring 24 | 85609 Munich | GERMANY Mob. +49 171 9710 149 www.openfoam.com | www.esi-group.com | mark.ole...@esi-group.com On 11/30/17 11:13, a...@yandex.ru wrote: Hi! I just finish my own CFD simulation on 256 processors. The program made graphics output in 2000 time points. So large number of outputs I need to prepare smooth animation. I use XML VTK ascii format. So I have now 2000 pvtu files and 2000*256=512000 vtu files. I have problems with such big number of files. Now I understand that may be it was not the best idea to use VTK format. I have the one fixed unstructured grid with ~10^7 cells, about ten CFD variables on the cells and several thousands output points. My program is MPI parallel and output time points are irregular over time. So I'd like to keep value of time in the files (as I know it is impossible in the VTK format). And I'd like to use ParaView for the visualization. What is the better format for storing big amount of CFD data that I be able to write in parallel and load in the ParaView? Thank in advance! ___ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the ParaView Wiki at: http://paraview.org/Wiki/ParaView Search the list archives at: http://markmail.org/search/?q=ParaView Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/paraview
[Paraview] Best file fomat for big data
Hi! I just finish my own CFD simulation on 256 processors. The program made graphics output in 2000 time points. So large number of outputs I need to prepare smooth animation. I use XML VTK ascii format. So I have now 2000 pvtu files and 2000*256=512000 vtu files. I have problems with such big number of files. Now I understand that may be it was not the best idea to use VTK format. I have the one fixed unstructured grid with ~10^7 cells, about ten CFD variables on the cells and several thousands output points. My program is MPI parallel and output time points are irregular over time. So I'd like to keep value of time in the files (as I know it is impossible in the VTK format). And I'd like to use ParaView for the visualization. What is the better format for storing big amount of CFD data that I be able to write in parallel and load in the ParaView? Thank in advance! ___ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the ParaView Wiki at: http://paraview.org/Wiki/ParaView Search the list archives at: http://markmail.org/search/?q=ParaView Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/paraview