Alright, three months have passed (there go the "two weeks" i thought
I'd start in :-) ) and I have the first version of incremental saving.
For now, the approach is simple - upon loading an MITK file I store the
file name and the original time stamps of loaded nodes.
When saving to the same file, all meta-data (i.e. all node and data
properties and node relations), new nodes as well as nodes that have
changed are written to temp folder, then I use Poco::ZipManipulator and
remove everything except the base-data files that belong to nodes that
haven't changed from the original mitk scene file. Then I pack the new
data into the mitk scene file - and done!
There are of course some problems. One of them is if the serializer
saved multiple files (e.g. think of mhd format which has image data in
.raw file). Obviously, this information needs to be passed to the
SceneIO in order to keep these additional files from being removed from
the mitk scene file. This problem is solved - take a look at the patch
in the attachment, but it would require some modifications from the MITK
users who want to use incremental saving for serializers that write more
than one file.
I tried to keep everything basically backwards-compatible so the old
scenes could be loaded easily. Also, the scenes created with the
incremental saver can be loaded with the old version.
I have also separated the zip operations to a separate thread to save
some more time.
In a scene with a large image, the save time with incremental saving the
saving time was cut from ~2 minutes to about 10s (in my case, the image
never changes and the rest of the data is relatively lightweight).
What needs to be done still - separate the Open/Save project actions
from Import action (which is the "open" action now). This will also
allow to get rid of static members that I had to put into SceneIO now.
The question then would be to the MITK developers - what would you see
as a good architecture for this? Is there something in the BlueBerry
framework which may be of help?
I would also like to hear from the MITK users - what do you think about it?
All best,
Rostislav.
On 04/12/2014 10:52, Rostislav Khlebnikov wrote:
I am planning to start working on it in the next couple of weeks. Was
waiting for 2014.10 release to start :)
Rostislav.
On 04/12/2014 10:50, Miklos Espak wrote:
I did not. Hopefully in January.
On 4 Dec 2014 10:28, "Martin Klemm" <[email protected]
<mailto:[email protected]>> wrote:
Hi list,
I am also interested in the auto-save feature. Did someone of you
already start working on it?
Best
Martin
On 10/17/2014 08:16 PM, Miklos Espak wrote:
On 17 October 2014 15:49, Rostislav Khlebnikov
<[email protected]
<mailto:[email protected]>> wrote:
Hi Miklos,
I understand that saving corrupted data to disk is not the
best approach. On the other hand, auto-save is likely a
crash recovery mechanism, so from my point of view this
would be the way to go as it doesn't eat up the memory or
slow down the interaction. Honestly, this is quite hard to
say which is a better approach without profiling the
application on real data and looking on how annoying are the
potential freezes during interaction. But as you say, it
would be possible to implement different approaches to this
problem so every MITK user could choose one which they like
the best.
I'm just afraid that if auto-saving hinders interaction, the
end users would either complain about it a lot or, given an
option, would completely turn off this feature and I feel
this would be very sad - it's better to have some corrupted
save than none at all. I'm thinking here about MS-word
autosave and that oftentimes the restored file is not
exactly perfect, but at least some of my work is saved. On
the other hand, I would be very very annoyed if MSWord would
hang even for couple of seconds during auto-save. But again,
this is a personal preference and we could consult the
end-users regarding that (or, perhaps, force something on
them and gather feedback as it is usually more effective :)).
No, the application should not hang during auto-save. It should
happen invisibly in the background. I do not know about MS Word
now, but I remember that about 10 years ago OpenOffice stopped
for the time of autosave, and with big documents and slow
computers it could take a few seconds. It was pretty annoying.
I would say that once we have these different approaches to
saving, it would be nice for people to test it on their data
in a real setting. Because it will depend on the data and
the workflow. For me, the images are relatively big
(~600^3), but they are never modified, but I work a lot with
large surface-like data and many small data objects, such as
planar figures. For you - it's mostly image data. Perhaps,
the behavior should be completely different for people
working with DTI data, etc.
My two cents regarding the locking mechanisms for other data
types. I think it would be really great, but I would also
say - don't do that. Even for me with only 5-6 different
BaseData subclasses, supporting this would be a big pain -
adding code for obtaining some kind of lock before accessing
the data in all the places seems like a lot of work and
error-prone work at that. And if each data access is
automatically surrounded by, say, mutex access - my gut
feeling is that this would be detrimental to the performance
of the whole application.
The way how it is implemented for images is completely
transparent, you do not need to add any lines of code. Maybe
there is a solution that does not need modification of client code.
Performance can be an issue, but I do not think it would be
significant. Of course, we cannot tell it until we test it.
My point is that you cannot exclude the possibility of an
eventual crash if you do not lock, you can only reduce the
chance of a crash. The cloning is faster than saving to disk,
but this just means that there is less chance that is data is
modified during that. It can potentially happen that a point is
removed from a pointset while it is being cloned by the auto
save process.
So, it seems that we agree in what we want to achieve, just you
are more concerned about the performance and I am more concerned
about the safety. Hopefully, we can find a solution that is
fast, does not block the GUI and is safe at the same time. The
only way to check this is if we try a couple of options and test
them.
The next MITK release has been feature freezed a week ago, so in
principle we can start the work on the current master.
all the best,
Miklos
All best,
Rostislav.
On 17/10/2014 15:21, Miklos Espak wrote:
Hi Rostislav,
I have not realised that this locking mechanism is only for
images. I work only with images at the moment but we would
need the autosave to work for other data as well, for other
projects.
What regards the images, I do not really like the shallow
clone approach. I do not think it is good to save an image
while it is being modified, even if the size of the image
does not change and it does not lead to a crash. Luckily,
the locking mechanism would prevent this. The autosave
thread would put a read lock on the image, i.e. no one
could make any modification on the image while it is being
saved. So, this is out of question.
The only exception is if the autosave thread creates the
read accessor with the IgnoreLock flag. But it is a kind of
undermining the whole locking mechanism, and we should not
do that. I am not sure that the size of the image can be
changed, but if yes, it can cause a crash, as you pointed
it out. Or it can cause that we save an invalid state
(shallow clone).
http://docs.mitk.org/nightly-qt4/classmitk_1_1ImageAccessorBase.html
If the saving to the disk is slow and we can also clone the
image in the memory, but the read lock would be applied in
this case as well, i.e. the image cannot be modified during
the cloning. All is safe.
How I see:
The locking mechanism should be extended for other kinds of
data as well. Without it, you cannot totally exclude the
possibility that that e.g. a planar polygon is modified
while it is being either saved or even just cloned.
In addition to this, there can be three possible autosave
policies, maybe depending on the data type or a property of
a data node:
1. The data is locked for reading by the auto-save process
while it is being saved to disk.
2. The data is locked for reading by the auto-save process
while it is being cloned in memory, and it is saved to disk
later.
3. The data is locked for reading by the auto-save process
with the IgnoreLock flag, and the thread starts listening
to modification of the data at the same time. If the data
is modified during the cloning or saving, it should be
interrupted or restarted a bit later.
What do you think?
Sascha,
if you are still reading. :) Are you planning #14866 for
the coming release? As I see, the last merge was around the
date when you did the feature freeze.
Do you think it would be complicated to introduce a
read/write locks for non-imaging data?
Cheers,
Miklos
On 17 October 2014 13:50, Rostislav Khlebnikov
<[email protected]
<mailto:[email protected]>> wrote:
Hi Miklos,
yes, indeed it's a good feature that would be really
nice to have.
However, I feel that we cannot really go for just one
of the approaches that you have described. I feel
several auto-save policies would have to be implemented
for the auto-save to always work correctly and quickly
enough.
There are several things that have to be considered
here. First and foremost, auto-save should never lead
to a crash. This means that a safe bet would be the
cloning approach. I would say that it would work for my
use case for a majority of data types invloved as the
data are relatively small compared to the image size.
However, it is quite clear that it wouldn't be a good
idea to use this approach for image data, so we would
have to consider a more involved approach here.
How I see it, it is necessary to define the changes to
the image data that would be considered "breaking". I
think that most likely these would be only the changes
in the image size that would lead to reallocation of
memory and thus a potential crash in the saving thread
which might try to read data from the free'd memory.
The changes to the pixel data would be fine. I mean,
the saved image might be corrupted in terms of the
actual data, but it's auto-save after all and if for
some reason app crashes, we will restore what can be
restored. If the app doesn't crash - then the next
auto-save or a proper save will overwrite the incorrect
data. This leads me to think that the good approach
here would be kind of a "shallow clone" of the image
data for auto-saving purposes. This will include the
image meta-data, but the pointer to the actual data
will remain intact during auto-save. If the image is
reallocated during autosave - then the old data will be
freed as soon as autosave finishes. I don't work a lot
with images and the implementation details regarding
locks on the image data that you describe is not quite
clear to me, but I think this should be quite doable.
For other data types, a full clone is likely a better
approach. Consider planar polygon for example. If the
user removes a point during auto-save - an exception or
a crash is very likely to happen.
I would say that there should be two general policies
(no auto-save or full clone) and a "custom" policy,
such as shallow clone that is specific for image data
type. With this, if someone needs an even more
sophisticated approach - such as auto-saving only the
portion of the image changed since last auto-save -
then it would be possible to implement this. I would
also make the auto-save policy as a BaseData or even
DataNode property. The reason for this is that for some
data objects, we may want to, e.g., disable the
auto-save completely. For example, this might be the
derived data that can be easily re-created - like a
surface that is extracted from a segmented image.
So I think you could start by looking into this
shallow-cloning of image data and a simle autosaver
class which would from time to time make a clone of the
data storage (with shallow clones where necessary) and
save it in separate thread.
In any case, I think we should only start working on
this when 2014.10 is out, especially given that there's
work being done on this bug:
http://bugs.mitk.org/show_bug.cgi?id=14866.
Hope this makes any sense,
Rostislav.
On 16/10/2014 19:03, Miklos Espak wrote:
Hi Rostislav,
it seems there is demand for this feature. :)
It's good that now the locking mechanism issues have
been sorted for this release because that would
prevent concurrent access to the data. Now we have
several options.
The data can be modified only when someone puts a
write lock on it. Until the lock is released, no-one
can get either read or write lock. I guess, if the
auto-saving would be scheduled for this time, that
thread would simply block until the write lock is
released. This can be good, but it assumes that the
applications lock the images only for the actual time
when they are modifying the data.
Other option is that the auto-save thread creates its
read accessors manually with the "IgnoreLock" option
(not using the ITK access functions). Then the
auto-save thread would not block, but it will need to
listen to the modifications of the input data and
restart the saving process when it happens. Or clone
the before the saving. Not much better, although there
less chance that the data is modified during cloning.
I would go for the first option, because that is
simpler. Then either people can fix there apps if the
blocking occurs, or we can go for the more complicated
and hacky way with the IgnoreLock flag and listening
to Modified events.
What you think?
Cheers,
Miklos
On 15 October 2014 13:15, Khlebnikov, Rostislav
<[email protected]
<mailto:[email protected]>> wrote:
Hi Miklos,
This goes in line with my earlier email regarding
the incremental saving (meaning saving only the
stuff that changed since the scene was opened). I
believe that implementing this is relatively
important before implementing auto-save as it will
speed up the saving process significantly and will
reduce load on the hard drive. At least in my case
where 80% of the project file is the original
image data that never changes.
I will start working on this very soon - I wanted
to wait until the new release but likely I will
just start working on this in the current master
if it builds correctly.
I guess we could work on this in parallel. It'd be
great if you could figure out how to handle new
changes to data nodes while they are being
auto-saved in a separate thread. Should a data
clone be made (likely too much memory
consumption)? Should the writers support
interruption of the saving process? What do you think?
I would then concentrate on supporting the
separate "open project" and "import data" actions,
support for time stamp tracking to detect what
really changed, and re-packing only changed data
on save.
How I saw the auto-save working then was - "open
project" - save the location of temp folder used
for scene loading as well as record that in the
persistent storage using QSetting-like mechanism.
During work - save the changes to this folder
(will also speed up the normal saving process as
only changes since last auto save would have to be
written to temp folder and packing would have to
be performed). On fresh start - check if the exit
was clean and if temp folder saved in persistent
storage is available - try to recover the scene
from there.
That's a big email I wrote :) Anyway - it'd be
nice to hear from you as well as MITK core
developers with thoughts on this.
Rostislav.
> On 15 Oct 2014, at 12:38, "Miklos Espak"
<[email protected] <mailto:[email protected]>> wrote:
>
> Hi All,
>
> is anybody interested in an auto-save feature
for MITK?
>
> I thought of saving the project at regular
interval and restore it if the application is
restarted after a "non-clean" termination. :)
>
> Regards,
> Miklos
>
>
------------------------------------------------------------------------------
> Comprehensive Server Monitoring with Site24x7.
> Monitor 10 servers for $9/Month.
> Get alerted through email, SMS, voice calls or
mobile push notifications.
> Take corrective actions from your mobile device.
> http://p.sf.net/sfu/Zoho
> _______________________________________________
> mitk-users mailing list
> [email protected]
<mailto:[email protected]>
>
https://lists.sourceforge.net/lists/listinfo/mitk-users
------------------------------------------------------------------------------
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
_______________________________________________
mitk-users mailing list
[email protected] <mailto:[email protected]>
https://lists.sourceforge.net/lists/listinfo/mitk-users
------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and
Dashboards
with Interactivity, Sharing, Native Excel Exports, App
Integration & more
Get technology previously reserved for billion-dollar
corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
mitk-users mailing list
[email protected]
<mailto:[email protected]>
https://lists.sourceforge.net/lists/listinfo/mitk-users
------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
mitk-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/mitk-users
------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
mitk-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/mitk-users
>From ab3b0c008982496ae8d15ef8a7d805eae848b849 Mon Sep 17 00:00:00 2001
From: Rostislav Khlebnikov <[email protected]>
Date: Thu, 26 Feb 2015 16:24:06 +0000
Subject: [PATCH 2/2] Incremental saving v0.1
---
Modules/SceneSerialization/mitkSceneIO.cpp | 260 ++++++++++++++++-----
Modules/SceneSerialization/mitkSceneIO.h | 15 ++
Modules/SceneSerialization/mitkSceneReader.cpp | 4 +-
Modules/SceneSerialization/mitkSceneReader.h | 12 +-
Modules/SceneSerialization/mitkSceneReaderV1.cpp | 40 +++-
Modules/SceneSerialization/mitkSceneReaderV1.h | 10 +-
.../mitkBaseDataSerializer.h | 3 +-
7 files changed, 272 insertions(+), 72 deletions(-)
diff --git a/Modules/SceneSerialization/mitkSceneIO.cpp
b/Modules/SceneSerialization/mitkSceneIO.cpp
index f3139f1..53db656 100644
--- a/Modules/SceneSerialization/mitkSceneIO.cpp
+++ b/Modules/SceneSerialization/mitkSceneIO.cpp
@@ -19,6 +19,9 @@ See LICENSE.txt or http://www.mitk.org for details.
#include <Poco/Delegate.h>
#include <Poco/Zip/Compress.h>
#include <Poco/Zip/Decompress.h>
+#include <Poco/Zip/ZipManipulator.h>
+#include <Poco/DirectoryIterator.h>
+#include <Poco/Thread.h>
#include "mitkSceneIO.h"
#include "mitkBaseDataSerializer.h"
@@ -41,6 +44,108 @@ See LICENSE.txt or http://www.mitk.org for details.
#include "itksys/SystemTools.hxx"
+std::map<const mitk::DataNode*, unsigned long>
mitk::SceneIO::m_NodeLoadTimeStamps;
+std::string mitk::SceneIO::m_LoadedProjectFileName;
+mitk::SceneReader::LoadedNodeFileNamesMap mitk::SceneIO::m_LoadedNodeFileNames;
+Poco::Timestamp mitk::SceneIO::m_FileTimeStamp;
+Poco::Thread mitk::SceneIO::m_CompressionThread;
+std::unique_ptr<mitk::CompressorTask> mitk::SceneIO::m_CompressorTask;
+
+namespace mitk {
+
+class CompressorTask : public Poco::Runnable {
+public:
+ CompressorTask(mitk::SceneIO::Pointer sceneIO, const std::string&
filename, bool incrementalSave, const std::set<std::string>& filesToMaintain)
+ : sceneIO(sceneIO)
+ , filename(filename)
+ , incrementalSave(incrementalSave)
+ , filesToMaintain(filesToMaintain)
+ {
+
+ }
+
+ virtual void run()
+ {
+ try
+ {
+ if (incrementalSave) {
+ MITK_INFO << "Saving incrementally";
+ Poco::Zip::ZipManipulator zipper(filename, true);
+
+ for (auto iter = zipper.originalArchive().fileInfoBegin();
iter != zipper.originalArchive().fileInfoEnd(); ++iter) {
+ if (filesToMaintain.find(iter->first) ==
filesToMaintain.end()) {
+ zipper.deleteFile(iter->first);
+ }
+ }
+
+ zipper.commit();
+
+ Poco::Zip::ZipManipulator zipper2(filename, false);
+
+ Poco::DirectoryIterator end;
+ for (Poco::DirectoryIterator
iter(sceneIO->m_WorkingDirectory); iter != end; ++iter) {
+ if (!iter->isDirectory()) {
+
zipper2.addFile(Poco::Path(iter->path()).getFileName(), iter->path(),
Poco::Zip::ZipCommon::CM_DEFLATE, Poco::Zip::ZipCommon::CL_SUPERFAST);
+ }
+ }
+
+ zipper2.commit();
+ }
+ else {
+ MITK_INFO << "Performing full save";
+ Poco::File deleteFile(filename.c_str());
+ if (deleteFile.exists())
+ {
+ deleteFile.remove();
+ }
+
+ // create zip at filename
+ std::ofstream file(filename.c_str(), std::ios::binary |
std::ios::out);
+ if (!file.good())
+ {
+ MITK_ERROR << "Could not open a zip file for writing: '"
<< filename << "'";
+ return;
+ }
+ else
+ {
+ Poco::Zip::Compress zipper(file, true);
+ Poco::Path tmpdir(sceneIO->m_WorkingDirectory);
+ zipper.addRecursive(tmpdir,
Poco::Zip::ZipCommon::CL_SUPERFAST);
+ zipper.close();
+ }
+ }
+ ProgressBar::GetInstance()->Progress();
+
+ try
+ {
+ Poco::File deleteDir(sceneIO->m_WorkingDirectory);
+ deleteDir.remove(true); // recursive
+ }
+ catch (...)
+ {
+ MITK_ERROR << "Could not delete temporary directory " <<
sceneIO->m_WorkingDirectory;
+ return; // ok?
+ }
+
+ }
+ catch (std::exception& e)
+ {
+ ProgressBar::GetInstance()->Progress();
+ MITK_ERROR << "Could not create ZIP file from " <<
sceneIO->m_WorkingDirectory << "\nReason: " << e.what();
+ return;
+ }
+ sceneIO->m_LoadedProjectFileName = filename;
+ sceneIO->m_FileTimeStamp = Poco::File(filename).getLastModified();
+ }
+
+ mitk::SceneIO::Pointer sceneIO;
+ std::string filename;
+ bool incrementalSave;
+ const std::set<std::string> filesToMaintain;
+};
+
+}
+
mitk::SceneIO::SceneIO()
:m_WorkingDirectory(""),
m_UnzipErrors(0)
@@ -49,8 +154,13 @@ mitk::SceneIO::SceneIO()
mitk::SceneIO::~SceneIO()
{
+ if (m_CompressionThread.isRunning()) {
+ MITK_INFO << "Waiting for save thread to complete...";
+ m_CompressionThread.join();
+ }
}
+
std::string mitk::SceneIO::CreateEmptyTempDirectory()
{
mitk::UIDGenerator uidGen("UID_",6);
@@ -88,6 +198,12 @@ mitk::DataStorage::Pointer mitk::SceneIO::LoadScene( const
std::string& filename
DataStorage* pStorage,
bool clearStorageFirst )
{
+ if (!m_LoadedProjectFileName.empty()) {
+ m_LoadedProjectFileName.clear();
+ m_NodeLoadTimeStamps.clear();
+ m_LoadedNodeFileNames.clear();
+ }
+
// prepare data storage
DataStorage::Pointer storage = pStorage;
if ( storage.IsNull() )
@@ -130,6 +246,8 @@ mitk::DataStorage::Pointer mitk::SceneIO::LoadScene( const
std::string& filename
return storage;
}
+ m_FileTimeStamp = Poco::File(filename).getLastModified();
+
// unzip all filenames contents to temp dir
m_UnzipErrors = 0;
Poco::Zip::Decompress unzipper( file, Poco::Path( m_WorkingDirectory ) );
@@ -153,11 +271,20 @@ mitk::DataStorage::Pointer mitk::SceneIO::LoadScene(
const std::string& filename
return storage;
}
+ TiXmlElement* versionObject = document.FirstChildElement("Version");
+ bool incrementalEnabledScene = versionObject &&
versionObject->Attribute("IncrementalEnabled");
+
+ if (incrementalEnabledScene) {
+ storage->AddNodeEvent.AddListener(mitk::MessageDelegate1<SceneIO, const
mitk::DataNode*>(this, &SceneIO::RecordLoadTimeStamp));
+ }
SceneReader::Pointer reader = SceneReader::New();
- if ( !reader->LoadScene( document, m_WorkingDirectory, storage ) )
+ if ( !reader->LoadScene( document, m_WorkingDirectory, storage,
&m_LoadedNodeFileNames ) )
{
MITK_ERROR << "There were errors while loading scene file " << filename <<
". Your data may be corrupted";
}
+ if (incrementalEnabledScene) {
+ storage->AddNodeEvent.RemoveListener(mitk::MessageDelegate1<SceneIO,
const mitk::DataNode*>(this, &SceneIO::RecordLoadTimeStamp));
+ }
// delete temp directory
try
@@ -170,13 +297,26 @@ mitk::DataStorage::Pointer mitk::SceneIO::LoadScene(
const std::string& filename
MITK_ERROR << "Could not delete temporary directory " <<
m_WorkingDirectory;
}
+ m_LoadedProjectFileName = filename;
+
// return new data storage, even if empty or uncomplete (return as much as
possible but notify calling method)
return storage;
}
+void mitk::SceneIO::RecordLoadTimeStamp(const mitk::DataNode* node)
+{
+ MITK_INFO << "Loaded " << node->GetName();
+ m_NodeLoadTimeStamps[node] = node->GetMTime();
+}
+
bool mitk::SceneIO::SaveScene( DataStorage::SetOfObjects::ConstPointer
sceneNodes, const DataStorage* storage,
const std::string& filename)
{
+ if (m_CompressionThread.isRunning()) {
+ MITK_INFO << "Waiting for save thread to complete...";
+ m_CompressionThread.join();
+ }
+
if (!sceneNodes)
{
MITK_ERROR << "No set of nodes given. Not possible to save scene.";
@@ -208,11 +348,14 @@ bool mitk::SceneIO::SaveScene(
DataStorage::SetOfObjects::ConstPointer sceneNode
version->SetAttribute("Writer", __FILE__ );
version->SetAttribute("Revision", "$Revision: 17055 $" );
version->SetAttribute("FileVersion", 1 );
+ version->SetAttribute("IncrementalEnabled", 1);
document.LinkEndChild(version);
//DataStorage::SetOfObjects::ConstPointer sceneNodes = storage->GetSubset(
predicate );
- if ( sceneNodes.IsNull() )
+ bool incrementalSave = Poco::File(filename).path() ==
Poco::File(m_LoadedProjectFileName).path() && m_FileTimeStamp ==
Poco::File(filename).getLastModified();
+ std::set<std::string> filesToMaintain;
+ if (sceneNodes.IsNull())
{
MITK_WARN << "Saving empty scene to " << filename;
}
@@ -232,7 +375,7 @@ bool mitk::SceneIO::SaveScene(
DataStorage::SetOfObjects::ConstPointer sceneNode
return false;
}
- ProgressBar::GetInstance()->AddStepsToDo( sceneNodes->size() );
+ ProgressBar::GetInstance()->AddStepsToDo( sceneNodes->size() + 1 ); // 1
is for compressor thread
// find out about dependencies
typedef std::map< DataNode*, std::string > UIDMapType;
@@ -248,6 +391,8 @@ bool mitk::SceneIO::SaveScene(
DataStorage::SetOfObjects::ConstPointer sceneNode
++iter)
{
DataNode* node = iter->GetPointer();
+
+
if (!node)
continue; // unlikely event that we get a NULL pointer as an object
for saving. just ignore
@@ -312,14 +457,46 @@ bool mitk::SceneIO::SaveScene(
DataStorage::SetOfObjects::ConstPointer sceneNode
}
// store basedata
- if ( BaseData* data = node->GetData() )
+ BaseData* data = node->GetData();
+ if (data)
{
- //std::string filenameHint( node->GetName() );
+ bool dataNeedsToBeWritten = (!incrementalSave ||
m_NodeLoadTimeStamps.find(node) == m_NodeLoadTimeStamps.end() ||
node->GetData()->GetMTime() > m_NodeLoadTimeStamps[node]);
+ //std::string filenameHint( node->GetName() );
bool error(false);
- TiXmlElement* dataElement( SaveBaseData( data, filenameHint, error
) ); // returns a reference to a file
- if (error)
- {
- m_FailedNodes->push_back( node );
+ TiXmlElement* dataElement;
+
+ if (dataNeedsToBeWritten) {
+ dataElement = SaveBaseData(data, filenameHint, error); //
returns a reference to a file
+ if (error)
+ {
+ m_FailedNodes->push_back(node);
+ }
+ else {
+ m_LoadedNodeFileNames[node].baseDataFiles.clear();
+
m_LoadedNodeFileNames[node].baseDataFiles.push_back(dataElement->Attribute("file"));
+
+ for (TiXmlElement* element =
dataElement->FirstChildElement("additionalFile"); element != NULL; element =
element->NextSiblingElement("additionalFile"))
+ {
+
m_LoadedNodeFileNames[node].baseDataFiles.push_back(element->Attribute("file"));
+ }
+
+ m_NodeLoadTimeStamps[node] = node->GetMTime();
+ }
+ MITK_INFO << "Saved " << node->GetName();
+ }
+ else {
+ // Create the element using stored data
+ dataElement = new TiXmlElement("data");
+ dataElement->SetAttribute("type", data->GetNameOfClass());
+ dataElement->SetAttribute("file",
m_LoadedNodeFileNames[node].baseDataFiles[0]);
+ for (size_t i = 1; i <
m_LoadedNodeFileNames[node].baseDataFiles.size(); ++i)
+ {
+ TiXmlElement* additionalFileElement = new
TiXmlElement("additionalFile");
+ additionalFileElement->SetAttribute("file",
m_LoadedNodeFileNames[node].baseDataFiles[i]);
+ dataElement->LinkEndChild(additionalFileElement);
+ }
+ std::copy(m_LoadedNodeFileNames[node].baseDataFiles.begin(),
m_LoadedNodeFileNames[node].baseDataFiles.end(), std::inserter(filesToMaintain,
filesToMaintain.end()));
+ MITK_INFO << "Skipped " << node->GetName();
}
// store basedata properties
@@ -327,6 +504,7 @@ bool mitk::SceneIO::SaveScene(
DataStorage::SetOfObjects::ConstPointer sceneNode
if (propertyList && !propertyList->IsEmpty() )
{
TiXmlElement* baseDataPropertiesElement( SavePropertyList(
propertyList, filenameHint + "-data") ); // returns a reference to a file
+ m_LoadedNodeFileNames[node].baseDataPropertiesFile =
baseDataPropertiesElement->Attribute("file");
dataElement->LinkEndChild( baseDataPropertiesElement );
}
@@ -334,6 +512,7 @@ bool mitk::SceneIO::SaveScene(
DataStorage::SetOfObjects::ConstPointer sceneNode
}
// store all renderwindow specific propertylists
+ m_LoadedNodeFileNames[node].nodePropertiesFiles.clear();
if (RenderingManager::IsInstantiated())
{
const RenderingManager::RenderWindowVector& allRenderWindows(
RenderingManager::GetInstance()->GetAllRegisteredRenderWindows() );
@@ -349,6 +528,9 @@ bool mitk::SceneIO::SaveScene(
DataStorage::SetOfObjects::ConstPointer sceneNode
if ( propertyList && !propertyList->IsEmpty() )
{
TiXmlElement* renderWindowPropertiesElement(
SavePropertyList( propertyList, filenameHint + "-" + renderWindowName) ); //
returns a reference to a file
+
+
m_LoadedNodeFileNames[node].nodePropertiesFiles[renderWindowName] =
renderWindowPropertiesElement->Attribute("file");
+
renderWindowPropertiesElement->SetAttribute("renderwindow",
renderWindowName);
nodeElement->LinkEndChild( renderWindowPropertiesElement );
}
@@ -361,7 +543,10 @@ bool mitk::SceneIO::SaveScene(
DataStorage::SetOfObjects::ConstPointer sceneNode
if ( propertyList && !propertyList->IsEmpty() )
{
TiXmlElement* propertiesElement( SavePropertyList( propertyList,
filenameHint + "-node") ); // returns a reference to a file
- nodeElement->LinkEndChild( propertiesElement );
+
+ m_LoadedNodeFileNames[node].nodePropertiesFiles[""] =
propertiesElement->Attribute("file");
+
+ nodeElement->LinkEndChild(propertiesElement);
}
document.LinkEndChild( nodeElement );
}
@@ -381,45 +566,9 @@ bool mitk::SceneIO::SaveScene(
DataStorage::SetOfObjects::ConstPointer sceneNode
}
else
{
- try
- {
- Poco::File deleteFile( filename.c_str() );
- if (deleteFile.exists())
- {
- deleteFile.remove();
- }
-
- // create zip at filename
- std::ofstream file( filename.c_str(), std::ios::binary |
std::ios::out);
- if (!file.good())
- {
- MITK_ERROR << "Could not open a zip file for writing: '" << filename
<< "'";
- return false;
- }
- else
- {
- Poco::Zip::Compress zipper( file, true );
- Poco::Path tmpdir( m_WorkingDirectory );
- zipper.addRecursive(tmpdir, Poco::Zip::ZipCommon::CL_SUPERFAST);
- zipper.close();
- }
- try
- {
- Poco::File deleteDir( m_WorkingDirectory );
- deleteDir.remove(true); // recursive
- }
- catch(...)
- {
- MITK_ERROR << "Could not delete temporary directory " <<
m_WorkingDirectory;
- return false; // ok?
- }
- }
- catch(std::exception& e)
- {
- MITK_ERROR << "Could not create ZIP file from " << m_WorkingDirectory
<< "\nReason: " << e.what();
- return false;
- }
- return true;
+ m_CompressorTask = std::make_unique<CompressorTask>(this, filename,
incrementalSave, filesToMaintain);
+ m_CompressionThread.start(*m_CompressorTask);
+ return true;
}
}
catch(std::exception& e)
@@ -462,9 +611,16 @@ TiXmlElement* mitk::SceneIO::SaveBaseData( BaseData* data,
const std::string& fi
serializer->SetWorkingDirectory( m_WorkingDirectory );
try
{
- std::string writtenfilename = serializer->Serialize();
- element->SetAttribute("file", writtenfilename);
- error = false;
+ std::vector<std::string> writtenfilenames = serializer->SerializeAll();
+ if (!writtenfilenames.empty()) {
+ element->SetAttribute("file", writtenfilenames[0]);
+ for (size_t i = 1; i < writtenfilenames.size(); ++i) {
+ TiXmlElement* additionalFileElement = new
TiXmlElement("additionalFile");
+ additionalFileElement->SetAttribute("file",
writtenfilenames[i]);
+ element->LinkEndChild(additionalFileElement);
+ }
+ error = false;
+ }
}
catch (std::exception& e)
{
diff --git a/Modules/SceneSerialization/mitkSceneIO.h
b/Modules/SceneSerialization/mitkSceneIO.h
index 2c0f771..af0bde0 100644
--- a/Modules/SceneSerialization/mitkSceneIO.h
+++ b/Modules/SceneSerialization/mitkSceneIO.h
@@ -18,11 +18,14 @@ See LICENSE.txt or http://www.mitk.org for details.
#define mitkSceneIO_h_included
#include <MitkSceneSerializationExports.h>
+#include <mitkSceneReader.h>
#include "mitkDataStorage.h"
#include "mitkNodePredicateBase.h"
#include <Poco/Zip/ZipLocalFileHeader.h>
+#include <Poco/Timestamp.h>
+#include <Poco/Thread.h>
class TiXmlElement;
@@ -111,6 +114,18 @@ class MitkSceneSerialization_EXPORT SceneIO : public
itk::Object
std::string m_WorkingDirectory;
unsigned int m_UnzipErrors;
+
+ friend class CompressorTask;
+
+ static std::map<const mitk::DataNode*, unsigned long> m_NodeLoadTimeStamps;
+ static std::string m_LoadedProjectFileName;
+ static SceneReader::LoadedNodeFileNamesMap m_LoadedNodeFileNames;
+ static Poco::Timestamp m_FileTimeStamp;
+ static Poco::Thread m_CompressionThread;
+ static std::unique_ptr<CompressorTask> m_CompressorTask;
+
+ void RecordLoadTimeStamp(const mitk::DataNode*);
+
};
}
diff --git a/Modules/SceneSerialization/mitkSceneReader.cpp
b/Modules/SceneSerialization/mitkSceneReader.cpp
index 4b9c267..10c974a 100644
--- a/Modules/SceneSerialization/mitkSceneReader.cpp
+++ b/Modules/SceneSerialization/mitkSceneReader.cpp
@@ -16,7 +16,7 @@ See LICENSE.txt or http://www.mitk.org for details.
#include "mitkSceneReader.h"
-bool mitk::SceneReader::LoadScene( TiXmlDocument& document, const std::string&
workingDirectory, DataStorage* storage )
+bool mitk::SceneReader::LoadScene(TiXmlDocument& document, const std::string&
workingDirectory, DataStorage* storage, LoadedNodeFileNamesMap*
nodeDataFileNameMap)
{
// find version node --> note version in some variable
int fileVersion = 1;
@@ -48,7 +48,7 @@ bool mitk::SceneReader::LoadScene( TiXmlDocument& document,
const std::string& w
{
if (SceneReader* reader = dynamic_cast<SceneReader*>( iter->GetPointer() )
)
{
- if ( !reader->LoadScene( document, workingDirectory, storage ) )
+ if ( !reader->LoadScene( document, workingDirectory, storage,
nodeDataFileNameMap ) )
{
MITK_ERROR << "There were errors while loading scene file " <<
workingDirectory + "/index.xml. Your data may be corrupted";
return false;
diff --git a/Modules/SceneSerialization/mitkSceneReader.h
b/Modules/SceneSerialization/mitkSceneReader.h
index 28b0dd9..a80650e 100644
--- a/Modules/SceneSerialization/mitkSceneReader.h
+++ b/Modules/SceneSerialization/mitkSceneReader.h
@@ -14,6 +14,8 @@ See LICENSE.txt or http://www.mitk.org for details.
===================================================================*/
+#pragma once
+
#include <MitkSceneSerializationExports.h>
#include <tinyxml.h>
@@ -33,7 +35,15 @@ class MitkSceneSerialization_EXPORT SceneReader : public
itk::Object
itkFactorylessNewMacro(Self)
itkCloneMacro(Self)
- virtual bool LoadScene( TiXmlDocument& document, const std::string&
workingDirectory, DataStorage* storage );
+ struct LoadedNodeFileNames {
+ std::vector<std::string> baseDataFiles; // First is the file to load,
the rest are additional files generated by serializer
+ std::string baseDataPropertiesFile;
+ std::map<std::string, std::string> nodePropertiesFiles; // Map from
renderer name to renderer-specific properties file. Default properties are
associated with empty string
+ };
+
+ typedef std::map<const mitk::DataNode*, LoadedNodeFileNames>
LoadedNodeFileNamesMap;
+
+ virtual bool LoadScene(TiXmlDocument& document, const std::string&
workingDirectory, DataStorage* storage, LoadedNodeFileNamesMap*
nodeDataFileNames = nullptr);
};
}
diff --git a/Modules/SceneSerialization/mitkSceneReaderV1.cpp
b/Modules/SceneSerialization/mitkSceneReaderV1.cpp
index db8e160..96233d8 100644
--- a/Modules/SceneSerialization/mitkSceneReaderV1.cpp
+++ b/Modules/SceneSerialization/mitkSceneReaderV1.cpp
@@ -25,7 +25,7 @@ See LICENSE.txt or http://www.mitk.org for details.
MITK_REGISTER_SERIALIZER(SceneReaderV1)
-bool mitk::SceneReaderV1::LoadScene( TiXmlDocument& document, const
std::string& workingDirectory, DataStorage* storage )
+bool mitk::SceneReaderV1::LoadScene(TiXmlDocument& document, const
std::string& workingDirectory, DataStorage* storage, LoadedNodeFileNamesMap*
nodeDataFileNameMap)
{
assert(storage);
bool error(false);
@@ -52,12 +52,13 @@ bool mitk::SceneReaderV1::LoadScene( TiXmlDocument&
document, const std::string&
for (TiXmlElement* element = document.FirstChildElement("node"); element !=
NULL; element = element->NextSiblingElement("node"))
{
-
DataNodes.push_back(LoadBaseDataFromDataTag(element->FirstChildElement("data"),
workingDirectory, error));
+ mitk::DataNode::Pointer newNode =
LoadBaseDataFromDataTag(element->FirstChildElement("data"), workingDirectory,
error, nodeDataFileNameMap);
+ DataNodes.push_back(newNode);
ProgressBar::GetInstance()->Progress();
}
OrderedLayers orderedLayers;
- this->GetLayerOrder(document, workingDirectory, DataNodes, orderedLayers);
+ this->GetLayerOrder(document, workingDirectory, DataNodes, orderedLayers,
nodeDataFileNameMap);
// iterate all nodes
@@ -74,7 +75,7 @@ bool mitk::SceneReaderV1::LoadScene( TiXmlDocument& document,
const std::string&
TiXmlElement *baseDataElement =
dataXmlElement->FirstChildElement("properties");
if ( node->GetData() )
{
- DecorateBaseDataWithProperties( node->GetData(), baseDataElement,
workingDirectory);
+ DecorateBaseDataWithProperties( node, baseDataElement,
workingDirectory, nodeDataFileNameMap);
}
else
{
@@ -102,7 +103,7 @@ bool mitk::SceneReaderV1::LoadScene( TiXmlDocument&
document, const std::string&
// - instantiate the appropriate PropertyListDeSerializer
// - use them to construct PropertyList objects
// - add these properties to the node (if necessary, use
renderwindow name)
- bool success = DecorateNodeWithProperties(node, element, workingDirectory);
+ bool success = DecorateNodeWithProperties(node, element, workingDirectory,
nodeDataFileNameMap);
if (!success)
{
MITK_ERROR << "Could not load properties for node.";
@@ -213,14 +214,14 @@ bool mitk::SceneReaderV1::LoadScene( TiXmlDocument&
document, const std::string&
return !error;
}
-void mitk::SceneReaderV1::GetLayerOrder(TiXmlDocument& document, const
std::string& workingDirectory, std::vector<mitk::DataNode::Pointer> DataNodes,
OrderedLayers& order)
+void mitk::SceneReaderV1::GetLayerOrder(TiXmlDocument& document, const
std::string& workingDirectory, std::vector<mitk::DataNode::Pointer> DataNodes,
OrderedLayers& order, LoadedNodeFileNamesMap* nodeDataFileNameMap)
{
typedef std::vector<mitk::DataNode::Pointer> DataNodeVector;
DataNodeVector::iterator nit = DataNodes.begin();
for( TiXmlElement* element = document.FirstChildElement("node"); element !=
NULL || nit != DataNodes.end(); element = element->NextSiblingElement("node"),
++nit )
{
DataNode::Pointer node = *nit;
- DecorateNodeWithProperties(node, element, workingDirectory);
+ DecorateNodeWithProperties(node, element, workingDirectory,
nodeDataFileNameMap);
int layer;
node->GetIntProperty("layer", layer);
@@ -242,7 +243,7 @@ void mitk::SceneReaderV1::GetLayerOrder(TiXmlDocument&
document, const std::stri
}
}
-mitk::DataNode::Pointer mitk::SceneReaderV1::LoadBaseDataFromDataTag(
TiXmlElement* dataElement, const std::string& workingDirectory, bool& error )
+mitk::DataNode::Pointer
mitk::SceneReaderV1::LoadBaseDataFromDataTag(TiXmlElement* dataElement, const
std::string& workingDirectory, bool& error, LoadedNodeFileNamesMap*
nodeDataFileNameMap)
{
DataNode::Pointer node;
@@ -260,6 +261,17 @@ mitk::DataNode::Pointer
mitk::SceneReaderV1::LoadBaseDataFromDataTag( TiXmlEleme
}
node = DataNode::New();
node->SetData(baseData.front());
+
+ if (nodeDataFileNameMap) {
+ (*nodeDataFileNameMap)[node].baseDataFiles.clear();
+ (*nodeDataFileNameMap)[node].baseDataFiles.push_back(filename);
+
+ for (TiXmlElement* element =
dataElement->FirstChildElement("additionalFile"); element != NULL; element =
element->NextSiblingElement("additionalFile"))
+ {
+
(*nodeDataFileNameMap)[node].baseDataFiles.push_back(element->Attribute("file"));
+ }
+
+ }
}
catch (std::exception& e)
{
@@ -284,7 +296,7 @@ mitk::DataNode::Pointer
mitk::SceneReaderV1::LoadBaseDataFromDataTag( TiXmlEleme
return node;
}
-bool mitk::SceneReaderV1::DecorateNodeWithProperties(DataNode* node,
TiXmlElement* nodeElement, const std::string& workingDirectory)
+bool mitk::SceneReaderV1::DecorateNodeWithProperties(DataNode* node,
TiXmlElement* nodeElement, const std::string& workingDirectory,
LoadedNodeFileNamesMap* nodeDataFileNameMap)
{
assert(node);
assert(nodeElement);
@@ -313,6 +325,9 @@ bool
mitk::SceneReaderV1::DecorateNodeWithProperties(DataNode* node, TiXmlElemen
if (readProperties.IsNotNull())
{
propertyList->ConcatenatePropertyList( readProperties, true ); // true =
replace
+ if (nodeDataFileNameMap) {
+ (*nodeDataFileNameMap)[node].nodePropertiesFiles[renderwindow] =
propertiesfile;
+ }
}
else
{
@@ -324,7 +339,7 @@ bool
mitk::SceneReaderV1::DecorateNodeWithProperties(DataNode* node, TiXmlElemen
return !error;
}
-bool mitk::SceneReaderV1::DecorateBaseDataWithProperties(BaseData::Pointer
data, TiXmlElement *baseDataNodeElem, const std::string &workingDir)
+bool mitk::SceneReaderV1::DecorateBaseDataWithProperties(DataNode* node,
TiXmlElement *baseDataNodeElem, const std::string &workingDir,
LoadedNodeFileNamesMap* nodeDataFileNameMap)
{
// check given variables, initialize error variable
assert(baseDataNodeElem);
@@ -350,7 +365,10 @@ bool
mitk::SceneReaderV1::DecorateBaseDataWithProperties(BaseData::Pointer data,
// store the read-in properties to the given node or throw error otherwise
if( inProperties.IsNotNull() )
{
- data->SetPropertyList( inProperties );
+ node->GetData()->SetPropertyList( inProperties );
+ if (nodeDataFileNameMap) {
+ (*nodeDataFileNameMap)[node].baseDataPropertiesFile =
baseDataPropertyFile;
+ }
}
else
{
diff --git a/Modules/SceneSerialization/mitkSceneReaderV1.h
b/Modules/SceneSerialization/mitkSceneReaderV1.h
index b7d7aab..ae70674 100644
--- a/Modules/SceneSerialization/mitkSceneReaderV1.h
+++ b/Modules/SceneSerialization/mitkSceneReaderV1.h
@@ -27,7 +27,7 @@ class SceneReaderV1 : public SceneReader
itkFactorylessNewMacro(Self)
itkCloneMacro(Self)
- virtual bool LoadScene( TiXmlDocument& document, const std::string&
workingDirectory, DataStorage* storage );
+ virtual bool LoadScene(TiXmlDocument& document, const std::string&
workingDirectory, DataStorage* storage, LoadedNodeFileNamesMap*
nodeDataFileNames = nullptr);
protected:
@@ -36,19 +36,19 @@ class SceneReaderV1 : public SceneReader
*/
DataNode::Pointer LoadBaseDataFromDataTag( TiXmlElement* dataElement,
const std::string&
workingDirectory,
- bool& error );
+ bool& error,
LoadedNodeFileNamesMap* nodeDataFileNameMap);
/**
\brief reads all the properties from the XML document and recreates them
in node
*/
- bool DecorateNodeWithProperties(DataNode* node, TiXmlElement* nodeElement,
const std::string& workingDirectory);
+ bool DecorateNodeWithProperties(DataNode* node, TiXmlElement* nodeElement,
const std::string& workingDirectory, LoadedNodeFileNamesMap*
nodeDataFileNameMap);
/**
\brief reads all properties assigned to a base data element and assigns
the list to the base data object
The baseDataNodeElem is supposed to be the <properties file="...">
element.
*/
- bool DecorateBaseDataWithProperties(BaseData::Pointer data, TiXmlElement*
baseDataNodeElem, const std::string& workingDir);
+ bool DecorateBaseDataWithProperties(DataNode* node, TiXmlElement*
baseDataNodeElem, const std::string& workingDir, LoadedNodeFileNamesMap*
nodeDataFileNameMap);
typedef std::multimap<int, std::string> UnorderedLayers;
typedef std::map<std::string, int> OrderedLayers;
@@ -57,7 +57,7 @@ class SceneReaderV1 : public SceneReader
typedef std::map<std::string, DataNode*> IDToNodeMappingType;
typedef std::map<DataNode*, std::string> NodeToIDMappingType;
- void GetLayerOrder(TiXmlDocument& document, const std::string&
workingDirectory, std::vector<mitk::DataNode::Pointer> DataNodes,
OrderedLayers& order);
+ void GetLayerOrder(TiXmlDocument& document, const std::string&
workingDirectory, std::vector<mitk::DataNode::Pointer> DataNodes,
OrderedLayers& order, LoadedNodeFileNamesMap* nodeDataFileNameMap);
UnorderedLayers m_UnorderedLayers;
OrderedLayers m_OrderedLayers;
diff --git a/Modules/SceneSerializationBase/mitkBaseDataSerializer.h
b/Modules/SceneSerializationBase/mitkBaseDataSerializer.h
index 18c8457..da4e754 100644
--- a/Modules/SceneSerializationBase/mitkBaseDataSerializer.h
+++ b/Modules/SceneSerializationBase/mitkBaseDataSerializer.h
@@ -57,9 +57,10 @@ class MitkSceneSerializationBase_EXPORT BaseDataSerializer :
public itk::Object
This should be overwritten by specific sub-classes.
*/
- virtual std::string Serialize();
+ virtual std::vector<std::string> SerializeAll() { return std::vector <
std::string > { Serialize() }; }
protected:
+ virtual std::string Serialize();
BaseDataSerializer();
virtual ~BaseDataSerializer();
--
1.8.4.msysgit.0
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
mitk-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/mitk-users
