[Dspace-tech] Export entire site manually
Hi guys, I have a serious problem with the machine where I had my DSpace installation. And I say 'had' because a few days that machine physically died... I had a lot of DSpace data here, so now I have this question: Assuming I have access to all the files of DSpace, and also to the postgresql database, is there a way to manually export the whole site so I can move all the data to a new machine where I fresh-installed DSpace? Commands like "./dspace ..." doesn't work at all because the system is corrupted. Thanks again, I hope there's a way... Regards, Miguel -- Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, analysis and resources for efficient Application Performance Management. Visit us today! http://pubads.g.doubleclick.net/gampad/clk?id=48897511&iu=/4140/ostg.clktrk___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Export entire site manually
Yes, it's possible to save it as long as the database is undamaged. Your first step should be recovering the database on a new machine. If it's intact, also copy the assetstore directory to a new installation of DSpace (same version). When you have it working, you can verify which bitstreams have been damaged by running "[dspace]/bin/dspace checker -l -v". https://wiki.duraspace.org/display/DSDOC3x/Validating+CheckSums+of+Bitstreams Regards, ~~helix84 Compulsory reading: DSpace Mailing List Etiquette https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, analysis and resources for efficient Application Performance Management. Visit us today! http://pubads.g.doubleclick.net/gampad/clk?id=48897511&iu=/4140/ostg.clktrk ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Export entire site manually
Hi miguel, sure there is... Just follow any of the procedures of migrating a version to another (of course... skipping the steps of code upgrading) Basically: 1. Export the BBDD (normal database procedures..) 2. copy the assetstore 3. copy the log files and the solr/statistics/data files (useful to keep/rebuid statistics...) 4. reinstall Dspace 5. import the BBDD 6. import the assetstore and other files 7. rebuild indexes.etc 8. rebuild statisitics (several procedures depending you version) best luck Emilio Lorenzo El 27/08/2013 10:41, Alumno Etsii escribió: Hi guys, I have a serious problem with the machine where I had my DSpace installation. And I say 'had' because a few days that machine physically died... I had a lot of DSpace data here, so now I have this question: Assuming I have access to all the files of DSpace, and also to the postgresql database, is there a way to manually export the whole site so I can move all the data to a new machine where I fresh-installed DSpace? Commands like "./dspace ..." doesn't work at all because the system is corrupted. Thanks again, I hope there's a way... Regards, Miguel -- Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, analysis and resources for efficient Application Performance Management. Visit us today! http://pubads.g.doubleclick.net/gampad/clk?id=48897511&iu=/4140/ostg.clktrk ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, analysis and resources for efficient Application Performance Management. Visit us today! http://pubads.g.doubleclick.net/gampad/clk?id=48897511&iu=/4140/ostg.clktrk___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Export entire site manually
Thanks for your help! The main thing is I'm not 100% sure the database is undamaged, as the system crashed unexpectedly, but I followed your instructions and here's the result: 1) Exported the PSQL files to the new system, I had to reset the xlog directory in order to let me run the PSQL database, but it worked. 2) Copied the whole assetstore directory as-it-was But once I access the xmlui I get this exception: org.postgresql.util.PSQLException: ERROR: current transaction is aborted, commands ignored until end of transaction block I guess PSQL is doing strange things, so unless you have any hint why could be this happening, I'll consider the DB is corrupted as a given... :( Thanks again! 2013/8/27 helix84 > Yes, it's possible to save it as long as the database is undamaged. > Your first step should be recovering the database on a new machine. If > it's intact, also copy the assetstore directory to a new installation > of DSpace (same version). When you have it working, you can verify > which bitstreams have been damaged by running "[dspace]/bin/dspace > checker -l -v". > > > https://wiki.duraspace.org/display/DSDOC3x/Validating+CheckSums+of+Bitstreams > > > Regards, > ~~helix84 > > Compulsory reading: DSpace Mailing List Etiquette > https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette > -- Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, analysis and resources for efficient Application Performance Management. Visit us today! http://pubads.g.doubleclick.net/gampad/clk?id=48897511&iu=/4140/ostg.clktrk___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Export entire site manually
> I'll consider the DB is corrupted as a given... :( Don't give up. If you don't have proper backups (I'm assuming you don't because otherwise you would have mentioned it) restoring the database is the critical step. Try doing pg_dump of your database. That might reveal errors in the database. If you're unsure how to proceed then, send me the dump. Regards, ~~helix84 Compulsory reading: DSpace Mailing List Etiquette https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, analysis and resources for efficient Application Performance Management. Visit us today! http://pubads.g.doubleclick.net/gampad/clk?id=48897511&iu=/4140/ostg.clktrk ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Export entire site manually
Fortunately I was able to recover a backup just a day prior to the crash I restored the assetstore from that backup and also the pgsql dump and it works perfectly now! Thank you so much for your help guys, you saved me a lot of work! :-) Regards. 2013/8/27 helix84 > > I'll consider the DB is corrupted as a given... :( > > Don't give up. If you don't have proper backups (I'm assuming you > don't because otherwise you would have mentioned it) restoring the > database is the critical step. > > Try doing pg_dump of your database. That might reveal errors in the > database. If you're unsure how to proceed then, send me the dump. > > > Regards, > ~~helix84 > > Compulsory reading: DSpace Mailing List Etiquette > https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette > -- Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, analysis and resources for efficient Application Performance Management. Visit us today! http://pubads.g.doubleclick.net/gampad/clk?id=48897511&iu=/4140/ostg.clktrk___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Export entire site manually
This may be a little off-topic but I'm wondering... I'm planning on using the rsync utility to keep a production and development machine in sync; copying the production machine to the development machine nightly, at least the dspace and pgsql (which is a mounted file system) directories to try to avoid this type of server crash problem. Additionally, both machines are RAID 1. Any problem with doing that? -- Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, analysis and resources for efficient Application Performance Management. Visit us today! http://pubads.g.doubleclick.net/gampad/clk?id=48897511&iu=/4140/ostg.clktrk ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Export entire site manually
On Tue, Aug 27, 2013 at 4:38 PM, Thomas Ronayne wrote: > Any problem with doing that? No, that's a perfectly good way to backup all the contents that changes often (there are also other ways but this one is easy and complete). Also remember to backup configuration, any customizations (like a theme) and [dspace]/solr/statistics if you're using Solr statistics, [dspace]/elasticsearch if you're using ES statistics or dspace.log.* if you're using legacy statistics. Regards, ~~helix84 Compulsory reading: DSpace Mailing List Etiquette https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more! Discover the easy way to master current and previous Microsoft technologies and advance your career. Get an incredible 1,500+ hours of step-by-step tutorial videos with LearnDevNow. Subscribe today and save! http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Export entire site manually
Helix84, Kinda what what I thought -- sure beats unloading data base files. I'm thinking just rsync /opt/dspace; ought to get everything (logs and all) and doing the same with /var/lib/pgsql -- files are just files after all else is said and done, eh? Stop PostgreSQL on both servers, do the rsync and restart the DBMS on both. Maybe stop Tomcat, too, but that might be overkill. Thanks! On 08/27/2013 12:28 PM, helix84 wrote: > On Tue, Aug 27, 2013 at 4:38 PM, Thomas Ronayne > wrote: >> Any problem with doing that? > No, that's a perfectly good way to backup all the contents that > changes often (there are also other ways but this one is easy and > complete). Also remember to backup configuration, any customizations > (like a theme) and [dspace]/solr/statistics if you're using Solr > statistics, [dspace]/elasticsearch if you're using ES statistics or > dspace.log.* if you're using legacy statistics. > > > Regards, > ~~helix84 > > Compulsory reading: DSpace Mailing List Etiquette > https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette > -- Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more! Discover the easy way to master current and previous Microsoft technologies and advance your career. Get an incredible 1,500+ hours of step-by-step tutorial videos with LearnDevNow. Subscribe today and save! http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Export entire site manually
On Tue, Aug 27, 2013 at 6:53 PM, Thomas Ronayne wrote: > doing the same with /var/lib/pgsql -- files are just files after all else is > said and done, eh? Stop PostgreSQL on both servers, do the rsync and restart > the DBMS on both. Maybe stop Tomcat, too, but that might be overkill. Oh, no, completely unnecessary downtime. Just do pg_dump on a running database and you'll have one nice sql file instead of database files specific to a particular postgres version. More importantly, as you noticed, you'd have to worry about hot backup - but why go through all that trouble? There is one theoretical issue - there might be an item submitted to DSpace between pg_dump finishes and rsync finishes (or vice versa), which would mean you backed up the database with the item record, but not its bitstream (or vice versa). It would affect only this item, the rest would work perfectly fine. In practice, the backup process will be a nightly cronjob that will finish pretty fast, so I wouldn't worry about it at all. Regards, ~~helix84 Compulsory reading: DSpace Mailing List Etiquette https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more! Discover the easy way to master current and previous Microsoft technologies and advance your career. Get an incredible 1,500+ hours of step-by-step tutorial videos with LearnDevNow. Subscribe today and save! http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Export entire site manually
Hi, Helix84, On 08/27/2013 04:51 PM, helix84 wrote: > On Tue, Aug 27, 2013 at 6:53 PM, Thomas Ronayne > wrote: >> doing the same with /var/lib/pgsql -- files are just files after all else is >> said and done, eh? Stop PostgreSQL on both servers, do the rsync and restart >> the DBMS on both. Maybe stop Tomcat, too, but that might be overkill. > Oh, no, completely unnecessary downtime. Well, it would be done with a cron job in the middle of the night -- don't like to do back up when folks have access to things. > > Just do pg_dump on a running database and you'll have one nice sql > file instead of database files specific to a particular postgres > version. More importantly, as you noticed, you'd have to worry about > hot backup - but why go through all that trouble? Yeah, good point. > > There is one theoretical issue - there might be an item submitted to > DSpace between pg_dump finishes and rsync finishes (or vice versa), > which would mean you backed up the database with the item record, but > not its bitstream (or vice versa). It would affect only this item, the > rest would work perfectly fine. In practice, the backup process will > be a nightly cronjob that will finish pretty fast, so I wouldn't worry > about it at all. Well, that's why I'd shut down Tomcat so nobody can get at anything in the middle of the night. But, yeah, rsync might just be overkill and pg_dump ought to do just fine. I generally keep copies of configuration files (in a tart file) so I can quick and easy do a complete software installation, copy the configuration file (just extract the tar file) on to the new install and be up and running quickly. Keeping a data base back up and loading from scratch has worked before and I'm pretty sure would work again (and it's easy). As long as both machines don't poop their diapers at the same time... life is good. And, of course, a copy of the back up is off on media too. > > > Regards, > ~~helix84 > > Compulsory reading: DSpace Mailing List Etiquette > https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette > Thanks for the thoughts. Thomas -- Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more! Discover the easy way to master current and previous Microsoft technologies and advance your career. Get an incredible 1,500+ hours of step-by-step tutorial videos with LearnDevNow. Subscribe today and save! http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
