Re: [Dhis2-devs] [Dhis2-users] DHIS2 Meta-Data and Data Import Issues
Hi Edwin Your problem seems to be some bad/duplicate data: Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint datavalue_pkey Detail: Key (dataelementid, periodid, sourceid, categoryoptioncomboid)=(19243, 28829, 4788, 15) already exists. I have seen this before and I guess it is not ideal behaviour. When there are two datavalues with the same Key (dataelementid, periodid, sourceid, categoryoptioncomboid) then the importer can't really know which to take as the real one, so instead it just crashes out a little unceremoniously. We should handle this better. But meanwhile if you cleanup that data before you import you should be fine. Bob On 18 December 2013 13:25, Edwin Mulwa eddi...@gmail.com wrote: Morten, Thank you! The in-memory db was the cause of one of my problems. So Issue 1 is resolved! :) I still cannot log in using an imported user. I suspect is it for the reason I mentioned in my original post i.e. importation of users was disabled? The other remaining problem is the importation of XML data. It is still giving a Failed to flush BatchHandler error when I try to import the data. I have attached the relevant section of the log file so that you check and see if you can make anything of it. Thanks alot Morten! Edwin On Wed, Dec 18, 2013 at 1:23 AM, Morten Olav Hansen morte...@gmail.comwrote: Hi, From what you are saying, it sounds like all these issues are caused by using an in-memory db.. Are you sure that you are connecting to a database, and that the user has the proper privileges? Try to look for dhis2 home references in the log.. it should be early in the startup -- Morten On Wed, Dec 18, 2013 at 10:09 AM, Edwin Mulwa eddi...@gmail.com wrote: Hi, I have some major problems when I am trying to import meta-data and data from one DHIS2 instance to another. These are as follows: Issue 1: I have a DHIS2 installation running on a VMWare Virtual Machine. Using the Meta-Data Export/Import features in DHIS2, I have imported data from another DHIS2 instance running on a laptop into the VM successfully. Successfully in quotes because the log says it was imported successfully. However, when I shut down the VM and start it again, all the imported data is missing from the VM. That is, it is as if I did not import any data. In the data import screen, Dry Run option is set to No. I did take a snapshot of the VM immediately after importation and reverted to the snapshot after firing up the VM but nothing doing. Any ideas on why the data is not persisting on the VM? Note: Nothing I am doing on the VM persists! For example, I looked at the default user I have been using to access DHIS2 (admin:district) and added all the roles to the user. These additions to the user account were not reflected after restarting the VM. Important Note: I have noticed, by looking at the logs, that every time the DHIS2 instance starts up, it is attempting to run alot of table altering SQL commands, most of which are failing. I suspect that it is these commands that are interfering with my data. If you would like to see a copy of the log file, I will be happy to provide it. Issue 2: After importing the data, I log out of DHIS2 and try logging in using a different user whose info has been imported into the VM. This does not work. This is BEFORE I even shut down the machine. When I log back in using the default admin user, I can see the user accounts in the system. Note: I have seen from an old posting on the forum that importation of users was disabled. I don't know if this is still the case and therefore why I am having this problem? Issue 3: After importing meta-data, I immediately tried to import DATA from the other instance. Data import fails with a message that it was unable to clear the BatchHandler. Any help on the above issues will be appreciated. Edwin ___ Mailing list: https://launchpad.net/~dhis2-devs Post to : dhis2-devs@lists.launchpad.net Unsubscribe : https://launchpad.net/~dhis2-devs More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~dhis2-users Post to : dhis2-us...@lists.launchpad.net Unsubscribe : https://launchpad.net/~dhis2-users More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~dhis2-devs Post to : dhis2-devs@lists.launchpad.net Unsubscribe : https://launchpad.net/~dhis2-devs More help : https://help.launchpad.net/ListHelp
Re: [Dhis2-devs] [Dhis2-users] DHIS2 Meta-Data and Data Import Issues
On 18 December 2013 13:30, Morten Olav Hansen morte...@gmail.com wrote: I still cannot log in using an imported user. I suspect is it for the reason I mentioned in my original post i.e. importation of users was disabled? Did you check to see that the users exists in the system? we do support import, but we null out the passwords.. for security reasons. So you will need to re-set the passwords. The other remaining problem is the importation of XML data. It is still giving a Failed to flush BatchHandler error when I try to import the data. I have attached the relevant section of the log file so that you check and see if you can make anything of it. I'm not sure about that one.. it says duplicate value, but I guess that should have been treated as an update.. could you file a bug report for it? Yes its definitely a bug or at least an ugly behaviour. I should have reported it earlier - we saw the same importing unclean data from iHRIS in Rwanda. Once we cleaned the data up all was fine. Updates only happen when these are part of separate batches. Duplicates in the same import batch get stuck in the throat. -- Morten Thanks alot Morten! Edwin On Wed, Dec 18, 2013 at 1:23 AM, Morten Olav Hansen morte...@gmail.comwrote: Hi, From what you are saying, it sounds like all these issues are caused by using an in-memory db.. Are you sure that you are connecting to a database, and that the user has the proper privileges? Try to look for dhis2 home references in the log.. it should be early in the startup -- Morten On Wed, Dec 18, 2013 at 10:09 AM, Edwin Mulwa eddi...@gmail.com wrote: Hi, I have some major problems when I am trying to import meta-data and data from one DHIS2 instance to another. These are as follows: Issue 1: I have a DHIS2 installation running on a VMWare Virtual Machine. Using the Meta-Data Export/Import features in DHIS2, I have imported data from another DHIS2 instance running on a laptop into the VM successfully. Successfully in quotes because the log says it was imported successfully. However, when I shut down the VM and start it again, all the imported data is missing from the VM. That is, it is as if I did not import any data. In the data import screen, Dry Run option is set to No. I did take a snapshot of the VM immediately after importation and reverted to the snapshot after firing up the VM but nothing doing. Any ideas on why the data is not persisting on the VM? Note: Nothing I am doing on the VM persists! For example, I looked at the default user I have been using to access DHIS2 (admin:district) and added all the roles to the user. These additions to the user account were not reflected after restarting the VM. Important Note: I have noticed, by looking at the logs, that every time the DHIS2 instance starts up, it is attempting to run alot of table altering SQL commands, most of which are failing. I suspect that it is these commands that are interfering with my data. If you would like to see a copy of the log file, I will be happy to provide it. Issue 2: After importing the data, I log out of DHIS2 and try logging in using a different user whose info has been imported into the VM. This does not work. This is BEFORE I even shut down the machine. When I log back in using the default admin user, I can see the user accounts in the system. Note: I have seen from an old posting on the forum that importation of users was disabled. I don't know if this is still the case and therefore why I am having this problem? Issue 3: After importing meta-data, I immediately tried to import DATA from the other instance. Data import fails with a message that it was unable to clear the BatchHandler. Any help on the above issues will be appreciated. Edwin ___ Mailing list: https://launchpad.net/~dhis2-devs Post to : dhis2-devs@lists.launchpad.net Unsubscribe : https://launchpad.net/~dhis2-devs More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~dhis2-users Post to : dhis2-us...@lists.launchpad.net Unsubscribe : https://launchpad.net/~dhis2-users More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~dhis2-devs Post to : dhis2-devs@lists.launchpad.net Unsubscribe : https://launchpad.net/~dhis2-devs More help : https://help.launchpad.net/ListHelp
Re: [Dhis2-devs] [Dhis2-users] DHIS2 Meta-Data and Data Import Issues
Hi Bob, I will take a look at the data and see where the duplicates may be. The data was generated using Data-Export feature in DHIS so there is certainly a problem with this feature. When I was importing, I had selected New and Updates. The Updates bit of it should have handled the duplicates. To test this, when I selected Updates Only, the import went through successfully. Morten, as for the user, yes I am sure the user exists in the database. I will reset password and see what happens. Thanks! Edwin On Wed, Dec 18, 2013 at 3:34 PM, Bob Jolliffe bobjolli...@gmail.com wrote: On 18 December 2013 13:30, Morten Olav Hansen morte...@gmail.com wrote: I still cannot log in using an imported user. I suspect is it for the reason I mentioned in my original post i.e. importation of users was disabled? Did you check to see that the users exists in the system? we do support import, but we null out the passwords.. for security reasons. So you will need to re-set the passwords. The other remaining problem is the importation of XML data. It is still giving a Failed to flush BatchHandler error when I try to import the data. I have attached the relevant section of the log file so that you check and see if you can make anything of it. I'm not sure about that one.. it says duplicate value, but I guess that should have been treated as an update.. could you file a bug report for it? Yes its definitely a bug or at least an ugly behaviour. I should have reported it earlier - we saw the same importing unclean data from iHRIS in Rwanda. Once we cleaned the data up all was fine. Updates only happen when these are part of separate batches. Duplicates in the same import batch get stuck in the throat. -- Morten Thanks alot Morten! Edwin On Wed, Dec 18, 2013 at 1:23 AM, Morten Olav Hansen morte...@gmail.comwrote: Hi, From what you are saying, it sounds like all these issues are caused by using an in-memory db.. Are you sure that you are connecting to a database, and that the user has the proper privileges? Try to look for dhis2 home references in the log.. it should be early in the startup -- Morten On Wed, Dec 18, 2013 at 10:09 AM, Edwin Mulwa eddi...@gmail.comwrote: Hi, I have some major problems when I am trying to import meta-data and data from one DHIS2 instance to another. These are as follows: Issue 1: I have a DHIS2 installation running on a VMWare Virtual Machine. Using the Meta-Data Export/Import features in DHIS2, I have imported data from another DHIS2 instance running on a laptop into the VM successfully. Successfully in quotes because the log says it was imported successfully. However, when I shut down the VM and start it again, all the imported data is missing from the VM. That is, it is as if I did not import any data. In the data import screen, Dry Run option is set to No. I did take a snapshot of the VM immediately after importation and reverted to the snapshot after firing up the VM but nothing doing. Any ideas on why the data is not persisting on the VM? Note: Nothing I am doing on the VM persists! For example, I looked at the default user I have been using to access DHIS2 (admin:district) and added all the roles to the user. These additions to the user account were not reflected after restarting the VM. Important Note: I have noticed, by looking at the logs, that every time the DHIS2 instance starts up, it is attempting to run alot of table altering SQL commands, most of which are failing. I suspect that it is these commands that are interfering with my data. If you would like to see a copy of the log file, I will be happy to provide it. Issue 2: After importing the data, I log out of DHIS2 and try logging in using a different user whose info has been imported into the VM. This does not work. This is BEFORE I even shut down the machine. When I log back in using the default admin user, I can see the user accounts in the system. Note: I have seen from an old posting on the forum that importation of users was disabled. I don't know if this is still the case and therefore why I am having this problem? Issue 3: After importing meta-data, I immediately tried to import DATA from the other instance. Data import fails with a message that it was unable to clear the BatchHandler. Any help on the above issues will be appreciated. Edwin ___ Mailing list: https://launchpad.net/~dhis2-devs Post to : dhis2-devs@lists.launchpad.net Unsubscribe : https://launchpad.net/~dhis2-devs More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~dhis2-users Post to : dhis2-us...@lists.launchpad.net Unsubscribe : https://launchpad.net/~dhis2-users More help : https://help.launchpad.net/ListHelp ___ Mailing list:
