Thanks Morten. Let us know if any other details are required from our end in this regard.
Thanks Aamer. On Mon, Sep 12, 2016 at 6:01 PM, Morten Olav Hansen <mor...@dhis2.org> wrote: > That's interesting. > > @Lars any suggestions why this happens? I guess there is something wrong > with our mapping. Will have a look tomorrow. > > -- > Morten Olav Hansen > Senior Engineer, DHIS 2 > University of Oslo > http://www.dhis2.org > > On Mon, Sep 12, 2016 at 5:42 PM, Victor Garcia <vgarcia...@gmail.com> > wrote: > >> Hi Morten, Sultan, >> >> I think we know the reason of duplicates: categoryOptionGroups are shared >> between categoryOptionGroupSets, for example, ">= 15" is used in three >> categoryOptionGroupSets (Paediatrics, HIV and general). In the API, >> categoryOptionGroups appear duplicated as many times as they are re-used. >> >> We don't know if sharing categoryOptionGroups is something to avoid, but >> it seems to be the cause of this issue. >> >> Regards, >> >> Víctor >> >> On 12 September 2016 at 09:41, Morten Olav Hansen <mor...@dhis2.org> >> wrote: >> >>> Hi again >>> >>> Would it be possible to share your database with me? I haven't seen this >>> issues anywhere else. >>> >>> -- >>> Morten Olav Hansen >>> Senior Engineer, DHIS 2 >>> University of Oslo >>> http://www.dhis2.org >>> >>> On Mon, Sep 12, 2016 at 2:28 PM, Sultanahamar Mohammad < >>> sult...@thoughtworks.com> wrote: >>> >>> > Hi Morten, >>> > >>> > This is what our table is showing up when we say \d dataelement. I >>> think >>> > there is nothing suspicious out here. >>> > >>> > >>> > >>> > Regards, >>> > Sultan Ahamar. >>> > >>> > On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen <mor...@dhis2.org> >>> > wrote: >>> > >>> >> I'm not sure what is causing it, but it shouldn't be allowed to begin >>> >> with. Is there any constraint on the UID column in your database? >>> >> >>> >> [image: Inline image 1] >>> >> >>> >> -- >>> >> Morten Olav Hansen >>> >> Senior Engineer, DHIS 2 >>> >> University of Oslo >>> >> http://www.dhis2.org >>> >> >>> >> On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad < >>> >> sult...@thoughtworks.com> wrote: >>> >> >>> >>> Hi Morten, >>> >>> >>> >>> Hope you are doing good. Let us know your thoughts on the above issue >>> >>> and please let us know if you need any information that you might >>> require. >>> >>> >>> >>> Regards, >>> >>> Sultan Ahamar. >>> >>> >>> >>> On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad < >>> >>> sult...@thoughtworks.com> wrote: >>> >>> >>> >>>> Hi Morten, >>> >>>> >>> >>>> We have pulled latest code from 2.24 and tried import / export >>> again. >>> >>>> We have some interesting observations >>> >>>> >>> >>>> *DB snapshot:* >>> >>>> >>> >>>> >>> >>>> >>> >>>> There are no duplicate items in DB. We double checked. >>> >>>> >>> >>>> >>> >>>> >>> >>>> >>> >>>> *Entity endpoint snapshot:* >>> >>> >>>> >>> >>>> Interestingly we found few duplicate entities when we try to fetch >>> >>>> this data through the API. In Maintenance app, we are not able to >>> see these >>> >>>> duplicate entities. Does it also explain why we get repeating >>> entries (with >>> >>>> same UID's) in the full export JSON as well? Let us know on how to >>> proceed >>> >>>> and if you need more input. >>> >>>> >>> >>>> Thanks for all the help in advance. >>> >>>> >>> >>>> Regards, >>> >>>> Sultan Ahamar. >>> >>>> >>> >>>> >>> >>>> On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen < >>> mor...@dhis2.org> >>> >>>> wrote: >>> >>>> >>> >>>>> Hi Vanya >>> >>>>> >>> >>>>> This should now have been fixed in master and 224. It will now do a >>> >>>>> global UID check first, and report back any duplicates, if it >>> finds a >>> >>>>> duplicate it will remove them from the import and report back an >>> >>>>> `ErrorReport` (as it would do with other validation issues). So if >>> it finds >>> >>>>> a duplicate, you will need to set atomic mode to NONE to have it >>> import and >>> >>>>> ignore the duplicates. >>> >>>>> >>> >>>>> (btw, there was several duplicates in your file, not sure how that >>> >>>>> happened.. maybe something needs to be cleaned in your source >>> instance) >>> >>>>> >>> >>>>> -- >>> >>>>> Morten Olav Hansen >>> >>>>> Senior Engineer, DHIS 2 >>> >>>>> University of Oslo >>> >>>>> http://www.dhis2.org >>> >>>>> >>> >>>>> On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen < >>> mor...@dhis2.org> >>> >>>>> wrote: >>> >>>>> >>> >>>>>> Hi Vanya >>> >>>>>> >>> >>>>>> This should have been caught by the importer... but there are >>> several >>> >>>>>> duplicates without the file you sent me, MaO4Ik8f34O is used in 3 >>> category >>> >>>>>> option groups, same with oqeVQ71LCgY.. >>> >>>>>> >>> >>>>>> I will look into making the validation process more robust.. but >>> at >>> >>>>>> least you should know that this file have issues >>> >>>>>> >>> >>>>>> -- >>> >>>>>> Morten Olav Hansen >>> >>>>>> Senior Engineer, DHIS 2 >>> >>>>>> University of Oslo >>> >>>>>> http://www.dhis2.org >>> >>>>>> >>> >>>>>> On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen < >>> mor...@dhis2.org> >>> >>>>>> wrote: >>> >>>>>> >>> >>>>>>> Ok, thanks Vanya, I'm looking into it now >>> >>>>>>> >>> >>>>>>> -- >>> >>>>>>> Morten Olav Hansen >>> >>>>>>> Senior Engineer, DHIS 2 >>> >>>>>>> University of Oslo >>> >>>>>>> http://www.dhis2.org >>> >>>>>>> >>> >>>>>>> On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth < >>> van...@thoughtworks.com> >>> >>>>>>> wrote: >>> >>>>>>> >>> >>>>>>>> Hi Morten >>> >>>>>>>> >>> >>>>>>>> Thanks for the response. >>> >>>>>>>> >>> >>>>>>>> We ran into another issue while trying to do a full export and >>> >>>>>>>> import. >>> >>>>>>>> >>> >>>>>>>> As per the previous conversations we took care of all data >>> >>>>>>>> migrations (as per the new constraints on the DB- pretty much >>> caught by the >>> >>>>>>>> Importer during the validation phase itself). >>> >>>>>>>> >>> >>>>>>>> Having done that we run into this issue: >>> >>>>>>>> The 'categoryoptiongroup' import is failing with constraint >>> >>>>>>>> violation for the UID. We checked the data and there is no >>> repeating UID in >>> >>>>>>>> the database. >>> >>>>>>>> >>> >>>>>>>> The error text is attached for your reference, as well the >>> payload >>> >>>>>>>> used for the import. >>> >>>>>>>> >>> >>>>>>>> Thanks for all the help in advance. >>> >>>>>>>> >>> >>>>>>>> Regards >>> >>>>>>>> Vanya >>> >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen < >>> >>>>>>>> mor...@dhis2.org> wrote: >>> >>>>>>>> >>> >>>>>>>>> Thanks Aamer, I'm looking at a similar bug right now.. seems >>> some >>> >>>>>>>>> objects don't get their deps properly attached.. >>> >>>>>>>>> >>> >>>>>>>>> -- >>> >>>>>>>>> Morten Olav Hansen >>> >>>>>>>>> Senior Engineer, DHIS 2 >>> >>>>>>>>> University of Oslo >>> >>>>>>>>> http://www.dhis2.org >>> >>>>>>>>> >>> >>>>>>>>> On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed < >>> >>>>>>>>> aam...@thoughtworks.com> wrote: >>> >>>>>>>>> >>> >>>>>>>>>> Hi Morten, >>> >>>>>>>>>> >>> >>>>>>>>>> Please find the payload for below request >>> >>>>>>>>>> *curl -H "Content-Type: application/json" -X GET -u >>> >>>>>>>>>> username:password >>> >>>>>>>>>> "http://localhost:8888/api/24/metadata?filter=created:gt:201 >>> 6-07 >>> >>>>>>>>>> <http://localhost:8888/api/24/metadata?filter=created:gt:201 >>> 6-07>" > >>> >>>>>>>>>> metadata.224.newapi.created.json* >>> >>>>>>>>>> >>> >>>>>>>>>> Import the payload in a fresh instance for import using >>> >>>>>>>>>> >>> >>>>>>>>>> *curl -H "Content-Type: application/json" -X POST --data >>> >>>>>>>>>> @metadata.224.newapi.created.json -u admin:district >>> >>>>>>>>>> “http://localhost:8080/api/24/metadata?atomicMode=NONE >>> >>>>>>>>>> <http://localhost:8080/api/24/metadata?atomicMode=NONE>” > >>> >>>>>>>>>> output_created_besteffort.txt* >>> >>>>>>>>>> >>> >>>>>>>>>> If the same payload is run with atomicMode=ALL, I am getting >>> >>>>>>>>>> 'Invalid references' errors which is acceptable. But if it is >>> run with >>> >>>>>>>>>> atomicMode=NONE, it is throwing the error for which the >>> complete stack >>> >>>>>>>>>> trace is attached earlier. filename: ' >>> >>>>>>>>>> *output_created_besteffort_trace.txt'* >>> >>> >>>>>>>>>> >>> >>>>>>>>>> Let me know if the complete payload or any other details are >>> >>>>>>>>>> required from my end. >>> >>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> Thanks >>> >>>>>>>>>> Aamer. >>> >>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen < >>> >>>>>>>>>> mor...@dhis2.org> wrote: >>> >>>>>>>>>> >>> >>>>>>>>>>> Hi Aamer >>> >>>>>>>>>>> >>> >>>>>>>>>>> Could you please share the payload of the object where this >>> >>>>>>>>>>> happen? >>> >>>>>>>>>>> >>> >>>>>>>>>>> -- >>> >>>>>>>>>>> Morten Olav Hansen >>> >>>>>>>>>>> Senior Engineer, DHIS 2 >>> >>>>>>>>>>> University of Oslo >>> >>>>>>>>>>> http://www.dhis2.org >>> >>>>>>>>>>> >>> >>>>>>>>>>> On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed < >>> >>>>>>>>>>> aam...@thoughtworks.com> wrote: >>> >>>>>>>>>>> >>> >>>>>>>>>>>> Hi Morten, >>> >>>>>>>>>>>> >>> >>>>>>>>>>>> The Importer was run with atomicMode=ALL. Though the >>> payload is >>> >>>>>>>>>>>> huge, only few errors were shown in ImportSummary. As per >>> errors in >>> >>>>>>>>>>>> previous ImportSummary, I have done below. >>> >>>>>>>>>>>> 1) ensured the 'short name' field is unique for each of the >>> >>>>>>>>>>>> 'data elements' and 'data element group' as well >>> >>>>>>>>>>>> 2) UID of admin, Super User, tracked entity was changed in >>> >>>>>>>>>>>> payload to be same to where it is getting imported. >>> >>>>>>>>>>>> I have manually resolved them and ran the importer again >>> with >>> >>>>>>>>>>>> atomicMode=ALL. >>> >>>>>>>>>>>> >>> >>>>>>>>>>>> I got exception as attached in trace file >>> >>>>>>>>>>>> 'output_besteffort_trace.txt' >>> >>>>>>>>>>>> Would these kind of issues be known only after the import >>> has >>> >>>>>>>>>>>> failed? and we need to resolve them one-by-one and >>> as-and-when the importer >>> >>>>>>>>>>>> throws them? Is there any note which has the details about >>> the constraints >>> >>>>>>>>>>>> being introduced in a new DHIS version? >>> >>>>>>>>>>>> >>> >>>>>>>>>>>> Alternatively, If we export a filtered set of metadata >>> using ' >>> >>>>>>>>>>>> /api/24/metadata?filter=lastUpdated:gt:2016-05' and run the >>> >>>>>>>>>>>> importer with atomicMode=NONE >>> >>>>>>>>>>>> >>> >>>>>>>>>>>> *curl -H "Content-Type: application/json" -X POST --data >>> >>>>>>>>>>>> @metadata.224.newapi.json -u admin:district >>> >>>>>>>>>>>> “http://localhost:8080/api/24/metadata?atomicMode=NONE >>> >>>>>>>>>>>> <http://localhost:8080/api/24/metadata?atomicMode=NONE>” > >>> >>>>>>>>>>>> output_created_besteffort_trace.txt* >>> >>>>>>>>>>>> >>> >>>>>>>>>>>> Getting below exception in trace. Complete trace also >>> attached. >>> >>>>>>>>>>>> ----- >>> >>>>>>>>>>>> * INFO 2016-08-26 15:58:38,217 (admin) Creating 57 >>> object(s) of >>> >>>>>>>>>>>> type ReportTable (DefaultObjectBundleService.java >>> >>>>>>>>>>>> [qtp289378424-12]) org.hibernate.TransientObjectException: >>> >>>>>>>>>>>> object references an unsaved transient instance - save the >>> transient >>> >>>>>>>>>>>> instance before flushing: org.hisp.dhis.indicator.Indicator >>> >>>>>>>>>>>> ------ >>> >>>>>>>>>>>> Any suggestions please. >>> >>>>>>>>>>>> >>> >>>>>>>>>>>> >>> >>>>>>>>>>>> Thanks >>> >>>>>>>>>>>> Aamer. >>> >>>>>>>>>>>> >>> >>>>>>>>>>>> >>> >>>>>>>>>>>> On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen < >>> >>>>>>>>>>>> mor...@dhis2.org> wrote: >>> >>>>>>>>>>>> >>> >>>>>>>>>>>>> Hi Aamer >>> >>>>>>>>>>>>> >>> >>>>>>>>>>>>> Probably what is happening is that the new database has >>> some >>> >>>>>>>>>>>>> constraints that the old one doesn't. We have had some >>> issues with >>> >>>>>>>>>>>>> hibernate in the past, and for certain databases that >>> means that many of >>> >>>>>>>>>>>>> the constraints was not applied.. >>> >>>>>>>>>>>>> >>> >>>>>>>>>>>>> If you look at the messages, you will see that e.g >>> `shortName` >>> >>>>>>>>>>>>> must be unique for data elements.. the list of errors is >>> not big though, >>> >>>>>>>>>>>>> probably you can go through >>> [truncated for moderation] >>> _______________________________________________ >>> Mailing list: https://launchpad.net/~dhis2-devs >>> Post to : dhis2-devs@lists.launchpad.net >>> Unsubscribe : https://launchpad.net/~dhis2-devs >>> More help : https://help.launchpad.net/ListHelp >>> >>> >> >
_______________________________________________ Mailing list: https://launchpad.net/~dhis2-devs Post to : dhis2-devs@lists.launchpad.net Unsubscribe : https://launchpad.net/~dhis2-devs More help : https://help.launchpad.net/ListHelp