Hi Aamer Lars knows what is wrong, he will provide a fix shortly :)
-- Morten Olav Hansen Senior Engineer, DHIS 2 University of Oslo http://www.dhis2.org On Thu, Sep 15, 2016 at 12:26 PM, Aamer Mohammed <aam...@thoughtworks.com> wrote: > Thanks Morten. Let us know if any other details are required from our end > in this regard. > > Thanks > Aamer. > > On Mon, Sep 12, 2016 at 6:01 PM, Morten Olav Hansen <mor...@dhis2.org> > wrote: > >> That's interesting. >> >> @Lars any suggestions why this happens? I guess there is something wrong >> with our mapping. Will have a look tomorrow. >> >> -- >> Morten Olav Hansen >> Senior Engineer, DHIS 2 >> University of Oslo >> http://www.dhis2.org >> >> On Mon, Sep 12, 2016 at 5:42 PM, Victor Garcia <vgarcia...@gmail.com> >> wrote: >> >>> Hi Morten, Sultan, >>> >>> I think we know the reason of duplicates: categoryOptionGroups are >>> shared between categoryOptionGroupSets, for example, ">= 15" is used in >>> three categoryOptionGroupSets (Paediatrics, HIV and general). In the API, >>> categoryOptionGroups appear duplicated as many times as they are re-used. >>> >>> We don't know if sharing categoryOptionGroups is something to avoid, but >>> it seems to be the cause of this issue. >>> >>> Regards, >>> >>> Víctor >>> >>> On 12 September 2016 at 09:41, Morten Olav Hansen <mor...@dhis2.org> >>> wrote: >>> >>>> Hi again >>>> >>>> Would it be possible to share your database with me? I haven't seen this >>>> issues anywhere else. >>>> >>>> -- >>>> Morten Olav Hansen >>>> Senior Engineer, DHIS 2 >>>> University of Oslo >>>> http://www.dhis2.org >>>> >>>> On Mon, Sep 12, 2016 at 2:28 PM, Sultanahamar Mohammad < >>>> sult...@thoughtworks.com> wrote: >>>> >>>> > Hi Morten, >>>> > >>>> > This is what our table is showing up when we say \d dataelement. I >>>> think >>>> > there is nothing suspicious out here. >>>> > >>>> > >>>> > >>>> > Regards, >>>> > Sultan Ahamar. >>>> > >>>> > On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen <mor...@dhis2.org> >>>> > wrote: >>>> > >>>> >> I'm not sure what is causing it, but it shouldn't be allowed to begin >>>> >> with. Is there any constraint on the UID column in your database? >>>> >> >>>> >> [image: Inline image 1] >>>> >> >>>> >> -- >>>> >> Morten Olav Hansen >>>> >> Senior Engineer, DHIS 2 >>>> >> University of Oslo >>>> >> http://www.dhis2.org >>>> >> >>>> >> On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad < >>>> >> sult...@thoughtworks.com> wrote: >>>> >> >>>> >>> Hi Morten, >>>> >>> >>>> >>> Hope you are doing good. Let us know your thoughts on the above >>>> issue >>>> >>> and please let us know if you need any information that you might >>>> require. >>>> >>> >>>> >>> Regards, >>>> >>> Sultan Ahamar. >>>> >>> >>>> >>> On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad < >>>> >>> sult...@thoughtworks.com> wrote: >>>> >>> >>>> >>>> Hi Morten, >>>> >>>> >>>> >>>> We have pulled latest code from 2.24 and tried import / export >>>> again. >>>> >>>> We have some interesting observations >>>> >>>> >>>> >>>> *DB snapshot:* >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> There are no duplicate items in DB. We double checked. >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> *Entity endpoint snapshot:* >>>> >>>> >>>> >>>> >>>> Interestingly we found few duplicate entities when we try to fetch >>>> >>>> this data through the API. In Maintenance app, we are not able to >>>> see these >>>> >>>> duplicate entities. Does it also explain why we get repeating >>>> entries (with >>>> >>>> same UID's) in the full export JSON as well? Let us know on how >>>> to proceed >>>> >>>> and if you need more input. >>>> >>>> >>>> >>>> Thanks for all the help in advance. >>>> >>>> >>>> >>>> Regards, >>>> >>>> Sultan Ahamar. >>>> >>>> >>>> >>>> >>>> >>>> On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen < >>>> mor...@dhis2.org> >>>> >>>> wrote: >>>> >>>> >>>> >>>>> Hi Vanya >>>> >>>>> >>>> >>>>> This should now have been fixed in master and 224. It will now do >>>> a >>>> >>>>> global UID check first, and report back any duplicates, if it >>>> finds a >>>> >>>>> duplicate it will remove them from the import and report back an >>>> >>>>> `ErrorReport` (as it would do with other validation issues). So >>>> if it finds >>>> >>>>> a duplicate, you will need to set atomic mode to NONE to have it >>>> import and >>>> >>>>> ignore the duplicates. >>>> >>>>> >>>> >>>>> (btw, there was several duplicates in your file, not sure how that >>>> >>>>> happened.. maybe something needs to be cleaned in your source >>>> instance) >>>> >>>>> >>>> >>>>> -- >>>> >>>>> Morten Olav Hansen >>>> >>>>> Senior Engineer, DHIS 2 >>>> >>>>> University of Oslo >>>> >>>>> http://www.dhis2.org >>>> >>>>> >>>> >>>>> On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen < >>>> mor...@dhis2.org> >>>> >>>>> wrote: >>>> >>>>> >>>> >>>>>> Hi Vanya >>>> >>>>>> >>>> >>>>>> This should have been caught by the importer... but there are >>>> several >>>> >>>>>> duplicates without the file you sent me, MaO4Ik8f34O is used in >>>> 3 category >>>> >>>>>> option groups, same with oqeVQ71LCgY.. >>>> >>>>>> >>>> >>>>>> I will look into making the validation process more robust.. but >>>> at >>>> >>>>>> least you should know that this file have issues >>>> >>>>>> >>>> >>>>>> -- >>>> >>>>>> Morten Olav Hansen >>>> >>>>>> Senior Engineer, DHIS 2 >>>> >>>>>> University of Oslo >>>> >>>>>> http://www.dhis2.org >>>> >>>>>> >>>> >>>>>> On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen < >>>> mor...@dhis2.org> >>>> >>>>>> wrote: >>>> >>>>>> >>>> >>>>>>> Ok, thanks Vanya, I'm looking into it now >>>> >>>>>>> >>>> >>>>>>> -- >>>> >>>>>>> Morten Olav Hansen >>>> >>>>>>> Senior Engineer, DHIS 2 >>>> >>>>>>> University of Oslo >>>> >>>>>>> http://www.dhis2.org >>>> >>>>>>> >>>> >>>>>>> On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth < >>>> van...@thoughtworks.com> >>>> >>>>>>> wrote: >>>> >>>>>>> >>>> >>>>>>>> Hi Morten >>>> >>>>>>>> >>>> >>>>>>>> Thanks for the response. >>>> >>>>>>>> >>>> >>>>>>>> We ran into another issue while trying to do a full export and >>>> >>>>>>>> import. >>>> >>>>>>>> >>>> >>>>>>>> As per the previous conversations we took care of all data >>>> >>>>>>>> migrations (as per the new constraints on the DB- pretty much >>>> caught by the >>>> >>>>>>>> Importer during the validation phase itself). >>>> >>>>>>>> >>>> >>>>>>>> Having done that we run into this issue: >>>> >>>>>>>> The 'categoryoptiongroup' import is failing with constraint >>>> >>>>>>>> violation for the UID. We checked the data and there is no >>>> repeating UID in >>>> >>>>>>>> the database. >>>> >>>>>>>> >>>> >>>>>>>> The error text is attached for your reference, as well the >>>> payload >>>> >>>>>>>> used for the import. >>>> >>>>>>>> >>>> >>>>>>>> Thanks for all the help in advance. >>>> >>>>>>>> >>>> >>>>>>>> Regards >>>> >>>>>>>> Vanya >>>> >>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen < >>>> >>>>>>>> mor...@dhis2.org> wrote: >>>> >>>>>>>> >>>> >>>>>>>>> Thanks Aamer, I'm looking at a similar bug right now.. seems >>>> some >>>> >>>>>>>>> objects don't get their deps properly attached.. >>>> >>>>>>>>> >>>> >>>>>>>>> -- >>>> >>>>>>>>> Morten Olav Hansen >>>> >>>>>>>>> Senior Engineer, DHIS 2 >>>> >>>>>>>>> University of Oslo >>>> >>>>>>>>> http://www.dhis2.org >>>> >>>>>>>>> >>>> >>>>>>>>> On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed < >>>> >>>>>>>>> aam...@thoughtworks.com> wrote: >>>> >>>>>>>>> >>>> >>>>>>>>>> Hi Morten, >>>> >>>>>>>>>> >>>> >>>>>>>>>> Please find the payload for below request >>>> >>>>>>>>>> *curl -H "Content-Type: application/json" -X GET -u >>>> >>>>>>>>>> username:password >>>> >>>>>>>>>> "http://localhost:8888/api/24/metadata?filter=created:gt:201 >>>> 6-07 >>>> >>>>>>>>>> <http://localhost:8888/api/24/metadata?filter=created:gt:201 >>>> 6-07>" > >>>> >>>>>>>>>> metadata.224.newapi.created.json* >>>> >>>>>>>>>> >>>> >>>>>>>>>> Import the payload in a fresh instance for import using >>>> >>>>>>>>>> >>>> >>>>>>>>>> *curl -H "Content-Type: application/json" -X POST --data >>>> >>>>>>>>>> @metadata.224.newapi.created.json -u admin:district >>>> >>>>>>>>>> “http://localhost:8080/api/24/metadata?atomicMode=NONE >>>> >>>>>>>>>> <http://localhost:8080/api/24/metadata?atomicMode=NONE>” > >>>> >>>>>>>>>> output_created_besteffort.txt* >>>> >>>>>>>>>> >>>> >>>>>>>>>> If the same payload is run with atomicMode=ALL, I am getting >>>> >>>>>>>>>> 'Invalid references' errors which is acceptable. But if it >>>> is run with >>>> >>>>>>>>>> atomicMode=NONE, it is throwing the error for which the >>>> complete stack >>>> >>>>>>>>>> trace is attached earlier. filename: ' >>>> >>>>>>>>>> *output_created_besteffort_trace.txt'* >>>> >>>> >>>>>>>>>> >>>> >>>>>>>>>> Let me know if the complete payload or any other details are >>>> >>>>>>>>>> required from my end. >>>> >>>>>>>>>> >>>> >>>>>>>>>> >>>> >>>>>>>>>> Thanks >>>> >>>>>>>>>> Aamer. >>>> >>>>>>>>>> >>>> >>>>>>>>>> >>>> >>>>>>>>>> >>>> >>>>>>>>>> On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen < >>>> >>>>>>>>>> mor...@dhis2.org> wrote: >>>> >>>>>>>>>> >>>> >>>>>>>>>>> Hi Aamer >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> Could you please share the payload of the object where this >>>> >>>>>>>>>>> happen? >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> -- >>>> >>>>>>>>>>> Morten Olav Hansen >>>> >>>>>>>>>>> Senior Engineer, DHIS 2 >>>> >>>>>>>>>>> University of Oslo >>>> >>>>>>>>>>> http://www.dhis2.org >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed < >>>> >>>>>>>>>>> aam...@thoughtworks.com> wrote: >>>> >>>>>>>>>>> >>>> >>>>>>>>>>>> Hi Morten, >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>> The Importer was run with atomicMode=ALL. Though the >>>> payload is >>>> >>>>>>>>>>>> huge, only few errors were shown in ImportSummary. As per >>>> errors in >>>> >>>>>>>>>>>> previous ImportSummary, I have done below. >>>> >>>>>>>>>>>> 1) ensured the 'short name' field is unique for each of the >>>> >>>>>>>>>>>> 'data elements' and 'data element group' as well >>>> >>>>>>>>>>>> 2) UID of admin, Super User, tracked entity was changed in >>>> >>>>>>>>>>>> payload to be same to where it is getting imported. >>>> >>>>>>>>>>>> I have manually resolved them and ran the importer again >>>> with >>>> >>>>>>>>>>>> atomicMode=ALL. >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>> I got exception as attached in trace file >>>> >>>>>>>>>>>> 'output_besteffort_trace.txt' >>>> >>>>>>>>>>>> Would these kind of issues be known only after the import >>>> has >>>> >>>>>>>>>>>> failed? and we need to resolve them one-by-one and >>>> as-and-when the importer >>>> >>>>>>>>>>>> throws them? Is there any note which has the details about >>>> the constraints >>>> >>>>>>>>>>>> being introduced in a new DHIS version? >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>> Alternatively, If we export a filtered set of metadata >>>> using ' >>>> >>>>>>>>>>>> /api/24/metadata?filter=lastUpdated:gt:2016-05' and run >>>> the >>>> >>>>>>>>>>>> importer with atomicMode=NONE >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>> *curl -H "Content-Type: application/json" -X POST --data >>>> >>>>>>>>>>>> @metadata.224.newapi.json -u admin:district >>>> >>>>>>>>>>>> “http://localhost:8080/api/24/metadata?atomicMode=NONE >>>> >>>>>>>>>>>> <http://localhost:8080/api/24/metadata?atomicMode=NONE>” > >>>> >>>>>>>>>>>> output_created_besteffort_trace.txt* >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>> Getting below exception in trace. Complete trace also >>>> attached. >>>> >>>>>>>>>>>> ----- >>>> >>>>>>>>>>>> * INFO 2016-08-26 15:58:38,217 (admin) Creating 57 >>>> object(s) of >>>> >>>>>>>>>>>> type ReportTable (DefaultObjectBundleService.java >>>> >>>>>>>>>>>> [qtp289378424-12]) org.hibernate.TransientObjectException: >>>> >>>>>>>>>>>> object references an unsaved transient instance - save the >>>> transient >>>> >>>>>>>>>>>> instance before flushing: org.hisp.dhis.indicator.Indica >>>> tor >>>> >>>>>>>>>>>> ------ >>>> >>>>>>>>>>>> Any suggestions please. >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>> Thanks >>>> >>>>>>>>>>>> Aamer. >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>> On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen < >>>> >>>>>>>>>>>> mor...@dhis2.org> wrote: >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>>> Hi Aamer >>>> >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> Probably what is happening is that the new database has >>>> some >>>> >>>>>>>>>>>>> constraints that the old one doesn't. We have had some >>>> issues with >>>> >>>>>>>>>>>>> hibernate in the past, and for certain databases that >>>> means that many of >>>> >>>>>>>>>>>>> the constraints was not applied.. >>>> >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> If you look at the messages, you will see that e.g >>>> `shortName` >>>> >>>>>>>>>>>>> must be unique for data elements.. the list of errors is >>>> not big though, >>>> >>>>>>>>>>>>> probably you can go through >>>> [truncated for moderation] >>>> _______________________________________________ >>>> Mailing list: https://launchpad.net/~dhis2-devs >>>> Post to : dhis2-devs@lists.launchpad.net >>>> Unsubscribe : https://launchpad.net/~dhis2-devs >>>> More help : https://help.launchpad.net/ListHelp >>>> >>>> >>> >> >
_______________________________________________ Mailing list: https://launchpad.net/~dhis2-devs Post to : dhis2-devs@lists.launchpad.net Unsubscribe : https://launchpad.net/~dhis2-devs More help : https://help.launchpad.net/ListHelp