[Dhis2-users] API 2.25: Return Full Property Set

[Uwe Wahser]

Dear all,

I am having a bit of an issue when querying metadata from the api in json 
format and further processing them in kettle (Pentaho DI): DHIS2 only returns 
those properties in the json-package that are not NULL. However, the json 
converter in kettle needs the full property-set in order to create a tabular 
structure from the data (don't know the reason for that).

Is there a way to force the api to return the full property set, e.g. a secret 
flag? I'd actually expect to get the full list - even if the property is NULL 
-, at least when I specify a list of properties in the field-filter ...

Of course, I could also ask the kettle-guys to be a bit more flexible, but 
normally they are not as agile as the DHIS2-DEVs ;-)

Best regards,

Uwe

___
Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-users] API 2.25: Return Full Property Set

[Jason Pickering]

Hi Uwe
Have a look here

https://ci.dhis2.org/docs/master/en/developer/html/webapi_metadata_field_filter.html

and specifically the use of  the ?fields=:all paramater.

Regards,
Jason


On Fri, Dec 16, 2016 at 7:58 AM, Uwe Wahser  wrote:

> Dear all,
>
> I am having a bit of an issue when querying metadata from the api in json
> format and further processing them in kettle (Pentaho DI): DHIS2 only
> returns those properties in the json-package that are not NULL. However,
> the json converter in kettle needs the full property-set in order to create
> a tabular structure from the data (don't know the reason for that).
>
> Is there a way to force the api to return the full property set, e.g. a
> secret flag? I'd actually expect to get the full list - even if the
> property is NULL -, at least when I specify a list of properties in the
> field-filter ...
>
> Of course, I could also ask the kettle-guys to be a bit more flexible, but
> normally they are not as agile as the DHIS2-DEVs ;-)
>
> Best regards,
>
> Uwe
>
> ___
> Mailing list: https://launchpad.net/~dhis2-users
> Post to : dhis2-users@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~dhis2-users
> More help   : https://help.launchpad.net/ListHelp
>



-- 
Jason P. Pickering
email: jason.p.picker...@gmail.com
tel:+46764147049
___
Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-users] API 2.25: Return Full Property Set

[Uwe Wahser]

Thanks a lot, Jason, I will try that. I tried 'fields=' expecting this would be 
the same as 'fields=:all' , which is obviously wrong. 

That's a very useful information - do you mind including it into the manual?

Regards, Uwe

---

> Jason Pickering  hat am 16. Dezember 2016 um 
> 10:13 geschrieben:
> 
> 
> Hi Uwe
> Have a look here
> 
> https://ci.dhis2.org/docs/master/en/developer/html/webapi_metadata_field_filter.html
> 
> and specifically the use of  the ?fields=:all paramater.
> 
> Regards,
> Jason
> 
> 
> On Fri, Dec 16, 2016 at 7:58 AM, Uwe Wahser  wrote:
> 
> > Dear all,
> >
> > I am having a bit of an issue when querying metadata from the api in json
> > format and further processing them in kettle (Pentaho DI): DHIS2 only
> > returns those properties in the json-package that are not NULL. However,
> > the json converter in kettle needs the full property-set in order to create
> > a tabular structure from the data (don't know the reason for that).
> >
> > Is there a way to force the api to return the full property set, e.g. a
> > secret flag? I'd actually expect to get the full list - even if the
> > property is NULL -, at least when I specify a list of properties in the
> > field-filter ...
> >
> > Of course, I could also ask the kettle-guys to be a bit more flexible, but
> > normally they are not as agile as the DHIS2-DEVs ;-)
> >
> > Best regards,
> >
> > Uwe
> >
> > ___
> > Mailing list: https://launchpad.net/~dhis2-users
> > Post to : dhis2-users@lists.launchpad.net
> > Unsubscribe : https://launchpad.net/~dhis2-users
> > More help   : https://help.launchpad.net/ListHelp
> >
> 
> 
> 
> -- 
> Jason P. Pickering
> email: jason.p.picker...@gmail.com
> tel:+46764147049

___
Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-users] API 2.25: Return Full Property Set

[Jason Pickering]

Hi Uwe,
I think the manual is correct, as it specifies how the paramater should be
called.

In general, we discourage actually using that "all" paramater, because
things can change from version to version (as you have noticed). Its better
to make explicit calls for the fields which you require for each object,
using something like /api/26/indicators?fields=id,name

where each required field is specified explicitly. This should also
decrease the payloads which the ETL process would need to work with. You
might want to have a look at that approach.

I would be happy to update the manual, but not really sure what is missing
in this case?

Regards,
Jason




On Fri, Dec 16, 2016 at 8:24 AM, Uwe Wahser  wrote:

> Thanks a lot, Jason, I will try that. I tried 'fields=' expecting this
> would be the same as 'fields=:all' , which is obviously wrong.
>
> That's a very useful information - do you mind including it into the
> manual?
>
> Regards, Uwe
>
> ---
>
> > Jason Pickering  hat am 16. Dezember 2016
> um 10:13 geschrieben:
> >
> >
> > Hi Uwe
> > Have a look here
> >
> > https://ci.dhis2.org/docs/master/en/developer/html/
> webapi_metadata_field_filter.html
> >
> > and specifically the use of  the ?fields=:all paramater.
> >
> > Regards,
> > Jason
> >
> >
> > On Fri, Dec 16, 2016 at 7:58 AM, Uwe Wahser  wrote:
> >
> > > Dear all,
> > >
> > > I am having a bit of an issue when querying metadata from the api in
> json
> > > format and further processing them in kettle (Pentaho DI): DHIS2 only
> > > returns those properties in the json-package that are not NULL.
> However,
> > > the json converter in kettle needs the full property-set in order to
> create
> > > a tabular structure from the data (don't know the reason for that).
> > >
> > > Is there a way to force the api to return the full property set, e.g. a
> > > secret flag? I'd actually expect to get the full list - even if the
> > > property is NULL -, at least when I specify a list of properties in the
> > > field-filter ...
> > >
> > > Of course, I could also ask the kettle-guys to be a bit more flexible,
> but
> > > normally they are not as agile as the DHIS2-DEVs ;-)
> > >
> > > Best regards,
> > >
> > > Uwe
> > >
> > > ___
> > > Mailing list: https://launchpad.net/~dhis2-users
> > > Post to : dhis2-users@lists.launchpad.net
> > > Unsubscribe : https://launchpad.net/~dhis2-users
> > > More help   : https://help.launchpad.net/ListHelp
> > >
> >
> >
> >
> > --
> > Jason P. Pickering
> > email: jason.p.picker...@gmail.com
> > tel:+46764147049
>



-- 
Jason P. Pickering
email: jason.p.picker...@gmail.com
tel:+46764147049
___
Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-users] API 2.25: Return Full Property Set

[Uwe Wahser]

Sorry, on a closer look it only seems to work for empty collections. Properties 
of the object itself (e.g. email, comments of organizationUnits) are still left 
out.

The floor is open again for other suggestions :-) 

Regards, Uwe

> Uwe Wahser  hat am 16. Dezember 2016 um 10:24 geschrieben:
> 
> 
> Thanks a lot, Jason, I will try that. I tried 'fields=' expecting this would 
> be the same as 'fields=:all' , which is obviously wrong. 
> 
> That's a very useful information - do you mind including it into the manual?
> 
> Regards, Uwe
> 
> ---
> 
> > Jason Pickering  hat am 16. Dezember 2016 um 
> > 10:13 geschrieben:
> > 
> > 
> > Hi Uwe
> > Have a look here
> > 
> > https://ci.dhis2.org/docs/master/en/developer/html/webapi_metadata_field_filter.html
> > 
> > and specifically the use of  the ?fields=:all paramater.
> > 
> > Regards,
> > Jason
> > 
> > 
> > On Fri, Dec 16, 2016 at 7:58 AM, Uwe Wahser  wrote:
> > 
> > > Dear all,
> > >
> > > I am having a bit of an issue when querying metadata from the api in json
> > > format and further processing them in kettle (Pentaho DI): DHIS2 only
> > > returns those properties in the json-package that are not NULL. However,
> > > the json converter in kettle needs the full property-set in order to 
> > > create
> > > a tabular structure from the data (don't know the reason for that).
> > >
> > > Is there a way to force the api to return the full property set, e.g. a
> > > secret flag? I'd actually expect to get the full list - even if the
> > > property is NULL -, at least when I specify a list of properties in the
> > > field-filter ...
> > >
> > > Of course, I could also ask the kettle-guys to be a bit more flexible, but
> > > normally they are not as agile as the DHIS2-DEVs ;-)
> > >
> > > Best regards,
> > >
> > > Uwe
> > >
> > > ___
> > > Mailing list: https://launchpad.net/~dhis2-users
> > > Post to : dhis2-users@lists.launchpad.net
> > > Unsubscribe : https://launchpad.net/~dhis2-users
> > > More help   : https://help.launchpad.net/ListHelp
> > >
> > 
> > 
> > 
> > -- 
> > Jason P. Pickering
> > email: jason.p.picker...@gmail.com
> > tel:+46764147049
> 
> ___
> Mailing list: https://launchpad.net/~dhis2-users
> Post to : dhis2-users@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~dhis2-users
> More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-users] API 2.25: Return Full Property Set

[Jason Pickering]

Hi Uwe,
Could you give a specific example from the demo server with an API call,
and what is not working as you expect?

Regards,
Jason


On Fri, Dec 16, 2016 at 8:48 AM, Uwe Wahser  wrote:

> Sorry, on a closer look it only seems to work for empty collections.
> Properties of the object itself (e.g. email, comments of organizationUnits)
> are still left out.
>
> The floor is open again for other suggestions :-)
>
> Regards, Uwe
>
> > Uwe Wahser  hat am 16. Dezember 2016 um 10:24
> geschrieben:
> >
> >
> > Thanks a lot, Jason, I will try that. I tried 'fields=' expecting this
> would be the same as 'fields=:all' , which is obviously wrong.
> >
> > That's a very useful information - do you mind including it into the
> manual?
> >
> > Regards, Uwe
> >
> > ---
> >
> > > Jason Pickering  hat am 16. Dezember
> 2016 um 10:13 geschrieben:
> > >
> > >
> > > Hi Uwe
> > > Have a look here
> > >
> > > https://ci.dhis2.org/docs/master/en/developer/html/
> webapi_metadata_field_filter.html
> > >
> > > and specifically the use of  the ?fields=:all paramater.
> > >
> > > Regards,
> > > Jason
> > >
> > >
> > > On Fri, Dec 16, 2016 at 7:58 AM, Uwe Wahser  wrote:
> > >
> > > > Dear all,
> > > >
> > > > I am having a bit of an issue when querying metadata from the api in
> json
> > > > format and further processing them in kettle (Pentaho DI): DHIS2 only
> > > > returns those properties in the json-package that are not NULL.
> However,
> > > > the json converter in kettle needs the full property-set in order to
> create
> > > > a tabular structure from the data (don't know the reason for that).
> > > >
> > > > Is there a way to force the api to return the full property set,
> e.g. a
> > > > secret flag? I'd actually expect to get the full list - even if the
> > > > property is NULL -, at least when I specify a list of properties in
> the
> > > > field-filter ...
> > > >
> > > > Of course, I could also ask the kettle-guys to be a bit more
> flexible, but
> > > > normally they are not as agile as the DHIS2-DEVs ;-)
> > > >
> > > > Best regards,
> > > >
> > > > Uwe
> > > >
> > > > ___
> > > > Mailing list: https://launchpad.net/~dhis2-users
> > > > Post to : dhis2-users@lists.launchpad.net
> > > > Unsubscribe : https://launchpad.net/~dhis2-users
> > > > More help   : https://help.launchpad.net/ListHelp
> > > >
> > >
> > >
> > >
> > > --
> > > Jason P. Pickering
> > > email: jason.p.picker...@gmail.com
> > > tel:+46764147049
> >
> > ___
> > Mailing list: https://launchpad.net/~dhis2-users
> > Post to : dhis2-users@lists.launchpad.net
> > Unsubscribe : https://launchpad.net/~dhis2-users
> > More help   : https://help.launchpad.net/ListHelp
>



-- 
Jason P. Pickering
email: jason.p.picker...@gmail.com
tel:+46764147049
___
Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-users] API 2.25: Return Full Property Set

[Uwe Wahser]

Hi Jason,

simple example: 
https://play.dhis2.org/demo/api/organisationUnits.json?paging=false&filter=level:eq:3&fields=name,comment

What I'd expect would be a json package with name & comment for ALL 
organisationUnits, even when the comment is NULL. What I get is a comment 
property for Badjia (which I created for the example), for all the other I only 
get the name property:
[{"name":"Badjia","comment":"This is a 
comment."},{"name":"Bagruwa"},{"name":"Baoma"}...]

Converting this package in kettle would result to an error message that 50 (or 
whatever) instances of name were found, but only 1 instance of comment. kettle 
expects always the complete set of properties for all objects. Not sure how R 
or others handle this ...

If there was a query parameter like returnNull=true , that could give back the 
null properties like {"name":"Bagruwa","comment":""}, it would be possible to 
work with these packages. Actually I had expected this to be the default 
behaviour, when specifying fields.

Best regards, 

Uwe

---

> Jason Pickering  hat am 16. Dezember 2016 um 
> 12:05 geschrieben:
> 
> 
> Hi Uwe,
> Could you give a specific example from the demo server with an API call,
> and what is not working as you expect?
> 
> Regards,
> Jason
> 
> 
> On Fri, Dec 16, 2016 at 8:48 AM, Uwe Wahser  wrote:
> 
> > Sorry, on a closer look it only seems to work for empty collections.
> > Properties of the object itself (e.g. email, comments of organizationUnits)
> > are still left out.
> >
> > The floor is open again for other suggestions :-)
> >
> > Regards, Uwe
> >
> > > Uwe Wahser  hat am 16. Dezember 2016 um 10:24
> > geschrieben:
> > >
> > >
> > > Thanks a lot, Jason, I will try that. I tried 'fields=' expecting this
> > would be the same as 'fields=:all' , which is obviously wrong.
> > >
> > > That's a very useful information - do you mind including it into the
> > manual?
> > >
> > > Regards, Uwe
> > >
> > > ---
> > >
> > > > Jason Pickering  hat am 16. Dezember
> > 2016 um 10:13 geschrieben:
> > > >
> > > >
> > > > Hi Uwe
> > > > Have a look here
> > > >
> > > > https://ci.dhis2.org/docs/master/en/developer/html/
> > webapi_metadata_field_filter.html
> > > >
> > > > and specifically the use of  the ?fields=:all paramater.
> > > >
> > > > Regards,
> > > > Jason
> > > >
> > > >
> > > > On Fri, Dec 16, 2016 at 7:58 AM, Uwe Wahser  wrote:
> > > >
> > > > > Dear all,
> > > > >
> > > > > I am having a bit of an issue when querying metadata from the api in
> > json
> > > > > format and further processing them in kettle (Pentaho DI): DHIS2 only
> > > > > returns those properties in the json-package that are not NULL.
> > However,
> > > > > the json converter in kettle needs the full property-set in order to
> > create
> > > > > a tabular structure from the data (don't know the reason for that).
> > > > >
> > > > > Is there a way to force the api to return the full property set,
> > e.g. a
> > > > > secret flag? I'd actually expect to get the full list - even if the
> > > > > property is NULL -, at least when I specify a list of properties in
> > the
> > > > > field-filter ...
> > > > >
> > > > > Of course, I could also ask the kettle-guys to be a bit more
> > flexible, but
> > > > > normally they are not as agile as the DHIS2-DEVs ;-)
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Uwe
> > > > >
> > > > > ___
> > > > > Mailing list: https://launchpad.net/~dhis2-users
> > > > > Post to : dhis2-users@lists.launchpad.net
> > > > > Unsubscribe : https://launchpad.net/~dhis2-users
> > > > > More help   : https://help.launchpad.net/ListHelp
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Jason P. Pickering
> > > > email: jason.p.picker...@gmail.com
> > > > tel:+46764147049
> > >
> > > ___
> > > Mailing list: https://launchpad.net/~dhis2-users
> > > Post to : dhis2-users@lists.launchpad.net
> > > Unsubscribe : https://launchpad.net/~dhis2-users
> > > More help   : https://help.launchpad.net/ListHelp
> >
> 
> 
> 
> -- 
> Jason P. Pickering
> email: jason.p.picker...@gmail.com
> tel:+46764147049

___
Mailing list: https://launchpad.net/~dhis2-users
Post to : dhis2-users@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-users
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-users] API 2.25: Return Full Property Set

[Jason Pickering]

Hi Uwe,

OK, I sort of see the problem you have, but am pretty sure its not a DHIS2
problem.  Are you saying that this is new behaviour, and it was not like
this before?  I just checked an old instance I have which is running 2.17,
and it is exactly the same, so not sure this has changed any time in the
recent past.

Maybe you should try and use CSV?

https://play.dhis2.org/demo/api/organisationUnits.csv?paging=false&filter=level:eq:3&fields=name,comment

does produce an extra column (with NULLS).

>From the R standpoint (as well as other languages I think), its really up
to you to parse the payload as you need it, but its handled quite easily
for this case in R. Other times, the nested lists need to be flattened, if
that is what you require, but I do not see this as  DHIS2 issue really. It
would make little sense to include all of these NULLs where they do not
exist, as it could drastically increase the size of the payload. That is
why they are not included, except where not null.

[image: Inline image 1]



On Fri, Dec 16, 2016 at 10:41 AM, Uwe Wahser  wrote:

> Hi Jason,
>
> simple example:
> https://play.dhis2.org/demo/api/organisationUnits.json?
> paging=false&filter=level:eq:3&fields=name,comment
>
> What I'd expect would be a json package with name & comment for ALL
> organisationUnits, even when the comment is NULL. What I get is a comment
> property for Badjia (which I created for the example), for all the other I
> only get the name property:
> [{"name":"Badjia","comment":"This is a comment."},{"name":"Bagruwa"},
> {"name":"Baoma"}...]
>
> Converting this package in kettle would result to an error message that 50
> (or whatever) instances of name were found, but only 1 instance of comment.
> kettle expects always the complete set of properties for all objects. Not
> sure how R or others handle this ...
>
> If there was a query parameter like returnNull=true , that could give back
> the null properties like {"name":"Bagruwa","comment":""}, it would be
> possible to work with these packages. Actually I had expected this to be
> the default behaviour, when specifying fields.
>
> Best regards,
>
> Uwe
>
> ---
>
> > Jason Pickering  hat am 16. Dezember 2016
> um 12:05 geschrieben:
> >
> >
> > Hi Uwe,
> > Could you give a specific example from the demo server with an API call,
> > and what is not working as you expect?
> >
> > Regards,
> > Jason
> >
> >
> > On Fri, Dec 16, 2016 at 8:48 AM, Uwe Wahser  wrote:
> >
> > > Sorry, on a closer look it only seems to work for empty collections.
> > > Properties of the object itself (e.g. email, comments of
> organizationUnits)
> > > are still left out.
> > >
> > > The floor is open again for other suggestions :-)
> > >
> > > Regards, Uwe
> > >
> > > > Uwe Wahser  hat am 16. Dezember 2016 um 10:24
> > > geschrieben:
> > > >
> > > >
> > > > Thanks a lot, Jason, I will try that. I tried 'fields=' expecting
> this
> > > would be the same as 'fields=:all' , which is obviously wrong.
> > > >
> > > > That's a very useful information - do you mind including it into the
> > > manual?
> > > >
> > > > Regards, Uwe
> > > >
> > > > ---
> > > >
> > > > > Jason Pickering  hat am 16. Dezember
> > > 2016 um 10:13 geschrieben:
> > > > >
> > > > >
> > > > > Hi Uwe
> > > > > Have a look here
> > > > >
> > > > > https://ci.dhis2.org/docs/master/en/developer/html/
> > > webapi_metadata_field_filter.html
> > > > >
> > > > > and specifically the use of  the ?fields=:all paramater.
> > > > >
> > > > > Regards,
> > > > > Jason
> > > > >
> > > > >
> > > > > On Fri, Dec 16, 2016 at 7:58 AM, Uwe Wahser  wrote:
> > > > >
> > > > > > Dear all,
> > > > > >
> > > > > > I am having a bit of an issue when querying metadata from the
> api in
> > > json
> > > > > > format and further processing them in kettle (Pentaho DI): DHIS2
> only
> > > > > > returns those properties in the json-package that are not NULL.
> > > However,
> > > > > > the json converter in kettle needs the full property-set in
> order to
> > > create
> > > > > > a tabular structure from the data (don't know the reason for
> that).
> > > > > >
> > > > > > Is there a way to force the api to return the full property set,
> > > e.g. a
> > > > > > secret flag? I'd actually expect to get the full list - even if
> the
> > > > > > property is NULL -, at least when I specify a list of properties
> in
> > > the
> > > > > > field-filter ...
> > > > > >
> > > > > > Of course, I could also ask the kettle-guys to be a bit more
> > > flexible, but
> > > > > > normally they are not as agile as the DHIS2-DEVs ;-)
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Uwe
> > > > > >
> > > > > > ___
> > > > > > Mailing list: https://launchpad.net/~dhis2-users
> > > > > > Post to : dhis2-users@lists.launchpad.net
> > > > > > Unsubscribe : https://launchpad.net/~dhis2-users
> > > > > > More help   : https://help.launchpad.net/ListHelp
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > >

Re: [Dhis2-users] API 2.25: Return Full Property Set

[Uwe Wahser]

Hi Jason,

It's not new - I had that problem already last year, but managed to work 
around.  Now I wanted to use more properties and the workaround doesn't 
work for that.


I had decided against csv back then, but forgot why. Must have been a good 
reason, because normally I prefer csv :-)


I also think that it is not a DHIS issue and that the kettle converter 
should be able to do that. But DHIS developers are usually more responsive :-)


I was hoping that it wouldn't be a big deal, as it works with csv. 
Implementing it as a parameter would also give the choice of using that 
feature if needed, thus avoiding inflated payloads per default.


Anyway, I think I'll enjoy our Xmas party now. Maybe I that gives me better 
ideas on Monday :-D


Thanks for looking at it,

Uwe





Sent with AquaMail for Android
http://www.aqua-mail.com


On 16 December 2016 12:54:17 Jason Pickering  
wrote:



Hi Uwe,

OK, I sort of see the problem you have, but am pretty sure its not a DHIS2
problem.  Are you saying that this is new behaviour, and it was not like
this before?  I just checked an old instance I have which is running 2.17,
and it is exactly the same, so not sure this has changed any time in the
recent past.

Maybe you should try and use CSV?

https://play.dhis2.org/demo/api/organisationUnits.csv?paging=false&filter=level:eq:3&fields=name,comment

does produce an extra column (with NULLS).

From the R standpoint (as well as other languages I think), its really up
to you to parse the payload as you need it, but its handled quite easily
for this case in R. Other times, the nested lists need to be flattened, if
that is what you require, but I do not see this as  DHIS2 issue really. It
would make little sense to include all of these NULLs where they do not
exist, as it could drastically increase the size of the payload. That is
why they are not included, except where not null.

[image: Inline image 1]



On Fri, Dec 16, 2016 at 10:41 AM, Uwe Wahser  wrote:


Hi Jason,

simple example:
https://play.dhis2.org/demo/api/organisationUnits.json?
paging=false&filter=level:eq:3&fields=name,comment

What I'd expect would be a json package with name & comment for ALL
organisationUnits, even when the comment is NULL. What I get is a comment
property for Badjia (which I created for the example), for all the other I
only get the name property:
[{"name":"Badjia","comment":"This is a comment."},{"name":"Bagruwa"},
{"name":"Baoma"}...]

Converting this package in kettle would result to an error message that 50
(or whatever) instances of name were found, but only 1 instance of comment.
kettle expects always the complete set of properties for all objects. Not
sure how R or others handle this ...

If there was a query parameter like returnNull=true , that could give back
the null properties like {"name":"Bagruwa","comment":""}, it would be
possible to work with these packages. Actually I had expected this to be
the default behaviour, when specifying fields.

Best regards,

Uwe

---

> Jason Pickering  hat am 16. Dezember 2016
um 12:05 geschrieben:
>
>
> Hi Uwe,
> Could you give a specific example from the demo server with an API call,
> and what is not working as you expect?
>
> Regards,
> Jason
>
>
> On Fri, Dec 16, 2016 at 8:48 AM, Uwe Wahser  wrote:
>
> > Sorry, on a closer look it only seems to work for empty collections.
> > Properties of the object itself (e.g. email, comments of
organizationUnits)
> > are still left out.
> >
> > The floor is open again for other suggestions :-)
> >
> > Regards, Uwe
> >
> > > Uwe Wahser  hat am 16. Dezember 2016 um 10:24
> > geschrieben:
> > >
> > >
> > > Thanks a lot, Jason, I will try that. I tried 'fields=' expecting
this
> > would be the same as 'fields=:all' , which is obviously wrong.
> > >
> > > That's a very useful information - do you mind including it into the
> > manual?
> > >
> > > Regards, Uwe
> > >
> > > ---
> > >
> > > > Jason Pickering  hat am 16. Dezember
> > 2016 um 10:13 geschrieben:
> > > >
> > > >
> > > > Hi Uwe
> > > > Have a look here
> > > >
> > > > https://ci.dhis2.org/docs/master/en/developer/html/
> > webapi_metadata_field_filter.html
> > > >
> > > > and specifically the use of  the ?fields=:all paramater.
> > > >
> > > > Regards,
> > > > Jason
> > > >
> > > >
> > > > On Fri, Dec 16, 2016 at 7:58 AM, Uwe Wahser  wrote:
> > > >
> > > > > Dear all,
> > > > >
> > > > > I am having a bit of an issue when querying metadata from the
api in
> > json
> > > > > format and further processing them in kettle (Pentaho DI): DHIS2
only
> > > > > returns those properties in the json-package that are not NULL.
> > However,
> > > > > the json converter in kettle needs the full property-set in
order to
> > create
> > > > > a tabular structure from the data (don't know the reason for
that).
> > > > >
> > > > > Is there a way to force the api to return the full property set,
> > e.g. a
> > > > > secret flag? I'd actually expect to get the full list - even if
the
> > > > > property is NUL

Re: [Dhis2-users] API 2.25: Return Full Property Set

[Ocaya Stephen]

Dear Uwe,

This is a json issue. Recently i had guidance from lars that the xml
returns even the empty properties.

Apparently, suggestions in the link below talks of leaving out properties
with null values.

https://google.github.io/styleguide/jsoncstyleguide.xml

The only work around:

1. Try using xml format ( if ok with U)
2. You will need to take back missing properties within your codes (
probably your previous work around)






On Dec 16, 2016 15:33, "Uwe Wahser"  wrote:

> Hi Jason,
>
> It's not new - I had that problem already last year, but managed to work
> around.  Now I wanted to use more properties and the workaround doesn't
> work for that.
>
> I had decided against csv back then, but forgot why. Must have been a good
> reason, because normally I prefer csv :-)
>
> I also think that it is not a DHIS issue and that the kettle converter
> should be able to do that. But DHIS developers are usually more responsive
> :-)
>
> I was hoping that it wouldn't be a big deal, as it works with csv.
> Implementing it as a parameter would also give the choice of using that
> feature if needed, thus avoiding inflated payloads per default.
>
> Anyway, I think I'll enjoy our Xmas party now. Maybe I that gives me
> better ideas on Monday :-D
>
> Thanks for looking at it,
>
> Uwe
>
>
>
> Sent with AquaMail for Android
> http://www.aqua-mail.com
>
> On 16 December 2016 12:54:17 Jason Pickering 
> wrote:
>
>> Hi Uwe,
>>
>> OK, I sort of see the problem you have, but am pretty sure its not a
>> DHIS2 problem.  Are you saying that this is new behaviour, and it was not
>> like this before?  I just checked an old instance I have which is running
>> 2.17, and it is exactly the same, so not sure this has changed any time in
>> the recent past.
>>
>> Maybe you should try and use CSV?
>>
>> https://play.dhis2.org/demo/api/organisationUnits.csv?
>> paging=false&filter=level:eq:3&fields=name,comment
>>
>> does produce an extra column (with NULLS).
>>
>> From the R standpoint (as well as other languages I think), its really up
>> to you to parse the payload as you need it, but its handled quite easily
>> for this case in R. Other times, the nested lists need to be flattened, if
>> that is what you require, but I do not see this as  DHIS2 issue really. It
>> would make little sense to include all of these NULLs where they do not
>> exist, as it could drastically increase the size of the payload. That is
>> why they are not included, except where not null.
>>
>> [image: Inline image 1]
>>
>>
>>
>> On Fri, Dec 16, 2016 at 10:41 AM, Uwe Wahser  wrote:
>>
>>> Hi Jason,
>>>
>>> simple example:
>>> https://play.dhis2.org/demo/api/organisationUnits.json?pagin
>>> g=false&filter=level:eq:3&fields=name,comment
>>>
>>> What I'd expect would be a json package with name & comment for ALL
>>> organisationUnits, even when the comment is NULL. What I get is a comment
>>> property for Badjia (which I created for the example), for all the other I
>>> only get the name property:
>>> [{"name":"Badjia","comment":"This is a comment."},{"name":"Bagruwa"},
>>> {"name":"Baoma"}...]
>>>
>>> Converting this package in kettle would result to an error message that
>>> 50 (or whatever) instances of name were found, but only 1 instance of
>>> comment. kettle expects always the complete set of properties for all
>>> objects. Not sure how R or others handle this ...
>>>
>>> If there was a query parameter like returnNull=true , that could give
>>> back the null properties like {"name":"Bagruwa","comment":""}, it would
>>> be possible to work with these packages. Actually I had expected this to be
>>> the default behaviour, when specifying fields.
>>>
>>> Best regards,
>>>
>>> Uwe
>>>
>>> ---
>>>
>>> > Jason Pickering  hat am 16. Dezember
>>> 2016 um 12:05 geschrieben:
>>> >
>>> >
>>> > Hi Uwe,
>>> > Could you give a specific example from the demo server with an API
>>> call,
>>> > and what is not working as you expect?
>>> >
>>> > Regards,
>>> > Jason
>>> >
>>> >
>>> > On Fri, Dec 16, 2016 at 8:48 AM, Uwe Wahser  wrote:
>>> >
>>> > > Sorry, on a closer look it only seems to work for empty collections.
>>> > > Properties of the object itself (e.g. email, comments of
>>> organizationUnits)
>>> > > are still left out.
>>> > >
>>> > > The floor is open again for other suggestions :-)
>>> > >
>>> > > Regards, Uwe
>>> > >
>>> > > > Uwe Wahser  hat am 16. Dezember 2016 um 10:24
>>> > > geschrieben:
>>> > > >
>>> > > >
>>> > > > Thanks a lot, Jason, I will try that. I tried 'fields=' expecting
>>> this
>>> > > would be the same as 'fields=:all' , which is obviously wrong.
>>> > > >
>>> > > > That's a very useful information - do you mind including it into
>>> the
>>> > > manual?
>>> > > >
>>> > > > Regards, Uwe
>>> > > >
>>> > > > ---
>>> > > >
>>> > > > > Jason Pickering  hat am 16.
>>> Dezember
>>> > > 2016 um 10:13 geschrieben:
>>> > > > >
>>> > > > >
>>> > > > > Hi Uwe
>>> > > > > Have a look here
>>> > > > >
>>> > > > > https://ci.dhis2.org/docs/master/en/dev

Re: [Dhis2-users] API 2.25: Return Full Property Set

[uwe wahser]

Hi Jason & Ocaya,

thanks for your input. Just to follow up: Kettle just released a new 
version (7.0) last month. According to their forum it includes a fix for 
the bug. Didn't have time to try this out, yet, but that seems to be the 
solution then.


For the record: the xml api doesn't seem to bring the full feature set 
either. Also I feel that the overhead of redundant strings is much 
higher than in json. I also decided against csv back then, because I 
couldn't figure out how to return deep structures (e.g. Parent ID of an 
OrgUnit).


Have a wonderful Christmas (if you have) and a great start into the new 
year!


Regards - Uwe

---

Am 16.12.2016 um 22:56 schrieb Ocaya Stephen:

Dear Uwe,

This is a json issue. Recently i had guidance from lars that the xml 
returns even the empty properties.


Apparently, suggestions in the link below talks of leaving out 
properties with null values.


https://google.github.io/styleguide/jsoncstyleguide.xml

The only work around:

1. Try using xml format ( if ok with U)
2. You will need to take back missing properties within your codes ( 
probably your previous work around)







On Dec 16, 2016 15:33, "Uwe Wahser" > wrote:


Hi Jason,

It's not new - I had that problem already last year, but managed
to work around.  Now I wanted to use more properties and the
workaround doesn't work for that.

I had decided against csv back then, but forgot why. Must have
been a good reason, because normally I prefer csv :-)

I also think that it is not a DHIS issue and that the kettle
converter should be able to do that. But DHIS developers are
usually more responsive :-)

I was hoping that it wouldn't be a big deal, as it works with csv.
Implementing it as a parameter would also give the choice of using
that feature if needed, thus avoiding inflated payloads per default.

Anyway, I think I'll enjoy our Xmas party now. Maybe I that gives
me better ideas on Monday :-D

Thanks for looking at it,

Uwe



Sent with AquaMail for Android
http://www.aqua-mail.com

On 16 December 2016 12:54:17 Jason Pickering
mailto:jason.p.picker...@gmail.com>>
wrote:


Hi Uwe,

OK, I sort of see the problem you have, but am pretty sure its
not a DHIS2 problem. Are you saying that this is new behaviour,
and it was not like this before?  I just checked an old instance
I have which is running 2.17, and it is exactly the same, so not
sure this has changed any time in the recent past.

Maybe you should try and use CSV?


https://play.dhis2.org/demo/api/organisationUnits.csv?paging=false&filter=level:eq:3&fields=name,comment



does produce an extra column (with NULLS).

From the R standpoint (as well as other languages I think), its
really up to you to parse the payload as you need it, but its
handled quite easily for this case in R. Other times, the nested
lists need to be flattened, if that is what you require, but I do
not see this as  DHIS2 issue really. It would make little sense
to include all of these NULLs where they do not exist, as it
could drastically increase the size of the payload. That is why
they are not included, except where not null.

Inline image 1



On Fri, Dec 16, 2016 at 10:41 AM, Uwe Wahser mailto:u...@wahser.de>> wrote:

Hi Jason,

simple example:

https://play.dhis2.org/demo/api/organisationUnits.json?paging=false&filter=level:eq:3&fields=name,comment



What I'd expect would be a json package with name & comment
for ALL organisationUnits, even when the comment is NULL.
What I get is a comment property for Badjia (which I created
for the example), for all the other I only get the name property:
[{"name":"Badjia","comment":"This is a
comment."},{"name":"Bagruwa"},{"name":"Baoma"}...]

Converting this package in kettle would result to an error
message that 50 (or whatever) instances of name were found,
but only 1 instance of comment. kettle expects always the
complete set of properties for all objects. Not sure how R or
others handle this ...

If there was a query parameter like returnNull=true , that
could give back the null properties like
{"name":"Bagruwa","comment":""}, it would be possible to work
with these packages. Actually I had expected this to be the
default behaviour, when specifying fields.

Best regards,

Uwe

---

> Jason Pickering mailto:jason.p.picker...@gmail.com>> hat am 16. Dezember
2016 um 12:05 geschrieben:
>
>
> Hi Uwe,
> Could you giv