Re: RFS: python-jdata and python-bjdata - JData/BJData encoders and decoders for python

[Qianqian Fang]

On 7/29/20 5:18 PM, Anton Gladky wrote:

Hi,

uploaded yesterday. Now it is waiting for review in NEW queue.



hi Anton

yes, I saw the email message. thank you so much for the help!

Qianqian




Best regards

Anton


Am Mo., 27. Juli 2020 um 22:15 Uhr schrieb Qianqian Fang 
mailto:fan...@gmail.com>>:


On 7/22/20 4:33 PM, Anton Gladky wrote:


> I currently have "|Testsuite: autopkgtest-pkg-python|" in
control and "|export PYBUILD_TEST_ARGS=test/|" in rules, the CI
pipeline seems to be ok with autopkgtest for pybj

Ah, OK. I missed it.

Please fix the binary inclusion (do not forget to rename the
tarball then).



hi Anton

I noticed that although the auto-test did run for the pyjdata
package, it reported 0 test. I migrated my test unit to use
unittest, and now it runs properly.

to make this update, I created a new upstream release (v0.3.6),
and imported it to salsa, see new commits here

https://salsa.debian.org/science-team/pyjdata/-/commits/master

let em know if you see anything else worth fixing.

Qianqian




Best regards

Anton


Am Mi., 22. Juli 2020 um 01:41 Uhr schrieb Qianqian Fang
mailto:fan...@gmail.com>>:

On 7/21/20 4:39 PM, Anton Gladky wrote:

Hi Qianqian,

some general notes to both packages:



thanks, see my below updates



- Please go through ALL files and put licenses/copyrights
into the d/copyright.


done



- Remove python2-binaries. This python version is not
supported any more.



done



- Remove all binaries from the code (ods-files)



forgive me, what are ods-files?



- pysdate - empty clean file is not needed



removed.



- Add DEP-8 autopkgstests



can you point me to an example project how this is done?

I currently have "|Testsuite: autopkgtest-pkg-python|" in
control and "|export PYBUILD_TEST_ARGS=test/|" in rules, the
CI pipeline seems to be ok with autopkgtest for pybj

https://salsa.debian.org/science-team/pybj/-/pipelines/158112


for pyjdata, two tests were failed due to the dependency to
python3-bjdata (which I believe can be fixed once both
packages are uploaded)

https://salsa.debian.org/science-team/pyjdata/-/pipelines/158115



Please pay attention, I did not compile and test your
packages. Please fix all lintian
errors and warnings, if they exist.



most of those should have been fixed, let me know if you see
something that worth fixing.

thanks


Qianqian




Best regards

Anton


    Am Fr., 17. Juli 2020 um 17:34 Uhr schrieb Qianqian Fang
mailto:fan...@gmail.com>>:

hi Anton

just to let you know that I've fixed the numpy-abi error
for pybj


https://salsa.debian.org/science-team/pybj/-/commit/818484c1eb462fa1abd80951132a95dcd048641d

https://mentors.debian.net/package/pybj

I also updated pyjdata dependency list:

https://mentors.debian.net/package/pyjdata

let me know if you have any additional questions
regarding these two packages.

    Qianqian

On 7/14/20 6:07 PM, Qianqian Fang wrote:

On 7/14/20 5:11 PM, Anton Gladky wrote:

Hi.

Thanks for your contribution to Debian. I have just
some doubts about
usefulness for Debian and possible popularity of those
two projects.



hi Anton

thanks for your comment. happy to explain. Changed
message title from "JSON/..." to "JData/BJData encoders
and decoders" to avoid further confusions.

see my self-introduction in a previous thread

https://lists.debian.org/debian-science/2020/06/msg6.html

https://bugs.debian.org/cgi-bin/pkgreport.cgi?submitter=fangqq%40gmail.com

I am working on packaging a number research software
produced from my lab and research projects. I have
already submitted 5 octave-related projects, mentored
by Rafael Laboissière (CCed) via the Debian Octave
Group. I intend to maintain these packages in the
future (already doing so for Fedora).

These two python modules are part of a bigger project
that I initiated last year (http://openjdata.org). They
allow python users to read/write JData-annotated data
files produced by my MATLAB toolbox JSONLab
(https://github.com/fangq/jsonlab , about 46000
downloads on Matlab file exchange and ~1000 clones/week
on github). This work is partly funded by my NIH
(N

Re: RFS: python-jdata and python-bjdata - JData/BJData encoders and decoders for python

[Qianqian Fang]

On 7/22/20 4:33 PM, Anton Gladky wrote:
>
> > I currently have "|Testsuite: autopkgtest-pkg-python|" in control
> and "|export PYBUILD_TEST_ARGS=test/|" in rules, the CI pipeline seems
> to be ok with autopkgtest for pybj
>
> Ah, OK. I missed it.
>
> Please fix the binary inclusion (do not forget to rename the tarball
> then).


hi Anton

I noticed that although the auto-test did run for the pyjdata package,
it reported 0 test. I migrated my test unit to use unittest, and now it
runs properly.

to make this update, I created a new upstream release (v0.3.6), and
imported it to salsa, see new commits here

https://salsa.debian.org/science-team/pyjdata/-/commits/master

let em know if you see anything else worth fixing.

Qianqian


>
> Best regards
>
> Anton
>
>
> Am Mi., 22. Juli 2020 um 01:41 Uhr schrieb Qianqian Fang
> mailto:fan...@gmail.com>>:
>
> On 7/21/20 4:39 PM, Anton Gladky wrote:
>> Hi Qianqian,
>>
>> some general notes to both packages:
>
>
> thanks, see my below updates
>
>
>> - Please go through ALL files and put licenses/copyrights into
>> the d/copyright.
>
> done
>
>
>> - Remove python2-binaries. This python version is not supported
>> any more.
>
>
> done
>
>
>> - Remove all binaries from the code (ods-files)
>
>
> forgive me, what are ods-files?
>
>
>> - pysdate - empty clean file is not needed
>
>
> removed.
>
>
>> - Add DEP-8 autopkgstests
>
>
> can you point me to an example project how this is done?
>
> I currently have "|Testsuite: autopkgtest-pkg-python|" in control
> and "|export PYBUILD_TEST_ARGS=test/|" in rules, the CI pipeline
> seems to be ok with autopkgtest for pybj
>
> https://salsa.debian.org/science-team/pybj/-/pipelines/158112
>
>
> for pyjdata, two tests were failed due to the dependency to
> python3-bjdata (which I believe can be fixed once both packages
> are uploaded)
>
> https://salsa.debian.org/science-team/pyjdata/-/pipelines/158115
>
>
>> Please pay attention, I did not compile and test your packages.
>> Please fix all lintian
>> errors and warnings, if they exist.
>
>
> most of those should have been fixed, let me know if you see
> something that worth fixing.
>
> thanks
>
>
> Qianqian
>
>
>>
>> Best regards
>>
>> Anton
>>
>>
>> Am Fr., 17. Juli 2020 um 17:34 Uhr schrieb Qianqian Fang
>> mailto:fan...@gmail.com>>:
>>
>> hi Anton
>>
>> just to let you know that I've fixed the numpy-abi error for pybj
>>
>>     
>> https://salsa.debian.org/science-team/pybj/-/commit/818484c1eb462fa1abd80951132a95dcd048641d
>>
>> https://mentors.debian.net/package/pybj
>>
>> I also updated pyjdata dependency list:
>>
>> https://mentors.debian.net/package/pyjdata
>>
>> let me know if you have any additional questions regarding
>> these two packages.
>>
>> Qianqian
>>
>> On 7/14/20 6:07 PM, Qianqian Fang wrote:
>>> On 7/14/20 5:11 PM, Anton Gladky wrote:
>>>> Hi.
>>>>
>>>> Thanks for your contribution to Debian. I have just some
>>>> doubts about
>>>> usefulness for Debian and possible popularity of those two
>>>> projects.
>>>
>>>
>>> hi Anton
>>>
>>> thanks for your comment. happy to explain. Changed message
>>> title from "JSON/..." to "JData/BJData encoders and
>>> decoders" to avoid further confusions.
>>>
>>> see my self-introduction in a previous thread
>>>
>>> https://lists.debian.org/debian-science/2020/06/msg6.html
>>> 
>>> https://bugs.debian.org/cgi-bin/pkgreport.cgi?submitter=fangqq%40gmail.com
>>>
>>> I am working on packaging a number research software
>>> produced from my lab and research projects. I have already
>>> submitted 5 octave-related projects, mentored by Rafael
>>> Laboissière (CCed) via the Debian Octave Group. I intend to
>>> maintain these packages in the future (already doing so for
>>> Fedora).
>>>
>>> These two python modules are part of a bigger project that I
>>>  

Re: RFS: python-jdata and python-bjdata - JData/BJData encoders and decoders for python

[Qianqian Fang]

On 7/23/20 2:01 PM, Anton Gladky wrote:

Up to you.



done. imported a new release (0.2.6) with the ods file removed.

https://salsa.debian.org/science-team/pybj/-/network/master




Qianqian Fang mailto:fan...@gmail.com>> schrieb am 
Do., 23. Juli 2020, 17:57:



just to clarify - do you mean do a new release? or do a repack to
this current release?

thanks





Anton

Re: RFS: python-jdata and python-bjdata - JData/BJData encoders and decoders for python

[Qianqian Fang]

On 7/23/20 11:39 AM, Anton Gladky wrote:
> Qianqian Fang mailto:fan...@gmail.com>> schrieb am
> Do., 23. Juli 2020, 01:17:
>
> https://github.com/Iotic-Labs/py-ubjson/tree/dev-contrib/test
>
> I can make a new release to get rid of this file, is that ok? or I
> need to do a repack?
>
> Yes, please.


just to clarify - do you mean do a new release? or do a repack to this
current release?

thanks


>
>
> Anton

Re: RFS: python-jdata and python-bjdata - JData/BJData encoders and decoders for python

[Qianqian Fang]

On 7/22/20 4:33 PM, Anton Gladky wrote:
> Hi,
>
> > forgive me, what are ods-files?
>
> https://salsa.debian.org/science-team/pybj/-/blob/master/test/perf_results.ods
>
> Please fix the binary inclusion (do not forget to rename the tarball
> then).


I see. it looks like this file came from the upstream repo that I forked
from

https://github.com/Iotic-Labs/py-ubjson/tree/dev-contrib/test

I can make a new release to get rid of this file, is that ok? or I need
to do a repack?

thanks, let me know

Qianqian


>
> Best regards
>
> Anton
>
>
> Am Mi., 22. Juli 2020 um 01:41 Uhr schrieb Qianqian Fang
> mailto:fan...@gmail.com>>:
>
> On 7/21/20 4:39 PM, Anton Gladky wrote:
>> Hi Qianqian,
>>
>> some general notes to both packages:
>
>
> thanks, see my below updates
>
>
>> - Please go through ALL files and put licenses/copyrights into
>> the d/copyright.
>
> done
>
>
>> - Remove python2-binaries. This python version is not supported
>> any more.
>
>
> done
>
>
>> - Remove all binaries from the code (ods-files)
>
>
> forgive me, what are ods-files?
>
>
>> - pysdate - empty clean file is not needed
>
>
> removed.
>
>
>> - Add DEP-8 autopkgstests
>
>
> can you point me to an example project how this is done?
>
> I currently have "|Testsuite: autopkgtest-pkg-python|" in control
> and "|export PYBUILD_TEST_ARGS=test/|" in rules, the CI pipeline
> seems to be ok with autopkgtest for pybj
>
> https://salsa.debian.org/science-team/pybj/-/pipelines/158112
>
>
> for pyjdata, two tests were failed due to the dependency to
> python3-bjdata (which I believe can be fixed once both packages
> are uploaded)
>
> https://salsa.debian.org/science-team/pyjdata/-/pipelines/158115
>
>
>> Please pay attention, I did not compile and test your packages.
>> Please fix all lintian
>>     errors and warnings, if they exist.
>
>
> most of those should have been fixed, let me know if you see
> something that worth fixing.
>
> thanks
>
>
> Qianqian
>
>
>>
>> Best regards
>>
>> Anton
>>
>>
>> Am Fr., 17. Juli 2020 um 17:34 Uhr schrieb Qianqian Fang
>> mailto:fan...@gmail.com>>:
>>
>> hi Anton
>>
>> just to let you know that I've fixed the numpy-abi error for pybj
>>
>> 
>> https://salsa.debian.org/science-team/pybj/-/commit/818484c1eb462fa1abd80951132a95dcd048641d
>>
>> https://mentors.debian.net/package/pybj
>>
>> I also updated pyjdata dependency list:
>>
>> https://mentors.debian.net/package/pyjdata
>>
>> let me know if you have any additional questions regarding
>> these two packages.
>>
>> Qianqian
>>
>> On 7/14/20 6:07 PM, Qianqian Fang wrote:
>>> On 7/14/20 5:11 PM, Anton Gladky wrote:
>>>> Hi.
>>>>
>>>> Thanks for your contribution to Debian. I have just some
>>>> doubts about
>>>> usefulness for Debian and possible popularity of those two
>>>> projects.
>>>
>>>
>>> hi Anton
>>>
>>> thanks for your comment. happy to explain. Changed message
>>> title from "JSON/..." to "JData/BJData encoders and
>>> decoders" to avoid further confusions.
>>>
>>> see my self-introduction in a previous thread
>>>
>>> https://lists.debian.org/debian-science/2020/06/msg6.html
>>> 
>>> https://bugs.debian.org/cgi-bin/pkgreport.cgi?submitter=fangqq%40gmail.com
>>>
>>> I am working on packaging a number research software
>>> produced from my lab and research projects. I have already
>>> submitted 5 octave-related projects, mentored by Rafael
>>> Laboissière (CCed) via the Debian Octave Group. I intend to
>>> maintain these packages in the future (already doing so for
>>> Fedora).
>>>
>>> These two python modules are part of a bigger project that I
>>> initiated last year (http://openjdata.org). They allow
>>> python users to read/write JData-annotated data files
>>> produced by my MATLAB toolbox JSONLab
>>> (https://github.com/fangq/json

Re: RFS: python-jdata and python-bjdata - JData/BJData encoders and decoders for python

[Qianqian Fang]

On 7/21/20 4:39 PM, Anton Gladky wrote:

Hi Qianqian,

some general notes to both packages:



thanks, see my below updates


- Please go through ALL files and put licenses/copyrights into the 
d/copyright.


done



- Remove python2-binaries. This python version is not supported any more.



done



- Remove all binaries from the code (ods-files)



forgive me, what are ods-files?



- pysdate - empty clean file is not needed



removed.



- Add DEP-8 autopkgstests



can you point me to an example project how this is done?

I currently have "|Testsuite: autopkgtest-pkg-python|" in control and 
"|export PYBUILD_TEST_ARGS=test/|" in rules, the CI pipeline seems to be 
ok with autopkgtest for pybj


https://salsa.debian.org/science-team/pybj/-/pipelines/158112


for pyjdata, two tests were failed due to the dependency to 
python3-bjdata (which I believe can be fixed once both packages are 
uploaded)


https://salsa.debian.org/science-team/pyjdata/-/pipelines/158115


Please pay attention, I did not compile and test your packages. Please 
fix all lintian

errors and warnings, if they exist.



most of those should have been fixed, let me know if you see something 
that worth fixing.


thanks


Qianqian




Best regards

Anton


Am Fr., 17. Juli 2020 um 17:34 Uhr schrieb Qianqian Fang 
mailto:fan...@gmail.com>>:


hi Anton

just to let you know that I've fixed the numpy-abi error for pybj


https://salsa.debian.org/science-team/pybj/-/commit/818484c1eb462fa1abd80951132a95dcd048641d

https://mentors.debian.net/package/pybj

I also updated pyjdata dependency list:

https://mentors.debian.net/package/pyjdata

let me know if you have any additional questions regarding these
two packages.

Qianqian

On 7/14/20 6:07 PM, Qianqian Fang wrote:

On 7/14/20 5:11 PM, Anton Gladky wrote:

Hi.

Thanks for your contribution to Debian. I have just some doubts
about
usefulness for Debian and possible popularity of those two projects.



hi Anton

thanks for your comment. happy to explain. Changed message title
from "JSON/..." to "JData/BJData encoders and decoders" to avoid
further confusions.

see my self-introduction in a previous thread

https://lists.debian.org/debian-science/2020/06/msg6.html
https://bugs.debian.org/cgi-bin/pkgreport.cgi?submitter=fangqq%40gmail.com

I am working on packaging a number research software produced
from my lab and research projects. I have already submitted 5
octave-related projects, mentored by Rafael Laboissière (CCed)
via the Debian Octave Group. I intend to maintain these packages
in the future (already doing so for Fedora).

These two python modules are part of a bigger project that I
initiated last year (http://openjdata.org). They allow python
users to read/write JData-annotated data files produced by my
MATLAB toolbox JSONLab (https://github.com/fangq/jsonlab , about
46000 downloads on Matlab file exchange and ~1000 clones/week on
github). This work is partly funded by my NIH (National Institute
of Health) grants and broader dissemination is part of the
project goals.



Do you know how many people can be interested in these two
libraries?
It looks like at least one of them duplicates the functionality
of the built-in
JSON module. Could you please shortly describe the benefits of both
of them before we start to evaluate it technically?



The *python-bjdata* project was extended from *python-ubjson* -
an existing Debian package. Unfortunately, the UBJSON spec
(http://ubjson.org), despite being broadly used, is no longer
actively maintained. I started a fork earlier this year to
continue the development of this specification, and python-bjdata
is a parser that is compliant to the BJData spec.

The jdata/bjdata framework is not a duplication to JSON -
instead, it defines a systematic way to encode basic data
structures into JSON/UBJSON/BJData serializable forms.

The detailed specifications, examples and rationales can be found at

http://openjdata.org/wiki/

in a way, the jdata module is similar to *json-tricks* but aimed
at a more systematic/standardized way to annotate complex data
(such as graphs, maps, ND arrays ...) for sharing, exchange and
reuse.

https://packages.debian.org/buster/python/python3-json-tricks

the bjdata module is a binary JSON format (similar to UBJSON, and
msgpack) to store binary and strongly typed hierarchical data.
The differences are highlighted in this github tracker

https://github.com/ubjson/universal-binary-json/issues/109

Although these two modules were recently developed, we are
beginning to integrate those in my other tools including
*iso2mesh* <http://iso2mesh.sf.net/>, *jsonlab*
<http://openjdata.org/jsonlab> and *mcx* <h

Re: RFS: python-jdata and python-bjdata - JData/BJData encoders and decoders for python

[Qianqian Fang]

hi Anton

just to let you know that I've fixed the numpy-abi error for pybj

https://salsa.debian.org/science-team/pybj/-/commit/818484c1eb462fa1abd80951132a95dcd048641d

https://mentors.debian.net/package/pybj

I also updated pyjdata dependency list:

https://mentors.debian.net/package/pyjdata

let me know if you have any additional questions regarding these two
packages.

Qianqian

On 7/14/20 6:07 PM, Qianqian Fang wrote:
> On 7/14/20 5:11 PM, Anton Gladky wrote:
>> Hi.
>>
>> Thanks for your contribution to Debian. I have just some doubts about
>> usefulness for Debian and possible popularity of those two projects.
>
>
> hi Anton
>
> thanks for your comment. happy to explain. Changed message title from
> "JSON/..." to "JData/BJData encoders and decoders" to avoid further
> confusions.
>
> see my self-introduction in a previous thread
>
> https://lists.debian.org/debian-science/2020/06/msg6.html
> https://bugs.debian.org/cgi-bin/pkgreport.cgi?submitter=fangqq%40gmail.com
>
> I am working on packaging a number research software produced from my
> lab and research projects. I have already submitted 5 octave-related
> projects, mentored by Rafael Laboissière (CCed) via the Debian Octave
> Group. I intend to maintain these packages in the future (already
> doing so for Fedora).
>
> These two python modules are part of a bigger project that I initiated
> last year (http://openjdata.org). They allow python users to
> read/write JData-annotated data files produced by my MATLAB toolbox
> JSONLab (https://github.com/fangq/jsonlab , about 46000 downloads on
> Matlab file exchange and ~1000 clones/week on github). This work is
> partly funded by my NIH (National Institute of Health) grants and
> broader dissemination is part of the project goals.
>
>
>> Do you know how many people can be interested in these two libraries?
>> It looks like at least one of them duplicates the functionality of
>> the built-in
>> JSON module. Could you please shortly describe the benefits of both
>> of them before we start to evaluate it technically?
>
>
> The *python-bjdata* project was extended from *python-ubjson* - an
> existing Debian package. Unfortunately, the UBJSON spec
> (http://ubjson.org), despite being broadly used, is no longer actively
> maintained. I started a fork earlier this year to continue the
> development of this specification, and python-bjdata is a parser that
> is compliant to the BJData spec.
>
> The jdata/bjdata framework is not a duplication to JSON - instead, it
> defines a systematic way to encode basic data structures into
> JSON/UBJSON/BJData serializable forms.
>
> The detailed specifications, examples and rationales can be found at
>
> http://openjdata.org/wiki/
>
> in a way, the jdata module is similar to *json-tricks* but aimed at a
> more systematic/standardized way to annotate complex data (such as
> graphs, maps, ND arrays ...) for sharing, exchange and reuse.
>
> https://packages.debian.org/buster/python/python3-json-tricks
>
> the bjdata module is a binary JSON format (similar to UBJSON, and
> msgpack) to store binary and strongly typed hierarchical data. The
> differences are highlighted in this github tracker
>
> https://github.com/ubjson/universal-binary-json/issues/109
>
> Although these two modules were recently developed, we are beginning
> to integrate those in my other tools including *iso2mesh*
> <http://iso2mesh.sf.net/>, *jsonlab* <http://openjdata.org/jsonlab>
> and *mcx* <http://mcx.space/> (~10,000 registered users combined). So
> packaging and maintaining these tools will greatly facilitate the data
> exchange among the user communities.
>
> let me know if I can provide any additional explanations.
>
> thanks
>
> Qianqian
>
>
>>
>> Best regards
>>
>>
>> Anton
>>
>>
>> Am Di., 14. Juli 2020 um 06:35 Uhr schrieb Qianqian Fang
>> mailto:fan...@gmail.com>>:
>>
>> Dear Science team,
>>
>> I just submitted two python module packages and wonder if anyone is
>> willing to take a look and sponsor these packages
>>
>> The python-jdata and python-bjdata packages aim to enable sharing
>> python
>> data with other programming environments (like MATLAB, C/C++) via
>> JSON/binary JSON encoded data files (i.e. the JData/Binary JData
>> specifications).
>>
>> The RFS and mentors links can be found in the below two links
>>
>> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=964993
>> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=964994
>>
>> both packaging files can be found at
>>
>> https://salsa.debian.org/science-team/pybj
>> https://salsa.debian.org/science-team/pyjdata
>>
>> Also need some input on removing the
>> missing-dependency-on-numpy-abi error.
>>
>> thanks
>>
>> Qianqian
>>

Re: RFS: python-jdata and python-bjdata - JData/BJData encoders and decoders for python

[Qianqian Fang]

On 7/14/20 5:11 PM, Anton Gladky wrote:
> Hi.
>
> Thanks for your contribution to Debian. I have just some doubts about
> usefulness for Debian and possible popularity of those two projects.


hi Anton

thanks for your comment. happy to explain. Changed message title from
"JSON/..." to "JData/BJData encoders and decoders" to avoid further
confusions.

see my self-introduction in a previous thread

https://lists.debian.org/debian-science/2020/06/msg6.html
https://bugs.debian.org/cgi-bin/pkgreport.cgi?submitter=fangqq%40gmail.com

I am working on packaging a number research software produced from my
lab and research projects. I have already submitted 5 octave-related
projects, mentored by Rafael Laboissière (CCed) via the Debian Octave
Group. I intend to maintain these packages in the future (already doing
so for Fedora).

These two python modules are part of a bigger project that I initiated
last year (http://openjdata.org). They allow python users to read/write
JData-annotated data files produced by my MATLAB toolbox JSONLab
(https://github.com/fangq/jsonlab , about 46000 downloads on Matlab file
exchange and ~1000 clones/week on github). This work is partly funded by
my NIH (National Institute of Health) grants and broader dissemination
is part of the project goals.


> Do you know how many people can be interested in these two libraries?
> It looks like at least one of them duplicates the functionality of the
> built-in
> JSON module. Could you please shortly describe the benefits of both
> of them before we start to evaluate it technically?


The *python-bjdata* project was extended from *python-ubjson* - an
existing Debian package. Unfortunately, the UBJSON spec
(http://ubjson.org), despite being broadly used, is no longer actively
maintained. I started a fork earlier this year to continue the
development of this specification, and python-bjdata is a parser that is
compliant to the BJData spec.

The jdata/bjdata framework is not a duplication to JSON - instead, it
defines a systematic way to encode basic data structures into
JSON/UBJSON/BJData serializable forms.

The detailed specifications, examples and rationales can be found at

http://openjdata.org/wiki/

in a way, the jdata module is similar to *json-tricks* but aimed at a
more systematic/standardized way to annotate complex data (such as
graphs, maps, ND arrays ...) for sharing, exchange and reuse.

https://packages.debian.org/buster/python/python3-json-tricks

the bjdata module is a binary JSON format (similar to UBJSON, and
msgpack) to store binary and strongly typed hierarchical data. The
differences are highlighted in this github tracker

https://github.com/ubjson/universal-binary-json/issues/109

Although these two modules were recently developed, we are beginning to
integrate those in my other tools including *iso2mesh*
<http://iso2mesh.sf.net/>, *jsonlab* <http://openjdata.org/jsonlab> and
*mcx* <http://mcx.space/> (~10,000 registered users combined). So
packaging and maintaining these tools will greatly facilitate the data
exchange among the user communities.

let me know if I can provide any additional explanations.

thanks

Qianqian


>
> Best regards
>
>
> Anton
>
>
> Am Di., 14. Juli 2020 um 06:35 Uhr schrieb Qianqian Fang
> mailto:fan...@gmail.com>>:
>
> Dear Science team,
>
> I just submitted two python module packages and wonder if anyone is
> willing to take a look and sponsor these packages
>
> The python-jdata and python-bjdata packages aim to enable sharing
> python
> data with other programming environments (like MATLAB, C/C++) via
> JSON/binary JSON encoded data files (i.e. the JData/Binary JData
> specifications).
>
> The RFS and mentors links can be found in the below two links
>
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=964993
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=964994
>
> both packaging files can be found at
>
> https://salsa.debian.org/science-team/pybj
> https://salsa.debian.org/science-team/pyjdata
>
> Also need some input on removing the
> missing-dependency-on-numpy-abi error.
>
> thanks
>
> Qianqian
>

RFS: python-jdata and python-bjdata - JSON/BJData encoders and decoders for python

[Qianqian Fang]

Dear Science team,

I just submitted two python module packages and wonder if anyone is 
willing to take a look and sponsor these packages


The python-jdata and python-bjdata packages aim to enable sharing python 
data with other programming environments (like MATLAB, C/C++) via 
JSON/binary JSON encoded data files (i.e. the JData/Binary JData 
specifications).


The RFS and mentors links can be found in the below two links

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=964993
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=964994

both packaging files can be found at

https://salsa.debian.org/science-team/pybj
https://salsa.debian.org/science-team/pyjdata

Also need some input on removing the missing-dependency-on-numpy-abi error.

thanks

Qianqian

Re: Looking for example package that uses multiple source tarballs

[Qianqian Fang]

On 6/26/20 12:00 PM, Pierre Gruet wrote:

As you guessed, the fact that you got the old version number 1.9.1 in
the name of the orig tarball is because "debian" is provided as the
third argument concerning the first tarball, as the above link explains:
it forces the use of the version number written in debian/changelog. You
may see, if you put the right number again in debian/changelog, that
running "uscan --force" will download the tarballs and give them the
right version number.

Yet, I guess this should not remain as is but I unfortunately do not
have the repacking skills to do more at that point.



the --force option worked like a charm!

the empty source package issue was eventually fixed by adjusting the 
repack script to deal with the oversionmangle.


one minor question: although uscan --force allowed me to download and 
build the MUT source package, when I tried "gbp import-orig --uscan" to 
create a git repo, it refused to download the packages again, and I was 
not able to find a place to tell uscan to use --force, is there a way to 
also import an MUT orig package to git?


thanks again

Qianqian

Re: Looking for example package that uses multiple source tarballs

[Qianqian Fang]

On 6/25/20 4:09 PM, Pierre Gruet wrote:

I think you should provide some options ("opts=..."): for instance, a
component name should be provided for the two last parts. You can look
at the manpage of uscan, it provides some examples of watch files for
multiple upstream tarballs.
As you are doing some repack, I guess you will add some suffix to the
version number (like "+dfsg"), and to do so you should need to provide
the "oversionmangle=..." option for the first tarball. Examples are
shown in uscan's manpage.
I recently packaged libaparapi-java, which is not yet in Debian but
which you can get from its Salsa repository

https://salsa.debian.org/med-team/libaparapi-java

Running "uscan --verbose" will allow you to get three tarballs : a main
one and two components.


thanks Pierre for the comments and sample project, very helpful!

after reading your watch file and uscan's manual page more carefully, I 
was able to download all 3 components using the below watch file


https://salsa.debian.org/pkg-octave-team/octave-iso2mesh/-/blob/master/debian/watch

after running *uscan --verbose*, a bunch of files/folders were created 
in the above directory, see


|drwxrwxr-x  5 fangq fangq 4096 Jun 25 22:03 
octave-iso2mesh-1.9.5+dfsg1||
||-rw-rw-r--  1 fangq fangq  409 Jun 25 22:03 
octave-iso2mesh_1.9.5+dfsg1-0ubuntu1.dsc||
||-rw-rw-r--  1 fangq fangq 5500 Jun 25 22:03 
octave-iso2mesh_1.9.5+dfsg1-0ubuntu1.debian.tar.xz||
||lrwxrwxrwx  1 fangq fangq   20 Jun 25 22:03 
octave-iso2mesh_1.9.5+dfsg1.orig-meshfix.tar.gz -> meshfix-1.2.2.tar.gz||

||-rw-rw-r--  1 fangq fangq  2092675 Jun 25 22:03 meshfix-1.2.2.tar.gz||
||-rw-rw-r--  1 fangq fangq  135 Jun 25 22:03 
octave-iso2mesh_1.9.5+dfsg1.orig.tar.gz||
||-rw-rw-r--  1 fangq fangq 5500 Jun 25 22:03 
octave-iso2mesh_1.9.1-1.debian.tar.xz||
||lrwxrwxrwx  1 fangq fangq   15 Jun 25 22:03 
octave-iso2mesh_1.9.5+dfsg1.orig-cork.tar.gz -> cork-0.9.tar.gz||

||-rw-rw-r--  1 fangq fangq   292728 Jun 25 22:03 cork-0.9.tar.gz||
||-rw-rw-r--  1 fangq fangq   941881 Jun 25 22:03 
octave-iso2mesh_1.9.1.orig.tar.gz||

||-rw-rw-r--  1 fangq fangq 28793215 Jun 25 22:03 iso2mesh-1.9.5.tar.gz|

the final merged source folder, however, only contains the source codes 
of the two components, but not from the main package, see


*|$~/space/git/Temp/pkg$ ls -lt octave-iso2mesh-1.9.5+dfsg1|*|
||total 12||
||drwxrwxr-x 4 fangq fangq 4096 Jun 25 22:03 debian||
||drwxrwxr-x 7 fangq fangq 4096 Jun 20 12:44 meshfix||
||drwxrwxr-x 5 fangq fangq 4096 Oct  1  2019 cork|

I am not sure if it is related, but i have to change my debian/changelog 
version to a lower version number (1.9.1) in order to let uscan download 
the new release (1.9.5). however, both .tar.xz files are empty (except 
the debian/ folder)


|octave-iso2mesh_1.9.1-1.debian.tar.xz
octave-iso2mesh_1.9.5+dfsg1-0ubuntu1.debian.tar.xz
|

but the *octave-iso2mesh_1.9.1.orig.tar.gz* file is correctly repacked 
and contains all needed files.


I am wondering if anyone notice something wrong in repack or watch 
script? my repack file was slightly modified from 
https://wiki.debian.org/BenFinney/software/repack


another minor thing is the "|-0ubuntu1|" suffix. I am testing the 
packaging in a Ubuntu 18.04 box, is there a way to remove that?


my packaging files are committed at 
https://salsa.debian.org/pkg-octave-team/octave-iso2mesh/ 
 



much appreciated!

Qianqian

Looking for example package that uses multiple source tarballs

[Qianqian Fang]

hi everyone,

I am working on packaging a matlab/octave toolbox that involves multiple 
source tarballs


https://salsa.debian.org/pkg-octave-team/octave-iso2mesh

this toolbox contains a number of built-in utilities that are linked to 
the source repo via submodules


https://github.com/fangq/iso2mesh/tree/master/tools

however, the github release tarball does not contain these submodules 
and I must combine those to build the package.


I previous had used a shell script 
 
to merge the source code trees and create a single .orig.tar.gz, 
however, I was told that using *debian/repack* and *uscan* is the 
preferred way to perform changes like this.


I notice that debian/watch file supports multiple source components, so 
I wrote the below watch file:


|
|

|version=3||
||
||opts="dversionmangle=s/\+dfsg\.\d+$//,filenamemangle=s/\S+\/v?(\S+)\.tar\.gz/iso2mesh-$1\.tar\.gz/" 
\||

||    https://github.com/fangq/iso2mesh/tags .*/v?(\d\S+)\.tar\.gz \||
||    debian debian/repack||
||
||https://github.com/fangq/cork/releases 
.*/v(\d[\d\.]*)\.(?:tar.gz|tar.bz2|tar.xz)||
||https://github.com/fangq/meshfix/releases 
.*/v(\d[\d\.]*)\.(?:tar.gz|tar.bz2|tar.xz)|



but when I ran*uscan --verbose*, it only downloaded/repacked the main 
source file, but refused to download the components


|...||
||uscan info: Matching target for filenamemangle: 
/fangq/iso2mesh/archive/v1.9.5.tar.gz||

||uscan info: Download filename (filenamemangled): iso2mesh-1.9.5.tar.gz||
||...||
||Removing non-DFSG-free files:||
||removed '/tmp/tmp.uNMHl78FqP/octave-iso2mesh-1.9.1.orig/bin/README.txt'||
||Rebuilding DFSG-free upstream source tarball:||
||Moving completed upstream tarball to 
‘../octave-iso2mesh_1.9.1.orig.tar.gz’:||

||...||
||uscan info: Newest upstream tarball version selected for download 
(uversionmangled): 0.9||

||uscan info: Download filename (filenamemangled): v0.9.tar.gz||
||*uscan info: Newest version of octave-iso2mesh on remote site is 0.9, 
local version is 1.9.1*||

||uscan info:    => Only older package available from||
||https://github.com/fangq/cork/archive/v0.9.tar.gz||
||...||
||uscan info: Scan finished|


I am wondering if you know any sample package that uses repack/uscan to 
combine multiple source packages? The uscan manpage mentioned about MUT, 
but I could not find a working example.


thanks, let me know if you have any suggestions.

Qianqian

Re: Becoming a package maintainer and best practices for porting RPMs from Fedora

[Qianqian Fang]

On 6/8/20 5:44 PM, Rebecca N. Palmer wrote:
Either including or not including the upstream files is allowed, but 
including them is usually preferred, unless they are very large.



thank you so much Rebecca. very helpful.

will start working on the packaging files, and send my packaging-related 
questions to the mentors/octave lists (also submitted my access request 
on salsa - although it did not ask me to write anything).


cheers

Qianqian




If they are included, the packaging repository should have a branch 
with upstream files only, and a branch with upstream+debian.  The 
upstream branch can be tarball imports (one commit per release) or a 
clone of the upstream repository (detailed history).


(And yes, it is a known issue that having so many choices can be 
confusing, particularly to beginners.)


The git-buildpackage tool can help you with this.

if I am willing to be the maintainer of my own packages, do I still 
need to find a package sponsor (given I am not a maintainer yet)?


Yes: the access levels are

sponsored maintainer (named in Maintainer: or Uploader: of a 
debian/control but can't upload)

->
Debian Maintainer (can upload their own packages)
->
Debian Developer (can upload any package, including as a sponsor)

https://wiki.debian.org/DebianMaintainer

from the above link, one of the paths is to "Join a packaging team", 
for example, how do I "join" debian-science or DebianMed?


https://salsa.debian.org/groups/science-team/-/group_members/request_access 




4. submit an ITP bug


Where possible, this should be the first step; otherwise correct.


sorry, forgive me for asking these basic questions.


It's arguably a bug that this information is so hard to find.

Re: Becoming a package maintainer and best practices for porting RPMs from Fedora

[Qianqian Fang]

On 6/8/20 3:00 PM, Rebecca N. Palmer wrote:


This is a library, so the binary package names should probably be 
libzmat0/libzmat-dev/octave-zmat.  I have not checked the rest of the 
package.


It's generally recommended, and in some teams required, that packaging 
repositories be hosted on https://salsa.debian.org, not with upstream.



got it. I just registered an account on salsa.


just a few quick follow up questions


from browsing some of the existing repos, it appears that the package 
repo contains all upstream source files (with the addition of the 
debian/ folder), which is different from Fedora (upstream source package 
is separate from the .spec and patch* files).


is there any preference in terms of how to manage upstream source files 
on salsa? for example, I can


1. clone my upstream git repo (including history) from github to salsa, 
and then add the debian/ files in a special packaging branch, and then 
keep syncing/merging with the upstream after each release, or


2. just upload a snapshot of each new release and keep it separate from 
the upstream source history, or


3. directly add those debian/ packaging files to upstream code tree and 
clone it to salsa.



Yes, and probably by posting to the team's mailing list: 
https://mentors.debian.net/sponsors (As you have a Git repository, you 
probably don't need to actually upload to mentors.debian.net - I never 
did.)



if I am willing to be the maintainer of my own packages, do I still need 
to find a package sponsor (given I am not a maintainer yet)?


from the above link, one of the paths is to "Join a packaging team", for 
example, how do I "join" debian-science or DebianMed? or the packaging 
team is separate? the "how to join" section of this webpage 
 loops back to this mailing list.



Yes, an ITP bug: see https://www.debian.org/devel/wnpp/ for how to do 
this.



just to straighten my thoughts, in general, to contribute a new package 
to Debian as a prospective new package maintainer, I need to


1. create drafted packaging files from the intended upstream release tarball

2. upload to salsa

3. email debian-mentors mailing list to get someone review it and fix 
the issues


4. submit an ITP bug

5. ask sponsor to submit the package on your behave

6. going through a separate process to become a maintainer? (or it 
coincides with this submission process and I will become a maintainer 
once the package is approved?)



sorry, forgive me for asking these basic questions.

Qianqian

Re: Becoming a package maintainer and best practices for porting RPMs from Fedora

[Qianqian Fang]

On 6/5/20 3:44 AM, Rebecca N. Palmer wrote:
Not that I'm aware of.  (The package 'alien' can install RPM binary 
packages on Debian, but does not convert source packages.)



any best practices guide for creating a package?
any links/steps on how to become a maintainer would be fantastic. Any 
one willing to mentor me would be great!


(Non-trust warning: wiki.debian.org is an anyone-can-edit site)



thanks Rebecca.

I went through the tutorials and also checked out a few existing 
packages to learn my way towards using the new packaging commands (did 
this long time ago when pre/postinst were used).


I picked one of my projects (https://github.com/fangq/zmat) to get 
started, currently, I managed to create all basic packaging files (3 
subpackages, zmat/zmat-dev/octave-zmat)


https://github.com/fangq/debpkg/tree/zmat

I have a couple of questions regarding finishing up the package, 
including the appropriate octave packaging commands, file permissions 
etc, but I am not sure if I should post those here or to other lists, 
debian-mentor for example.


Moving forward from here, I would like to know

1. is there a package review process? how do I initiate it?

2. do I need to create a bug request on prospective packages 
(https://www.debian.org/devel/wnpp/) ?


3. related to the above question, is there an overview of a new package 
submission procedure online?


forgive me if these were answered somewhere in the links you provided.

cheers

Qianqian


General (possibly out of date):
https://wiki.debian.org/Packaging/Intro
https://www.debian.org/doc/manuals/maint-guide/
https://www.debian.org/devel/

Relevant teams:
https://wiki.debian.org/DebianScience
https://wiki.debian.org/Teams/DebianOctaveGroup
https://neuro.debian.net/ (possibly inactive)

Becoming a package maintainer and best practices for porting RPMs from Fedora

[Qianqian Fang]

Dear Debianers,

I have been maintaining a number of packages created by my lab for 
Fedora (as part of NeuroFedora) - most of these tools are for 
optical/neuro imaging, modeling and data processing. I would like to 
contribute these packages to Debian and want to get some pointers where 
to start.


The packages I am currently maintaining include

https://fedoraproject.org/wiki/User:Fangq#Maintained_packages

Many of these are MATLAB/Octave compatible toolboxes, some are C-based 
Monte Carlo photon simulators (OpenCL based).


I am wondering if someone can let me know

1. is there an established procedure to convert an official Fedora RPM 
to a deb package? any best practices guide for creating a package?


2. I am interested in maintaining these packages if accepted, any 
links/steps on how to become a maintainer would be fantastic. Any one 
willing to mentor me would be great!


thanks

Qianqian

Re: science application + opencl/rocm + AMD GPU?

[Qianqian Fang]

hi Mo

we have some experience with OpenCL+AMD GPUs, but pretty much
limited to Monte Carlo (MC) photon transport simulations. My lab developed
a few GPU-accelerated photon simulators, both in CUDA and OpenCL,
see http://mcx.space. This work is funded by the NIH.

In the past, we have heavily optimized our codes for different GPUs
from different generations and vendors. If you are interested, please
check out two of our papers on the OpenCL codes (MCX-CL and MMCL)

https://doi.org/10.1117/1.JBO.23.1.010504
http://dx.doi.org/10.1117/1.JBO.24.11.115002

the first one was on a voxel-based MC code - from Fig. 2, you can see
the benchmark speed comparisons across different processors. We used
amdgpu-pro 16.30.3 on a Ubuntu 14.04. Overall, the performance on
the AMD GPU was pretty decent - although I expect to see more (if
you compare with the corresponding CUDA code on NVIDIA GPUs in
the inset).

a more comprehensive benchmark list can be found at
http://mcx.space/wiki/?Speed

the 2nd paper just came out last week, it is a more accurate MC
algorithm but needs more memory operations. From Fig. 3, you can
see AMD GPUs clearly left behind (Vega II, Vega64) compared to either
the NVIDIA cards of the same tier, or the voxel-based MC on the same
card (see Fig. 3b). We had some thoughts on what happened in the
Discussion section. These results were consistent from amdgpu/rocm drivers.

Regarding ROCm, we have been trying to get it to work since early
last year, but the experience was a bit complicated. please see
this thread

https://github.com/RadeonOpenCompute/ROCm-OpenCL-Runtime/issues/43

initially, it did not work, but after several updates, after v1.9, my
code started working, but the speed is several fold slower than
amdgpu-pro (16.xx, 17.xx), then, both newer amdgpu-pro/rocm
landed at the lower end of the speed because amdgpu-pro started
to share compilers with rocm see

https://github.com/RadeonOpenCompute/ROCm-OpenCL-Runtime/issues/43#issuecomment-421805412.

from what we saw in our latest paper, the slow speed from ROCm
+AMD GPU was a result of agressive register allocation - my cl kernel
only needs 69 registers by the NVIDIA ocl compiler, but allocates
over 200 vector registers by the ROCm driver. This greatly limits the
"wavefronts" that can run simultaneously.

Regarding open-source ocl driver, yes, it works, at least for my
application. I recently packaged my simulators for Fedora and
by listing "opencl-filesystem" package as the dependency, my
simulators ran out of box on a completely open-source environment,

https://src.fedoraproject.org/rpms/octave-mcxlab/blob/master/f/octave-mcxlab.spec#_13

however, to get good speed we still ask users to install GPU drivers
to use the vendor-optimized opencl library. A ROCm integrated
platform would be wonderful.

happy to share more if it is relevant.

Qianqian


PS: I recently subscribed to this list with a hope to learn packaging
procedures so I can make these tools available to Debian.
these packages are listed here

https://fedoraproject.org/wiki/User:Fangq#Maintained_packages


On 11/23/19 11:34 AM, Mo Zhou wrote:

Hi science team,

I'd like to request the team for sharing some experience on this topic:
"scientific application + opencl/rocm + AMD GPU".  Any experience will
be very helpful to me in terms of the being-investigated ROCm
integration[1] to Debian.

I'd like to ask you the following questions:

1. how's everything going without the amdgpu-pro[2] driver, especially
the opencl/rocm programs? Most importantly, can OpenCL work without
any non-free component?

2. how does the consumer-grade AMD gpu perform in terms of scientific
computing or other opencl/rocm applications?

3. do you think AMD GPU can be a practical alternative to Nvidia/CUDA
in at least a few applications?


[1] https://salsa.debian.org/rocm-team
[2] non-free, https://www.amd.com/en/support/gpu-pro-eula