Re: [NEW] devel/py-pandas
Find attached the new version with just missing whitespaces between the variables of the Makefile. 2018-06-12 11:07 GMT-03:00 Elias M. Mariani : > and the attached tar.gz ? > Missed by that much... > > 2018-06-12 11:06 GMT-03:00 Elias M. Mariani : >> Sending the new version for test. >> The Makefile uses two flavors: >> python3 and test >> So, if you want to install you just run: >> env FLAVOR="" make install >> or >> env FLAVOR="python3" make install >> >> That generates everything only once. >> >> if you want to test: >> env FLAVOR="test" make test >> or >> env FLAVOR="test python3" make test >> also building just once but the result is non installable. >> >> I think is the best choice, if you want to test you build once, if you >> want to install you build once, if you want both... well. just build >> twice... >> Results of the regression tests (using textproc/py-xlrd as a >> dependency test witch I just sent to ports@): >> python2.7 = 2 failed, 24523 passed, 1675 skipped, 79 xfailed, 24 >> xpassed, 86 warnings in 76400.37 seconds >> python3.6 = 3 failed, 24619 passed, 1573 skipped, 80 xfailed, 26 >> xpassed, 197 warnings in 6313.57 seconds >> both OpenBSD-current, amd64. >> 2 failures are because the tester tries to invoke "python" as a >> command, witch it doesn't (necessarily) exist... Just ignore those >> 2... >> Also notice the difference in test time between py2 and py3... is >> better to use the py3 version to do tests... >> Some skips are hardcoded (wt...) others are because we are missing >> some test-dependencies, keep in mind that this tests cover pretty much >> everything that py-pandas does, just look at the RUN_DEPENDS vs >> TEST_DEPENDS and you get an idea of the magnitude of the test. >> >> So Edd kindly offered to check it out. >> Ignore the regression tests, the ideal would be to test it in a real >> application to see if really works as the regression tests say. >> Also I have been working on ports for some weeks now, but this is the >> first with some real logic inside, if Stuart or someone can check the >> Makefile to see if some corrections are needed would be great. >> >> Thanks to all, this pre-port-release was already been made with help >> of several people with examples, advises, warnings, etc. >> Alexandr Shadchin give me his permission to merge our versions in >> openbsd-wip, so, credits to him as well. >> >> Cheers. >> Elias. >> >> 2018-06-09 19:17 GMT-03:00 Elias M. Mariani : >>> Okey, last update: >>> - patched the docs building system. >>> - while building the docs yo need LOTS of depends and for some reason >>> a X session running (yes... I know...). So, either I am doing >>> something very wrong or this is a nasty way of building docs. I guess >>> that something in the source code uses the backend-gtk part of >>> Matplotlib and that triggers the creation of a cursor... because the >>> hole process uses the actual code working to build the docs... I get >>> lost in this nonsense... >>> My Solution will be: build without any docs, use >>> http://pandas.pydata.org/pandas-docs/stable/ for that. >>> Build py-pandas normally with setuptools and python3 flavor. >>> Regression tests will be done by re-building the port, so yes, we >>> build the port twice but only for testing, normal build will be done >>> just once. >>> Any ideas or/and OK ? >>> >>> Cheers. >>> Elias. >>> >>> >>> 2018-06-09 15:59 GMT-03:00 Elias M. Mariani : > Not sure if this helps, but the version in openbsd-wip has a (commented) > block which looks like it does a second (in-place) build to make the > docs: > https://github.com/jasperla/openbsd-wip/blob/cb917e8bcdf2b912209f97652eca484494138fa3/math/py-pandas/Makefile#L65-L68 > > If we really can't get the docs until the package is installed, you > could use SUBDIRs, and have a doc subdir BUILD_DEPEND on the main > package. Yes, but building the same package twice does not sound nice at all. I will add the py-pandas dependency to build py-pandas-docs, that seems the best solution overall... Cheers. Elias. py-pandas.tar.gz Description: GNU Zip compressed data
Re: [NEW] devel/py-pandas
and the attached tar.gz ? Missed by that much... 2018-06-12 11:06 GMT-03:00 Elias M. Mariani : > Sending the new version for test. > The Makefile uses two flavors: > python3 and test > So, if you want to install you just run: > env FLAVOR="" make install > or > env FLAVOR="python3" make install > > That generates everything only once. > > if you want to test: > env FLAVOR="test" make test > or > env FLAVOR="test python3" make test > also building just once but the result is non installable. > > I think is the best choice, if you want to test you build once, if you > want to install you build once, if you want both... well. just build > twice... > Results of the regression tests (using textproc/py-xlrd as a > dependency test witch I just sent to ports@): > python2.7 = 2 failed, 24523 passed, 1675 skipped, 79 xfailed, 24 > xpassed, 86 warnings in 76400.37 seconds > python3.6 = 3 failed, 24619 passed, 1573 skipped, 80 xfailed, 26 > xpassed, 197 warnings in 6313.57 seconds > both OpenBSD-current, amd64. > 2 failures are because the tester tries to invoke "python" as a > command, witch it doesn't (necessarily) exist... Just ignore those > 2... > Also notice the difference in test time between py2 and py3... is > better to use the py3 version to do tests... > Some skips are hardcoded (wt...) others are because we are missing > some test-dependencies, keep in mind that this tests cover pretty much > everything that py-pandas does, just look at the RUN_DEPENDS vs > TEST_DEPENDS and you get an idea of the magnitude of the test. > > So Edd kindly offered to check it out. > Ignore the regression tests, the ideal would be to test it in a real > application to see if really works as the regression tests say. > Also I have been working on ports for some weeks now, but this is the > first with some real logic inside, if Stuart or someone can check the > Makefile to see if some corrections are needed would be great. > > Thanks to all, this pre-port-release was already been made with help > of several people with examples, advises, warnings, etc. > Alexandr Shadchin give me his permission to merge our versions in > openbsd-wip, so, credits to him as well. > > Cheers. > Elias. > > 2018-06-09 19:17 GMT-03:00 Elias M. Mariani : >> Okey, last update: >> - patched the docs building system. >> - while building the docs yo need LOTS of depends and for some reason >> a X session running (yes... I know...). So, either I am doing >> something very wrong or this is a nasty way of building docs. I guess >> that something in the source code uses the backend-gtk part of >> Matplotlib and that triggers the creation of a cursor... because the >> hole process uses the actual code working to build the docs... I get >> lost in this nonsense... >> My Solution will be: build without any docs, use >> http://pandas.pydata.org/pandas-docs/stable/ for that. >> Build py-pandas normally with setuptools and python3 flavor. >> Regression tests will be done by re-building the port, so yes, we >> build the port twice but only for testing, normal build will be done >> just once. >> Any ideas or/and OK ? >> >> Cheers. >> Elias. >> >> >> 2018-06-09 15:59 GMT-03:00 Elias M. Mariani : Not sure if this helps, but the version in openbsd-wip has a (commented) block which looks like it does a second (in-place) build to make the docs: https://github.com/jasperla/openbsd-wip/blob/cb917e8bcdf2b912209f97652eca484494138fa3/math/py-pandas/Makefile#L65-L68 If we really can't get the docs until the package is installed, you could use SUBDIRs, and have a doc subdir BUILD_DEPEND on the main package. >>> >>> Yes, but building the same package twice does not sound nice at all. >>> I will add the py-pandas dependency to build py-pandas-docs, that >>> seems the best solution overall... >>> >>> Cheers. >>> Elias. py-pandas.tar.gz Description: GNU Zip compressed data
Re: [NEW] devel/py-pandas
Sending the new version for test. The Makefile uses two flavors: python3 and test So, if you want to install you just run: env FLAVOR="" make install or env FLAVOR="python3" make install That generates everything only once. if you want to test: env FLAVOR="test" make test or env FLAVOR="test python3" make test also building just once but the result is non installable. I think is the best choice, if you want to test you build once, if you want to install you build once, if you want both... well. just build twice... Results of the regression tests (using textproc/py-xlrd as a dependency test witch I just sent to ports@): python2.7 = 2 failed, 24523 passed, 1675 skipped, 79 xfailed, 24 xpassed, 86 warnings in 76400.37 seconds python3.6 = 3 failed, 24619 passed, 1573 skipped, 80 xfailed, 26 xpassed, 197 warnings in 6313.57 seconds both OpenBSD-current, amd64. 2 failures are because the tester tries to invoke "python" as a command, witch it doesn't (necessarily) exist... Just ignore those 2... Also notice the difference in test time between py2 and py3... is better to use the py3 version to do tests... Some skips are hardcoded (wt...) others are because we are missing some test-dependencies, keep in mind that this tests cover pretty much everything that py-pandas does, just look at the RUN_DEPENDS vs TEST_DEPENDS and you get an idea of the magnitude of the test. So Edd kindly offered to check it out. Ignore the regression tests, the ideal would be to test it in a real application to see if really works as the regression tests say. Also I have been working on ports for some weeks now, but this is the first with some real logic inside, if Stuart or someone can check the Makefile to see if some corrections are needed would be great. Thanks to all, this pre-port-release was already been made with help of several people with examples, advises, warnings, etc. Alexandr Shadchin give me his permission to merge our versions in openbsd-wip, so, credits to him as well. Cheers. Elias. 2018-06-09 19:17 GMT-03:00 Elias M. Mariani : > Okey, last update: > - patched the docs building system. > - while building the docs yo need LOTS of depends and for some reason > a X session running (yes... I know...). So, either I am doing > something very wrong or this is a nasty way of building docs. I guess > that something in the source code uses the backend-gtk part of > Matplotlib and that triggers the creation of a cursor... because the > hole process uses the actual code working to build the docs... I get > lost in this nonsense... > My Solution will be: build without any docs, use > http://pandas.pydata.org/pandas-docs/stable/ for that. > Build py-pandas normally with setuptools and python3 flavor. > Regression tests will be done by re-building the port, so yes, we > build the port twice but only for testing, normal build will be done > just once. > Any ideas or/and OK ? > > Cheers. > Elias. > > > 2018-06-09 15:59 GMT-03:00 Elias M. Mariani : >>> Not sure if this helps, but the version in openbsd-wip has a (commented) >>> block which looks like it does a second (in-place) build to make the >>> docs: >>> https://github.com/jasperla/openbsd-wip/blob/cb917e8bcdf2b912209f97652eca484494138fa3/math/py-pandas/Makefile#L65-L68 >>> >>> If we really can't get the docs until the package is installed, you >>> could use SUBDIRs, and have a doc subdir BUILD_DEPEND on the main >>> package. >> >> Yes, but building the same package twice does not sound nice at all. >> I will add the py-pandas dependency to build py-pandas-docs, that >> seems the best solution overall... >> >> Cheers. >> Elias.
Re: [NEW] devel/py-pandas
Okey, last update: - patched the docs building system. - while building the docs yo need LOTS of depends and for some reason a X session running (yes... I know...). So, either I am doing something very wrong or this is a nasty way of building docs. I guess that something in the source code uses the backend-gtk part of Matplotlib and that triggers the creation of a cursor... because the hole process uses the actual code working to build the docs... I get lost in this nonsense... My Solution will be: build without any docs, use http://pandas.pydata.org/pandas-docs/stable/ for that. Build py-pandas normally with setuptools and python3 flavor. Regression tests will be done by re-building the port, so yes, we build the port twice but only for testing, normal build will be done just once. Any ideas or/and OK ? Cheers. Elias. 2018-06-09 15:59 GMT-03:00 Elias M. Mariani : >> Not sure if this helps, but the version in openbsd-wip has a (commented) >> block which looks like it does a second (in-place) build to make the >> docs: >> https://github.com/jasperla/openbsd-wip/blob/cb917e8bcdf2b912209f97652eca484494138fa3/math/py-pandas/Makefile#L65-L68 >> >> If we really can't get the docs until the package is installed, you >> could use SUBDIRs, and have a doc subdir BUILD_DEPEND on the main >> package. > > Yes, but building the same package twice does not sound nice at all. > I will add the py-pandas dependency to build py-pandas-docs, that > seems the best solution overall... > > Cheers. > Elias.
Re: [NEW] devel/py-pandas
> Not sure if this helps, but the version in openbsd-wip has a (commented) > block which looks like it does a second (in-place) build to make the > docs: > https://github.com/jasperla/openbsd-wip/blob/cb917e8bcdf2b912209f97652eca484494138fa3/math/py-pandas/Makefile#L65-L68 > > If we really can't get the docs until the package is installed, you > could use SUBDIRs, and have a doc subdir BUILD_DEPEND on the main > package. Yes, but building the same package twice does not sound nice at all. I will add the py-pandas dependency to build py-pandas-docs, that seems the best solution overall... Cheers. Elias.
Re: [NEW] devel/py-pandas
On Thu, Jun 07, 2018 at 03:46:48PM -0300, Elias M. Mariani wrote: > If the normal build style is acceptable with building docs separately > and testing itself is OK I will proceed to merge the py-pandas in wip. > I could not reach shadchin@ to talk this with him and according to > github that repository is the for a year now... so... I like > aggressive takeovers. :) Not sure if this helps, but the version in openbsd-wip has a (commented) block which looks like it does a second (in-place) build to make the docs: https://github.com/jasperla/openbsd-wip/blob/cb917e8bcdf2b912209f97652eca484494138fa3/math/py-pandas/Makefile#L65-L68 If we really can't get the docs until the package is installed, you could use SUBDIRs, and have a doc subdir BUILD_DEPEND on the main package. -- Best Regards Edd Barrett http://www.theunixzoo.co.uk
Re: [NEW] devel/py-pandas
Okey, I need advice from Stuart or someone :) The problem is the following: Pandas can be built "inplace" for testing and for building the docs. If you do it "inplace" yo do not have the "install" option on setup.py to automatically get a nice and clean install, one should "get the files" from in between the source... not nice... and not something that I want to maintain that way... If you do it with the normal build-install approach you get py-pandas ready to go, but without docs. You can build the docs and run the tests once py-pandas is installed, the docs can be built quickly separately, so no problem there, I would just make py-pandas-docs and that's it. The problem is with the test, the test only works well when testing itself (that is, testing against site-packages/pandas), this is not nice ether because it leaves a horrible trail of files in the package directory and if you run uninstall will get the "directory not empty" problem. But test pass pretty good, I would need someone to really test the "live?" version of py-pandas, maybe Edd can help me with this. If the normal build style is acceptable with building docs separately and testing itself is OK I will proceed to merge the py-pandas in wip. I could not reach shadchin@ to talk this with him and according to github that repository is the for a year now... so... I like aggressive takeovers. :) Cheers. Elias. 2018-06-06 16:17 GMT-03:00 Elias M. Mariani : >> You should be able to fix this by setting TZ=UTC in the environment. > Great idea. > >>> Did you know that py-pandas already was in openbsd-wip? You can find it >>> at https://github.com/jasperla/openbsd-wip/tree/master/math/py-pandas > I did not realized that py-pandas was already in there, my mind > discarded that because I consider the devel category only, my bad... > >>> >>> That said, I had a quick look at your Makefile, and have some comments: >>> - I think CATEGORIES should be math >>> - DEPS should be sorted >> >> ack. > > Will do. > >>> - Having py-pandas as test dependency doesn't make sense. Please have >>> a look at shadchin@'s port as mentioned above >> >> It's not uncommon with python things, though perhaps just adding a make >> dependency like this is even more common: >> >> do-test: fake >> >>> My suggestion would be to contact shadchin@ and try to combine efforts. >>> >>> BTW pandas would make a great addition to ports. >>> >> >> +1 > > Yes, I will contact shadchin@ to merge both versions, already working on it. > In fact, he sorted the testing part much better than I did. > > Cheers. > Elias.
Re: [NEW] devel/py-pandas
> You should be able to fix this by setting TZ=UTC in the environment. Great idea. >> Did you know that py-pandas already was in openbsd-wip? You can find it >> at https://github.com/jasperla/openbsd-wip/tree/master/math/py-pandas I did not realized that py-pandas was already in there, my mind discarded that because I consider the devel category only, my bad... >> >> That said, I had a quick look at your Makefile, and have some comments: >> - I think CATEGORIES should be math >> - DEPS should be sorted > > ack. Will do. >> - Having py-pandas as test dependency doesn't make sense. Please have >> a look at shadchin@'s port as mentioned above > > It's not uncommon with python things, though perhaps just adding a make > dependency like this is even more common: > > do-test: fake > >> My suggestion would be to contact shadchin@ and try to combine efforts. >> >> BTW pandas would make a great addition to ports. >> > > +1 Yes, I will contact shadchin@ to merge both versions, already working on it. In fact, he sorted the testing part much better than I did. Cheers. Elias.
Re: [NEW] devel/py-pandas
On 2018/06/06 11:31, Björn Ketelaars wrote:
> On Mon 28/05/2018 23:50, Elias M. Mariani wrote:
> > Decided to change the way in witch the regression test is performed.
> > New version attached.
> >
> > 2018-05-27 14:31 GMT-03:00 Elias M. Mariani :
> > > pandas is a Python package providing fast, flexible, and expressive
> > > data structures designed to make working with "relational" or
> > > "labeled" data both easy and intuitive. It aims to be the fundamental
> > > high-level building block for doing practical, real world data
> > > analysis in Python. Additionally, it has the broader goal of becoming
> > > the most powerful and flexible open source data analysis /
> > > manipulation tool available in any language. It is already well on its
> > > way toward this goal.
> > >
> > > Here is the result of a fast pytest test:
> > > 7 failed, 23637 passed, 2470 skipped, 77 xfailed, 22 xpassed, 4
> > > warnings in 17223.28 seconds
> > >
> > > I added some dependencies to TEST_DEPENDS to avoid some of the skipped
> > > tests (it takes to long to run a full test suite, but the code is
> > > clearly working).
> > >
> > > The only problem that I encountered is that the "make test" should be
> > > done after "make fake", against the fake directory, because the port
> > > builds some C libraries that are needed in place, if this is a big
> > > problem (clearly at least an ugly one...) I can make a test flavor to
> > > do a different kind of "in place" build to allow the test to be done
> > > without the fake framework but mixing this with the python3 flavor
> > > seems adding complexity for nothing in my opinion.
> > > if you run:
> > > make fake
> > > make test
> > > The tests run fine and return the results without a problem if you
> > > need to see the regression tests in an interactive way.
> > > I do not know if the test are done in bulk or something that might be
> > > a problem with this.
> > >
> > > Beside that the fails are minor and it tests A LOT of thinks that 7
> > > failures are because timezone in the test machine is not set to UTC
You should be able to fix this by setting TZ=UTC in the environment.
> > > and for the lack of some test dependencies that I already added
> > > (py-html5lib) for example... the documentation about the test suite is
> > > kind of lacking in the project buy it covers pretty everything.
> > >
> > > Looking for opinions, OKs and someone to commit.
> > > :)
> > >
> > > Attached and available in openbsd-wip if someone else want to see/edit
> > > the files online.
> > >
> > > Cheers.
> > > Elias.
>
> Did you know that py-pandas already was in openbsd-wip? You can find it
> at https://github.com/jasperla/openbsd-wip/tree/master/math/py-pandas
>
>
> That said, I had a quick look at your Makefile, and have some comments:
> - I think CATEGORIES should be math
> - DEPS should be sorted
ack.
> - Instead of having math/py-numpy${MODPY_FLAVOR}>=1.9.0 in RDEP you
> could use ${BUILD_DEPENDS}
Please don't include BUILD_DEPENDS in RUN_DEPENDS, it's prone to picking
up unwanted things. In this case, it will pick up ccache if USE_CCACHE
is set, but in other ports you'll end up adding things like autoconf
as RUN_DEPENDS (possibly not initially, maybe just after somebody
later goes to fix a bug and changes CONFIGURE_STYLE..)
> - Having py-pandas as test dependency doesn't make sense. Please have
> a look at shadchin@'s port as mentioned above
It's not uncommon with python things, though perhaps just adding a make
dependency like this is even more common:
do-test: fake
> My suggestion would be to contact shadchin@ and try to combine efforts.
>
> BTW pandas would make a great addition to ports.
>
+1
Re: [NEW] devel/py-pandas
On Mon 28/05/2018 23:50, Elias M. Mariani wrote:
> Decided to change the way in witch the regression test is performed.
> New version attached.
>
> 2018-05-27 14:31 GMT-03:00 Elias M. Mariani :
> > pandas is a Python package providing fast, flexible, and expressive
> > data structures designed to make working with "relational" or
> > "labeled" data both easy and intuitive. It aims to be the fundamental
> > high-level building block for doing practical, real world data
> > analysis in Python. Additionally, it has the broader goal of becoming
> > the most powerful and flexible open source data analysis /
> > manipulation tool available in any language. It is already well on its
> > way toward this goal.
> >
> > Here is the result of a fast pytest test:
> > 7 failed, 23637 passed, 2470 skipped, 77 xfailed, 22 xpassed, 4
> > warnings in 17223.28 seconds
> >
> > I added some dependencies to TEST_DEPENDS to avoid some of the skipped
> > tests (it takes to long to run a full test suite, but the code is
> > clearly working).
> >
> > The only problem that I encountered is that the "make test" should be
> > done after "make fake", against the fake directory, because the port
> > builds some C libraries that are needed in place, if this is a big
> > problem (clearly at least an ugly one...) I can make a test flavor to
> > do a different kind of "in place" build to allow the test to be done
> > without the fake framework but mixing this with the python3 flavor
> > seems adding complexity for nothing in my opinion.
> > if you run:
> > make fake
> > make test
> > The tests run fine and return the results without a problem if you
> > need to see the regression tests in an interactive way.
> > I do not know if the test are done in bulk or something that might be
> > a problem with this.
> >
> > Beside that the fails are minor and it tests A LOT of thinks that 7
> > failures are because timezone in the test machine is not set to UTC
> > and for the lack of some test dependencies that I already added
> > (py-html5lib) for example... the documentation about the test suite is
> > kind of lacking in the project buy it covers pretty everything.
> >
> > Looking for opinions, OKs and someone to commit.
> > :)
> >
> > Attached and available in openbsd-wip if someone else want to see/edit
> > the files online.
> >
> > Cheers.
> > Elias.
Did you know that py-pandas already was in openbsd-wip? You can find it
at https://github.com/jasperla/openbsd-wip/tree/master/math/py-pandas
That said, I had a quick look at your Makefile, and have some comments:
- I think CATEGORIES should be math
- DEPS should be sorted
- Instead of having math/py-numpy${MODPY_FLAVOR}>=1.9.0 in RDEP you
could use ${BUILD_DEPENDS}
- Having py-pandas as test dependency doesn't make sense. Please have
a look at shadchin@'s port as mentioned above
My suggestion would be to contact shadchin@ and try to combine efforts.
BTW pandas would make a great addition to ports.
Re: [NEW] devel/py-pandas
Decided to change the way in witch the regression test is performed. New version attached. 2018-05-27 14:31 GMT-03:00 Elias M. Mariani : > pandas is a Python package providing fast, flexible, and expressive > data structures designed to make working with "relational" or > "labeled" data both easy and intuitive. It aims to be the fundamental > high-level building block for doing practical, real world data > analysis in Python. Additionally, it has the broader goal of becoming > the most powerful and flexible open source data analysis / > manipulation tool available in any language. It is already well on its > way toward this goal. > > Here is the result of a fast pytest test: > 7 failed, 23637 passed, 2470 skipped, 77 xfailed, 22 xpassed, 4 > warnings in 17223.28 seconds > > I added some dependencies to TEST_DEPENDS to avoid some of the skipped > tests (it takes to long to run a full test suite, but the code is > clearly working). > > The only problem that I encountered is that the "make test" should be > done after "make fake", against the fake directory, because the port > builds some C libraries that are needed in place, if this is a big > problem (clearly at least an ugly one...) I can make a test flavor to > do a different kind of "in place" build to allow the test to be done > without the fake framework but mixing this with the python3 flavor > seems adding complexity for nothing in my opinion. > if you run: > make fake > make test > The tests run fine and return the results without a problem if you > need to see the regression tests in an interactive way. > I do not know if the test are done in bulk or something that might be > a problem with this. > > Beside that the fails are minor and it tests A LOT of thinks that 7 > failures are because timezone in the test machine is not set to UTC > and for the lack of some test dependencies that I already added > (py-html5lib) for example... the documentation about the test suite is > kind of lacking in the project buy it covers pretty everything. > > Looking for opinions, OKs and someone to commit. > :) > > Attached and available in openbsd-wip if someone else want to see/edit > the files online. > > Cheers. > Elias. py-pandas.tar.gz Description: GNU Zip compressed data
