Re: [Yade-dev] stability & compatibility between newer and older versions of yade

[Anton Gladky]

Hi all,

please do not forget, that Yade has already a rich opportunity
to create semi-unit-tests (yade --test) and nice semi-integration-tests
(yade --check). Sure, one can create unit tests for pure C++-functions
using boost::unit_test, cppunit or googletest etc., but I would propose
first to extend existing tests, not to spread an energy for the connecting
the new framework into the Yade (+new dependencies, +new cmake-change,
+integration overhead).

And if the existing "yade --test" or "yade --check" are really not enough,
one need to analyze the problem deeper and then make a decision for the
connecting of newer frameworks.

Regards

Anton

Am Mo., 7. Jan. 2019 um 18:26 Uhr schrieb Janek Kozicki :
>
> Bruno Chareyre said: (by the date of Sun, 6 Jan 2019 17:08:12 +0100)
>
> > Thanks for raising this major issue. It would be great to populate unit
> > tests indeed.
> >
> > I recently
> > https://github.com/yade/trunk/commit/b5fbefc6463294f580296cb5727dbbfd733fa8a0
> > introduced a regression test for the utils module and I would like
> > to advertise it here. It is testing only one function from utils
> > currently.
>
> Very interesting, thank you!
>
> I have in mind writing testing code in C++, so that's a little different.
>
> > It needs volunteers to expand it (which can be done simply by reproducing
> > the logic of testUserCreatedInteraction() in more functions). If nobody is
> > going to add tests systematically - the ideal case - I would suggest at
> > least that:
> > *when a bug is fixed a unit test is added simultaneously*.
> > Fixing a bug usually gives a fresh vision of the behavior, which makes
> > writing a unit test easier.
> > Ultimately we could even collectively agree that a bug is not fixed if
> > there is no test proving it.
>
> I would go a bit further: assume that current yade version is the
> reference version. Then I would add test to every C++ function and class
> method. Store the results of tests as a reference result. When a bug
> is found, then the reference result would change. Or worse: it would
> turn out that although function is tested, the test did not catch the
> bug. And that would be great incentive to add a test case where this
> bug happened.
>
> And also this would be a lot easier for others: the code testing
> this function is already written, only another set of input
> parameters must be added to test this case where bug appeared.
>
> Yes, I know that it is a crazy amount of work.
>
> > About 1: would be great provided that it doesn't end-up in simply removing
> > examples which do not work.
>
> Definitely not. The main goal is to really fix all examples. This is
> a perfect opportunity for me to see the latest additions to yade! :)
>
> > Classifying examples is also an important point and I would discourage the
> > previous approach of moving failing scripts to a special
> > "examples/not-working" folder since it breaks the classification in
> > subfolders. Better rename them (something like *.py.fail) while keeping
> > them in their original location.
> > It is less clear if/how you intend to implement the "all examples must
> > work" policy. It is difficult to automatize testing of examples since they
> > are very heterogeneous. For instance some examples don't have a O.run() as
> > user is supposed to click "play" instead.
> > If the error happens after playing the error will not be detected. I
> > suspect many other special situations like this one.
>
> Maybe I would be able to implement this idea in following manner:
> run yade on each example with extra flag --test-example or such.
> This flag would mean that O.run() must be invoked anyways. If user
> has to click it, then yade instead does it. Some parts of examples
> would be untestable like interaction with GUI, in such cases a dummy
> function would be called instead (the point is that example.py need
> not be modified, the --test-example flag should take care of that). If
> examples produce some output file, than that is checked too. I am not
> sure how it will turn up. That's just a general idea.
>
>
> > About 2. I support the idea of investigating new techniques yet I don't
> > understand the suggestion very well. My impression is that all plugins are
> > already eligible for unit tests. For instance, testing a function from
> > utils in [1] did not need any change to the utils module itslef. All it
> > needs is to effectively design and write the unit tests for each other
> > function of each other class/module. That's indeed hundreds - if not
> > thousands - of tests.
>
> Well, time for me to learn what boost::unit_tests has to offer ;)
>
> The general idea within the framework is that it would be able to
> print a list of all publicly accessible C++ methods (not necessarily
> all of them being exported to python) which do not have an
> accompanying test.
>
> I don't know how to achieve this now. That's just an idea.
>
> Then using that list we would know the test coverage ;) If this list
> 

Re: [Yade-dev] Migrating to GitLab

[Anton Gladky]

Hello all,

last several years I did Yade releases and the process was
the following. Before the release was done I created the corresponding
release-branch (for example 0.60 [1]) and just tagged the new
Yade version there. It worked relatively good.

--> develop
  \\
   \ \
\Release 1 \Release 2
 \Release 1.1

I can remember only a couple of times, where the hot-fixes were
needed to be integrated into the release-branch due to some serious
stability or functionality issues.

Last years I have an experience of work with master-develop
approach. It is not bad. But you really need to know, why do
you need it.

I am fully support the feature-branch + merge request way of working.
It can really help to double check the code and implement some
additional tests.

[1] https://github.com/yade/trunk/tree/0.60

My 2cts

Regards

Anton

Am Mo., 7. Jan. 2019 um 17:39 Uhr schrieb Janek Kozicki :
>
> Bruno Chareyre said: (by the date of Mon, 7 Jan 2019 16:59:53 +0100)
>
> > Daily builds would be based on the develop branch.
>
> good, that answer my question from other mail.
>
> > > (by the way, with a tighter control on development, would we still
> > > need a distinction between "yade" and "yadedaily" packages ?..)
> >
> > Yade is stable release, not updated very often, included in main
> > debian/ubuntu repos.
> > Yadedaily is updated  more than daily, after each change to the source
> > code, not included in debian/ubuntu repos.
> > They are very different things and I think we need both.
>
> agreed.
>
> > > Also, with "develop" and "master", I guess any proposal for code
> > > modification would have to be closely examined and validated twice :
> > > - once to make it into "develop"
> > > - and once, to make it from "develop" into "master"
> > > ?...
> > There is no reason to check the develop->master merge if everything in
> > develop is already validated by regtests + human review.
> > Our github/master corresponds to "develop" more or less.
> > Merging develop into master in the new model would correspond to Anton
> > calling for update and releasing 2018.b. More or less.
>
> agred.
>
> > We probably need a liberal, truly unstable repo on the top of this, at
> > least in a transitory phase, so that everyone can play with gitlab a bit
> > and break everything with no limit. For instance to compare --no-ff,
> > --only-ff, and other variants.
>
> how about calling it experimental ? :-))
>
> And yes, we definitely need something like that.
> Where git reset --hard is nothing to be afraid of.
>
> --
> Janek Kozicki
>
> ___
> Mailing list: https://launchpad.net/~yade-dev
> Post to : yade-dev@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~yade-dev
> More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~yade-dev
Post to : yade-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yade-dev
More help   : https://help.launchpad.net/ListHelp

Re: [Yade-dev] Yade is not compatible with CGAL_4.13

[Anton Gladky]

Hello Janek,

thanks for the fix! I have checked it and it really resolves the problem
with the newer CGAL. I will reactivate CGAL-function for the
Debian-build again.

Regards

Anton

Am Mo., 7. Jan. 2019 um 13:44 Uhr schrieb Janek Kozicki :
>
> Bruno Chareyre said: (by the date of Sun, 6 Jan 2019 15:34:11 +0100)
>
> > > Hi Janek,
> > > I think that was the right fix, thanks.
> > > Note that yade is not using the periodic triangulation implemented in CGAL
> > > [1] so there is no question on that point afaik.
> > > Bruno
>
> good, great to know. During the migration I am not committing
> anything to gitlab. I also will wait until you synchronize my commits
> on github to gitlab :)
>
> > > [1] that's because yade included periodicity years before cgal. I actually
> > > implemented a periodicity based on cgal's non-periodic triangulation. Cgal
> > > devs made periodic regular triangulation available more recently -
> > > triggered by me for a part - but it's still not used in yade. Hopefully it
> > > will be used one day but the refactoring it implies is a bit daunting.
>
> Before we even think about doing this, let's first make sure that
> whatever we modify has working tests :)
>
> > I forgot to answer the test coverage question: no, move() is not covered by
> > any test, even indirectly.
> > The function is not used in other parts of the code currently.
>
> That is one of the test cases which I want to discuss in the
> compatibility thread :)
>
> --
> Janek Kozicki
>
> ___
> Mailing list: https://launchpad.net/~yade-dev
> Post to : yade-dev@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~yade-dev
> More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~yade-dev
Post to : yade-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yade-dev
More help   : https://help.launchpad.net/ListHelp

Re: [Yade-dev] stability & compatibility between newer and older versions of yade

[Janek Kozicki]

Bruno Chareyre said: (by the date of Sun, 6 Jan 2019 17:08:12 +0100)

> Thanks for raising this major issue. It would be great to populate unit
> tests indeed.
> 
> I recently
> https://github.com/yade/trunk/commit/b5fbefc6463294f580296cb5727dbbfd733fa8a0
> introduced a regression test for the utils module and I would like
> to advertise it here. It is testing only one function from utils
> currently.

Very interesting, thank you!

I have in mind writing testing code in C++, so that's a little different.

> It needs volunteers to expand it (which can be done simply by reproducing
> the logic of testUserCreatedInteraction() in more functions). If nobody is
> going to add tests systematically - the ideal case - I would suggest at
> least that:
> *when a bug is fixed a unit test is added simultaneously*.
> Fixing a bug usually gives a fresh vision of the behavior, which makes
> writing a unit test easier.
> Ultimately we could even collectively agree that a bug is not fixed if
> there is no test proving it.

I would go a bit further: assume that current yade version is the
reference version. Then I would add test to every C++ function and class
method. Store the results of tests as a reference result. When a bug
is found, then the reference result would change. Or worse: it would
turn out that although function is tested, the test did not catch the
bug. And that would be great incentive to add a test case where this
bug happened.

And also this would be a lot easier for others: the code testing
this function is already written, only another set of input
parameters must be added to test this case where bug appeared.

Yes, I know that it is a crazy amount of work.

> About 1: would be great provided that it doesn't end-up in simply removing
> examples which do not work.

Definitely not. The main goal is to really fix all examples. This is
a perfect opportunity for me to see the latest additions to yade! :)

> Classifying examples is also an important point and I would discourage the
> previous approach of moving failing scripts to a special
> "examples/not-working" folder since it breaks the classification in
> subfolders. Better rename them (something like *.py.fail) while keeping
> them in their original location.
> It is less clear if/how you intend to implement the "all examples must
> work" policy. It is difficult to automatize testing of examples since they
> are very heterogeneous. For instance some examples don't have a O.run() as
> user is supposed to click "play" instead.
> If the error happens after playing the error will not be detected. I
> suspect many other special situations like this one.

Maybe I would be able to implement this idea in following manner:
run yade on each example with extra flag --test-example or such.
This flag would mean that O.run() must be invoked anyways. If user
has to click it, then yade instead does it. Some parts of examples
would be untestable like interaction with GUI, in such cases a dummy
function would be called instead (the point is that example.py need
not be modified, the --test-example flag should take care of that). If
examples produce some output file, than that is checked too. I am not
sure how it will turn up. That's just a general idea. 

 
> About 2. I support the idea of investigating new techniques yet I don't
> understand the suggestion very well. My impression is that all plugins are
> already eligible for unit tests. For instance, testing a function from
> utils in [1] did not need any change to the utils module itslef. All it
> needs is to effectively design and write the unit tests for each other
> function of each other class/module. That's indeed hundreds - if not
> thousands - of tests.

Well, time for me to learn what boost::unit_tests has to offer ;)

The general idea within the framework is that it would be able to
print a list of all publicly accessible C++ methods (not necessarily
all of them being exported to python) which do not have an
accompanying test.

I don't know how to achieve this now. That's just an idea.

Then using that list we would know the test coverage ;) If this list
would someday become empty then we could say with confidence that we
have 100% test coverage. If someone writes a new public function in
some class, then even without exporting it to python, it would be
caught and printed as a warning, that it has no accompanying test.

Your _Tesselation::VertexHandle _Tesselation::move(…)
should be caught automatically.

I hope that it is possible. Maybe only a slight modification to
YADE_PLUGIN or similar macro would be just enough? I don't know yet.
Or maybe use some code for reading the library objects: it would go
though all functions inside the binary library file, and try{}catch{}
attempt to test them. I know that it is possible to read library
symbols, I need to check how to do that.
In that case each instance of _Tesselation::move(…) for all TT
that ended up in the library file would be caught.

-- 
Janek Kozicki

Re: [Yade-dev] Migrating to GitLab

[Janek Kozicki]

Bruno Chareyre said: (by the date of Mon, 7 Jan 2019 16:59:53 +0100)

> Daily builds would be based on the develop branch.

good, that answer my question from other mail.
 
> > (by the way, with a tighter control on development, would we still 
> > need a distinction between "yade" and "yadedaily" packages ?..)  
> 
> Yade is stable release, not updated very often, included in main 
> debian/ubuntu repos.
> Yadedaily is updated  more than daily, after each change to the source 
> code, not included in debian/ubuntu repos.
> They are very different things and I think we need both.

agreed.
 
> > Also, with "develop" and "master", I guess any proposal for code 
> > modification would have to be closely examined and validated twice :
> > - once to make it into "develop"
> > - and once, to make it from "develop" into "master"
> > ?...  
> There is no reason to check the develop->master merge if everything in 
> develop is already validated by regtests + human review.
> Our github/master corresponds to "develop" more or less.
> Merging develop into master in the new model would correspond to Anton 
> calling for update and releasing 2018.b. More or less.

agred.

> We probably need a liberal, truly unstable repo on the top of this, at 
> least in a transitory phase, so that everyone can play with gitlab a bit 
> and break everything with no limit. For instance to compare --no-ff, 
> --only-ff, and other variants.

how about calling it experimental ? :-))

And yes, we definitely need something like that.
Where git reset --hard is nothing to be afraid of.

-- 
Janek Kozicki

___
Mailing list: https://launchpad.net/~yade-dev
Post to : yade-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yade-dev
More help   : https://help.launchpad.net/ListHelp

Re: [Yade-dev] [Bug 1810283] Re: Wiki homepage broken

[Janek Kozicki]

Bruno Chareyre said: (by the date of Mon, 07 Jan 2019 15:23:11 -)

> To be honest that wiki hosted on some 3SR server will get less and less 
> support with Rémi gone.
> I don't even know which computer exactly is hosting this.
> 
> Finding a replacement would be a good thing. Gitlab is hosting wiki pages 
> indeed (even Github actually) but I don't if it is possible to just dump 
> everything and upload it elsewhere...
> If the bug appear randomly I suspect an issue on the host computer, like full 
> HD or exhausted RAM. 

I propose to not edit wiki until the migration process to gitlab is
finished. Maybe Rémi could find the hosting computer and send us the
compressed directory with it?

Then we would investigate possible ways to import it into gitlab?

-- 
Janek Kozicki

___
Mailing list: https://launchpad.net/~yade-dev
Post to : yade-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yade-dev
More help   : https://help.launchpad.net/ListHelp

Re: [Yade-dev] Migrating to GitLab

[Janek Kozicki]

Bruno Chareyre said: (by the date of Sun, 6 Jan 2019 23:04:13 +0100)

> Hi Janek, I think we are on the same line more or less. We only differ on
> practical details, mainly the notion of merge request in gitlab.

yes, I noticed the same thing too. It seems that my suggestion can be
applied in miniature to each repo. While repos act as "larger
branches", and the same strategy works between them, just on a larger
scale :)

> On Sat, 5 Jan 2019 at 00:15, Janek Kozicki  wrote:
> >
> > I just noticed that with my CGAL 4.13 attempt to fix I did a push to
> > master. Which felt uneasy.
> >  
> Why? Do you refer to github or gitlab repo here? (remember that gitlab repo
> is not known by any other dev yet)

I refer to github. I felt uneasy because I would prefer to push new
stuff to develop. Not master. Master, at least for me, feels like
"it's certified to work" :)
If you run into problems on develop branch, you always know, that you
can compare with master and get reproducible results. Maybe `master`
could be called `stable` to reflect how I feel this.

> > I suppose that using
> > https://nvie.com/posts/a-successful-git-branching-model/
> > would help a lot. (short summary on
> > https://nvie.com/files/Git-branching-model.pdf )
> 
> It sounds good.
> 
> > I propose that:
> > 1. we will use feature branches with a specific name to hot-develop
> > features. (e.g. fluid)
> > 2. when feature is "almost ready" we merge it into develop branch
> > 3. when develop branch is stable we merge it into master using a release
> > branch
> 
> I agree. Only the last part "using a release branch" is not completely
> clear to me.

The "release branch" is actually useful, at least for me. I have a
short cheat-sheet for that, here it goes:
(it is from https://nvie.com/posts/a-successful-git-branching-model/ )

===
RELEASE BRANCH:
Release branches are created from the develop branch:

$ git checkout -b release-1.2 develop
$ ./bump-version.sh 1.2   # or whatever edits are necessary to change version 
number
$ git commit -a -m "Bumped version number to 1.2"

Finishing a release branch: merge to master

$ git checkout master
$ git merge --no-ff release-1.2
$ git tag -a 1.2

Merge to develop

$ git checkout develop
$ git merge --no-ff release-1.2

Delete unused branch:
$ git branch -d release-1.2
===

It is a little fiddling between the two branches. The point is, that
whatever conflicts arise during the merge, they do not mess up with
develop branch, and not mess up with master branch. They get solved
inside release branch. Which is deleted after merge is a success.

Most of the time, it's just 'nothing', when stuff goes smoothly. But
in some cases when you get conflicts, you suddenly are glad that you
are not on develop or master branch :) For example if you have problems
with this line:
$ git merge --no-ff release-1.2
Then you abort the merge (git reset --merge ORIG_HEAD
https://medium.com/@porteneuve/mastering-git-reset-commit-alchemy-ba3a83bdfddc 
funny read btw ;)).
Everything is reset on master, and you go back to release-1.2 to fix there.


Actually I have more of this cheat-sheet, for hotfixes it goes like that:
===
HOTFIX BRANCH:
Creating the hotfix branch:

$ git checkout -b hotfix-1.2.1 master
$ ./bump-version.sh 1.2.1
$ git commit -a -m "Bumped version number to 1.2.1"

Fix stuff, then:
$ git commit -m "Fixed severe production problem"

Finishing a hotfix branch, update master and tag the release:
$ git checkout master
$ git merge --no-ff hotfix-1.2.1
$ git tag -a 1.2.1

Merge into develop:
$ git checkout develop
$ git merge --no-ff hotfix-1.2.1

Example if git history messes up:
$ git reset --merge ORIG_HEAD
Then for example only pick specific commits:
$ git cherry-pick 81f7007
$ git cherry-pick 5bf2e3a

Delete unused branch:
$ git branch -d hotfix-1.2.1
===


> > The main branches
> > * master
> > * develop
> 
> Definitely.
> 
> Other branches:
> > * Feature branches
> >+ branch from develop (with command: git checkout -b myfeature develop)
> >+ merge to develop (with command: git checkout develop ; git merge
> > --no-ff myfeature )
> >+ naming: anything except `master`, `develop`, `release-*`, or
> > `hotfix-*`
> >  
> 
> Yes.
> Until now feature branches (e.g. yade-mpi [1]) are not centralized. They
> remain under personal github accounts.
> Whether those branches are/not pushed into the upstream repo as stateful
> branches makes no real difference in practice (in my view at least).
> github.com/bchareyre/yade-mpi/master is exactly one of the feature branches
> as they appear in your model. It will be merged into
> github.com/yade/trunk/master  when ready (or
> into a "develop" branch).
> Equivalently I could have pushed a new "mpi" branch to yade/trunk, then
> start working on it, and later merge to yade/master.
> It makes nearly no difference in the final result but the later has
> downsides:
> - more noise on yade-dev (notifications for each push)
> - less flexibility  

Re: [Yade-dev] Migrating to GitLab

[Bruno Chareyre]

On 1/7/19 1:12 PM, Jerome Duriez wrote:

I could see just one point having an additional "develop": in the case 
where eg yadedaily packages would be built from master only, and not 
from develop. Is this a plan ?


Daily builds would be based on the develop branch.

(by the way, with a tighter control on development, would we still 
need a distinction between "yade" and "yadedaily" packages ?..)


Yade is stable release, not updated very often, included in main 
debian/ubuntu repos.
Yadedaily is updated  more than daily, after each change to the source 
code, not included in debian/ubuntu repos.

They are very different things and I think we need both.

Also, with "develop" and "master", I guess any proposal for code 
modification would have to be closely examined and validated twice :

- once to make it into "develop"
- and once, to make it from "develop" into "master"
?...
There is no reason to check the develop->master merge if everything in 
develop is already validated by regtests + human review.

Our github/master corresponds to "develop" more or less.
Merging develop into master in the new model would correspond to Anton 
calling for update and releasing 2018.b. More or less.


We probably need a liberal, truly unstable repo on the top of this, at 
least in a transitory phase, so that everyone can play with gitlab a bit 
and break everything with no limit. For instance to compare --no-ff, 
--only-ff, and other variants.


Bruno



___
Mailing list: https://launchpad.net/~yade-dev
Post to : yade-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yade-dev
More help   : https://help.launchpad.net/ListHelp

[Yade-dev] [Bug 1810283] Re: Wiki homepage broken

[Bruno Chareyre]

To be honest that wiki hosted on some 3SR server will get less and less support 
with Rémi gone.
I don't even know which computer exactly is hosting this.

Finding a replacement would be a good thing. Gitlab is hosting wiki pages 
indeed (even Github actually) but I don't if it is possible to just dump 
everything and upload it elsewhere...
If the bug appear randomly I suspect an issue on the host computer, like full 
HD or exhausted RAM. 

Bruno

-- 
You received this bug notification because you are a member of Yade
developers, which is subscribed to Yade.
https://bugs.launchpad.net/bugs/1810283

Title:
  Wiki homepage broken

Status in Yade:
  Invalid

Bug description:
  For the record, https://www.yade-dem.org/wiki/Yade currently just
  outputs here:

  ***
  Database error

  From Yade
  A database query syntax error has occurred. This may indicate a bug in the 
software. The last attempted database query was:

  (SQL query hidden)

  from within function "SqlBagOStuff::set". Database returned error "1114: The 
table 'yade_objectcache' is full (localhost)".
  ***

  No idea whether it's a problem on wikimedia's, or our side ?

To manage notifications about this bug go to:
https://bugs.launchpad.net/yade/+bug/1810283/+subscriptions

___
Mailing list: https://launchpad.net/~yade-dev
Post to : yade-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yade-dev
More help   : https://help.launchpad.net/ListHelp