Re: [sage-devel] python3 report (end of august 2018)
Another easy diagnostic: look at /proc/$PID/fd/* and see which file(-descriptors) are open. On Monday, August 27, 2018 at 7:21:45 AM UTC+2, Snark wrote: > > That probably means the doctesting framework is leaking file > descriptors, ie: it opens each and every file but never closes them. > > I would search: > (1) either a single piece of code which does all the opening and never > closes anything during the whole run (the least likely scenario) > (2) or a piece of code forking correct workers (ie: they would > open+close correctly) but leaking them (so the closing part doesn't > happen) (the most likely scenario). > > Someone familiar with the code should probably have a look and would be > more efficient at spotting the problem. > > jpuydt on irc.debian.org > -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] python3 report (end of august 2018)
On Sun, Aug 26, 2018 at 5:47 PM Frédéric Chapoton wrote: > > Dear all, here is a short python3 status report. > > There is a patchbot report with python3 for 8.4.b2 somewhere on the page : > https://patchbot.sagemath.org/ticket/0/ > (@ patchbot breeders, please update your patchbot so that we know if it > builds and runs sage with python2 or python3.) > > We are now hopefully getting close to making the documentation (html and pdf) > build with python3. One good step in this direction would be to merge the 4 > waiting pull requests in sagenb (and then upgrade the sagenb spkg, see > https://trac.sagemath.org/ticket/25837). > > Otherwise, some progress has been made on the hash issue > (https://trac.sagemath.org/ticket/24551). There remains in particular the > controversial question of the hash issue for groups. There is no general > agreement on the meaning of equality for groups, and this will probably make > the definition of hash difficult. > > The patchbot report on sage+python3 for 8.4.b2 has still many (and I mean > many) failing doctests. But it also stops badly in the way shown below, > complaining that too many files are open. Could somebody investigate this > issue, please ? I'm not sure what's going on there--I haven't seen that before. But I also haven't merged 8.4.beta2 into my python3 branch yet. I think it's still based on 8.3.something. For comparison though, on my python3 branch, which admittedly I haven't updated in 2 or 3 weeks, but which has many fixes merged into it that are not yet merged into develop, I have: $ grep -R "doctests\? failed" logs/ptest.log | wc -l 465 or ~465 modules with failing doctests. Of those, most of them do not have hundreds of failures as in your report. Here are the top 10 in number of failures: $ grep 'doctests\? failed' logs/ptest.log | sort -k 5 -nr | head -10 sage -t src/sage/misc/explain_pickle.py # 71 doctests failed sage -t src/sage/graphs/generic_graph.py # 25 doctests failed sage -t src/sage/combinat/diagram_algebras.py # 25 doctests failed sage -t src/sage/combinat/finite_state_machine.py # 24 doctests failed sage -t src/sage/combinat/rigged_configurations/kr_tableaux.py # 23 doctests failed sage -t src/sage/matroids/matroid.pyx # 20 doctests failed sage -t src/sage/categories/finite_dimensional_algebras_with_basis.py # 18 doctests failed sage -t src/sage/modular/modsym/ambient.py # 17 doctests failed sage -t src/sage/categories/quantum_group_representations.py # 17 doctests failed sage -t src/sage/quadratic_forms/genera/normal_form.py # 16 doctests failed Of these, I know many of the failures to be due to slight ordering differences in the output, such as of things like sets, or results from graph algorithms that are not exactly deterministic (at least between Python versions). Of course, there are also still a number of outright exceptions: $ grep -R "Exception raised" logs/ptest.log | wc -l 613 To be clear, this was also with just `make ptest`, not `make ptestlong`. The numbers are probably a bit higher with some `# long time` tests. But I think what this shows is that we're in better shape on Python 3 than I think the OP suggests, at least in theory. We just need to get more of the existing fixes cleaned up so that they can be merged. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] python3 report (end of august 2018)
Hi, On 26/08/2018 17:47, Frédéric Chapoton wrote: The patchbot report on sage+python3 for 8.4.b2 has still *many* (and I mean *_many_*) failing doctests. But it also stops badly in the way shown below, complaining that too many files are open. Could somebody investigate this issue, please ? That probably means the doctesting framework is leaking file descriptors, ie: it opens each and every file but never closes them. I would search: (1) either a single piece of code which does all the opening and never closes anything during the whole run (the least likely scenario) (2) or a piece of code forking correct workers (ie: they would open+close correctly) but leaking them (so the closing part doesn't happen) (the most likely scenario). Someone familiar with the code should probably have a look and would be more efficient at spotting the problem. jpuydt on irc.debian.org -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] python3 report (end of august 2018)
Dear all, here is a short python3 status report. There is a patchbot report with python3 for 8.4.b2 somewhere on the page : https://patchbot.sagemath.org/ticket/0/ (@ patchbot breeders, please update your patchbot so that we know if it builds and runs sage with python2 or python3.) We are now hopefully getting close to making the documentation (html and pdf) build with python3. One good step in this direction would be to merge the 4 waiting pull requests in sagenb (and then upgrade the sagenb spkg, see https://trac.sagemath.org/ticket/25837). Otherwise, some progress has been made on the hash issue (https://trac.sagemath.org/ticket/24551). There remains in particular the controversial question of the hash issue for groups. There is no general agreement on the meaning of equality for groups, and this will probably make the definition of hash difficult. The patchbot report on sage+python3 for 8.4.b2 has still *many* (and I mean *many*) failing doctests. But it also stops badly in the way shown below, complaining that too many files are open. Could somebody investigate this issue, please ? And I would like to know if some buildbot is using the python3 make target that is now available ? Frédéric sage -t --long src/sage/combinat/crystals/kac_modules.py # 3 doctests failed sage -t --long src/sage/matroids/matroids_plot_helpers.py # 1 doctest failed sage -t --long src/sage/libs/gap/element.pyx # 2 doctests failed sage -t --long src/sage/groups/conjugacy_classes.py # 2 doctests failed sage -t --long src/sage/docs/instancedoc.pyx # 1 doctest failed sage -t --long src/sage/rings/padics/factory.py # 104 doctests failed Doctests interrupted: 843/3700 files tested -- Total time for all tests: 6360.5 seconds cpu time: 10897.2 seconds cumulative wall time: 12855.4 seconds Traceback (most recent call last): File "/home/u1/chapoton/sage3/src/bin/sage-runtests", line 156, in err = DC.run() File "/home/u1/chapoton/sage3/local/lib/python3.6/site-packages/sage/doctest/control.py" , line 1187, in run self.run_doctests() File "/home/u1/chapoton/sage3/local/lib/python3.6/site-packages/sage/doctest/control.py" , line 915, in run_doctests self.dispatcher.dispatch() File "/home/u1/chapoton/sage3/local/lib/python3.6/site-packages/sage/doctest/forker.py" , line 1912, in dispatch self.parallel_dispatch() File "/home/u1/chapoton/sage3/local/lib/python3.6/site-packages/sage/doctest/forker.py" , line 1809, in parallel_dispatch w.start() # This might take some time File "/home/u1/chapoton/sage3/local/lib/python3.6/site-packages/sage/doctest/forker.py" , line 2095, in start super(DocTestWorker, self).start() File "/home/u1/chapoton/sage3/local/lib/python3.6/multiprocessing/process.py", line 105, in start self._popen = self._Popen(self) File "/home/u1/chapoton/sage3/local/lib/python3.6/multiprocessing/context.py", line 223, in _Popen return _default_context.get_context().Process._Popen(process_obj) File "/home/u1/chapoton/sage3/local/lib/python3.6/multiprocessing/context.py", line 277, in _Popen return Popen(process_obj) File "/home/u1/chapoton/sage3/local/lib/python3.6/multiprocessing/popen_fork.py", line 19, in __init__ self._launch(process_obj) File "/home/u1/chapoton/sage3/local/lib/python3.6/multiprocessing/popen_fork.py", line 65, in _launch parent_r, child_w = os.pipe() OSError: [Errno 24] Too many open files By the way, here are the worst offenders, those having hundreds of failing doctests: sage -t --long src/sage/schemes/elliptic_curves/ell_number_field.py # 107 doctests failed sage -t --long src/sage/modular/pollack_stevens/modsym.py # 199 doctests failed sage -t --long src/sage/schemes/hyperelliptic_curves/ hyperelliptic_finite_field.py # 126 doctests failed sage -t --long src/sage/modular/modform/element.py # 346 doctests failed sage -t --long src/sage/rings/padics/padic_lattice_element.py # 167 doctests failed sage -t --long src/sage/rings/valuation/augmented_valuation.py # 256 doctests failed sage -t --long src/sage/graphs/generic_graph.py # 136 doctests failed sage -t --long src/sage/combinat/partition_kleshchev.py # 116 doctests failed sage -t --long src/sage/dynamics/arithmetic_dynamics/projective_ds.py # 121 doctests failed sage -t --long src/sage/algebras/steenrod/steenrod_algebra.py # 225 doctests failed sage -t --long src/sage/schemes/elliptic_curves/ell_curve_isogeny.py # 214 doctests failed sage -t --long src/sage/combinat/tiling.py # 144 doctests failed sage -t --long src/sage/schemes/toric/sheaf/klyachko.py # 133 doctests failed sage -t --long src/sage/modular/abvar/abvar.py # 606 doctests failed sage -t --long src/sage/modular/modsym/ambient.py # 352 doctests failed sage -t --long src/sage/schemes/toric/chow_group.py # 195 doctests failed sage -t --long src/sage/rings/asymptotic/asymptotic_ring.py
