Hi!
Thanks Anne and Travis for starting the discussion!
On Sun, Nov 03, 2013 at 05:48:19PM -0800, Anne Schilling wrote:
> Here is an initial list of discussion points regarding the Sage-combinat
> queue:
>
> - Should we use a fork for sage-combinat? Or branches for the various patches
> or projects?
> * If we do use a fork, how to sync the combinat fork with the main sage
> fork? In particular, once branches are merged in the main fork, can they
> merge in the fork as well?
> * If we don't fork, then is there some way we can tie all of the
> combinat branches together? Specifically can we list all branches
> which don't have ticket numbers.
At the beginning we were a relatively small community and there were
lots of interactions across the whole "combinat" code base, because we
had to build the foundations. Hence there was a relatively clear
notion of "combinat patch" (which somehow included categories) and it
was good to have a relatively centralized workflow. Now the core is
stabilizing and the community has grown quite much both in terms of
number of contributors and of breadth of topics. It's thus naturally
evolving from a tightly knit community into a continuum of overlapping
subcommunities of Sage developers that gather together at the occasion
of specific projects. And that's great!
Now, we want the workflow to reflect that. The dichotomy "in
Sage-Combinat or not" that the queue enforces has become a bother,
especially for borderline people like Simon.
With that in mind, I would aim for just using a bunch of branches in
the sage git repository, with a way to group those in collections that
would model our subcommunities: combinat / cluster_algebras /
categories / tableaux / crystals / dynamics / semigroups / ...
As Simon points out, some tags mechanism would be great for this.
Here is what feels like the largest hurdles we might face; I might be
completely wrong though, and we probably can only know for sure
through experimenting at a large scale with the new workflow:
- Switching from one branch to the other.
As Simon pointed out, this will be easy. However whether it's
practical or not will depend on how much time we will have to wait
for recompilation in average. Part of the answer will be in the
"compilation cache"[1] infrastructure that Andrew & co have been
exploring, but I don't know how far this project progressed.
- Mitigation of the divergence between parallel development branches.
An important feature of the workflow is to be able to request things
like ``pull on my local Sage all the experimental features about
tableaux, categories, and semigroups''.
Typical applications: I want to run a particular calculation for
which I need a bunch of different in-development features. Or I want
to share with a non-programmer colleague the latest semigroup
code. That's what "sage -combinat install" was about; granted it
very was far from 100% robust, but it worked relatively often.
In theory, this is easy: it's just about creating a temporary merge
point for a bunch of branches. But will this be robust in practice?
That is will we be able to maintain the state of our branches in a
consistent enough state that the most-often-needed combinations of
branches will actually merge without conflicts?
Of course the new workflow, and the fact that the development is
progressively moving from the core to the periphery, is likely to
reduce the life span of branches, which in turn will reduce the need
for the above.
- Dependency handling
> - How do we transform the queue into git branches,
> how do we handle dependencies?
Back in March, I wrote a script which was based on the sage
development tools and did just this (attached; see the
import_mercurial_queue function). With it, I was able to import most
of the Sage-Combinat queue [2] as a bunch of branches with
dependencies (see the graph attached to [2]).
So, in principle that should be easy technically. However:
- The sage dev tools have evolved quite much since then, and I have
not played with the script since then. So it certainly needs to be
rebased on the latest dev tools. And it also deserves some cleanup.
- A good part of the work was to annotate the series file with
explicit dependencies information. Part of this information is on
trac, but in practice it's often outdated or incomplete, especially
for the more experimental patches. Thus that required quite some
manual trial-and-error work (oh, it does not apply! hmm, let's see
why ...). And now this information needs to be updated again
w.r.t. all the changes in the queue in the past six months. Note
that this is not wasted time, as we need to update this dependency
information on trac anyway at some point.
- We need to decide whether we do the migration all at once, or
instead we let the patch developers migrate progressively their own
patches. It's probably easier to do it all at once, if we can afford
it (essentially if there is a clear line between those patches to
finalize right now in mercurial, and those patches that we want to
develop further in git).
> - When will the change occur?
> * Will we freeze the hg queue at that point?
Probably yes? It would be a nice feature if we could leave the queue
in a state where it would install smoothly on, say, Sage 5.12 and Sage
5.13, so that the Sage-Combinat users could keep playing with it for a
while.
Maybe we should leave it open for tiny changes like the finalization
of patches that are just about to go into Sage 5.13 (typically if the
release manager notices a glitch in a positively reviewed patch, we
don't want this to bump that patch to 5.14). I am of course thinking
about #10963 here in case the migration would occur soon.
Looking forward the progress this week! Have fun!
Cheers,
Nicolas
[1] http://fr.wikipedia.org/wiki/Ccache
[2] https://groups.google.com/d/msg/sage-combinat-devel/hGsalRNWZ3U/c3Uk_1RK6OQJ
--
Nicolas M. Thiéry "Isil" <nthi...@users.sf.net>
http://Nicolas.Thiery.name/
--
You received this message because you are subscribed to the Google Groups
"sage-combinat-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-combinat-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-combinat-devel@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-combinat-devel.
For more options, visit https://groups.google.com/groups/opt_out.
import io
import os
import re
import sagedev
import subprocess
"""
sage: import sagedev
sage: %attach sage-combinat.py
sage: cd /opt/sage-git
sage: patch_dir = "/opt/sage/devel/sage/.hg/patches/"
sage: s = sagedev.SageDev()
sage: git = s.git
sage: git.checkout("master")
sage: git.branch("-D", "t/9107")
Deleted branch t/9107 (was 031ee52).
sage: s.import_patch(local_file=patch_dir+"trac9107_nesting_nested_classes.patch", ticketnum=9107)
sage: s.import_patch(local_file=patch_dir+"trac_9107_fix_cross_reference.patch", ticketnum=9107)
sage: git.checkout("master")
# Breaks due to the removal of new lines
sage: git.branch("-D", "t/14140")
sage: s.import_patch(local_file=patch_dir+"trac_14140-remove_cc_set_partitions-ts.patch", ticketnum=14140)
sage: git.checkout("master")
sage: s.import_patch(local_file=patch_dir+"trac_14248-global_options_case-ts.patch", ticketnum=14248)
sage: git.checkout("master")
sage: s.import_patch(local_file=patch_dir+"trac_14299-gelfand_tsetlin_patterns-ts.patch", ticketnum=14299)
sage: git.checkout("master")
sage: s.import_patch(local_file=patch_dir+"trac_13624-dot2tex-verb_workaround-nt.patch", ticketnum=13624)
"""
sage_hg_dir="/opt/sage/devel/sage-combinat/"
sage_git_dir="/opt/sage-git/"
def ticket_and_branch_of_patch(name):
ticket = get_ticket(name)
if ticket:
branch = "t/%s"%ticket
else:
branch = "p/%s"%name
return ticket, branch
def branch_of_patch(name):
return ticket_and_branch_of_patch(name)[1]
def import_patch(patches, depends=[], patch_dir=sage_hg_dir+".hg/patches/"):
"""
EXAMPLES::
sage: %attach sage-combinat.py
sage: import sagedev
sage: cd /opt/sage-git
sage: destroy_all_branches()
sage: import_patch("trac9107_nesting_nested_classes.patch")
sage: import_patch("trac_12876_category-fix_abstract_class-nt-rel11521.patch")
sage: import_patch("trac_12876_category-fix_abstract_class-nt-rel11521-review-nt.patch", append=True)
sage: import_patch("trac11935_weak_pickling_by_construction_rel11943-nt.patch", depends=[12876])
sage: import_patch("trac11935_share_on_base_category.patch", append=True)
sage: import_patch("trac_12894-classcall_setter-nt.patch")
"""
print patches
assert patches
if isinstance(patches, str):
name = patches
patches = [name]
else:
name = patches[0]
print "== Importing %s "%short_name(name) + "="*max(60-len(name), 0)
ticket, branch = ticket_and_branch_of_patch(name)
s = sagedev.SageDev()
s.git.checkout("master")
if s.git.branch_exists(branch):
s.git.branch("-D", branch)
#s.git.create_branch(branch)
s.git.create_branch(branch, remote_branch=False)
if ticket:
s.git._ticket[branch] = ticket
assert s.git.branch_exists(branch)
s.git.checkout(branch)
for dependency in depends:
print "* Applying dependency: %s"%dependency
branch = branch_of_patch(dependency)
s.git.merge(branch)
for patch in patches:
if os.path.getsize(patch_dir+patch) > 0:
print "* Applying patch %s"%patch
s.import_patch(local_file=patch_dir+patch)
else:
print "* Empty patch; nothing to do!"
def destroy_all_branches():
s = sagedev.SageDev()
branches = s.git.local_branches()
s.git.checkout("master")
for branch in branches:
if branch != "master":
s.git.branch("-D", branch)
def git_reset():
s = sagedev.SageDev()
s.git.am("--abort")
s.git.stash()
s.git.checkout("master")
s.git.clean("-f", "-d")
HG_SERIES_REGEXP="(\S+)\s*#.*?deps: (.*?)(?:#.*)?$"
"""
sage: re.match(HG_SERIES_REGEXP, "bla.patch #+1 depends: 1111 trac_111.patch").groups()[0].split()
['1111', 'trac_111.patch']
"""
def ticket_or_patch(s):
"""
EXAMPLES::
sage: ticket_or_patch(u"123")
123
sage: ticket_or_patch(u"trac_123.patch")
u'trac_123.patch'
"""
try:
return int(s)
except ValueError:
return s
def get_dependencies(sage_hg_dir=sage_hg_dir):
"""
Extract the dependencies encoded in the series file of a mercurial patch queue
The dependencies should be expressed by adding in the series file
a comment after the patch of the form ``deps: dep1 ... ``. For
example:
``blah.patch # deps: 12344 blih.patch # trivial``
``blah.patch #+foo deps: 12344 blih.patch # trivial``
EXAMPLES::
sage: get_dependencies("/opt/sage/devel/sage/")
{u'12940': ['8392'],
u'14143': ['4327', '14252'],
u'14192': ['14252', '14143']}
"""
sage_patch_dir = sage_hg_dir+"/.hg/patches/"
series = list(io.open(sage_patch_dir+"series"))
dependencies = {}
for line in series:
match = re.match(HG_SERIES_REGEXP, line)
if match:
patch, deps = match.groups()
name = short_name(patch)
deps = map(short_name, deps.split())
if name in dependencies:
dependencies[name].extend(deps)
else:
dependencies[name] = deps
return dependencies
def get_patches(sage_hg_dir=sage_hg_dir):
"""
Return the list of active patches (applied or not).
EXAMPLES::
sage: get_patches()
['trac9107_nesting_nested_classes.patch',
'trac_9107_fix_cross_reference.patch',
...
'trac_13855_planar_binary_trees_hopf_algebra-EliX-jbp.patch']
"""
return (subprocess.check_output(["hg", "qapplied", "--cwd", sage_hg_dir]) +
subprocess.check_output(["hg", "qunapplied", "--cwd", sage_hg_dir])).split()
def get_ticket(patch):
"""
Return the ticket number for this patch, or None
EXAMPLES::
sage: get_ticket("trac_13423_bla-vt.patch")
13423
sage: get_ticket("bla-vt.patch")
"""
match = re.match(".*?(\d\d\d\d+)",patch)
if match:
return int(match.groups()[0])
return None
def short_name(patch):
"""
Return a short name for the patch
If the patch corresponds to a ticket, return the ticket number;
otherwise return the patch name.
EXAMPLES::
sage: short_name("trac_13423_bla-vt.patch")
'13423'
sage: short_name("bla-vt.patch")
'bla-vt.patch'
"""
ticket = get_ticket(patch)
if ticket:
return str(ticket)
else:
return patch
def dependency_graph(sage_hg_dir=sage_hg_dir):
"""
Return the dependency graph for the given mercurial patch queue
EXAMPLES::
sage: dependency_graph()
"""
G = DiGraph()
G.add_vertices(short_name(patch) for patch in get_patches(sage_hg_dir))
for name, deps in get_dependencies(sage_hg_dir).iteritems():
for dep in deps:
G.add_edge(name, dep)
return G
def import_mercurial_queue(sage_hg_dir=sage_hg_dir, sage_git_dir=sage_git_dir, resume=False):
"""
Import the patches in a mercurial queue
INPUT:
- ``resume`` -- a boolean
Whether to skip existing branches, or recreate them from scratch
EXAMPLES::
sage: %attach sage-combinat.py
sage: cd /opt/sage-git
sage: import_mercurial_queue(sage_hg_dir="/opt/sage/devel/sage-combinat/")
"""
os.chdir(sage_git_dir)
#destroy_all_branches()
s = sagedev.SageDev()
git_reset()
dependencies = get_dependencies()
def handle(patches):
if not patches:
return
name = patches[0]
ticket, branch = ticket_and_branch_of_patch(name)
if s.git.branch_exists(branch):
if resume:
return
else:
s.git.branch("-D", branch)
deps = dependencies.get(short_name(name), [])
import_patch(patches, depends=deps, patch_dir=sage_hg_dir+".hg/patches/")
patches = []
previous_ticket = None
for patch in get_patches():
ticket = get_ticket(patch)
if ticket is not None and ticket == previous_ticket:
patches.append(patch)
previous_ticket = ticket
else:
handle(patches)
patches = [patch]
previous_ticket = ticket
handle(patches)
return
#patches = (subprocess.check_output(["hg", "qapplied", "--cwd", sage_hg_dir]) +
# subprocess.check_output(["hg", "qapplied", "--cwd", sage_hg_dir])).split()
import_patch("trac9107_nesting_nested_classes.patch")
import_patch("trac_2023-dynkin_graphs-ts.patch")
# Applies on second thought
import_patch("trac_14252-KRLS-as.patch", )
import_patch("trac_14094-partition_iterator-mh.patch")
import_patch("trac_14094-partition_iterator-review-ts.patch", append=True)
import_patch("trac_14145-fix_contains_tableau-ts.patch")
import_patch("trac_8392-check_permutation-ts.patch")
import_patch("trac_13871-virtual_cartan_type-ts.patch")
import_patch("trac_13838-virtual_kleber_alg-ts.patch")
import_patch("trac_13872-RC_bijections-ts.patch", depends=[13838])
import_patch("trac9107_nesting_nested_classes.patch")
import_patch("trac_9107_fix_cross_reference.patch", append=True)
import_patch("trac_14140-remove_cc_set_partitions-ts.patch")
import_patch("trac_14094-partition_iterator-mh.patch")
import_patch("trac_14094-partition_iterator-review-ts.patch", append=True)
import_patch("trac_14248-global_options_case-ts.patch")
import_patch("trac_14299-gelfand_tsetlin_patterns-ts.patch")
import_patch("trac_13624-dot2tex-verb_workaround-nt.patch")
import_patch("trac_10054-parent_gen_words-ts.patch")
import_patch("trac_14141-knutson_tao_puzzles-fs.patch")
import_patch("trac_14223-plot-aspect_ratio-nt.patch")
import_patch("trac_4327-root_system_plot_refactor-nt.patch")
import_patch("trac_14143-alcove-path-al.patch", depends=[14252])
import_patch("trac_14192-infinity_crystal-bs.patch", depends=[14252,14143]) # Trivial textual dependencies
import_patch("trac_10170-bell_number_improvements-ts.patch")
import_patch("dynamics-iet-tutorial.patch") # causes problem?
import_patch("trac_10193-graded_enumerated_sets-vd_no_more_nt.patch")
import_patch("trac_10193-review-nb.patch", append=True)
import_patch("trac_10193-more-vd.patch", append=True)
import_patch("trac_10194-factories_policy-fh.patch")
import_patch("trac_12940_affine_permutations-td.patch", depends=[8392]) # Trivial dependency in sage.combinat.all
import_patch("trac_12876_category-fix_abstract_class-nt-rel11521.patch") # Does not apply yet (end-of-file whitespace)
import_patch("trac_12876_category-fix_abstract_class-nt-rel11521-review-nt.patch", append=True)
import_patch("trac11935_weak_pickling_by_construction_rel11943-nt.patch", depends=[12876])
import_patch("trac11935_share_on_base_category.patch", append=True)
import_patch("trac_12894-classcall_setter-nt.patch")
import_patch("trac_12895-subcategory-methods-nt.patch", depends=[11935,12894])
import_patch("trac_13580-map_reduce-old-fh.patch") # Does not apply yet (end-of-file whitespace)
import_patch("trac_13433-lazy_power_serie_gen_fix-fh.patch")
import_patch("finite_set_map-isomorphic_cartesian_product-nt.patch")
import_patch("trac_12848-posets-order_ideal_complement_generators_fix-nt.patch")
import_patch("trac_12920-is_test_methods-nt.patch")
import_patch("doc_underscore-fh.patch")
import_patch("trac_8703-trees-fh.patch", depends=[8392]) # in permutations.py
import_patch("trac_13987_mary_trees-vp.patch", depends=[8703])
import_patch("trac_11407-list_clone_improve-fh.patch", depends=[8703])
import_patch("mutator-fh.patch", depends=[11407])
import_patch("trac_9280-graded-algebras-example-fs.patch")
import_patch("coercion_graph-nt.patch") # No ticket
import_patch("finiteenumset_random_improve-fh.patch")
import_patch("trac_12250-ktableaux-as.patch")
import_patch("missing-doc-includes-nt.patch")
import_patch("dyck_word_to_binary_tree-fh.patch")
import_patch("trac_10963-more_functorial_constructions-nt.patch", depends=[10193,12895,9280])
import_patch("trac_14102-nonsymmetric-macdonald.patch", depends=[4327,14143,10963])
import_patch("ncsf-qsym-new-bases-fs.patch")
import_patch("crystal_isomorphism-ts.patch")
import_patch("hall_littlewood_yt-ts.patch", depends=[14141])
import_patch("trac_11285-decompose_vecspace-ts.patch")
import_patch("12630_quivers.patch")
import_patch("12630_quivers_review-fs.patch")
import_patch("qpa_interface-fs.patch")
import_patch("dynamic-fh.patch")
import_patch("element_compare_consistency-fh.patch")
import_patch("trees_symmetry_factor-fh.patch", depends=[8703])
import_patch("trac_10950-hash_matrices-nt.patch")
import_patch("trac_13232-plot_latex-nt.patch")
import_patch("kschur-as.patch")
import_patch("trac_8678-module_morphisms-nt.patch", depends=[10963])
import_patch("trac_13317-species_unique_representation.patch")
import_patch("trac_10227-species_fixes-mh.patch", depends=[13317])
import_patch("categories-tutorial.patch")
import_patch("trac_11111-finite_dimensional_modules-nt.patch", depends=[10963,8678]) # still causing problem
import_patch("trac_8822-family_constructor-fh.patch")
import_patch("trac_6484-ranker-improvements-nt.patch")
import_patch("selector-fh.patch")
import_patch("trac_11529-rooted_trees-fh.patch", depends=[8703,13987]) # causing problem
import_patch("shape_tree-fc.patch", depends=[11529])
import_patch("shuffle_overlap_generic-fh.patch")
import_patch("operads-fh.patch", depends=[8703])
import_patch("operads_more-fc.patch",depends=[11529, "operads-fh.patch"])
import_patch("mupad-interface-improve-fh.patch")
import_patch("combinat-quickref-jb.patch")
import_patch("partition_k_boundary_speedup-fh.patch")
import_patch("partition_leg_length_speedup-fh.patch")
import_patch("kshape-om.patch")
import_patch("bintrees_leaf_paths-fh.patch", depends=[8703])
import_patch("trac_11109-stable-grothendieck-polynomials-nt.patch", depends=[10963]) # really depends on it?
import_patch("add_cache-nt.patch", depends=[10963])
import_patch("games_dao-nt.patch")
import_patch("finite-subquotients-nt.patch", depends=[10963])
import_patch("finite_set_map_mul-nt.patch")
import_patch("automatic_monoid-nt.patch", depends=[10963]) # Trivial dependency in sage.combinat.all
import_patch("discrete_function-nt.patch", depends=["automatic_monoid-nt.patch"])
import_patch("discrete_function_exper-fh.patch", depends=["discrete_function-nt.patch"])
import_patch("finite_semigroup-nt.patch", depends=[10963])
import_patch("finite_semigroup-subcategory-methods-nt.patch", depends=["finite_semigroup-nt.patch"])
import_patch("digraphs-as-automatons-nt.patch")
import_patch("category-symmetric_groups-nt.patch")
import_patch("ndpf_mult_side-fh.patch", depends=["finite_semigroup-nt.patch"])
import_patch("graph-latex-nt.patch")
import_patch("weyl_characters-nt.patch")
import_patch("test_len_object-fh.patch")
import_patch("invariant_ring_permutation_group-nb.patch") # failing
import_patch("permutation_inverse-vd.patch")
import_patch("conjugacy_class_iterator-vd.patch")
import_patch("wang_tile_set-tm.patch")
import_patch("trac_9439-hyperbolic_geometry-vd.patch")
import_patch("trac_9557-fundamental_domains-vd.patch", depends=[9439])
import_patch("trac_9806-constellations-vd.patch", depends=[9557])
import_patch("trac_9806-constellations-doc-patch-fc.patch", append=True)
import_patch("permutation_groups_stabilizer_chains-rm.patch")
import_patch("trac_7983_tableau_fixes-jb.patch")
import_patch("refactor_sf-jb.patch")
import_patch("trac_8581_multivariate_schubert_step_1-nb.patch", depends=["invariant_ring_permutation_group-nb.patch"]) # trivial dependency in setup.py
import_patch("trac_6629_abstract_ring_of_multivariate_polynomials_with_several_bases_vp.patch", depends=["trac_8581_multivariate_schubert_step_1-nb.patch"])
import_patch("trac_12460_polynomial_module_on_sym-nb-vp.patch", depends=["trac_6629_abstract_ring_of_multivariate_polynomials_with_several_bases_vp.patch"])
# roughly got up to there ...
import_patch("sage-demos-and-tutorials-nt.patch", depends=[10963,"combinat-quickref-jb.patch"])
import_patch("descents_composition_of_empty_permutation_jyt.patch")
import_patch("exterior_algebra-vd.patch")
import_patch("nested_lists_jb.patch")
import_patch("trac_7980-multiple-realizations-extra_do_not_merge-nt.patch")
import_patch("hopf_algebra_of_supercharacters-fs.patch")
import_patch("permutations_descent_values-fh.patch")
import_patch("sf_principal_specialization-mr.patch")
import_patch("trac_9123-schur-algebra-and-gln-characters-ht.patch")
import_patch("trac_11386_bracelet_class-dr.patch")
import_patch("trac_11571_catalan_objects-nm.patch")
import_patch("catalan_quasi_symmetric-fc.patch")
import_patch("catalan_quasi_symmetric-rebase-cs.patch")
import_patch("tableaux-combinatorics-am.patch")
import_patch("cartesian_product_improvements-nt.patch")
import_patch("extended_affine_weyl_groups_sd40.patch")
import_patch("affine_iwahori_hecke_algebras.patch")
import_patch("q_tree_factorial-fc.patch")
import_patch("trac_12916_completion_by_cuts-fc.patch")
import_patch("algebras_over_operads-fc.patch", depends=["operads_more-fc.patch"])
import_patch("shuffle-operads-fc.patch",depends=["algebras_over_operads-fc.patch"])
import_patch("trac_Kleshchev-partitions-am.patch")
import_patch("hgignore_eclipse_project-EliX-jbp.patch")
import_patch("trac_13935_coercion_of_coproduct_of_Hopf_algebra-EliX-jbp.patch")
import_patch("trac_13793-some-hopf-algebra-f-w-pqsym-EliX-jbp.patch", depends=[11571,13935]) # Trivial dependency upon #11571 in setup.py Trivial dependency on invariant_ring_permutation_group-nb.patch in doc/en/reference/combinat/index.rst
import_patch("trac_14104--html_display-am.patch")
import_patch("trac_14103--labelled_matrices-am.patch")
import_patch("trac_13855_planar_binary_trees_hopf_algebra-EliX-jbp.patch", depends=[8703,13793]) # and q_tree_factorial-fc.patch and trees_symmetry_factor-fh.patch
