Re: [sage-devel] Re: sage -br rebuilds things that were not changed

[Johan S . H . Rosenkilde]

Jeroen Demeyer writes:
>> https://ccache.samba.org/
>
> And I forgot to mention that I have 2 complete Sage installation trees 
> so I can let one compile while I use the other.

Thanks, that's useful. Do you need any special setup or care for making
it work in the complicated Sage compilation system, or do you just
override gcc as suggested in the manual? How much does it speed up
compilation?

It would seem to me you could often come in the case that *both*
copies are/needs compiling: You want to work on ticket A and so checkout
that on Sage1. While it's doing a 1h compilation due to jumping between
releases, you checkout ticket B on Sage2. But that's also a release
jump, so compilation ensues. Perhaps ccache makes this moot?

Does it ever get clumsy wrt. tracking which tree is in which version and
has which ticket checked out?

I also have two Sage installations, but it's a stable one I use for
work, and a devel one. That way, if devel screws up for whatever reason,
I can still always continue working. I also use it for checking Sage
behaviour without my ticket or if devel is compiling.

Daniel Krenn writes:
> Having a version of each stable/rc/beta SageMath in a separate
> directory. Every now and then I delete an old rc/beta. For working on a
> ticket I use exactly the version of the branch of this ticket.
> Recompiliation time is very low in this way (usually).
>
> I have one additional directory "merge", which is only the git
> repository without anything created for
> - looking at tickets if I am in doubt about the current version (git
> describe) and
> - for merging in a newer stable/rc/beta into a branch.

Thanks.
Do you feel this can get clumsy?
Like, how do you find which release have which ticket checked out?
When you merge in a new release into a branch you will have high
compilation time - but perhaps you don't do this too often?



I seriously think a lot of beginners are struggling with compilation
time when working with Sage, inadvertently jumping a lot between
releases. Apparently, my workflow is not popular, but could we still
add something to the Development manual to help people?

Best,
Johan

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] Re: sage -br rebuilds things that were not changed

[Jeroen Demeyer]

On 2016-09-11 10:48, Johan S. H. Rosenkilde wrote:


Jeroen Demeyer writes:

https://ccache.samba.org/


And I forgot to mention that I have 2 complete Sage installation trees
so I can let one compile while I use the other.


Thanks, that's useful. Do you need any special setup or care for making
it work in the complicated Sage compilation system, or do you just
override gcc as suggested in the manual?


I am using system-wide ccache and my OS sets up the environment 
variables correctly. But a Sage-installed ccache should also "just work".



How much does it speed up
compilation?


A *lot* for everything involving compiling C code. So in the case of the 
Sage library, the Cythonizing still needs to be done, but the second 
step can be very fast. For packages, the most annoying ones are those 
which do a lot of tuning.



Does it ever get clumsy wrt. tracking which tree is in which version and
has which ticket checked out?


Well, there is "git log" or "git status" for that.


I also have two Sage installations, but it's a stable one I use for
work, and a devel one. That way, if devel screws up for whatever reason,
I can still always continue working. I also use it for checking Sage
behaviour without my ticket or if devel is compiling.


In the two Sage trees that I have, I also try to use one as "stable" 
release for real work. However, it does not matter much which branch 
that one is on (as long as it's not too outdated). If 7.3 works for real 
work, then 7.4.beta4 + some unrelated ticket most likely works too.


--
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] Re: sage -br rebuilds things that were not changed

[Johan S . H . Rosenkilde]

leif writes:
> Well, probably it's just me, but I always (also) review tickets on trac
> (i.e., via what git-o-lite gives); the whole branch, but also each
> individual commit.  You just need a browser to do so.

I use git-o-lite a lot too, but often switch back to my Emacs and git
log using Magit :-P

> Having a Sage installation at hand, it's even more tedious to do so on
> the command line when the relevant commits are "out-of-order" / mixed
> with unrelated commits.

I mostly take the commit id from trac. But yes, without git-o-lite it
can get difficult to find the commits you need. I'm sure there are "git
log" flags for it, but I'm not an expert. It just hasn't come up enough
for me to really try and find a solution.

>> "whatever head" is always develop, so I'm always pulling into the newest
>> beta release.
>
> Well, that doesn't make much of a difference, unless the ticket is
> incidentally based on the same (which is rarely the case, since most
> tickets live longer than a single beta-release cycle).

I just wanted to stress that it's not a random choice. Indeed, barring
the "history becomes difficult to read", I don't see any argument
against all devs always working on latest beta, in which case it would
always make sense to update all tickets to that. We could even
automatically upgrade all tickets to latest beta on releases. Though
based on this discussion, I'd guess many of you would get angry due to
"history becomes more difficult to read".

> For trivial changes, you can also put inline patches onto trac the owner
> of the ticket / branch can easily apply, and for more complex changes,
> you can always create a new branch on trac *without* changing the
> ticket's branch to that.  Then the owner of the original branch can
> cherry-pick from your branch, if it's based on something else.

This is a good idea that would allow me to continue using my current
workflow, without annoying other devs.

> (With different Sage installations, you could do similar locally of
> course, i.e., "backport" the changes you made on top of your develop to
> master+fetched original branch, say, and push the resulting branch.  I
> personally often first make changes in some random Sage installation,
> then moving them around, later pushing to trac from a "suited" one.)

Aha, sort of like what Daniel wrote, except that fetching the branch
into  "random Sage installation" means you don't get the low compilation
time benefits.

>> I don't see why all developers shouldn't always be on the latest beta.
>
> Time?  Resources?
>
> Note that many, I guess, have Sage installations here and there, and
> most likely don't or cannot keep each of them always what you'd call
> "up-to-date", but rather at one of the previous "stable" releases, or
> something inbetween.


What time? What resources? My dev model needs only a single Sage
installation and no extra software to achieve overall very low
compilation time. The other suggestions I've heard need multiple Sages,
extra software and/or does not minimise compilation time. Checking out a
ticket takes a whopping extra 3 seconds (if it takes longer it's because
of merge errors which I would have to fix anyway).

Note I have one extra Sage tree to ensure that I have a working Sage at
any time. It's on latest stable and I never develop on it.

> [Random insertion:  Wasn't it you who [had to] upgrade[d] to Ubuntu
> 16.04 just to get the next version of Emacs? ;-) ]

Hmm, no. I've used Arch linux for 3 years now because Ubuntu has too old
software. But I've twice had major compilation issues due to Arch
upgrading to a new release of some dependency (e.g. ranlib or gcc) long
before any other distro.


> Obviously, there are more "recent" beta releases than stable ones.  And
> as mentioned, tickets and their branches age.
>
> So you'll never have all tickets (not even a large percentage) based on
> the latest beta.
>
> It's easier if you keep one Sage installation at master, and another at
> some later beta.  The problem (if tickets are based on the latest beta)
> IMHO is that one is too often forced to "upgrade" for no reason, at
> least if one forgets to first check what release a branch was based on.
> If people keep pushing back "rebased" branches, the situation just worsens.

OK, but I just still don't really see the problem of being "forced to
upgrade". I don't see how it hurts me that I'm always working on the
latest beta - my patch has to work with that anyway. Except for the
"history is more difficult to understand" argument. But is that really
enough of an argument to jump through hoops and rings for achieving low
compilation time in another way?


> More seriously, different Sage installations, trying to pick the most
> suited when going to actually build a fetched branch.  But sometimes at
> least, you also want to test on a specific machine (hardware, OS/distro,
> compiler, whatever).

OK, thanks for describing your workflow.

Best,
Johan

-- 
You received this message beca

[sage-devel] Re: Suggestion for main configure script to give package names for Latex.

[Dr. David Kirkby (Kirkby Microwave Ltd)]

On 6 September 2016 at 13:45, Dima Pasechnik  wrote:

> Hi David,
>


>
> On Tuesday, September 6, 2016 at 9:08:25 AM UTC, Dr. David Kirkby (Kirkby
> Microwave Ltd) wrote:
>>
>> I'm trying to migrate away from Solaris to Linux, given the takeover by
>> Oracle. I decided to install the latest Debian (8.5 == Jessie) , and Sage.
>> I have not succeeded yet, but some of the issues seem to be ones where the
>> installation could be made easier for popular linux distributions.
>>
>>
> with your Unix knowledge you'd feel Debian is a jail of sorts; I'd
> recommend Arch.
>

Hi Dima

Why do you say Debian is a jail of sorts?

I'm looking for a distribution, that perhaps does not exist

* Reasonably well set up for things like propriety video drivers (my Nvidia
FX3800 seems to present a few hassles for any distribution) . I tried Mint,
and quite looked the front end and the fact it installed drivers for my
hardware. I like open-source, but when closed-source hardware drivers are
better, I will use them.

* Be reasonably flexible I'm not sure how this will work on Mint, but I
read something on a Scientific Linux mailing list which made me think Mint
was a poor choice. The comment there was that the Mint forums were more
"enthusiastic hobbyists, and not professionals". As you know, I'm pretty
decent at Unix, having long since used Solaris, and having also used AIX,
HP-UX etc. I think its fair to say I would probably need help with more
complex things, that maybe "enthusiastic hobbyists" would not be
sufficiently competent.

* I don't want to keep re-installing operating systems regularly. I have
not updated my Solaris distribution for about 7 years. I run a business and
want a computer that works - not one I am going to have to spend all my
time maintaining.

In fact, my only real reason for ditching Solaris now is that one of my
SPARC machines is being unreliable with perplexing hardware errors. Since I
use that for data collection from test equipment, I want to move the GPIB
card that collects the data to a more modern machine, which should be more
reliable.

I have Centros on a couple of IBM servers which I have not used for the
last year or so. My reason for going to Centros was that I had some
commercial software that was supported only on Redhat. I did work on
Centros, whereas any other distribution was likely to lead to issues.

One thing I do want to do is video editing. Blender seems to be the best
software for that.



Dave

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

[sage-devel] Re: memory management issue: deleted variables not released

[Denis]

Thanks for the idea to post the example on the cloud. I will try to do 
that, but can't begin before Sept. 20.

Denis

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

[sage-devel] Re: sage -br rebuilds things that were not changed

[leif]

Johan S. H. Rosenkilde wrote:
> 
> leif writes:
>> Well, probably it's just me, but I always (also) review tickets on trac
>> (i.e., via what git-o-lite gives); the whole branch, but also each
>> individual commit.  You just need a browser to do so.
> 
> I use git-o-lite a lot too, but often switch back to my Emacs and git
> log using Magit :-P
> 
>> Having a Sage installation at hand, it's even more tedious to do so on
>> the command line when the relevant commits are "out-of-order" / mixed
>> with unrelated commits.
> 
> I mostly take the commit id from trac. But yes, without git-o-lite it
> can get difficult to find the commits you need. I'm sure there are "git
> log" flags for it, but I'm not an expert. It just hasn't come up enough
> for me to really try and find a solution.
> 
>>> "whatever head" is always develop, so I'm always pulling into the newest
>>> beta release.
>>
>> Well, that doesn't make much of a difference, unless the ticket is
>> incidentally based on the same (which is rarely the case, since most
>> tickets live longer than a single beta-release cycle).
> 
> I just wanted to stress that it's not a random choice. Indeed, barring
> the "history becomes difficult to read", I don't see any argument
> against all devs always working on latest beta, in which case it would
> always make sense to update all tickets to that. We could even
> automatically upgrade all tickets to latest beta on releases. Though
> based on this discussion, I'd guess many of you would get angry due to
> "history becomes more difficult to read".
> 
>> For trivial changes, you can also put inline patches onto trac the owner
>> of the ticket / branch can easily apply, and for more complex changes,
>> you can always create a new branch on trac *without* changing the
>> ticket's branch to that.  Then the owner of the original branch can
>> cherry-pick from your branch, if it's based on something else.
> 
> This is a good idea that would allow me to continue using my current
> workflow, without annoying other devs.
> 
>> (With different Sage installations, you could do similar locally of
>> course, i.e., "backport" the changes you made on top of your develop to
>> master+fetched original branch, say, and push the resulting branch.  I
>> personally often first make changes in some random Sage installation,
>> then moving them around, later pushing to trac from a "suited" one.)
> 
> Aha, sort of like what Daniel wrote, except that fetching the branch
> into  "random Sage installation" means you don't get the low compilation
> time benefits.

Well, making changes doesn't necessarily imply fetching branches, nor
[immediate] (re)compilation.  Also, occasionally I just "manually" grab
changes, i.e., patches or even only hunks, and apply them temporarily,
later creating a "clean" branch, probably elsewhere, in order to push or
just properly do final testing.

What would have to get rebuilt of course also varies broadly, depending
on what you change (or what a fetched or pulled branch brings with it).


>>> I don't see why all developers shouldn't always be on the latest beta.
>>
>> Time?  Resources?
>>
>> Note that many, I guess, have Sage installations here and there, and
>> most likely don't or cannot keep each of them always what you'd call
>> "up-to-date", but rather at one of the previous "stable" releases, or
>> something inbetween.
> 
> 
> What time? What resources? My dev model needs only a single Sage
> installation and no extra software to achieve overall very low
> compilation time. The other suggestions I've heard need multiple Sages,
> extra software and/or does not minimise compilation time. Checking out a
> ticket takes a whopping extra 3 seconds (if it takes longer it's because
> of merge errors which I would have to fix anyway).

One Sage installation means one machine (you "always" have access to /
can currently use), with a specific processor, operating system/distro
version, and a single tool chain.


> Note I have one extra Sage tree to ensure that I have a working Sage at
> any time. It's on latest stable and I never develop on it.
> 
>> [Random insertion:  Wasn't it you who [had to] upgrade[d] to Ubuntu
>> 16.04 just to get the next version of Emacs? ;-) ]
> 
> Hmm, no. I've used Arch linux for 3 years now because Ubuntu has too old
> software. But I've twice had major compilation issues due to Arch
> upgrading to a new release of some dependency (e.g. ranlib or gcc) long
> before any other distro.

Then you know what "upgrades" /can/ bring.


>> Obviously, there are more "recent" beta releases than stable ones.  And
>> as mentioned, tickets and their branches age.
>>
>> So you'll never have all tickets (not even a large percentage) based on
>> the latest beta.
>>
>> It's easier if you keep one Sage installation at master, and another at
>> some later beta.  The problem (if tickets are based on the latest beta)
>> IMHO is that one is too often forced to "upgrade" for no reason, at
>> least if one forgets 

[sage-devel] Re: sage -br rebuilds things that were not changed

[leif]

Johan S. H. Rosenkilde wrote:
> I seriously think a lot of beginners are struggling with compilation
> time when working with Sage, inadvertently jumping a lot between
> releases. Apparently, my workflow is not popular, but could we still
> add something to the Development manual to help people?

Well, jumping between different tickets / switching branches too often
is certainly not something for a beginner.

They'll presumably though learn quickly what kind of changes are
"dangerous" w.r.t. rebuild time...


Interestingly, so far nobody mentioned what the patch/buildbots do.

When Sage was "relocatable", it was much easier to maintain different
Sage installations.  For temporary / scratch installations just to test
tickets and throw them away afterwards, without much rebuild effort,
it's still possible to create tarballs of built vanilla releases (kind
of your own bdists), but because of the hardcoded absolute paths, this
(and other things) have become a headache.


-leif


-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

[sage-devel] 3d graphics error in Jupyter notebook while legacy Sage notebook is OK

[Eric Gourgoulhon]

Hi,

While migrating the legacy Sage notebooks at 
http://sagemanifolds.obspm.fr/examples.html
to Jupyter notebooks (thanks to Volker's notebook exporter introduced in 
Sage 7.3, cf. #19877 ), I got the 
error

error: Error -3 while decompressing: invalid distance too far back


when displaying some 3d graphic object in the Jupyter notebook. The minimal 
worksheet reproducing this error is displayed at
http://nbviewer.jupyter.org/github/egourgoulhon/SageMathTest/blob/master/Worksheets/test_sphere.ipynb

The point is that the legacy Sage notebook with exactly the same input does 
not show such an error and displays the object correctly:
http://sagemanifolds.obspm.fr/examples/html/test_sphere.html

I suspect this is related to the size of the 3d object: it is formed by 
adding a plot of a vector field on S^2 to a plot of the spherical 
coordinate grid, each of these two objects having a correct display. 

These worksheets are based on SageManifolds, so that to reproduce the 
calculation, one has to pull the branch 
public/manifolds/Schouten_Cotton_York in any version of Sage >= 7.3 (in my 
case, it was Sage 7.4.beta1). 

Eric.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-devel] Re: sage -br rebuilds things that were not changed

[Paul Masson]

> But a Sage-installed ccache should also "just work". 

Jeroen, could you elaborate on this? How does one use ccache most simply 
with Sage?

If this is currently the only way to speed up rebuilding when switching 
between branches, then I would like to add a concise description to the 
appropriate documentation file.


On Sunday, September 11, 2016 at 1:58:18 AM UTC-7, Jeroen Demeyer wrote:
>
> On 2016-09-11 10:48, Johan S. H. Rosenkilde wrote: 
> > 
> > Jeroen Demeyer writes: 
> >>> https://ccache.samba.org/ 
> >> 
> >> And I forgot to mention that I have 2 complete Sage installation trees 
> >> so I can let one compile while I use the other. 
> > 
> > Thanks, that's useful. Do you need any special setup or care for making 
> > it work in the complicated Sage compilation system, or do you just 
> > override gcc as suggested in the manual? 
>
> I am using system-wide ccache and my OS sets up the environment 
> variables correctly. But a Sage-installed ccache should also "just work". 
>
> > How much does it speed up 
> > compilation? 
>
> A *lot* for everything involving compiling C code. So in the case of the 
> Sage library, the Cythonizing still needs to be done, but the second 
> step can be very fast. For packages, the most annoying ones are those 
> which do a lot of tuning. 
>
> > Does it ever get clumsy wrt. tracking which tree is in which version and 
> > has which ticket checked out? 
>
> Well, there is "git log" or "git status" for that. 
>
> > I also have two Sage installations, but it's a stable one I use for 
> > work, and a devel one. That way, if devel screws up for whatever reason, 
> > I can still always continue working. I also use it for checking Sage 
> > behaviour without my ticket or if devel is compiling. 
>
> In the two Sage trees that I have, I also try to use one as "stable" 
> release for real work. However, it does not matter much which branch 
> that one is on (as long as it's not too outdated). If 7.3 works for real 
> work, then 7.4.beta4 + some unrelated ticket most likely works too. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

[sage-devel] Re: sage -br rebuilds things that were not changed

[leif]

Paul Masson wrote:
>> But a Sage-installed ccache should also "just work". 
> 
> Jeroen, could you elaborate on this? How does one use ccache most simply
> with Sage?

./sage -i ccache

That's all, unless you want to configure it differently.

(After installation, see local/etc/ccache.conf.)


-leif


-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.