[sage-devel] Re: pickle jar is checked into git

[Simon King]

Hi Andrew,

On 2014-01-17, Andrew  wrote:
>> Or do you suggest to "update the pickle jar" in the sense of "remove old 
>> pickles and *replace* them by new ones"? 
>>
>
> I was mainly thinking of adding new pickles rather than removing old ones. 

OK, then we agree here.

> My main point is that at the moment (almost?) no new pickles are being 
> added to the pickle jar so it not currently protecting us from the angry 
> user you warn about below.

Good point. And the question is: What new classes should contribute a sample
instance for the pickle jar? If you have a plain Python class, then
probably it is not needed to put anything into the pickle jar. But if
you have a class with a non-trivial __reduce__ or __setstate__ method,
then it probably should be added to the pickle jar.

> I think that within reason sage should maintain backwards compatibility 
> least some one finds that her 2008 files no longer work. On the other hand, 
> I don't think that we should do this at any cost -- for example, some 
> pickles related to k-Schur functions will soon be removed from the pickle 
> jar -- and that it should be possible to remove pickles given a reasonable 
> depreciation [sic] period.

Hm. I think one should at least try hard to keep the pickles, before
removing them. One should come up with good arguments before removing
old pickles from the jar.

Best regards,
Simon


-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [sage-devel] Re: pickle jar is checked into git

[William Stein]

On Jan 17, 2014 10:10 AM, "Andrew"  wrote:
>
>
>
> On Friday, 17 January 2014 15:07:26 UTC+1, Nathann Cohen wrote:
>>
>>
>> Well. Don't we keep old versions of Sage too ? The guy can save his data
by reinstalling the version he used just before updating his Sage install.
Even Sage 2.0 if he feels like it.
>>
> Well, in principle yes, but how easy do you think it will be to compile
sage 2.0 in 2020? My guess is that this will be more difficult than writing
new code to read old pickles -- should any of them no longer be supported:)

All old Ubuntu distros are archived and easy to install into virtual box,
qemu, vmware, etc., and sage 2.0 will easily build on Ubuntu 8.04 (say).  I
predict that in 2020 it will be very easy to build sage 2.0 from source,
since virtualization is getting even more pervasive.

> Andrew
>
> --
> You received this message because you are subscribed to the Google Groups
"sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
email to sage-devel+unsubscr...@googlegroups.com.
> To post to this group, send email to sage-devel@googlegroups.com.
> Visit this group at http://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/groups/opt_out.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [sage-devel] Re: pickle jar is checked into git

[Nils Bruin]

On Friday, January 17, 2014 7:14:58 AM UTC-8, Nathann Cohen wrote:
>
> Well, that's what would happen if 2.0 was the latest version able to 
> open those pickles, and that the guy never updated his version of Sage 
> until today.


That's not how bitrot tends to work. The more likely scenario is that the 
particular *pickle* doesn't get used for more than a decade and then 
suddenly the data turns out to be relevant again (perhaps an attachment to 
a paper on ArXiv about the complete classification of X, and now suddenly 
someone needs that classification).

Of course, it's questionable if even with the pickle jar in place, sage 
pickle is an archival-grade file format, but we can try.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [sage-devel] Re: pickle jar is checked into git

[Nathann Cohen]

> Well, in principle yes, but how easy do you think it will be to compile sage
> 2.0 in 2020? My guess is that this will be more difficult than writing new
> code to read old pickles -- should any of them no longer be supported:)

Well, that's what would happen if 2.0 was the latest version able to
open those pickles, and that the guy never updated his version of Sage
until today. If he works on a version of Sage on which the pickles
works, then updates Sage to find out the pickles cannot be read
anymore, well, he had a working version of Sage (that can be installed
on his computer) just before he did that and he can probably
re-install it again !

Perhaps archiving old versions of Sage is all the deprecation we need
for pickles.

Nathann

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/groups/opt_out.

[sage-devel] Re: pickle jar is checked into git

[Andrew]

On Friday, 17 January 2014 14:53:22 UTC+1, Simon King wrote:
>
> Hi Andrew, 
>
> On 2014-01-17, Andrew > wrote: 
>
> Do you suggest to "update the pickle jar" in the sense of "whenever 
> someone implements a new data structure or changes an existing data 
> structure, then an example of the new/changed data structure should be 
> *added* to the pickle jar? 
>
> Or do you suggest to "update the pickle jar" in the sense of "remove old 
> pickles and *replace* them by new ones"? 
>

I was mainly thinking of adding new pickles rather than removing old ones. 
As Robert suggests perhaps, especially given Volkers concerns about the 
potential growth of the pickle jar, only "important" or "fundamental" 
objects should live in the pickle jar. 

My main point is that at the moment (almost?) no new pickles are being 
added to the pickle jar so it not currently protecting us from the angry 
user you warn about below.
 

>
> But I think it is absolutely not acceptable to say: "OK, let's remove 
> the pickle jar and keep going until some angry user tells us that his 
> seminal 2008 work that took him 10 months of computation time can not 
> be accessed with the latest version of Sage." 
>
> I think that within reason sage should maintain backwards compatibility 
least some one finds that her 2008 files no longer work. On the other hand, 
I don't think that we should do this at any cost -- for example, some 
pickles related to k-Schur functions will soon be removed from the pickle 
jar -- and that it should be possible to remove pickles given a reasonable 
depreciation [sic] period. If sage is still going in 2020, as I hope it 
will be, then I personally would not be upset if pickles from 2008 no 
longer unpickled.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/groups/opt_out.

[sage-devel] Re: pickle jar is checked into git

[Andrew]

On Friday, 17 January 2014 15:07:26 UTC+1, Nathann Cohen wrote:
>
>
> Well. Don't we keep old versions of Sage too ? The guy can save his data 
> by reinstalling the version he used just before updating his Sage install. 
> Even Sage 2.0 if he feels like it.
>
> Well, in principle yes, but how easy do you think it will be to compile 
sage 2.0 in 2020? My guess is that this will be more difficult than writing 
new code to read old pickles -- should any of them no longer be supported:)

Andrew 

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/groups/opt_out.

[sage-devel] Re: pickle jar is checked into git

[Nathann Cohen]

>
> But I think it is absolutely not acceptable to say: "OK, let's remove 
> the pickle jar and keep going until some angry user tells us that his 
> seminal 2008 work that took him 10 months of computation time can not 
> be accessed with the latest version of Sage." 


Well. Don't we keep old versions of Sage too ? The guy can save his data by 
reinstalling the version he used just before updating his Sage install. 
Even Sage 2.0 if he feels like it.

Nathann 

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/groups/opt_out.

[sage-devel] Re: pickle jar is checked into git

[Simon King]

Hi Andrew,

On 2014-01-17, Andrew  wrote:
> I would like to suggest that we either:
>
>1. Maintain the pickle jar properly, by which I mean that the jar is 
>updated regularly (and probably tracked by git).
>2. Remove the pickle jar entirely.
>
> In principle, having a pickle jar is a good way of ensuring backward 
> compatibility. In practice, if the pickles in the jar are not 
> representative all of the objects in the latest version of sage then the 
> pickle jar is providing us with a false sense of security.

Do you suggest to "update the pickle jar" in the sense of "whenever
someone implements a new data structure or changes an existing data
structure, then an example of the new/changed data structure should be
*added* to the pickle jar?

Or do you suggest to "update the pickle jar" in the sense of "remove old
pickles and *replace* them by new ones"?

> If no one is 
> storing old pickles then the pickle jar is not really needed. On these 
> grounds my preference is for option 2.

You can't know what data people may have stored on their hard disk
drive. So, to be on the safe side, we *must* assume that some user has
created some database in the good (?) old sage-2.x times. We should aim
at keeping these data unpicklable. As it was pointed out in another
answer, a pragmatic approach would be to have a focus on "basic",
"ubiquitous" data structures unpicklable (such as integers, rationals
and so on).

But I think it is absolutely not acceptable to say: "OK, let's remove
the pickle jar and keep going until some angry user tells us that his
seminal 2008 work that took him 10 months of computation time can not
be accessed with the latest version of Sage."

Best regards,
Simon


-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [sage-devel] Re: pickle jar is checked into git

[Robert Bradshaw]

On Fri, Jan 17, 2014 at 2:35 AM, Andrew  wrote:
> I'd be interested in knowing how many people actually use/store pickles and
> whether this is enough to justify maintaining the pickle jar. Speaking for
> myself, I never use pickles and the only time that I have ever looked at
> them is when I was upgrading various classes and this broke some of the old
> pickles in the pickle jar. (The first time this happened there was almost no
> documentation on the pickle jar, and how to fix broken pickles, so I wasted
> a lot of time upgrading the pickles only to be told that this was not sage
> policy -- and then I wrote the documentation in the manual to hopefully save
> others grief...)
>
> Part of the reason why the pickles in the pickle jar are updated so rarely
> is because this is cumbersome to do because the jar is a compressed tar
> file.  The other reason is that, as Volker said, there is no automated
> process for updating the pickle jar.
>
> On #10768 Volker has asked whether we really want the pickle jar to be
> tracked by the git repo as he estimates this will add about ~10M per year to
> the repo. On the other hand, I think that the current state of the pickle
> jar is close to useless because it is so out of date (the pickle are
> rotting?).

The pickle jar should rarely modified, and occasionally get added to.
Everything in the pickle jar is still fresh, otherwise tests wouldn't
pass.

> I would like to suggest that we either:
>
> Maintain the pickle jar properly, by which I mean that the jar is updated
> regularly (and probably tracked by git).
> Remove the pickle jar entirely.

The whole point of the pickle jar is to maintain stability of
serialized formats. We don't want to be constantly updating it.

> In principle, having a pickle jar is a good way of ensuring backward
> compatibility. In practice, if the pickles in the jar are not representative
> all of the objects in the latest version of sage then the pickle jar is
> providing us with a false sense of security. If no one is storing old
> pickles then the pickle jar is not really needed. On these grounds my
> preference is for option 2.

The pickle jar does not have to be representative of all the objects
in Sage to be useful. In fact, in my mind the most important objects
are the basic ones (integers, polynomials, matrices), and those are
represented. So I'd vote for keeping the pickle jar (perhaps as a
directory rather than a monolithic binary blob) and also making it
easy to add items to it (but with strong warnings about removing
anything).

This is like saying we don't have enough tests, so they're providing a
false sense of security and we should throw out the ones we have.

- Robert

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/groups/opt_out.

[sage-devel] Re: pickle jar is checked into git

[Andrew]

I'd be interested in knowing how many people actually use/store pickles and 
whether this is enough to justify maintaining the pickle jar. Speaking for 
myself, I never use pickles and the only time that I have ever looked at 
them is when I was upgrading various classes and this broke some of the old 
pickles in the pickle jar. (The first time this happened there was almost 
no documentation on the pickle jar, and how to fix broken pickles, so I 
wasted a lot of time upgrading the pickles only to be told that this was 
not sage policy -- and then I wrote the documentation in the manual to 
hopefully save others grief...)

Part of the reason why the pickles in the pickle jar are updated so rarely 
is because this is cumbersome to do because the jar is a compressed tar 
file.  The other reason is that, as Volker said, there is no automated 
process for updating the pickle jar. 

On #10768  Volker has 
asked whether we really want the pickle jar to be tracked by the git repo 
as he estimates this will add about ~10M per year to the repo. On the other 
hand, I think that the current state of the pickle jar is close to useless 
because it is so out of date (the pickle are rotting?).

I would like to suggest that we either:

   1. Maintain the pickle jar properly, by which I mean that the jar is 
   updated regularly (and probably tracked by git).
   2. Remove the pickle jar entirely.

In principle, having a pickle jar is a good way of ensuring backward 
compatibility. In practice, if the pickles in the jar are not 
representative all of the objects in the latest version of sage then the 
pickle jar is providing us with a false sense of security. If no one is 
storing old pickles then the pickle jar is not really needed. On these 
grounds my preference is for option 2.

Andrew

On Friday, 17 January 2014 03:46:00 UTC+1, Volker Braun wrote:

> Any changes are very infrequent and there is no process to update it 
> regularly:
>
> $ git log --oneline src/ext/pickle_jar/
> 3d1bf45 Trac #7922: Categories for Weyl character rings and weight rings
> 38475a0 Trac #10354: Remove deprecated word objects from the pickle jar
> 6dcaf62 4.4.4.alpha1
> 6ce1bde 4.1.2.rc1.alpha3
> 7e586f9 Update the pickle jar for 4.0.alpha0
> 77359f7 Adjust pickle_jar due to removal of extended rationals and 
> integers (#5735)
> d1c28ff Update pickle jar for Sage 3.4
> 20a3237 Remove failing pickles from #4653
> cdaa569 added the first pickle jar.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/groups/opt_out.

[sage-devel] Re: pickle jar is checked into git

[Volker Braun]

Any changes are very infrequent and there is no process to update it 
regularly:

$ git log --oneline src/ext/pickle_jar/
3d1bf45 Trac #7922: Categories for Weyl character rings and weight rings
38475a0 Trac #10354: Remove deprecated word objects from the pickle jar
6dcaf62 4.4.4.alpha1
6ce1bde 4.1.2.rc1.alpha3
7e586f9 Update the pickle jar for 4.0.alpha0
77359f7 Adjust pickle_jar due to removal of extended rationals and integers 
(#5735)
d1c28ff Update pickle jar for Sage 3.4
20a3237 Remove failing pickles from #4653
cdaa569 added the first pickle jar.


-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/groups/opt_out.

[sage-devel] Re: pickle jar is checked into git

[Stefan]

How often is the pickle jar updated? My impression was that it would be 
with each new release. But I just unzipped it, and I don't see any matroids 
in there, while sage.matroids has been part of Sage since 5.12.

--Stefan.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/groups/opt_out.

[sage-devel] Re: pickle jar is checked into git

[Andrew]

There is an old trac ticket #10768 that 
proposes that the pickle_jar should be tracked as files rather than as 
a compressed tar file. I think that that this is a good idea. The ticket 
refers to hg but is otherwise sound.

Andrew

On Thursday, 16 January 2014 06:21:24 UTC+1, Volker Braun wrote:
>
> The pickle jar is 600kb bzip2-compressed (hence probably un-diffable by 
> git). So every change to it inflates our repo size by about 1%. IMHO we 
> should either untar/decompress it to track its content, or turn it into a 
> downloadable tarball. Suggestions? Volunteers?
>
> This comes up in http://trac.sagemath.org/ticket/15575 which changed the 
> pickle jar...
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/groups/opt_out.

[sage-devel] Re: Pickle jar

[Nicolas M. Thiery]

Oops, I am so sorry. This mail was only meant to be sent to sage-devel
and sage-combinat-devel. Please moderate if still at all possible.

On Thu, Feb 10, 2011 at 06:44:11PM +0100, Nicolas M. Thiery wrote:
> On Wed, Feb 09, 2011 at 04:51:14AM -0800, Anne Schilling wrote:
> > For #8911 the new pickle jar was just attached. So I added this file
> > to #10632. Probably you can do the same for #7922 (but then the tickets
> > might not commute if both change the pickles).
> 
> Yeah, that pickle jar procedure is not perfect yet. I have just
> created a ticket with a suggestion for improving it:
> 
>   http://trac.sagemath.org/sage_trac/ticket/10768
> 
> Cheers,
Nicolas
--
Nicolas M. Thiéry "Isil" 
http://Nicolas.Thiery.name/

-- 
To post to this group, send an email to sage-devel@googlegroups.com
To unsubscribe from this group, send an email to 
sage-devel+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URL: http://www.sagemath.org