Re: [Tutor] Pickles and Shelves Concept
Thank you all for explaining me the above concept. On Sat, Jun 8, 2019 at 5:27 PM Alan Gauld via Tutor wrote: > On 08/06/2019 01:02, Alan Gauld via Tutor wrote: > > > keys to your data and then restore it in any sequence, or > > only restore some of it, you can. > > Apologies for the Yoda-speak at the end! > A bit of editing that didn't quite work out as intended... > > -- > Alan G > Author of the Learn to Program web site > http://www.alan-g.me.uk/ > http://www.amazon.com/author/alan_gauld > Follow my photo-blog on Flickr at: > http://www.flickr.com/photos/alangauldphotos > > > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor > ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Pickles and Shelves Concept
On 08/06/2019 01:02, Alan Gauld via Tutor wrote: > keys to your data and then restore it in any sequence, or > only restore some of it, you can. Apologies for the Yoda-speak at the end! A bit of editing that didn't quite work out as intended... -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Pickles and Shelves Concept
Hello, >I am not getting the concept of pickle and shelves in python, I >mean what's the use of both the concepts, when to use them in code >instead of using file read and write operations. > >Could anyone please explain me the concepts. I see you already have two answers (from David Rock and Alan Gauld). I will add a slightly different answer and try also to explain some of the history (at a very high level). * Programs need to take input from "somewhere". * Programs need data structures in memory on which to operate. There are many different ways to store data "somewhere" and in order to create the data structures in memory (on which your program will operate), you need to have code that knows how to read that data from "somewhere". So, there's data that has been written to disk. I often call this the serialized form form of the data [0]. There different usages for serialization, but ones I'll talk about are the serialized formats that we typically write to disk to store data for a program to read. Here are a few such serialized formats: * JSON or XML (homage: SGML) * pickles and shelves * GNU dbm, ndbm, cdb, rocksdb and probably 1 million others * custom binary formats * plain text files (be careful with your encoding...) Digression: You might ask... what about SQL? Technically, the serialization is something that the SQL database software takes care of and your application doesn't. So no need to know about the serialized format. This can be freeing, at some complexity. But, back to your question. Every one of the serialized formats comes with some advantages and some disadvantages. Some are easy. Some are flexible. Other formats are structured with bindings in many languages. Some are tied closely to a single language or even specific language versions. Some formats are even defined by a single application or program that somebody has written. What about pickle and shelve? Where do they fit? Both pickle and shelve are well-maintained and older Python-specific formats that allow you to serialize Python objects and data structures to disk. This is extremely convenient if you are unlikely to change Python versions or to change your data structures. Need your program to "remember" something from a prior run? When it starts up, it can read a ./state.pickle straight into memory, pick up where it left off and perform some operation, and then, when complete, save the dat astructure back to ./state (or more safely to a new file ./state.$timestamp) and exit. This is a convenient way to store Python objects and data structures. Advantage: Native Python. Dead simple to use (you still have to be careful about file-writing logic, overwriting old files can be bad, but it's a bit up to you). You can dump many Python data structures and objects to disk. Disadvantages: Files are only be readable by Python (excluding motivated implementers in other languages). If you would like to use pickle or shelve, please ask again on this list for specific advice on these. The shelve module is intended to make it easy to have a data structure in memory that is backed by a data file on disk. This is very similar to what the dbm module also offers. The pickle module is more geared toward loading an entire data structure from the disk into memory. There are other options, that have been used for decades (see below my sig for an incomplete and light-hearted history of serialization in the digital world). The option for serialization formats and accessing them from Python are many. Pickle and shelve are very Python specific, but will be very easy to use and will be more forgiving if you happen to try to store some code as well as "pure" data. If you are going to need to exchange data with other programs, consider JSON. Reading and writing to JSON format is as easy as reading and writing to a shelve (which is a Python pickle format under the hood). Here's a two liner that will take the environment of a running program and dump that into a human- and machine-readable JSON format. Step A: import os, sys, json json.dump(dict(os.environ), sys.stdout, indent=4, sort_keys=True) Now, let's say that you want to read that in another program (and I'll demonstrate just dumping the in-memory representation to your terminal). Step B: import sys, json, pprint pprint.pprint(json.load(sys.stdin)) So, going back to your original question. >I am not getting the concept of pickle and shelves in python, I >mean what's the use of both the concepts, when to use them in code >instead of using file read and write operations. You can, of course, use file read / write operations whenever you need to load data into memory from the disk (or "somewhere") or to write data from memory into the disk (or "somewhere"). The idea behind tools and libraries like... pickle and shelve (which are Python-specific), JSON (flexible and used by
Re: [Tutor] Pickles and Shelves Concept
On 07/06/2019 18:42, Gursimran Maken wrote: > I am not getting the concept of pickle and shelves in python, I mean what's > the use of both the concepts, when to use them in code instead of using > file read and write operations. You are right in that you could save all your data to a file using read/write primitives. Of course you'd need to convert them all to/from strings. And all collections would need to be iterated over (and if the collection contained other collections they'd need to be iterated too). But with pickle you can just save the entire data structure with all its contents, in a single operation and it can be a collection (of any type and depth) and the contents can be any kind of mixed data. You don't need to care, pickle takes care of it all for you. And you can read it back again in a single operation and all the data will be restored to its original type and location. But with pickle you still need to store things in sequence and read them back out in the sequence that you stored them. If all you are doing is saving the state of your program and restoring it that isn't a problem. But if you only want to retrieve one piece of your data later then its a bit of a nuisance. That's where shelve comes in. By acting like a persistent dictionary you can assign keys to your data and then restore it in any sequence, or only restore some of it, you can. Does that help? -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Pickles and Shelves Concept
> On Jun 7, 2019, at 12:42, Gursimran Maken wrote: > > Hi All, > > I am not getting the concept of pickle and shelves in python, I mean what's > the use of both the concepts, when to use them in code instead of using > file read and write operations. > > Could anyone please explain me the concepts. The simplest way to look at it is they are ways to save python data objects to disk. That way, you can have a dataset in an already-usable format for later on use. By contrast, file read/write is usually for raw data in/out in an unprocessed form that is not readily usable in your program. So you would see something like this: read data from file store data in a dictionary pickle and shelve the dictionary for later Then later on… grab the shelved pickle access the dict that was shelved If the data you are working with will always be fresh/new, the file reads is probably more usable. The shelving of data is most useful for easier retrieval later so you don’t have to re-process the raw data every time. — David Rock da...@graniteweb.com ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] Pickles and Shelves Concept
Hi All, I am not getting the concept of pickle and shelves in python, I mean what's the use of both the concepts, when to use them in code instead of using file read and write operations. Could anyone please explain me the concepts. Thank you, Gursimran. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor