Re: [PyMOL] dump_binary and PyMOL 1.8 PSE files
To finish this thread -- Jmol and JSmol can now read (basic) PyMOL 1.8 PSE files that have set dump_binary, 1 for faster loading and smaller file sizes. Jmol only implements a small, basic range of PyMOL functionality, of course. https://sourceforge.net/projects/jmol/files/Jmol/Version%2014.6/Version%2014.6.1/Jmol-14.6.1_2016.08.09-binary.zip/download Thank you, Thomas, for pointing me in the right direction on this. Bob Hanson -- What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic patterns at an interface-level. Reveals which users, apps, and protocols are consuming the most bandwidth. Provides multi-vendor support for NetFlow, J-Flow, sFlow and other flows. Make informed decisions using capacity planning reports. http://sdm.link/zohodev2dev___ PyMOL-users mailing list (PyMOL-users@lists.sourceforge.net) Info Page: https://lists.sourceforge.net/lists/listinfo/pymol-users Archives: http://www.mail-archive.com/pymol-users@lists.sourceforge.net
Re: [PyMOL] dump_binary and PyMOL 1.8 PSE files
I'll continue this off-list with Bob. Anyone who's interested in following along please drop me an email :) On 08 Aug 2016, at 17:38, Robert Hansonwrote: > Almost have it. Having trouble fitting all that into 120 bytes, though. I'm > good through bytes 96-99, visrep. Somehow, though I count 124 bytes total, > not 120 (which is clearly the case in the PSE file). What is > sizeof(AtomInfo_1_8_1) ? I calculate: > > ... > 100-105 (six booleans) > 106-111 (six characters) > 112-116 (5 char for ElemName) > 117-118 (2 char for SSType) > 119-120 (2 char for Chain) > 121 stereo > 123 chemFlag > 123 protekted > > I need this to end at 119, not 123, to be consistent with the pickle data. -- Thomas Holder PyMOL Principal Developer Schrödinger, Inc. -- What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic patterns at an interface-level. Reveals which users, apps, and protocols are consuming the most bandwidth. Provides multi-vendor support for NetFlow, J-Flow, sFlow and other flows. Make informed decisions using capacity planning reports. http://sdm.link/zohodev2dev ___ PyMOL-users mailing list (PyMOL-users@lists.sourceforge.net) Info Page: https://lists.sourceforge.net/lists/listinfo/pymol-users Archives: http://www.mail-archive.com/pymol-users@lists.sourceforge.net
Re: [PyMOL] dump_binary and PyMOL 1.8 PSE files
Almost have it. Having trouble fitting all that into 120 bytes, though. I'm good through bytes 96-99, visrep. Somehow, though I count 124 bytes total, not 120 (which is clearly the case in the PSE file). What is sizeof(AtomInfo_1_8_1) ? I calculate: ... 100-105 (six booleans) 106-111 (six characters) 112-116 (5 char for ElemName) 117-118 (2 char for SSType) 119-120 (2 char for Chain) 121 stereo 123 chemFlag 123 protekted I need this to end at 119, not 123, to be consistent with the pickle data. -- What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic patterns at an interface-level. Reveals which users, apps, and protocols are consuming the most bandwidth. Provides multi-vendor support for NetFlow, J-Flow, sFlow and other flows. Make informed decisions using capacity planning reports. http://sdm.link/zohodev2dev___ PyMOL-users mailing list (PyMOL-users@lists.sourceforge.net) Info Page: https://lists.sourceforge.net/lists/listinfo/pymol-users Archives: http://www.mail-archive.com/pymol-users@lists.sourceforge.net
Re: [PyMOL] dump_binary and PyMOL 1.8 PSE files
Ah, that is what I needed. For some reason that file is not in the download I unzipped. Much more sensible! Bob On Mon, Aug 8, 2016 at 3:18 PM, Thomas Holderwrote: > On 08 Aug 2016, at 16:01, Robert Hanson wrote: > > > I see this in 1.8.2.1: > > > > #define _PyMOL_VERSION "1.8.2.1" > > > > and in AtomInfo.h: > > > > #if _PyMOL_VERSION_int < 1770 > > #define AtomInfoVERSION 176 > > #define BondInfoVERSION 176 > > #elif _PyMOL_VERSION_int < 1810 > > #define AtomInfoVERSION 177 > > #define BondInfoVERSION 177 > > #else > > #define AtomInfoVERSION 181 > > #define BondInfoVERSION 181 > > #endif > > > > So that must be where that 181 version indicator is coming from in a PSE > file saved by 1.8.2.3, right? > > Correct. > > > But when you look at AtomInfoType in that same file, we see: > > > > typedef struct AtomInfoType { > > union { > > float * anisou; // only allocate with get_anisou > > int64_t dummyanisou; > > }; > > > > lexidx_t segi; > > lexidx_t chain; > > lexidx_t resn; > > lexidx_t name; > > lexidx_t textType; > > lexidx_t custom; > > lexidx_t label; > > ... > > > > What am I missing? > > You're looking at the wrong struct. When doing the binary dump, structs > from layer2/AtomInfoHistory.h are used. See https://sourceforge.net/p/ > pymol/code/HEAD/tree/trunk/pymol/layer2/AtomInfoHistory.h#l187 > > Cheers, > Thomas > > -- > Thomas Holder > PyMOL Principal Developer > Schrödinger, Inc. > > -- Robert M. Hanson Larson-Anderson Professor of Chemistry St. Olaf College Northfield, MN http://www.stolaf.edu/people/hansonr If nature does not answer first what we want, it is better to take what answer we get. -- Josiah Willard Gibbs, Lecture XXX, Monday, February 5, 1900 -- What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic patterns at an interface-level. Reveals which users, apps, and protocols are consuming the most bandwidth. Provides multi-vendor support for NetFlow, J-Flow, sFlow and other flows. Make informed decisions using capacity planning reports. http://sdm.link/zohodev2dev___ PyMOL-users mailing list (PyMOL-users@lists.sourceforge.net) Info Page: https://lists.sourceforge.net/lists/listinfo/pymol-users Archives: http://www.mail-archive.com/pymol-users@lists.sourceforge.net
Re: [PyMOL] dump_binary and PyMOL 1.8 PSE files
On 08 Aug 2016, at 16:01, Robert Hansonwrote: > I see this in 1.8.2.1: > > #define _PyMOL_VERSION "1.8.2.1" > > and in AtomInfo.h: > > #if _PyMOL_VERSION_int < 1770 > #define AtomInfoVERSION 176 > #define BondInfoVERSION 176 > #elif _PyMOL_VERSION_int < 1810 > #define AtomInfoVERSION 177 > #define BondInfoVERSION 177 > #else > #define AtomInfoVERSION 181 > #define BondInfoVERSION 181 > #endif > > So that must be where that 181 version indicator is coming from in a PSE file > saved by 1.8.2.3, right? Correct. > But when you look at AtomInfoType in that same file, we see: > > typedef struct AtomInfoType { > union { > float * anisou; // only allocate with get_anisou > int64_t dummyanisou; > }; > > lexidx_t segi; > lexidx_t chain; > lexidx_t resn; > lexidx_t name; > lexidx_t textType; > lexidx_t custom; > lexidx_t label; > ... > > What am I missing? You're looking at the wrong struct. When doing the binary dump, structs from layer2/AtomInfoHistory.h are used. See https://sourceforge.net/p/pymol/code/HEAD/tree/trunk/pymol/layer2/AtomInfoHistory.h#l187 Cheers, Thomas -- Thomas Holder PyMOL Principal Developer Schrödinger, Inc. -- What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic patterns at an interface-level. Reveals which users, apps, and protocols are consuming the most bandwidth. Provides multi-vendor support for NetFlow, J-Flow, sFlow and other flows. Make informed decisions using capacity planning reports. http://sdm.link/zohodev2dev ___ PyMOL-users mailing list (PyMOL-users@lists.sourceforge.net) Info Page: https://lists.sourceforge.net/lists/listinfo/pymol-users Archives: http://www.mail-archive.com/pymol-users@lists.sourceforge.net
Re: [PyMOL] dump_binary and PyMOL 1.8 PSE files
I see this in 1.8.2.1: #define _PyMOL_VERSION "1.8.2.1" and in AtomInfo.h: #if _PyMOL_VERSION_int < 1770 #define AtomInfoVERSION 176 #define BondInfoVERSION 176 #elif _PyMOL_VERSION_int < 1810 #define AtomInfoVERSION 177 #define BondInfoVERSION 177 #else #define AtomInfoVERSION 181 #define BondInfoVERSION 181 #endif So that must be where that 181 version indicator is coming from in a PSE file saved by 1.8.2.3, right? But when you look at AtomInfoType in that same file, we see: typedef struct AtomInfoType { union { float * anisou; // only allocate with get_anisou int64_t dummyanisou; }; lexidx_t segi; lexidx_t chain; lexidx_t resn; lexidx_t name; lexidx_t textType; lexidx_t custom; lexidx_t label; ... What am I missing? Bob On Mon, Aug 8, 2016 at 10:51 AM, Thomas Holder < thomas.hol...@schrodinger.com> wrote: > look here: > https://sourceforge.net/p/pymol/code/HEAD/tree/trunk/ > pymol/layer2/AtomInfoHistory.h > > The AtomInfoType_1_8_1 version (used in 1.8.2) stores ANISOU as shorts > (PDB precision). The AtomInfoType_1_7_7 version (used in 1.8.0) doesn't > support ANISOU (will indeed write an unused NULL pointer), at the time it > was work in progress for a specific use case which didn't require ANISOU. > > Cheers, > Thomas > > On 08 Aug 2016, at 11:32, Robert Hansonwrote: > > > I do like the compactness of that and the quicker loading, but I do see > how this could be a nightmare. I'm looking at a file that lists the dump > version as 1.8.1, but I don't even see how that would ever work, since in > that AtomInfo structure the ANISOU data is just a 64-bit pointer. That > would end up just a useless reference, wouldn't it? > > > > On Mon, Aug 8, 2016 at 10:05 AM, Thomas Holder < > thomas.hol...@schrodinger.com> wrote: > > Hi Bob, > > > > I assume those are PSE files with setting "pse_binary_dump=on". This is > not the default. You can load such files into PyMOL, set > pse_binary_dump=off and save it as a regular serialized PSE. I definitely > recommend the conversion step over reimplementing the "pse_binary_dump" > decoding. Right now there are 3 binary atom layouts (for 1.7.6, 1.8.0 and > 1.8.2). Grep the PyMOL code for "AtomInfoTypeConverter" if you're > interested in the details. > > > > The default (in 1.8.2) is to save PyMOL 1.7.6 compatible PSE files, and > with the "pse_export_version" setting you can also export for older PyMOL > versions. > > > > Cheers, > > Thomas > > > > On 08 Aug 2016, at 07:57, Robert Hanson wrote: > > > > > Thomas, > > > > > > In reading PyMOL PSE files into Jmol I have recently run into the > problem with some of these files (version 1.8.2.3). Jmol has no problem > reading the standard Pickle format, but there seems to be an option in > PyMOL to serialize binary data structures as BINSTRING or SHORT_BINSTRING > format along with other standard strings. This sounds like a great idea; I > have no doubt that this will lead to considerably smaller PSE files and > also faster loading. However, I'm struggling with atom information > deserialization. > > > > > > Q: Is this the default format now for saving PyMOL PSE files? > > > > > > Q: Is there a way to turn that off? > > > > > > Q: In order to do this properly, how many versions of the atom info > data structure do I need to be able to process? > > > > > > Bob > > > > -- > > Robert M. Hanson > > Larson-Anderson Professor of Chemistry > > St. Olaf College > > Northfield, MN > > http://www.stolaf.edu/people/hansonr > > > > > > If nature does not answer first what we want, > > it is better to take what answer we get. > > > > -- Josiah Willard Gibbs, Lecture XXX, Monday, February 5, 1900 > > -- > Thomas Holder > PyMOL Principal Developer > Schrödinger, Inc. > > -- Robert M. Hanson Larson-Anderson Professor of Chemistry St. Olaf College Northfield, MN http://www.stolaf.edu/people/hansonr If nature does not answer first what we want, it is better to take what answer we get. -- Josiah Willard Gibbs, Lecture XXX, Monday, February 5, 1900 -- What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic patterns at an interface-level. Reveals which users, apps, and protocols are consuming the most bandwidth. Provides multi-vendor support for NetFlow, J-Flow, sFlow and other flows. Make informed decisions using capacity planning reports. http://sdm.link/zohodev2dev___ PyMOL-users mailing list (PyMOL-users@lists.sourceforge.net) Info Page: https://lists.sourceforge.net/lists/listinfo/pymol-users Archives: http://www.mail-archive.com/pymol-users@lists.sourceforge.net
Re: [PyMOL] dump_binary and PyMOL 1.8 PSE files
look here: https://sourceforge.net/p/pymol/code/HEAD/tree/trunk/pymol/layer2/AtomInfoHistory.h The AtomInfoType_1_8_1 version (used in 1.8.2) stores ANISOU as shorts (PDB precision). The AtomInfoType_1_7_7 version (used in 1.8.0) doesn't support ANISOU (will indeed write an unused NULL pointer), at the time it was work in progress for a specific use case which didn't require ANISOU. Cheers, Thomas On 08 Aug 2016, at 11:32, Robert Hansonwrote: > I do like the compactness of that and the quicker loading, but I do see how > this could be a nightmare. I'm looking at a file that lists the dump version > as 1.8.1, but I don't even see how that would ever work, since in that > AtomInfo structure the ANISOU data is just a 64-bit pointer. That would end > up just a useless reference, wouldn't it? > > On Mon, Aug 8, 2016 at 10:05 AM, Thomas Holder > wrote: > Hi Bob, > > I assume those are PSE files with setting "pse_binary_dump=on". This is not > the default. You can load such files into PyMOL, set pse_binary_dump=off and > save it as a regular serialized PSE. I definitely recommend the conversion > step over reimplementing the "pse_binary_dump" decoding. Right now there are > 3 binary atom layouts (for 1.7.6, 1.8.0 and 1.8.2). Grep the PyMOL code for > "AtomInfoTypeConverter" if you're interested in the details. > > The default (in 1.8.2) is to save PyMOL 1.7.6 compatible PSE files, and with > the "pse_export_version" setting you can also export for older PyMOL versions. > > Cheers, > Thomas > > On 08 Aug 2016, at 07:57, Robert Hanson wrote: > > > Thomas, > > > > In reading PyMOL PSE files into Jmol I have recently run into the problem > > with some of these files (version 1.8.2.3). Jmol has no problem reading the > > standard Pickle format, but there seems to be an option in PyMOL to > > serialize binary data structures as BINSTRING or SHORT_BINSTRING format > > along with other standard strings. This sounds like a great idea; I have no > > doubt that this will lead to considerably smaller PSE files and also faster > > loading. However, I'm struggling with atom information deserialization. > > > > Q: Is this the default format now for saving PyMOL PSE files? > > > > Q: Is there a way to turn that off? > > > > Q: In order to do this properly, how many versions of the atom info data > > structure do I need to be able to process? > > > > Bob > > -- > Robert M. Hanson > Larson-Anderson Professor of Chemistry > St. Olaf College > Northfield, MN > http://www.stolaf.edu/people/hansonr > > > If nature does not answer first what we want, > it is better to take what answer we get. > > -- Josiah Willard Gibbs, Lecture XXX, Monday, February 5, 1900 -- Thomas Holder PyMOL Principal Developer Schrödinger, Inc. -- What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic patterns at an interface-level. Reveals which users, apps, and protocols are consuming the most bandwidth. Provides multi-vendor support for NetFlow, J-Flow, sFlow and other flows. Make informed decisions using capacity planning reports. http://sdm.link/zohodev2dev ___ PyMOL-users mailing list (PyMOL-users@lists.sourceforge.net) Info Page: https://lists.sourceforge.net/lists/listinfo/pymol-users Archives: http://www.mail-archive.com/pymol-users@lists.sourceforge.net
Re: [PyMOL] dump_binary and PyMOL 1.8 PSE files
I do like the compactness of that and the quicker loading, but I do see how this could be a nightmare. I'm looking at a file that lists the dump version as 1.8.1, but I don't even see how that would ever work, since in that AtomInfo structure the ANISOU data is just a 64-bit pointer. That would end up just a useless reference, wouldn't it? On Mon, Aug 8, 2016 at 10:05 AM, Thomas Holder < thomas.hol...@schrodinger.com> wrote: > Hi Bob, > > I assume those are PSE files with setting "pse_binary_dump=on". This is > not the default. You can load such files into PyMOL, set > pse_binary_dump=off and save it as a regular serialized PSE. I definitely > recommend the conversion step over reimplementing the "pse_binary_dump" > decoding. Right now there are 3 binary atom layouts (for 1.7.6, 1.8.0 and > 1.8.2). Grep the PyMOL code for "AtomInfoTypeConverter" if you're > interested in the details. > > The default (in 1.8.2) is to save PyMOL 1.7.6 compatible PSE files, and > with the "pse_export_version" setting you can also export for older PyMOL > versions. > > Cheers, > Thomas > > On 08 Aug 2016, at 07:57, Robert Hansonwrote: > > > Thomas, > > > > In reading PyMOL PSE files into Jmol I have recently run into the > problem with some of these files (version 1.8.2.3). Jmol has no problem > reading the standard Pickle format, but there seems to be an option in > PyMOL to serialize binary data structures as BINSTRING or SHORT_BINSTRING > format along with other standard strings. This sounds like a great idea; I > have no doubt that this will lead to considerably smaller PSE files and > also faster loading. However, I'm struggling with atom information > deserialization. > > > > Q: Is this the default format now for saving PyMOL PSE files? > > > > Q: Is there a way to turn that off? > > > > Q: In order to do this properly, how many versions of the atom info data > structure do I need to be able to process? > > > > Bob > > > > -- > > Robert M. Hanson > > Larson-Anderson Professor of Chemistry > > St. Olaf College > > Northfield, MN > > http://www.stolaf.edu/people/hansonr > > > > > > If nature does not answer first what we want, > > it is better to take what answer we get. > > > > -- Josiah Willard Gibbs, Lecture XXX, Monday, February 5, 1900 > > -- > Thomas Holder > PyMOL Principal Developer > Schrödinger, Inc. > > -- Robert M. Hanson Larson-Anderson Professor of Chemistry St. Olaf College Northfield, MN http://www.stolaf.edu/people/hansonr If nature does not answer first what we want, it is better to take what answer we get. -- Josiah Willard Gibbs, Lecture XXX, Monday, February 5, 1900 -- What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic patterns at an interface-level. Reveals which users, apps, and protocols are consuming the most bandwidth. Provides multi-vendor support for NetFlow, J-Flow, sFlow and other flows. Make informed decisions using capacity planning reports. http://sdm.link/zohodev2dev___ PyMOL-users mailing list (PyMOL-users@lists.sourceforge.net) Info Page: https://lists.sourceforge.net/lists/listinfo/pymol-users Archives: http://www.mail-archive.com/pymol-users@lists.sourceforge.net
Re: [PyMOL] dump_binary and PyMOL 1.8 PSE files
Hi Bob, I assume those are PSE files with setting "pse_binary_dump=on". This is not the default. You can load such files into PyMOL, set pse_binary_dump=off and save it as a regular serialized PSE. I definitely recommend the conversion step over reimplementing the "pse_binary_dump" decoding. Right now there are 3 binary atom layouts (for 1.7.6, 1.8.0 and 1.8.2). Grep the PyMOL code for "AtomInfoTypeConverter" if you're interested in the details. The default (in 1.8.2) is to save PyMOL 1.7.6 compatible PSE files, and with the "pse_export_version" setting you can also export for older PyMOL versions. Cheers, Thomas On 08 Aug 2016, at 07:57, Robert Hansonwrote: > Thomas, > > In reading PyMOL PSE files into Jmol I have recently run into the problem > with some of these files (version 1.8.2.3). Jmol has no problem reading the > standard Pickle format, but there seems to be an option in PyMOL to serialize > binary data structures as BINSTRING or SHORT_BINSTRING format along with > other standard strings. This sounds like a great idea; I have no doubt that > this will lead to considerably smaller PSE files and also faster loading. > However, I'm struggling with atom information deserialization. > > Q: Is this the default format now for saving PyMOL PSE files? > > Q: Is there a way to turn that off? > > Q: In order to do this properly, how many versions of the atom info data > structure do I need to be able to process? > > Bob > > -- > Robert M. Hanson > Larson-Anderson Professor of Chemistry > St. Olaf College > Northfield, MN > http://www.stolaf.edu/people/hansonr > > > If nature does not answer first what we want, > it is better to take what answer we get. > > -- Josiah Willard Gibbs, Lecture XXX, Monday, February 5, 1900 -- Thomas Holder PyMOL Principal Developer Schrödinger, Inc. -- ___ PyMOL-users mailing list (PyMOL-users@lists.sourceforge.net) Info Page: https://lists.sourceforge.net/lists/listinfo/pymol-users Archives: http://www.mail-archive.com/pymol-users@lists.sourceforge.net
[PyMOL] dump_binary and PyMOL 1.8 PSE files
Thomas, In reading PyMOL PSE files into Jmol I have recently run into the problem with some of these files (version 1.8.2.3). Jmol has no problem reading the standard Pickle format, but there seems to be an option in PyMOL to serialize binary data structures as BINSTRING or SHORT_BINSTRING format along with other standard strings. This sounds like a great idea; I have no doubt that this will lead to considerably smaller PSE files and also faster loading. However, I'm struggling with atom information deserialization. Q: Is this the default format now for saving PyMOL PSE files? Q: Is there a way to turn that off? Q: In order to do this properly, how many versions of the atom info data structure do I need to be able to process? Bob -- Robert M. Hanson Larson-Anderson Professor of Chemistry St. Olaf College Northfield, MN http://www.stolaf.edu/people/hansonr If nature does not answer first what we want, it is better to take what answer we get. -- Josiah Willard Gibbs, Lecture XXX, Monday, February 5, 1900 -- ___ PyMOL-users mailing list (PyMOL-users@lists.sourceforge.net) Info Page: https://lists.sourceforge.net/lists/listinfo/pymol-users Archives: http://www.mail-archive.com/pymol-users@lists.sourceforge.net