Re: [PyMOL] dump_binary and PyMOL 1.8 PSE files

2016-08-10 Thread Robert Hanson
To finish this thread -- Jmol and JSmol can now read (basic) PyMOL 1.8 PSE
files that have

set dump_binary, 1

for faster loading and smaller file sizes. Jmol only implements a small,
basic range of PyMOL functionality, of course.

https://sourceforge.net/projects/jmol/files/Jmol/Version%2014.6/Version%2014.6.1/Jmol-14.6.1_2016.08.09-binary.zip/download

Thank you, Thomas, for pointing me in the right direction on this.

Bob Hanson





​
--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity 
planning reports. http://sdm.link/zohodev2dev___
PyMOL-users mailing list (PyMOL-users@lists.sourceforge.net)
Info Page: https://lists.sourceforge.net/lists/listinfo/pymol-users
Archives: http://www.mail-archive.com/pymol-users@lists.sourceforge.net

Re: [PyMOL] dump_binary and PyMOL 1.8 PSE files

2016-08-08 Thread Thomas Holder
I'll continue this off-list with Bob. Anyone who's interested in following 
along please drop me an email :)

On 08 Aug 2016, at 17:38, Robert Hanson  wrote:

> Almost have it. Having trouble fitting all that into 120 bytes, though. I'm 
> good through bytes 96-99, visrep. Somehow, though I count 124 bytes total, 
> not 120 (which is clearly the case in the PSE file). What is 
> sizeof(AtomInfo_1_8_1) ? I calculate:
> 
> ...
> 100-105 (six booleans)
> 106-111 (six characters)
> 112-116 (5 char for ElemName)
> 117-118 (2 char for SSType)
> 119-120 (2 char for Chain)
> 121 stereo
> 123 chemFlag
> 123 protekted
> 
> I need this to end at 119, not 123, to be consistent with the pickle data.

-- 
Thomas Holder
PyMOL Principal Developer
Schrödinger, Inc.


--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity 
planning reports. http://sdm.link/zohodev2dev
___
PyMOL-users mailing list (PyMOL-users@lists.sourceforge.net)
Info Page: https://lists.sourceforge.net/lists/listinfo/pymol-users
Archives: http://www.mail-archive.com/pymol-users@lists.sourceforge.net


Re: [PyMOL] dump_binary and PyMOL 1.8 PSE files

2016-08-08 Thread Robert Hanson
Almost have it. Having trouble fitting all that into 120 bytes, though. I'm
good through bytes 96-99, visrep. Somehow, though I count 124 bytes total,
not 120 (which is clearly the case in the PSE file). What is
sizeof(AtomInfo_1_8_1) ? I calculate:

...
100-105 (six booleans)
106-111 (six characters)
112-116 (5 char for ElemName)
117-118 (2 char for SSType)
119-120 (2 char for Chain)
121 stereo
123 chemFlag
123 protekted

I need this to end at 119, not 123, to be consistent with the pickle data.


​
--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity 
planning reports. http://sdm.link/zohodev2dev___
PyMOL-users mailing list (PyMOL-users@lists.sourceforge.net)
Info Page: https://lists.sourceforge.net/lists/listinfo/pymol-users
Archives: http://www.mail-archive.com/pymol-users@lists.sourceforge.net

Re: [PyMOL] dump_binary and PyMOL 1.8 PSE files

2016-08-08 Thread Robert Hanson
Ah, that is what I needed. For some reason that file is not in the download
I unzipped. Much more sensible!

Bob

On Mon, Aug 8, 2016 at 3:18 PM, Thomas Holder  wrote:

> On 08 Aug 2016, at 16:01, Robert Hanson  wrote:
>
> > I see this in 1.8.2.1:
> >
> > #define _PyMOL_VERSION "1.8.2.1"
> >
> > and in AtomInfo.h:
> >
> > #if _PyMOL_VERSION_int < 1770
> > #define AtomInfoVERSION  176
> > #define BondInfoVERSION  176
> > #elif _PyMOL_VERSION_int < 1810
> > #define AtomInfoVERSION  177
> > #define BondInfoVERSION  177
> > #else
> > #define AtomInfoVERSION  181
> > #define BondInfoVERSION  181
> > #endif
> >
> > So that must be where that 181 version indicator is coming from in a PSE
> file saved by 1.8.2.3, right?
>
> Correct.
>
> > But when you look at AtomInfoType in that same file, we see:
> >
> > typedef struct AtomInfoType {
> >   union {
> > float * anisou;   // only allocate with get_anisou
> > int64_t dummyanisou;
> >   };
> >
> >   lexidx_t segi;
> >   lexidx_t chain;
> >   lexidx_t resn;
> >   lexidx_t name;
> >   lexidx_t textType;
> >   lexidx_t custom;
> >   lexidx_t label;
> > ...
> >
> > What am I missing?
>
> You're looking at the wrong struct. When doing the binary dump, structs
> from layer2/AtomInfoHistory.h are used. See https://sourceforge.net/p/
> pymol/code/HEAD/tree/trunk/pymol/layer2/AtomInfoHistory.h#l187
>
> Cheers,
>   Thomas
>
> --
> Thomas Holder
> PyMOL Principal Developer
> Schrödinger, Inc.
>
>


-- 
Robert M. Hanson
Larson-Anderson Professor of Chemistry
St. Olaf College
Northfield, MN
http://www.stolaf.edu/people/hansonr


If nature does not answer first what we want,
it is better to take what answer we get.

-- Josiah Willard Gibbs, Lecture XXX, Monday, February 5, 1900
--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity 
planning reports. http://sdm.link/zohodev2dev___
PyMOL-users mailing list (PyMOL-users@lists.sourceforge.net)
Info Page: https://lists.sourceforge.net/lists/listinfo/pymol-users
Archives: http://www.mail-archive.com/pymol-users@lists.sourceforge.net

Re: [PyMOL] dump_binary and PyMOL 1.8 PSE files

2016-08-08 Thread Thomas Holder
On 08 Aug 2016, at 16:01, Robert Hanson  wrote:

> I see this in 1.8.2.1:
> 
> #define _PyMOL_VERSION "1.8.2.1"
> 
> and in AtomInfo.h:
> 
> #if _PyMOL_VERSION_int < 1770
> #define AtomInfoVERSION  176
> #define BondInfoVERSION  176
> #elif _PyMOL_VERSION_int < 1810
> #define AtomInfoVERSION  177
> #define BondInfoVERSION  177
> #else
> #define AtomInfoVERSION  181
> #define BondInfoVERSION  181
> #endif
> 
> So that must be where that 181 version indicator is coming from in a PSE file 
> saved by 1.8.2.3, right?

Correct.

> But when you look at AtomInfoType in that same file, we see:
> 
> typedef struct AtomInfoType {
>   union {
> float * anisou;   // only allocate with get_anisou
> int64_t dummyanisou;
>   };
> 
>   lexidx_t segi;
>   lexidx_t chain;
>   lexidx_t resn;
>   lexidx_t name;
>   lexidx_t textType;
>   lexidx_t custom;
>   lexidx_t label;
> ...
> 
> What am I missing?

You're looking at the wrong struct. When doing the binary dump, structs from 
layer2/AtomInfoHistory.h are used. See 
https://sourceforge.net/p/pymol/code/HEAD/tree/trunk/pymol/layer2/AtomInfoHistory.h#l187

Cheers,
  Thomas

-- 
Thomas Holder
PyMOL Principal Developer
Schrödinger, Inc.


--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity 
planning reports. http://sdm.link/zohodev2dev
___
PyMOL-users mailing list (PyMOL-users@lists.sourceforge.net)
Info Page: https://lists.sourceforge.net/lists/listinfo/pymol-users
Archives: http://www.mail-archive.com/pymol-users@lists.sourceforge.net


Re: [PyMOL] dump_binary and PyMOL 1.8 PSE files

2016-08-08 Thread Robert Hanson
I see this in 1.8.2.1:

#define _PyMOL_VERSION "1.8.2.1"

and in AtomInfo.h:

#if _PyMOL_VERSION_int < 1770
#define AtomInfoVERSION  176
#define BondInfoVERSION  176
#elif _PyMOL_VERSION_int < 1810
#define AtomInfoVERSION  177
#define BondInfoVERSION  177
#else
#define AtomInfoVERSION  181
#define BondInfoVERSION  181
#endif

So that must be where that 181 version indicator is coming from in a PSE
file saved by 1.8.2.3, right?

But when you look at AtomInfoType in that same file, we see:

typedef struct AtomInfoType {
  union {
float * anisou;   // only allocate with get_anisou
int64_t dummyanisou;
  };

  lexidx_t segi;
  lexidx_t chain;
  lexidx_t resn;
  lexidx_t name;
  lexidx_t textType;
  lexidx_t custom;
  lexidx_t label;
...

What am I missing?

Bob





On Mon, Aug 8, 2016 at 10:51 AM, Thomas Holder <
thomas.hol...@schrodinger.com> wrote:

> look here:
> https://sourceforge.net/p/pymol/code/HEAD/tree/trunk/
> pymol/layer2/AtomInfoHistory.h
>
> The AtomInfoType_1_8_1 version (used in 1.8.2) stores ANISOU as shorts
> (PDB precision). The AtomInfoType_1_7_7 version (used in 1.8.0) doesn't
> support ANISOU (will indeed write an unused NULL pointer), at the time it
> was work in progress for a specific use case which didn't require ANISOU.
>
> Cheers,
>   Thomas
>
> On 08 Aug 2016, at 11:32, Robert Hanson  wrote:
>
> > I do like the compactness of that and the quicker loading, but I do see
> how this could be a nightmare. I'm looking  at a file that lists the dump
> version as 1.8.1, but I don't even see how that would ever work, since in
> that AtomInfo structure the ANISOU data is just a 64-bit pointer. That
> would end up just a useless reference, wouldn't it?
> >
> > On Mon, Aug 8, 2016 at 10:05 AM, Thomas Holder <
> thomas.hol...@schrodinger.com> wrote:
> > Hi Bob,
> >
> > I assume those are PSE files with setting "pse_binary_dump=on". This is
> not the default. You can load such files into PyMOL, set
> pse_binary_dump=off and save it as a regular serialized PSE. I definitely
> recommend the conversion step over reimplementing the "pse_binary_dump"
> decoding. Right now there are 3 binary atom layouts (for 1.7.6, 1.8.0 and
> 1.8.2). Grep the PyMOL code for "AtomInfoTypeConverter" if you're
> interested in the details.
> >
> > The default (in 1.8.2) is to save PyMOL 1.7.6 compatible PSE files, and
> with the "pse_export_version" setting you can also export for older PyMOL
> versions.
> >
> > Cheers,
> >   Thomas
> >
> > On 08 Aug 2016, at 07:57, Robert Hanson  wrote:
> >
> > > Thomas,
> > >
> > > In reading PyMOL PSE files into Jmol I have recently run into the
> problem with some of these files (version 1.8.2.3). Jmol has no problem
> reading the standard Pickle format, but there seems to be an option in
> PyMOL to serialize binary data structures as BINSTRING or SHORT_BINSTRING
> format along with other standard strings. This sounds like a great idea; I
> have no doubt that this will lead to considerably smaller PSE files and
> also faster loading. However, I'm struggling with atom information
> deserialization.
> > >
> > > Q: Is this the default format now for saving PyMOL PSE files?
> > >
> > > Q: Is there a way to turn that off?
> > >
> > > Q: In order to do this properly, how many versions of the atom info
> data structure do I need to be able to process?
> > >
> > > Bob
> >
> > --
> > Robert M. Hanson
> > Larson-Anderson Professor of Chemistry
> > St. Olaf College
> > Northfield, MN
> > http://www.stolaf.edu/people/hansonr
> >
> >
> > If nature does not answer first what we want,
> > it is better to take what answer we get.
> >
> > -- Josiah Willard Gibbs, Lecture XXX, Monday, February 5, 1900
>
> --
> Thomas Holder
> PyMOL Principal Developer
> Schrödinger, Inc.
>
>


-- 
Robert M. Hanson
Larson-Anderson Professor of Chemistry
St. Olaf College
Northfield, MN
http://www.stolaf.edu/people/hansonr


If nature does not answer first what we want,
it is better to take what answer we get.

-- Josiah Willard Gibbs, Lecture XXX, Monday, February 5, 1900
--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity 
planning reports. http://sdm.link/zohodev2dev___
PyMOL-users mailing list (PyMOL-users@lists.sourceforge.net)
Info Page: https://lists.sourceforge.net/lists/listinfo/pymol-users
Archives: http://www.mail-archive.com/pymol-users@lists.sourceforge.net

Re: [PyMOL] dump_binary and PyMOL 1.8 PSE files

2016-08-08 Thread Thomas Holder
look here:
https://sourceforge.net/p/pymol/code/HEAD/tree/trunk/pymol/layer2/AtomInfoHistory.h

The AtomInfoType_1_8_1 version (used in 1.8.2) stores ANISOU as shorts (PDB 
precision). The AtomInfoType_1_7_7 version (used in 1.8.0) doesn't support 
ANISOU (will indeed write an unused NULL pointer), at the time it was work in 
progress for a specific use case which didn't require ANISOU.

Cheers,
  Thomas

On 08 Aug 2016, at 11:32, Robert Hanson  wrote:

> I do like the compactness of that and the quicker loading, but I do see how 
> this could be a nightmare. I'm looking  at a file that lists the dump version 
> as 1.8.1, but I don't even see how that would ever work, since in that 
> AtomInfo structure the ANISOU data is just a 64-bit pointer. That would end 
> up just a useless reference, wouldn't it?
> 
> On Mon, Aug 8, 2016 at 10:05 AM, Thomas Holder 
>  wrote:
> Hi Bob,
> 
> I assume those are PSE files with setting "pse_binary_dump=on". This is not 
> the default. You can load such files into PyMOL, set pse_binary_dump=off and 
> save it as a regular serialized PSE. I definitely recommend the conversion 
> step over reimplementing the "pse_binary_dump" decoding. Right now there are 
> 3 binary atom layouts (for 1.7.6, 1.8.0 and 1.8.2). Grep the PyMOL code for 
> "AtomInfoTypeConverter" if you're interested in the details.
> 
> The default (in 1.8.2) is to save PyMOL 1.7.6 compatible PSE files, and with 
> the "pse_export_version" setting you can also export for older PyMOL versions.
> 
> Cheers,
>   Thomas
> 
> On 08 Aug 2016, at 07:57, Robert Hanson  wrote:
> 
> > Thomas,
> >
> > In reading PyMOL PSE files into Jmol I have recently run into the problem 
> > with some of these files (version 1.8.2.3). Jmol has no problem reading the 
> > standard Pickle format, but there seems to be an option in PyMOL to 
> > serialize binary data structures as BINSTRING or SHORT_BINSTRING format 
> > along with other standard strings. This sounds like a great idea; I have no 
> > doubt that this will lead to considerably smaller PSE files and also faster 
> > loading. However, I'm struggling with atom information deserialization.
> >
> > Q: Is this the default format now for saving PyMOL PSE files?
> >
> > Q: Is there a way to turn that off?
> >
> > Q: In order to do this properly, how many versions of the atom info data 
> > structure do I need to be able to process?
> >
> > Bob
> 
> -- 
> Robert M. Hanson
> Larson-Anderson Professor of Chemistry
> St. Olaf College
> Northfield, MN
> http://www.stolaf.edu/people/hansonr
> 
> 
> If nature does not answer first what we want,
> it is better to take what answer we get. 
> 
> -- Josiah Willard Gibbs, Lecture XXX, Monday, February 5, 1900

-- 
Thomas Holder
PyMOL Principal Developer
Schrödinger, Inc.


--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity 
planning reports. http://sdm.link/zohodev2dev
___
PyMOL-users mailing list (PyMOL-users@lists.sourceforge.net)
Info Page: https://lists.sourceforge.net/lists/listinfo/pymol-users
Archives: http://www.mail-archive.com/pymol-users@lists.sourceforge.net


Re: [PyMOL] dump_binary and PyMOL 1.8 PSE files

2016-08-08 Thread Robert Hanson
I do like the compactness of that and the quicker loading, but I do see how
this could be a nightmare. I'm looking  at a file that lists the dump
version as 1.8.1, but I don't even see how that would ever work, since in
that AtomInfo structure the ANISOU data is just a 64-bit pointer. That
would end up just a useless reference, wouldn't it?

On Mon, Aug 8, 2016 at 10:05 AM, Thomas Holder <
thomas.hol...@schrodinger.com> wrote:

> Hi Bob,
>
> I assume those are PSE files with setting "pse_binary_dump=on". This is
> not the default. You can load such files into PyMOL, set
> pse_binary_dump=off and save it as a regular serialized PSE. I definitely
> recommend the conversion step over reimplementing the "pse_binary_dump"
> decoding. Right now there are 3 binary atom layouts (for 1.7.6, 1.8.0 and
> 1.8.2). Grep the PyMOL code for "AtomInfoTypeConverter" if you're
> interested in the details.
>
> The default (in 1.8.2) is to save PyMOL 1.7.6 compatible PSE files, and
> with the "pse_export_version" setting you can also export for older PyMOL
> versions.
>
> Cheers,
>   Thomas
>
> On 08 Aug 2016, at 07:57, Robert Hanson  wrote:
>
> > Thomas,
> >
> > In reading PyMOL PSE files into Jmol I have recently run into the
> problem with some of these files (version 1.8.2.3). Jmol has no problem
> reading the standard Pickle format, but there seems to be an option in
> PyMOL to serialize binary data structures as BINSTRING or SHORT_BINSTRING
> format along with other standard strings. This sounds like a great idea; I
> have no doubt that this will lead to considerably smaller PSE files and
> also faster loading. However, I'm struggling with atom information
> deserialization.
> >
> > Q: Is this the default format now for saving PyMOL PSE files?
> >
> > Q: Is there a way to turn that off?
> >
> > Q: In order to do this properly, how many versions of the atom info data
> structure do I need to be able to process?
> >
> > Bob
> >
> > --
> > Robert M. Hanson
> > Larson-Anderson Professor of Chemistry
> > St. Olaf College
> > Northfield, MN
> > http://www.stolaf.edu/people/hansonr
> >
> >
> > If nature does not answer first what we want,
> > it is better to take what answer we get.
> >
> > -- Josiah Willard Gibbs, Lecture XXX, Monday, February 5, 1900
>
> --
> Thomas Holder
> PyMOL Principal Developer
> Schrödinger, Inc.
>
>


-- 
Robert M. Hanson
Larson-Anderson Professor of Chemistry
St. Olaf College
Northfield, MN
http://www.stolaf.edu/people/hansonr


If nature does not answer first what we want,
it is better to take what answer we get.

-- Josiah Willard Gibbs, Lecture XXX, Monday, February 5, 1900
--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity 
planning reports. http://sdm.link/zohodev2dev___
PyMOL-users mailing list (PyMOL-users@lists.sourceforge.net)
Info Page: https://lists.sourceforge.net/lists/listinfo/pymol-users
Archives: http://www.mail-archive.com/pymol-users@lists.sourceforge.net

Re: [PyMOL] dump_binary and PyMOL 1.8 PSE files

2016-08-08 Thread Thomas Holder
Hi Bob,

I assume those are PSE files with setting "pse_binary_dump=on". This is not the 
default. You can load such files into PyMOL, set pse_binary_dump=off and save 
it as a regular serialized PSE. I definitely recommend the conversion step over 
reimplementing the "pse_binary_dump" decoding. Right now there are 3 binary 
atom layouts (for 1.7.6, 1.8.0 and 1.8.2). Grep the PyMOL code for 
"AtomInfoTypeConverter" if you're interested in the details.

The default (in 1.8.2) is to save PyMOL 1.7.6 compatible PSE files, and with 
the "pse_export_version" setting you can also export for older PyMOL versions.

Cheers,
  Thomas

On 08 Aug 2016, at 07:57, Robert Hanson  wrote:

> Thomas,
> 
> In reading PyMOL PSE files into Jmol I have recently run into the problem 
> with some of these files (version 1.8.2.3). Jmol has no problem reading the 
> standard Pickle format, but there seems to be an option in PyMOL to serialize 
> binary data structures as BINSTRING or SHORT_BINSTRING format along with 
> other standard strings. This sounds like a great idea; I have no doubt that 
> this will lead to considerably smaller PSE files and also faster loading. 
> However, I'm struggling with atom information deserialization. 
> 
> Q: Is this the default format now for saving PyMOL PSE files?
> 
> Q: Is there a way to turn that off? 
> 
> Q: In order to do this properly, how many versions of the atom info data 
> structure do I need to be able to process? 
> 
> Bob
> 
> -- 
> Robert M. Hanson
> Larson-Anderson Professor of Chemistry
> St. Olaf College
> Northfield, MN
> http://www.stolaf.edu/people/hansonr
> 
> 
> If nature does not answer first what we want,
> it is better to take what answer we get. 
> 
> -- Josiah Willard Gibbs, Lecture XXX, Monday, February 5, 1900

-- 
Thomas Holder
PyMOL Principal Developer
Schrödinger, Inc.


--
___
PyMOL-users mailing list (PyMOL-users@lists.sourceforge.net)
Info Page: https://lists.sourceforge.net/lists/listinfo/pymol-users
Archives: http://www.mail-archive.com/pymol-users@lists.sourceforge.net


[PyMOL] dump_binary and PyMOL 1.8 PSE files

2016-08-08 Thread Robert Hanson
Thomas,

In reading PyMOL PSE files into Jmol I have recently run into the problem
with some of these files (version 1.8.2.3). Jmol has no problem reading the
standard Pickle format, but there seems to be an option in PyMOL to
serialize binary data structures as BINSTRING or SHORT_BINSTRING format
along with other standard strings. This sounds like a great idea; I have no
doubt that this will lead to considerably smaller PSE files and also faster
loading. However, I'm struggling with atom information deserialization.

Q: Is this the default format now for saving PyMOL PSE files?

Q: Is there a way to turn that off?

Q: In order to do this properly, how many versions of the atom info data
structure do I need to be able to process?



Bob

-- 
Robert M. Hanson
Larson-Anderson Professor of Chemistry
St. Olaf College
Northfield, MN
http://www.stolaf.edu/people/hansonr


If nature does not answer first what we want,
it is better to take what answer we get.

-- Josiah Willard Gibbs, Lecture XXX, Monday, February 5, 1900
--
___
PyMOL-users mailing list (PyMOL-users@lists.sourceforge.net)
Info Page: https://lists.sourceforge.net/lists/listinfo/pymol-users
Archives: http://www.mail-archive.com/pymol-users@lists.sourceforge.net