Re: New Leo "Package" File Format?

[Offray Vladimir Luna Cárdenas]

Hi,

What we do, is that we package the associated files to a data narrative 
as a Fossil[1] repository, with versioned and unversioned files. 
Versioned files are used for the ones where we want to track the history 
and the unversioned are used for raster files (for example, PDF outputs 
for the data stories or PNG/JPG) images. This gives us a pretty simple 
infrastructure to exchange and publish our resources, that can travel in 
a single file. It's like .zip but with history.


[1] https://fossil-scm.org

I think that, via Fossil, SQLite as an application format is a pretty 
powerful tool. Some related docs and videos below:


 * https://www.youtube.com/watch?v=8y_ABXwYtuc
 * https://www.youtube.com/watch?v=Jib2AmRb_rk
 * https://www.sqlite.org/appfileformat.html

My 100 pesos,

Offray

On 29/03/23 12:48, Thomas Passin wrote:
There is a perennial problem when one wants to give a  Leo outline to 
someone else.  It happens when an outline contains external files, or 
images to display, or any other data files that might be needed.  For 
example, an article written with the Viewrendered3 plugin in mind, or 
for a Sphinx document, must have its resources available or it cannot 
work.


If the outline contains @file trees and these external files aren't 
included in, e.g., a zip file, those files will be blank when the 
recipient open the outline.  Yes, one can change the @files to @clean 
and re-save them all.  But that is awkward, and negates the reason for 
having them be @files in the first place.


Otherwise, one is forced to create a package file - usually a zip file 
- that contains the outline and any required external files and 
subdirectories.


Current software, such as LibreOffice or Word, handle this by saving 
their files as archives that contain all the external resources a 
document needs.  I suggest that Leo needs a similar capability.  This 
would not replace the existing Leo file format nor the existing 
outline save commands.  It would add new /Save/Open Archive/ commands.


How might this work?  For a save, Leo would check each external file 
and each @rst tree to get their paths, and then compress the external 
resources and at-files into the archive. For other resources, such as 
images in, say, an /images/ directory, An outline could have a new 
kind of node, perhaps with an /@resources/ headline, that specifies 
what subdirectories and files to include.  Perhaps there could be more 
than one /@resources/ node in the document.


To open an archive, Leo would expand the archive, which would create 
all the directories and files. Then it would load the .leo outline 
contained in the archive.

--
You received this message because you are subscribed to the Google 
Groups "leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/516d7db0-e2e9-4fdf-9197-0a10f869edfbn%40googlegroups.com 
.


--
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/bf351b2e-f8ef-4c9a-9238-4a0dad478c8c%40riseup.net.

Re: New Leo "Package" File Format?

[David Szent-Györgyi]

Now this is interesting!  I wasn't considering huge image files, most just 
"ordinary" ones like photos, screenshots, or graphs that would be common 
images to want to include with, for example, a markdown document.  I'll 
read up on your links.  Thanks!


A consideration that didn't occur to me initially: the archive file format 
might have a limit on file size. The archive file library might also have a 
limit on that. The original ZIP file format limited archive size to 4 GB; I 
read that the ZIP64 format extension 
 raises that limit 
to 16 exabytes(!), and that Windows Vista and its successors build support 
for ZIP64-sized archives into Windows Explorer/File Explorer, and that 
macOS Sierra's built-in Archive Utility does not support ZIP64. 

Limits imposed by the file system can also be a problem. NTFS has no 
issues, but FAT32 limits file size to 4GB 
; I 
reformat FAT32 thumb drives drives to exFAT format, which lifts that to 
2^64-1 bytes (!). 

I don't want to think about cross-platform issues handling metadata such as 
NTFS attributes. 

I find that Windows 7 Windows Explorer and Windows 10 File Explorer clutter 
the Windows file cache when I ask them to process Zip archives of many 
hundreds of files or multiple gigabyte Zip files; basic operations slow to 
a crawl as a result, leading me to reboot Windows to recover the 
performance lost. I don't see such performance losses when I use 7-Zip 
instead. 

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/d691f7df-ad28-47df-82dd-ec43ba61b292n%40googlegroups.com.

Re: New Leo "Package" File Format?

[Edward K. Ream]

On Sat, Apr 1, 2023 at 4:04 PM David Szent-Györgyi  wrote:

> I have seen this function referred to as "Pack and Go" in other software
> that is widely used - SolidWorks, AutoCAD for starters.


See #3238 . I'm going
to keep this simple for now. The new command will only add the .leo file
and its external files to the .zip file.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/CAMF8tS0ChekM5qXi%2BniAuCUt6_D7mWRG4w7J85WPBwR62C6zuw%40mail.gmail.com.

Re: New Leo "Package" File Format?

[Thomas Passin]

Now this is interesting!  I wasn't considering huge image files, most just 
"ordinary" ones like photos, screenshots, or graphs that would be common 
images to want to include with, for example, a markdown document.  I'll 
read up on your links.  Thanks!

On Sunday, April 2, 2023 at 5:18:24 PM UTC-4 David Szent-Györgyi wrote:

> On Sunday, April 2, 2023 at 3:55:31 PM UTC-4 tbp1...@gmail.com wrote:
>
> It seems to me that the main challenge would be for Leo to know just what 
> to have in the package.  External files would be easy, but for example 
> image files - how to know about them could be a real challenge.  I'm 
> thinking that an outline could contain an @resources node, where the user 
> could add anything that Leo didn't know about.  Not ideal, but perhaps 
> necessary.
>
>  
> If you limit the scope of your work to compression and decompression of 
> files, you might consider the libraries available for 7-Zip 
>  - support for operating ystems 
> other than Windows requires one of the variants described there. If you 
> care about handling individual images or metadata from them, your task is 
> much greater and a great challenge. 
>
> I know something of that challenge, since I earn my living supporting 
> software for life science microscopy. The number of formats used in that 
> field is enormous, the requirements that must be during acquisition are 
> distinct from those required thereafter for retrieval and analysis. 
> Acquisition can involve a great number of individual images, enough that 
> efficient writing to disk and reading back from disk can require a number 
> of individual files, with a separate file that describes the entire 
> dataset. 
>
> Not that you would necessarily wish to use the formats designed for life 
> science microscopy of the open source software available for reading and 
> writing them, but here are links that might be of interest. 
>
> OME-TIFF and OME-Big-TIFF: these support individual files with a great 
> number of images; the OME-Big-TIFF variant supports files larger than four 
> gigabytes. These, among others, are described under "OME Model and File 
> Formats ". Information 
> specific to OME-TIFF 
>  is 
> available; documentation for the OME-TIFF file structure 
> 
>  
> is available also. 
>
> Bio-Formats  is a 
> standalone Java library for reading and writing life sciences image file 
> formats. It is capable of parsing both pixels and metadata for a large 
> number of formats, as well as writing to several formats. C++ code is 
> available; I cannot speak to its condition and compliance with the current 
> standard for the format. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/bef585eb-b1bd-4b57-95dc-9c7a2884a27bn%40googlegroups.com.

Re: New Leo "Package" File Format?

[David Szent-Györgyi]

On Sunday, April 2, 2023 at 3:55:31 PM UTC-4 tbp1...@gmail.com wrote:

It seems to me that the main challenge would be for Leo to know just what 
to have in the package.  External files would be easy, but for example 
image files - how to know about them could be a real challenge.  I'm 
thinking that an outline could contain an @resources node, where the user 
could add anything that Leo didn't know about.  Not ideal, but perhaps 
necessary.

 
If you limit the scope of your work to compression and decompression of 
files, you might consider the libraries available for 7-Zip 
 - support for operating ystems other 
than Windows requires one of the variants described there. If you care 
about handling individual images or metadata from them, your task is much 
greater and a great challenge. 

I know something of that challenge, since I earn my living supporting 
software for life science microscopy. The number of formats used in that 
field is enormous, the requirements that must be during acquisition are 
distinct from those required thereafter for retrieval and analysis. 
Acquisition can involve a great number of individual images, enough that 
efficient writing to disk and reading back from disk can require a number 
of individual files, with a separate file that describes the entire 
dataset. 

Not that you would necessarily wish to use the formats designed for life 
science microscopy of the open source software available for reading and 
writing them, but here are links that might be of interest. 

OME-TIFF and OME-Big-TIFF: these support individual files with a great 
number of images; the OME-Big-TIFF variant supports files larger than four 
gigabytes. These, among others, are described under "OME Model and File 
Formats ". Information 
specific to OME-TIFF 
 is 
available; documentation for the OME-TIFF file structure 
 
is available also. 

Bio-Formats  is a standalone 
Java library for reading and writing life sciences image file formats. It 
is capable of parsing both pixels and metadata for a large number of 
formats, as well as writing to several formats. C++ code is available; I 
cannot speak to its condition and compliance with the current standard for 
the format. 

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/2df6065c-b24a-4400-af40-7b21f2b45c09n%40googlegroups.com.

Re: New Leo "Package" File Format?

[Edward K. Ream]

On Sun, Apr 2, 2023 at 2:55 PM Thomas Passin  wrote:

> That's good to know.  It seems to me that the main challenge would be for
> Leo to know just what to have in the package.  External files would be
> easy, but for example image files - how to know about them could be a real
> challenge.  I'm thinking that an outline could contain an @resources node,
> where the user could add anything that Leo didn't know about.  Not ideal,
> but perhaps necessary.
>

I'll create a new command soon. A first draft will include all @
nodes.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/CAMF8tS1m8qBTsi38brcN_atObtrtt73Qj3s%3D34ph8aKVdGK0wQ%40mail.gmail.com.

Re: New Leo "Package" File Format?

[Thomas Passin]

That's good to know.  It seems to me that the main challenge would be for 
Leo to know just what to have in the package.  External files would be 
easy, but for example image files - how to know about them could be a real 
challenge.  I'm thinking that an outline could contain an @resources node, 
where the user could add anything that Leo didn't know about.  Not ideal, 
but perhaps necessary.

On Sunday, April 2, 2023 at 3:24:02 PM UTC-4 David Szent-Györgyi wrote:

> On Saturday, April 1, 2023 at 10:06:06 PM UTC-4 mys...@gmail.com wrote:
>
> Having a zip file of all file assets would work as well with some 
> implementation of a VFS by the program utilizing it I presume.
>
>
> Given use of the appropriate codec, Zip files are easily packed or 
> unpacked. 
>
> The zlib codec  provides for superior compression, and 
> is open source; it can be built for operating systems that do not bundle 
> it. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/ebac23cf-c2b6-4dd0-a42e-043342aeab8cn%40googlegroups.com.

Re: New Leo "Package" File Format?

[David Szent-Györgyi]

On Saturday, April 1, 2023 at 10:06:06 PM UTC-4 mys...@gmail.com wrote:

Having a zip file of all file assets would work as well with some 
implementation of a VFS by the program utilizing it I presume.


Given use of the appropriate codec, Zip files are easily packed or 
unpacked. 

The zlib codec  provides for superior compression, and is 
open source; it can be built for operating systems that do not bundle it. 

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/e620c1a7-9c36-4b97-925c-2d2790397990n%40googlegroups.com.

Re: New Leo "Package" File Format?

[Mike Hodson]

On Wed, Mar 29, 2023, 11:48 Thomas Passin  wrote:

> There is a perennial problem when one wants to give a  Leo outline to
> someone else.  It happens when an outline contains external files, or
> images to display, or any other data files that might be needed.  For
> example, an article written with the Viewrendered3 plugin in mind, or for a
> Sphinx document, must have its resources available or it cannot work.




This is the main reason why Audacity decided to move to SQLite as its file
saving format in v3.0. Someone would have an AUP file and not every one of
the .au additional files to go along with it and wonder where their project
went when they tried to move it.

I assume they wanted to make it something that was simply usable in the
program as an extent API, however this does have the issue of not releasing
space from the save file until an unsaved session is completely closed.
[Truth be told I'm not a huge fan of the SQLite format myself, and would
much prefer a mysql server in almost every use case.]

It seems it took audacity ~2 years to implement this as a lasting format
change from the point it was mentioned as a pull request until it became
cemented in version three of the program.

Having a zip file of all file assets would work as well with some
implementation of a VFS by the program utilizing it I presume.



Mike

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/CAN%2B%2B4hFhqEhg_RCXrBetDm%2BD6J%3DTv880cQROF2DTLpmS26whFQ%40mail.gmail.com.

Re: New Leo "Package" File Format?

[David Szent-Györgyi]

I have seen this function referred to as "Pack and Go" in other software 
that is widely used - SolidWorks, AutoCAD for starters. 

On Wednesday, March 29, 2023 at 1:48:09 PM UTC-4 tbp1...@gmail.com wrote:

> There is a perennial problem when one wants to give a  Leo outline to 
> someone else.  It happens when an outline contains external files, or 
> images to display, or any other data files that might be needed.  For 
> example, an article written with the Viewrendered3 plugin in mind, or for a 
> Sphinx document, must have its resources available or it cannot work.
>
> If the outline contains @file trees and these external files aren't 
> included in, e.g., a zip file, those files will be blank when the recipient 
> open the outline.  Yes, one can change the @files to @clean and re-save 
> them all.  But that is awkward, and negates the reason for having them be 
> @files in the first place.  
>
> Otherwise, one is forced to create a package file - usually a zip file - 
> that contains the outline and any required external files and 
> subdirectories.
>
> Current software, such as LibreOffice or Word, handle this by saving their 
> files as archives that contain all the external resources a document 
> needs.  I suggest that Leo needs a similar capability.  This would not 
> replace the existing Leo file format nor the existing outline save 
> commands.  It would add new *Save/Open Archive* commands.
>
> How might this work?  For a save, Leo would check each external file and 
> each @rst tree to get their paths, and then compress the external resources 
> and at-files into the archive.  For other resources, such as images in, 
> say, an *images* directory, An outline could have a new kind of node, 
> perhaps with an *@resources* headline, that specifies what subdirectories 
> and files to include.  Perhaps there could be more than one *@resources* 
> node in the document.
>
> To open an archive, Leo would expand the archive, which would create all 
> the directories and files. Then it would load the .leo outline contained in 
> the archive.
>

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/a30f5050-562b-43f1-8bba-88077223d8acn%40googlegroups.com.

New Leo "Package" File Format?

[Thomas Passin]

There is a perennial problem when one wants to give a  Leo outline to 
someone else.  It happens when an outline contains external files, or 
images to display, or any other data files that might be needed.  For 
example, an article written with the Viewrendered3 plugin in mind, or for a 
Sphinx document, must have its resources available or it cannot work.

If the outline contains @file trees and these external files aren't 
included in, e.g., a zip file, those files will be blank when the recipient 
open the outline.  Yes, one can change the @files to @clean and re-save 
them all.  But that is awkward, and negates the reason for having them be 
@files in the first place.  

Otherwise, one is forced to create a package file - usually a zip file - 
that contains the outline and any required external files and 
subdirectories.

Current software, such as LibreOffice or Word, handle this by saving their 
files as archives that contain all the external resources a document 
needs.  I suggest that Leo needs a similar capability.  This would not 
replace the existing Leo file format nor the existing outline save 
commands.  It would add new *Save/Open Archive* commands.

How might this work?  For a save, Leo would check each external file and 
each @rst tree to get their paths, and then compress the external resources 
and at-files into the archive.  For other resources, such as images in, 
say, an *images* directory, An outline could have a new kind of node, 
perhaps with an *@resources* headline, that specifies what subdirectories 
and files to include.  Perhaps there could be more than one *@resources* 
node in the document.

To open an archive, Leo would expand the archive, which would create all 
the directories and files. Then it would load the .leo outline contained in 
the archive.

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/516d7db0-e2e9-4fdf-9197-0a10f869edfbn%40googlegroups.com.