[julia-users] Re: [ANN] JuliaIO and FileIO

[MA Laforge]

Thanks for that Simon.

I think I did see the query() function...  But it did not register until 
now.

Indeed, I would prefer hierarchy in my file types... but experience tells 
me that's not trivial.  Sometimes there are more than one useful 
hierarchies, so building a useful one can be difficult (I gave it my best 
shot for now).

The "Union" solution:
I am sort of reluctant to use unions (I don't quite find it as elegant as 
the inheritance system).  Also, there is a change in Union syntax from 
v0.3->v0.4.  But, since Julia's current release is finally at v0.4, I 
should really migrate ASAP.

That being said, I still like your suggestion:  The good thing about your 
Union solution is that it allows the *user* to select which filetypes are 
grouped, and how.  I must give this serious thought.

As for the dynamic loading:  I can't really comment yet.  I will have to 
get used to that system a bit more before I can have a real opinion.

Regards,
MA

[julia-users] Re: [ANN] JuliaIO and FileIO

[Simon Danisch]

So your biggest concern doesn't seem to be a problem:
You can just do query("file.ext") and get back the format without loading the 
file. I see that in the way you want to use it, a hierarchy would be a nice 
addition.
You could just do this:
myload(path) = myload(query(path))
myload{T <: VectorGraphic}(::File{T}) = ...
Right now, VectorGraphic would need to be a union of the formats, though.

We do dynamically load libraries to completly decouple the loading from the IO 
library, so that a new user doesn't need to know what the best IO library is 
for his format. Since he doesn't know, FileIO must know.

When you have an IO library already loaded, FileIO might use it. So far it 
depends on the priority of that library, if FileIO loads another one. Maybe I 
should change that, so that FileIO determines first if any of the IO candidates 
is already loaded and if so, it just uses that library.

[julia-users] Re: [ANN] JuliaIO and FileIO

[MA Laforge]

Hi Simon,

Thanks for the comments.

>In the end, I would rather like to have the hierarchy in the Julia objects 
that get returned by load (e.g. AbstractString <: UTF8String etc)

Hmm... Did not think of this... I guess because an encoding is not 
technically a string, but maybe that does not matter...

>- There are many IO libraries which should get dynamically loaded when the 
format is detected (first argument for registry)

I don't think I really understand this part... I usually like to explicitly 
pick which libraries I want my program to use...  But it looks like FileIO 
auto-loads libraries.  I'm not saying this is necessarily a bad thing... 
I'm just not used to thinking that way.

>- we utilize magic bytes to recognize file formats (second argument for 
registry)

Indeed that part is great!  I just don't need it at the moment - because 
with the code I am writing (right know) I already know the file type.

That said: I *know* I can make use of the magic byte detection in the 
future.  Thanks for that!

>- there are multiple IO libraries for the same formats and we need to 
choose one dynamically.

Maybe that's one of the things I don't get fully... I would expect that if 
you have two PNG reader libraries:

using MyPNGLib
or
using ThisGuysEvenBetterPNGLib


Then, whichever library was "using"d should have registered itself with 
FileIO, and would then be used when the user calls:
img = load(File{format"PNG"}("thisimage.png"))

>>but something seems a little off about how I use/generate objects
>I'm not really sure what you mean by that.

Indeed... I'm not sure either... I'm trying to figure that out:

At the moment, one thing I can think of are issues I expect when building 
an image editor program:

module MyImageEditor #Think of something like GIMP

#This would be a high-level open function that does everything necessary to
#"open" the file to a new canvas:
function open{T<:PixelImageFormat}(file::File{T})
data = FileIO.load(file)
#Create new canvas
#Populate canvas with data
#Show canvas
end

function open{T<:VectorImageFormat}(file::File{T})
#Ask user what resolution they want for the rendered bitmap
data = FileIO.load(file)
#Create new canvas
#Render vector image to canvas
#Populate canvas with data
#Show canvas
end

#For any other file:
function open{T}(file::File{T})
throw("Cannot open this kind of file... not a (recognized) image")
end

#When the user clicks "ok":
function imgopendlg_ok(dlg::ImgOpenDlg)
#I assume FileIO could have an "autodetect" function that tries to
#determine the most likely file type from the extension, then confirm
#everything is ok using magic byes
file = FileIO.autodetect(dlg.filepath)
#Now file is a File object with type information.

try
open(file)
catch e
display_message(e)
end
end

end #module

As far as I can tell: If I were to try to open a large movie instead of a 
pixel image, I expect the following issues:

1) The current implementation of FileIO will try to auto-load the movie 
libraries I don't want to include with my program (because the code to 
register file readers is actually *inside* the FileIO module?? - I think?)

2) I would have to wait until my program has loaded the movie file before I 
can detect it was a movie file, and *then* I can finally catch it as an 
exception:

function open(file::File) #In MyImageEditor
data = FileIO.load(file)
if !isimage(data)
throw("Cannot open this kind of file... not a (recognized) image")
end
#Create new canvas
#Populate canvas with data
#Show canvas
end

isimage(data::PNGData) = true
isimage(data::BMPData) = true
...
isimage(unknowndata) = false

...Or something similar.

To be fair: I am almost certain the the biggest reason for my unease with 
FileIO is that I have not seen enough examples on how to use the module 
properly.

Can you point me to good example projects/other modules that use FileIO as 
intended?  I am only getting a vague feel for its use by reading the 
included tests.

Otherwise: I guess I will wait until more examples are available - or keep 
trying to play around with FileIO's capabilities when I have time.

Best Regards,

MA

[julia-users] Re: [ANN] JuliaIO and FileIO

[Simon Danisch]

@MA Laforge

This looks a little bit like our first prototypes, with the addition of an 
hierarchical format system.
I'm not sure if we really need a hierarchy. It seems hard to program 
against some abstract type in this area, since you get so many edge cases.
It might be useful in some cases, but at least for now, FileIO works pretty 
well without it.
In the end, I would rather like to have the hierarchy in the Julia objects 
that get returned by load (e.g. AbstractString <: UTF8String etc)

Please consider the following points, that have lead to the current design:
- There are many IO libraries which should get dynamically loaded when the 
format is detected (first argument for registry)
- we utilize magic bytes to recognize file formats (second argument for 
registry)
- there are multiple IO libraries for the same formats and we need to 
choose one dynamically
- we use dispatch in the same way, but just inside one module, to better 
resolve ambiguities

>but something seems a little off about how I use/generate objects

I'm not really sure what you mean by that.

Best,
Simon

Am Samstag, 4. April 2015 17:41:14 UTC+2 schrieb Simon Danisch:
>
> Hi there,
>
> FileIO has the aim to make it very easy to read any arbitrary file.
> I hastily copied together a proof of concept by taking code from Images.jl.
>
> JuliaIO is the umbrella group, which takes IO packages with no home. If 
> someone wrote an IO package, but doesn't have time to implement the FileIO 
> interface, giving it to JuliaIO might be a good idea in order to keep the 
> package usable.
>
> Concept of FileIO is described in the readme:
>
> Meta package for FileIO. Purpose is to open a file and return the 
> respective Julia object, without doing any research on how to open the file.
>
> f = file"test.jpg" # -> File{:jpg}read(f) # -> Imageread(file"test.obj") # -> 
> Meshread(file"test.csv") # -> DataFrame
>
> So far only Images are supported and MeshIO is on the horizon.
>
> It is structured the following way: There are three levels of abstraction, 
> first FileIO, defining the file_str macro etc, then a meta package for a 
> certain class of file, e.g. Images or Meshes. This meta package defines the 
> Julia datatype (e.g. Mesh, Image) and organizes the importer libraries. 
> This is also a good place to define IO library independant tests for 
> different file formats. Then on the last level, there are the low-level 
> importer libraries, which do the actual IO. They're included via Mike Innes 
> Requires  package, so 
> that it doesn't introduce extra load time if not needed. This way, using 
> FileIO without reading/writing anything should have short load times.
>
> As an implementation example please look at FileIO -> ImageIO -> 
> ImageMagick. This should already work as a proof of concept. Try:
>
> using FileIO # should be very fast, thanks to Mike Innes Requires 
> packageread(file"test.jpg") # takes a little longer as it needs to load the 
> IO libraryread(file"test.jpg") # should be fastread(File("documents", 
> "images", "myimage.jpg") # automatic joinpath via File constructor
>
> Please open issues if things are not clear or if you find flaws in the 
> concept/implementation.
>
> If you're interested in working on this infrastructure I'll be pleased to 
> add you to the group JuliaIO.
>
>
> Best,
>
> Simon
>

Re: [julia-users] Re: [ANN] JuliaIO and FileIO

[MA Laforge]

Hi all,

I myself have met a similar need for a file object.  That way, I can use 
the dispatch engine to overload the open function:
f = File(format"test.jpg")
s = open(f)

instead of what I consider to be less attractive solutions:
f = "test.jpg"
s = open_jpeg(f)
#or
s = MyModule.open(f) #My module's JPG reader

I like the idea of the FileIO module, but I am less a fan of having to 
register new filetypes with the module.  ...Yet I must admit there is 
something nice about how FileIO appears to automate type creation (Sorry: I 
do not fully understand the module yet).

I played a bit with FileIO, but something seems a little off about how I 
use/generate objects (I'm not sure why that is, though).

Anyways, I tried getting similar functionality (attempts at solving my own 
problems dispatching read/open on different file types) by relying more 
heavily on the Julia type system:
https://github.com/ma-laforge/FileIO2.jl

Comments:

   - FileIO2 does not have as many bells & whistles as FileIO - but I think 
   it has potential for that.
   - FileIO2 even has facilities to dispatch on different file encodings 
   (ex: binary, UTF8, ASCII, ...) if one would desire such a thing (Though I 
   prefer not to do so, in most cases).

Sample Code:
#Generate a reference to a text file:
file = File{TextFmt}("generictextfile.txt")

#Easily display the entire contents of an ASCII file:
typealias ASCIIFile File{ASCIIFmt} #For convenience
print(read(ASCIIFile(@__FILE__)))

...So I figured I would just put this out there in case the FileIO group 
finds something of value in the FileIO2 solution.

MA

Re: [julia-users] Re: [ANN] JuliaIO and FileIO

[Tim Holy]

No criticism intended, I just wanted to save you (or anyone else) time :-).

I agree that it's also a great opportunity to refactor, and (without taking 
the time to think about it too deeply) your suggestions seem very reasonable.

Best,
--Tim

On Wednesday, April 08, 2015 07:23:40 AM Simon Danisch wrote:
> Excuse my ignorance Tim! I just don't allow myself to waste too much time
> on this, so that I'm not looking into things very deeply.
> This indeed solves the issue in a slightly different way.
> I think we can definitely reuse more of what Images is currently doing.
> It makes a lot of sense to switch out symbols in favor of types to better
> model hierarchies.
> Maybe this should come with a mime type overhaul, as they're closely
> related.
> 
> Am Samstag, 4. April 2015 17:41:14 UTC+2 schrieb Simon Danisch:
> > Hi there,
> > 
> > FileIO has the aim to make it very easy to read any arbitrary file.
> > I hastily copied together a proof of concept by taking code from
> > Images.jl.
> > 
> > JuliaIO is the umbrella group, which takes IO packages with no home. If
> > someone wrote an IO package, but doesn't have time to implement the FileIO
> > interface, giving it to JuliaIO might be a good idea in order to keep the
> > package usable.
> > 
> > Concept of FileIO is described in the readme:
> > 
> > Meta package for FileIO. Purpose is to open a file and return the
> > respective Julia object, without doing any research on how to open the
> > file.
> > 
> > f = file"test.jpg" # -> File{:jpg}read(f) # -> Imageread(file"test.obj") #
> > -> Meshread(file"test.csv") # -> DataFrame
> > 
> > So far only Images are supported and MeshIO is on the horizon.
> > 
> > It is structured the following way: There are three levels of abstraction,
> > first FileIO, defining the file_str macro etc, then a meta package for a
> > certain class of file, e.g. Images or Meshes. This meta package defines
> > the
> > Julia datatype (e.g. Mesh, Image) and organizes the importer libraries.
> > This is also a good place to define IO library independant tests for
> > different file formats. Then on the last level, there are the low-level
> > importer libraries, which do the actual IO. They're included via Mike
> > Innes
> > Requires  package, so
> > that it doesn't introduce extra load time if not needed. This way, using
> > FileIO without reading/writing anything should have short load times.
> > 
> > As an implementation example please look at FileIO -> ImageIO ->
> > ImageMagick. This should already work as a proof of concept. Try:
> > 
> > using FileIO # should be very fast, thanks to Mike Innes Requires
> > packageread(file"test.jpg") # takes a little longer as it needs to load
> > the IO libraryread(file"test.jpg") # should be fastread(File("documents",
> > "images", "myimage.jpg") # automatic joinpath via File constructor
> > 
> > Please open issues if things are not clear or if you find flaws in the
> > concept/implementation.
> > 
> > If you're interested in working on this infrastructure I'll be pleased to
> > add you to the group JuliaIO.
> > 
> > 
> > Best,
> > 
> > Simon

[julia-users] Re: [ANN] JuliaIO and FileIO

[Simon Danisch]

Excuse my ignorance Tim! I just don't allow myself to waste too much time 
on this, so that I'm not looking into things very deeply.
This indeed solves the issue in a slightly different way.
I think we can definitely reuse more of what Images is currently doing.
It makes a lot of sense to switch out symbols in favor of types to better 
model hierarchies.
Maybe this should come with a mime type overhaul, as they're closely 
related. 


Am Samstag, 4. April 2015 17:41:14 UTC+2 schrieb Simon Danisch:
>
> Hi there,
>
> FileIO has the aim to make it very easy to read any arbitrary file.
> I hastily copied together a proof of concept by taking code from Images.jl.
>
> JuliaIO is the umbrella group, which takes IO packages with no home. If 
> someone wrote an IO package, but doesn't have time to implement the FileIO 
> interface, giving it to JuliaIO might be a good idea in order to keep the 
> package usable.
>
> Concept of FileIO is described in the readme:
>
> Meta package for FileIO. Purpose is to open a file and return the 
> respective Julia object, without doing any research on how to open the file.
>
> f = file"test.jpg" # -> File{:jpg}read(f) # -> Imageread(file"test.obj") # -> 
> Meshread(file"test.csv") # -> DataFrame
>
> So far only Images are supported and MeshIO is on the horizon.
>
> It is structured the following way: There are three levels of abstraction, 
> first FileIO, defining the file_str macro etc, then a meta package for a 
> certain class of file, e.g. Images or Meshes. This meta package defines the 
> Julia datatype (e.g. Mesh, Image) and organizes the importer libraries. 
> This is also a good place to define IO library independant tests for 
> different file formats. Then on the last level, there are the low-level 
> importer libraries, which do the actual IO. They're included via Mike Innes 
> Requires  package, so 
> that it doesn't introduce extra load time if not needed. This way, using 
> FileIO without reading/writing anything should have short load times.
>
> As an implementation example please look at FileIO -> ImageIO -> 
> ImageMagick. This should already work as a proof of concept. Try:
>
> using FileIO # should be very fast, thanks to Mike Innes Requires 
> packageread(file"test.jpg") # takes a little longer as it needs to load the 
> IO libraryread(file"test.jpg") # should be fastread(File("documents", 
> "images", "myimage.jpg") # automatic joinpath via File constructor
>
> Please open issues if things are not clear or if you find flaws in the 
> concept/implementation.
>
> If you're interested in working on this infrastructure I'll be pleased to 
> add you to the group JuliaIO.
>
>
> Best,
>
> Simon
>

Re: [julia-users] Re: [ANN] JuliaIO and FileIO

[Tim Holy]

Images has (I think) basically already solved this problem. The way it uses 
magic numbers is described at
https://github.com/timholy/Images.jl/blob/master/doc/extendingIO.md

For the example (a SIF) file, the magic bytes are enclosed in that
b"Andor Technology Multi-Channel File"
string.

--Tim

On Wednesday, April 08, 2015 05:58:11 AM Simon Danisch wrote:
> This is half true.
> Magic numbers are only used in ImageMagick.jl per "accident", as that's how
> the underlying C-library does things.
> To be honest, I haven't looked at the details yet myself, as I mostly copy
> and pasted code to get a first prototype running.
> I sketch out in FileIO.jl#3 
> how I plan on using magic numbers.
> If I find some time, I will make a schematic of some more advanced features
> to handle ambiguities and different endings which map to the same type.
> This should also give a better overview of the architecture and what a
> package creator must do to get integrated into FileIO.
> 
> Best,
> Simon
> 
> Am Samstag, 4. April 2015 17:41:14 UTC+2 schrieb Simon Danisch:
> > Hi there,
> > 
> > FileIO has the aim to make it very easy to read any arbitrary file.
> > I hastily copied together a proof of concept by taking code from
> > Images.jl.
> > 
> > JuliaIO is the umbrella group, which takes IO packages with no home. If
> > someone wrote an IO package, but doesn't have time to implement the FileIO
> > interface, giving it to JuliaIO might be a good idea in order to keep the
> > package usable.
> > 
> > Concept of FileIO is described in the readme:
> > 
> > Meta package for FileIO. Purpose is to open a file and return the
> > respective Julia object, without doing any research on how to open the
> > file.
> > 
> > f = file"test.jpg" # -> File{:jpg}read(f) # -> Imageread(file"test.obj") #
> > -> Meshread(file"test.csv") # -> DataFrame
> > 
> > So far only Images are supported and MeshIO is on the horizon.
> > 
> > It is structured the following way: There are three levels of abstraction,
> > first FileIO, defining the file_str macro etc, then a meta package for a
> > certain class of file, e.g. Images or Meshes. This meta package defines
> > the
> > Julia datatype (e.g. Mesh, Image) and organizes the importer libraries.
> > This is also a good place to define IO library independant tests for
> > different file formats. Then on the last level, there are the low-level
> > importer libraries, which do the actual IO. They're included via Mike
> > Innes
> > Requires  package, so
> > that it doesn't introduce extra load time if not needed. This way, using
> > FileIO without reading/writing anything should have short load times.
> > 
> > As an implementation example please look at FileIO -> ImageIO ->
> > ImageMagick. This should already work as a proof of concept. Try:
> > 
> > using FileIO # should be very fast, thanks to Mike Innes Requires
> > packageread(file"test.jpg") # takes a little longer as it needs to load
> > the IO libraryread(file"test.jpg") # should be fastread(File("documents",
> > "images", "myimage.jpg") # automatic joinpath via File constructor
> > 
> > Please open issues if things are not clear or if you find flaws in the
> > concept/implementation.
> > 
> > If you're interested in working on this infrastructure I'll be pleased to
> > add you to the group JuliaIO.
> > 
> > 
> > Best,
> > 
> > Simon

[julia-users] Re: [ANN] JuliaIO and FileIO

[Simon Danisch]

This is half true.
Magic numbers are only used in ImageMagick.jl per "accident", as that's how 
the underlying C-library does things.
To be honest, I haven't looked at the details yet myself, as I mostly copy 
and pasted code to get a first prototype running.
I sketch out in FileIO.jl#3  
how 
I plan on using magic numbers.
If I find some time, I will make a schematic of some more advanced features 
to handle ambiguities and different endings which map to the same type.
This should also give a better overview of the architecture and what a 
package creator must do to get integrated into FileIO.

Best,
Simon


Am Samstag, 4. April 2015 17:41:14 UTC+2 schrieb Simon Danisch:
>
> Hi there,
>
> FileIO has the aim to make it very easy to read any arbitrary file.
> I hastily copied together a proof of concept by taking code from Images.jl.
>
> JuliaIO is the umbrella group, which takes IO packages with no home. If 
> someone wrote an IO package, but doesn't have time to implement the FileIO 
> interface, giving it to JuliaIO might be a good idea in order to keep the 
> package usable.
>
> Concept of FileIO is described in the readme:
>
> Meta package for FileIO. Purpose is to open a file and return the 
> respective Julia object, without doing any research on how to open the file.
>
> f = file"test.jpg" # -> File{:jpg}read(f) # -> Imageread(file"test.obj") # -> 
> Meshread(file"test.csv") # -> DataFrame
>
> So far only Images are supported and MeshIO is on the horizon.
>
> It is structured the following way: There are three levels of abstraction, 
> first FileIO, defining the file_str macro etc, then a meta package for a 
> certain class of file, e.g. Images or Meshes. This meta package defines the 
> Julia datatype (e.g. Mesh, Image) and organizes the importer libraries. 
> This is also a good place to define IO library independant tests for 
> different file formats. Then on the last level, there are the low-level 
> importer libraries, which do the actual IO. They're included via Mike Innes 
> Requires  package, so 
> that it doesn't introduce extra load time if not needed. This way, using 
> FileIO without reading/writing anything should have short load times.
>
> As an implementation example please look at FileIO -> ImageIO -> 
> ImageMagick. This should already work as a proof of concept. Try:
>
> using FileIO # should be very fast, thanks to Mike Innes Requires 
> packageread(file"test.jpg") # takes a little longer as it needs to load the 
> IO libraryread(file"test.jpg") # should be fastread(File("documents", 
> "images", "myimage.jpg") # automatic joinpath via File constructor
>
> Please open issues if things are not clear or if you find flaws in the 
> concept/implementation.
>
> If you're interested in working on this infrastructure I'll be pleased to 
> add you to the group JuliaIO.
>
>
> Best,
>
> Simon
>

Re: [julia-users] Re: [ANN] JuliaIO and FileIO

[Tim Holy]

I haven't yet had time to look at FileIO, but if the code has been lifted from 
Images, then it uses the magic bytes in preference to the file extension (when 
the magic bytes are recognized).

Best,
--Tim

On Monday, April 06, 2015 06:30:40 AM Sebastian Good wrote:
> Very interesting project. I've worked a little on problems like this in my
> projects and have found that filename extensions aren't always enough to
> distinguish filetypes. It's not uncommon to have to scan the first few
> (hundred) bytes looking for distinctive patterns, magic cookies, etc. to
> determine just what type a file is. Do you anticipate having a registry of
> canonical file types, e.g. related to MIME types if available?
> 
> On Saturday, April 4, 2015 at 11:41:14 AM UTC-4, Simon Danisch wrote:
> > Hi there,
> > 
> > FileIO has the aim to make it very easy to read any arbitrary file.
> > I hastily copied together a proof of concept by taking code from
> > Images.jl.
> > 
> > JuliaIO is the umbrella group, which takes IO packages with no home. If
> > someone wrote an IO package, but doesn't have time to implement the FileIO
> > interface, giving it to JuliaIO might be a good idea in order to keep the
> > package usable.
> > 
> > Concept of FileIO is described in the readme:
> > 
> > Meta package for FileIO. Purpose is to open a file and return the
> > respective Julia object, without doing any research on how to open the
> > file.
> > 
> > f = file"test.jpg" # -> File{:jpg}read(f) # -> Imageread(file"test.obj") #
> > -> Meshread(file"test.csv") # -> DataFrame
> > 
> > So far only Images are supported and MeshIO is on the horizon.
> > 
> > It is structured the following way: There are three levels of abstraction,
> > first FileIO, defining the file_str macro etc, then a meta package for a
> > certain class of file, e.g. Images or Meshes. This meta package defines
> > the
> > Julia datatype (e.g. Mesh, Image) and organizes the importer libraries.
> > This is also a good place to define IO library independant tests for
> > different file formats. Then on the last level, there are the low-level
> > importer libraries, which do the actual IO. They're included via Mike
> > Innes
> > Requires  package, so
> > that it doesn't introduce extra load time if not needed. This way, using
> > FileIO without reading/writing anything should have short load times.
> > 
> > As an implementation example please look at FileIO -> ImageIO ->
> > ImageMagick. This should already work as a proof of concept. Try:
> > 
> > using FileIO # should be very fast, thanks to Mike Innes Requires
> > packageread(file"test.jpg") # takes a little longer as it needs to load
> > the IO libraryread(file"test.jpg") # should be fastread(File("documents",
> > "images", "myimage.jpg") # automatic joinpath via File constructor
> > 
> > Please open issues if things are not clear or if you find flaws in the
> > concept/implementation.
> > 
> > If you're interested in working on this infrastructure I'll be pleased to
> > add you to the group JuliaIO.
> > 
> > 
> > Best,
> > 
> > Simon

[julia-users] Re: [ANN] JuliaIO and FileIO

[Sebastian Good]

Very interesting project. I've worked a little on problems like this in my 
projects and have found that filename extensions aren't always enough to 
distinguish filetypes. It's not uncommon to have to scan the first few 
(hundred) bytes looking for distinctive patterns, magic cookies, etc. to 
determine just what type a file is. Do you anticipate having a registry of 
canonical file types, e.g. related to MIME types if available?

On Saturday, April 4, 2015 at 11:41:14 AM UTC-4, Simon Danisch wrote:
>
> Hi there,
>
> FileIO has the aim to make it very easy to read any arbitrary file.
> I hastily copied together a proof of concept by taking code from Images.jl.
>
> JuliaIO is the umbrella group, which takes IO packages with no home. If 
> someone wrote an IO package, but doesn't have time to implement the FileIO 
> interface, giving it to JuliaIO might be a good idea in order to keep the 
> package usable.
>
> Concept of FileIO is described in the readme:
>
> Meta package for FileIO. Purpose is to open a file and return the 
> respective Julia object, without doing any research on how to open the file.
>
> f = file"test.jpg" # -> File{:jpg}read(f) # -> Imageread(file"test.obj") # -> 
> Meshread(file"test.csv") # -> DataFrame
>
> So far only Images are supported and MeshIO is on the horizon.
>
> It is structured the following way: There are three levels of abstraction, 
> first FileIO, defining the file_str macro etc, then a meta package for a 
> certain class of file, e.g. Images or Meshes. This meta package defines the 
> Julia datatype (e.g. Mesh, Image) and organizes the importer libraries. 
> This is also a good place to define IO library independant tests for 
> different file formats. Then on the last level, there are the low-level 
> importer libraries, which do the actual IO. They're included via Mike Innes 
> Requires  package, so 
> that it doesn't introduce extra load time if not needed. This way, using 
> FileIO without reading/writing anything should have short load times.
>
> As an implementation example please look at FileIO -> ImageIO -> 
> ImageMagick. This should already work as a proof of concept. Try:
>
> using FileIO # should be very fast, thanks to Mike Innes Requires 
> packageread(file"test.jpg") # takes a little longer as it needs to load the 
> IO libraryread(file"test.jpg") # should be fastread(File("documents", 
> "images", "myimage.jpg") # automatic joinpath via File constructor
>
> Please open issues if things are not clear or if you find flaws in the 
> concept/implementation.
>
> If you're interested in working on this infrastructure I'll be pleased to 
> add you to the group JuliaIO.
>
>
> Best,
>
> Simon
>

[julia-users] Re: [ANN] JuliaIO and FileIO

[Simon Danisch]

Oh brilliant, this is looks very similar :D 
I should follow Julia's issues more closely...
Looking forward to your feedback! There are still quite some problems to 
figure out!

Am Samstag, 4. April 2015 17:41:14 UTC+2 schrieb Simon Danisch:
>
> Hi there,
>
> FileIO has the aim to make it very easy to read any arbitrary file.
> I hastily copied together a proof of concept by taking code from Images.jl.
>
> JuliaIO is the umbrella group, which takes IO packages with no home. If 
> someone wrote an IO package, but doesn't have time to implement the FileIO 
> interface, giving it to JuliaIO might be a good idea in order to keep the 
> package usable.
>
> Concept of FileIO is described in the readme:
>
> Meta package for FileIO. Purpose is to open a file and return the 
> respective Julia object, without doing any research on how to open the file.
>
> f = file"test.jpg" # -> File{:jpg}read(f) # -> Imageread(file"test.obj") # -> 
> Meshread(file"test.csv") # -> DataFrame
>
> So far only Images are supported and MeshIO is on the horizon.
>
> It is structured the following way: There are three levels of abstraction, 
> first FileIO, defining the file_str macro etc, then a meta package for a 
> certain class of file, e.g. Images or Meshes. This meta package defines the 
> Julia datatype (e.g. Mesh, Image) and organizes the importer libraries. 
> This is also a good place to define IO library independant tests for 
> different file formats. Then on the last level, there are the low-level 
> importer libraries, which do the actual IO. They're included via Mike Innes 
> Requires  package, so 
> that it doesn't introduce extra load time if not needed. This way, using 
> FileIO without reading/writing anything should have short load times.
>
> As an implementation example please look at FileIO -> ImageIO -> 
> ImageMagick. This should already work as a proof of concept. Try:
>
> using FileIO # should be very fast, thanks to Mike Innes Requires 
> packageread(file"test.jpg") # takes a little longer as it needs to load the 
> IO libraryread(file"test.jpg") # should be fastread(File("documents", 
> "images", "myimage.jpg") # automatic joinpath via File constructor
>
> Please open issues if things are not clear or if you find flaws in the 
> concept/implementation.
>
> If you're interested in working on this infrastructure I'll be pleased to 
> add you to the group JuliaIO.
>
>
> Best,
>
> Simon
>

[julia-users] Re: [ANN] JuliaIO and FileIO

[Paulo Castro]

That' s a very nice idea. Having a common way to load files with different 
backends is very neat and very useful. Even the idea of having a file_str 
macro is a very Julian way to do thing, I believe.

Maybe FastaIO could benefit from this model (and also other parsers for 
biological data). We should contact BioJulia and FastaIO guys to see what 
can be done.

Em sábado, 4 de abril de 2015 12:41:14 UTC-3, Simon Danisch escreveu:
>
> Hi there,
>
> FileIO has the aim to make it very easy to read any arbitrary file.
> I hastily copied together a proof of concept by taking code from Images.jl.
>
> JuliaIO is the umbrella group, which takes IO packages with no home. If 
> someone wrote an IO package, but doesn't have time to implement the FileIO 
> interface, giving it to JuliaIO might be a good idea in order to keep the 
> package usable.
>
> Concept of FileIO is described in the readme:
>
> Meta package for FileIO. Purpose is to open a file and return the 
> respective Julia object, without doing any research on how to open the file.
>
> f = file"test.jpg" # -> File{:jpg}read(f) # -> Imageread(file"test.obj") # -> 
> Meshread(file"test.csv") # -> DataFrame
>
> So far only Images are supported and MeshIO is on the horizon.
>
> It is structured the following way: There are three levels of abstraction, 
> first FileIO, defining the file_str macro etc, then a meta package for a 
> certain class of file, e.g. Images or Meshes. This meta package defines the 
> Julia datatype (e.g. Mesh, Image) and organizes the importer libraries. 
> This is also a good place to define IO library independant tests for 
> different file formats. Then on the last level, there are the low-level 
> importer libraries, which do the actual IO. They're included via Mike Innes 
> Requires  package, so 
> that it doesn't introduce extra load time if not needed. This way, using 
> FileIO without reading/writing anything should have short load times.
>
> As an implementation example please look at FileIO -> ImageIO -> 
> ImageMagick. This should already work as a proof of concept. Try:
>
> using FileIO # should be very fast, thanks to Mike Innes Requires 
> packageread(file"test.jpg") # takes a little longer as it needs to load the 
> IO libraryread(file"test.jpg") # should be fastread(File("documents", 
> "images", "myimage.jpg") # automatic joinpath via File constructor
>
> Please open issues if things are not clear or if you find flaws in the 
> concept/implementation.
>
> If you're interested in working on this infrastructure I'll be pleased to 
> add you to the group JuliaIO.
>
>
> Best,
>
> Simon
>