[julia-users] HDF5 file id biger then txt. What wrong?
I have data in txt file, some milons like this:
0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0
0,0,0,0,0,0,0,2,0,0,0,2,0,0,0,0,1
0,0,0,0,1,0,0,0,1,0,0,0,0,0,0,0,1
Coding win1250.
size of dane.txt is 1.3 GB
D=readcsv("dane.txt")
k,l=size(D)
using HDF5, JLD
hfi=h5open("D.h5","w")
close(hfi)
fid = h5open("D.h5","r+")
g = fid["/"]
dset1 = d_create(g, "/D", datatype(Int64), dataspace(k,l))
dset1[:,:]=D
close(fid)
After save to h5 file the file has 6.3 GB ? Why new file is 4 times biger?
Paul
Re: [julia-users] HDF5 file id biger then txt. What wrong?
In your example data, each value is represented with two bytes: one for the
value, one for a comma or newline. Each Int64 value is 8 bytes. If all your
values are between 0 and 255, you could use UInt8 to represent them and cut
the size in half.
On Tue, Jul 21, 2015 at 1:16 PM, paul analyst wrote:
> I have data in txt file, some milons like this:
> 0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0
> 0,0,0,0,0,0,0,2,0,0,0,2,0,0,0,0,1
> 0,0,0,0,1,0,0,0,1,0,0,0,0,0,0,0,1
>
> Coding win1250.
>
> size of dane.txt is 1.3 GB
>
> D=readcsv("dane.txt")
> k,l=size(D)
>
> using HDF5, JLD
> hfi=h5open("D.h5","w")
> close(hfi)
>
> fid = h5open("D.h5","r+")
> g = fid["/"]
> dset1 = d_create(g, "/D", datatype(Int64), dataspace(k,l))
> dset1[:,:]=D
> close(fid)
>
> After save to h5 file the file has 6.3 GB ? Why new file is 4 times biger?
> Paul
>
Re: [julia-users] HDF5 file id biger then txt. What wrong?
HDF5 file support compression. This is enabled via a flag when writing the
file; when reading, it is automatically decompressed. I assume that
compression would greatly reduce the file size.
-erik
On Tue, Jul 21, 2015 at 1:21 PM, Stefan Karpinski
wrote:
> In your example data, each value is represented with two bytes: one for
> the value, one for a comma or newline. Each Int64 value is 8 bytes. If all
> your values are between 0 and 255, you could use UInt8 to represent them
> and cut the size in half.
>
> On Tue, Jul 21, 2015 at 1:16 PM, paul analyst
> wrote:
>
>> I have data in txt file, some milons like this:
>> 0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0
>> 0,0,0,0,0,0,0,2,0,0,0,2,0,0,0,0,1
>> 0,0,0,0,1,0,0,0,1,0,0,0,0,0,0,0,1
>>
>> Coding win1250.
>>
>> size of dane.txt is 1.3 GB
>>
>> D=readcsv("dane.txt")
>> k,l=size(D)
>>
>> using HDF5, JLD
>> hfi=h5open("D.h5","w")
>> close(hfi)
>>
>> fid = h5open("D.h5","r+")
>> g = fid["/"]
>> dset1 = d_create(g, "/D", datatype(Int64), dataspace(k,l))
>> dset1[:,:]=D
>> close(fid)
>>
>> After save to h5 file the file has 6.3 GB ? Why new file is 4 times biger?
>> Paul
>>
>
>
--
Erik Schnetter
http://www.perimeterinstitute.ca/personal/eschnetter/
Re: [julia-users] HDF5 file id biger then txt. What wrong?
Yes, that could be even more effective.
On Tue, Jul 21, 2015 at 2:09 PM, Erik Schnetter wrote:
> HDF5 file support compression. This is enabled via a flag when writing the
> file; when reading, it is automatically decompressed. I assume that
> compression would greatly reduce the file size.
>
> -erik
>
> On Tue, Jul 21, 2015 at 1:21 PM, Stefan Karpinski
> wrote:
>
>> In your example data, each value is represented with two bytes: one for
>> the value, one for a comma or newline. Each Int64 value is 8 bytes. If all
>> your values are between 0 and 255, you could use UInt8 to represent them
>> and cut the size in half.
>>
>> On Tue, Jul 21, 2015 at 1:16 PM, paul analyst
>> wrote:
>>
>>> I have data in txt file, some milons like this:
>>> 0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0
>>> 0,0,0,0,0,0,0,2,0,0,0,2,0,0,0,0,1
>>> 0,0,0,0,1,0,0,0,1,0,0,0,0,0,0,0,1
>>>
>>> Coding win1250.
>>>
>>> size of dane.txt is 1.3 GB
>>>
>>> D=readcsv("dane.txt")
>>> k,l=size(D)
>>>
>>> using HDF5, JLD
>>> hfi=h5open("D.h5","w")
>>> close(hfi)
>>>
>>> fid = h5open("D.h5","r+")
>>> g = fid["/"]
>>> dset1 = d_create(g, "/D", datatype(Int64), dataspace(k,l))
>>> dset1[:,:]=D
>>> close(fid)
>>>
>>> After save to h5 file the file has 6.3 GB ? Why new file is 4 times
>>> biger?
>>> Paul
>>>
>>
>>
>
>
> --
> Erik Schnetter
> http://www.perimeterinstitute.ca/personal/eschnetter/
>
