Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
On Wednesday, January 07, 2015 01:42:24 PM Paul Analyst wrote:
> one vector 10^9 it is about 16 GB,
...and that explains it. (Compare against your RAM size.)
--Tim
> Paul
>
> W dniu 2015-01-07 o 13:13, Tim Holy pisze:
> > Did you try calculating how much memory the right hand side of that
> > assignment requires? (Remember that sizeof(Float64) == 8).
> >
> > --Tim
> >
> > On Wednesday, January 07, 2015 03:25:54 AM paul analyst wrote:
> >> _ _ _(_)_ | A fresh approach to technical computing
> >>
> >>(_) | (_) (_)| Documentation: http://docs.julialang.org
> >>
> >> _ _ _| |_ __ _ | Type "help()" for help.
> >>
> >>| | | | | | |/ _` | |
> >>| | |
> >>| | |_| | | | (_| | | Version 0.3.3 (2014-10-21 20:18 UTC)
> >>
> >> _/ |\__'_|_|_|\__'_| | Official http://julialang.org release
> >> |
> >> |__/ | i686-linux-gnu
> >>
> >>
> >>
> >> dset[1:10^8,1]=rand(10^8);
> >>
> >> julia> size(dset)
> >> (10,1000)
> >>
> >> julia> dset[1:10^9,1]=rand(10^9);
> >> ERROR: invalid Array size
> >>
> >> in rand at random.jl:123
> >>
> >> julia>
> >>
> >> Julia now is using 32% of ram, 2.5 GB of 7.8 GB aviable..
> >> Ubuntu 14.10
> >> Why ? free size of HD is now 30GB
> >> Paul
> >>
> >>
> >> W dniu wtorek, 6 stycznia 2015 21:04:53 UTC+1 użytkownik paul analyst
> >>
> >> napisał:
> 2.if sum k and l > 9 Julia cant work. Is it to big size for hdf5 or for
>
> > system (Win7 64 Home Premium) ?
>
> Not sure. It works for me (Kubuntu Linux 14.04).
> >>>
> I checked on Ubuntu :
> >>> If sum of "k" or/and "l" is more then 7 I have problem with reading
> >>> vectors (cols)
> >>>
> samsung2@samsung2:~$ julia
>
> _
>
> _ _ _(_)_ | A fresh approach to technical computing
>
> (_) | (_) (_)| Documentation: http://docs.julialang.org
>
> _ _ _| |_ __ _ | Type "help()" for help.
>
> | | | | | | |/ _` | |
> | | |
> | | |_| | | | (_| | | Version 0.3.3 (2014-10-21 20:18 UTC)
>
> _/ |\__'_|_|_|\__'_| | Official http://julialang.org release
> |
> |__/ | i686-linux-gnu
>
> julia> using HDF5
>
> julia> hfi=h5open("F_big.h5","w");close(hfi)
>
> julia> k,l=8,8;
>
> julia> fid = h5open("F_big.h5","r+")
> HDF5 data file: F_big.h5
>
> julia> fid["mygroup/A"]=rand(2)#niepotrzebny
>
> 2-element Array{Float64,1}:
> 0.318459
> 0.258055
>
> julia> g = fid["mygroup"]
> HDF5 group: /mygroup (file: F_big.h5)
>
> julia> dset = d_create(g, "F", datatype(Float64),
> dataspace(10^k,1*10^l))
> HDF5 dataset: /mygroup/F (file: F_big.h5)
>
> julia> h5read("F_big.h5","mygroup/F",(:,1))
>
> 1x1 Array{Float64,2}:
> 0.0
> 0.0
> 0.0
> 0.0
> 0.0
> 0.0
> 0.0
> 0.0
> 0.0
> 0.0
> ⋮
> 0.0
> 0.0
> 0.0
> 0.0
> 0.0
> 0.0
> 0.0
> 0.0
> 0.0
>
> julia> close(fid)
>
> julia> hfi=h5open("F_big.h5","w");close(hfi)
>
> julia> k,l=9,8;
>
> julia> fid = h5open("F_big.h5","r+")
> HDF5 data file: F_big.h5
>
> julia> fid["mygroup/A"]=rand(2)#niepotrzebny
>
> 2-element Array{Float64,1}:
> 0.129214
> 0.4785
>
> julia> g = fid["mygroup"]
> HDF5 group: /mygroup (file: F_big.h5)
>
> julia> dset = d_create(g, "F", datatype(Float64),
> dataspace(10^k,1*10^l))
> HDF5 dataset: /mygroup/F (file: F_big.h5)
>
> julia> h5read("F_big.h5","mygroup/F",(:,1))
> ERROR: invalid Array size
>
> in Array at base.jl:223
> in _getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1557
> in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1550
> in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1620
> in h5read at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:602
>
> julia> close(fid)
>
> julia> h5read("F_big.h5","mygroup/F",(1:2,:))
>
> 2x1 Array{Float64,2}:
> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 … 0.0 0.0 0.0 0.0 0.0
> 0.0
>
> 0.0
>
> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
> 0.0
>
> 0.0
>
> julia> h5read("F_big.h5","mygroup/F",(:,1))
> ERROR: invalid Array size
>
> in Array at base.jl:223
> in _getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1557
> in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1550
> in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1620
> in h5
Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
one vector 10^9 it is about 16 GB,
Paul
W dniu 2015-01-07 o 13:13, Tim Holy pisze:
Did you try calculating how much memory the right hand side of that assignment
requires? (Remember that sizeof(Float64) == 8).
--Tim
On Wednesday, January 07, 2015 03:25:54 AM paul analyst wrote:
_ _ _(_)_ | A fresh approach to technical computing
(_) | (_) (_)| Documentation: http://docs.julialang.org
_ _ _| |_ __ _ | Type "help()" for help.
| | | | | | |/ _` | |
| | |
| | |_| | | | (_| | | Version 0.3.3 (2014-10-21 20:18 UTC)
_/ |\__'_|_|_|\__'_| | Official http://julialang.org release
|__/ | i686-linux-gnu
dset[1:10^8,1]=rand(10^8);
julia> size(dset)
(10,1000)
julia> dset[1:10^9,1]=rand(10^9);
ERROR: invalid Array size
in rand at random.jl:123
julia>
Julia now is using 32% of ram, 2.5 GB of 7.8 GB aviable..
Ubuntu 14.10
Why ? free size of HD is now 30GB
Paul
W dniu wtorek, 6 stycznia 2015 21:04:53 UTC+1 użytkownik paul analyst
napisał:
2.if sum k and l > 9 Julia cant work. Is it to big size for hdf5 or for
system (Win7 64 Home Premium) ?
Not sure. It works for me (Kubuntu Linux 14.04).
I checked on Ubuntu :
If sum of "k" or/and "l" is more then 7 I have problem with reading
vectors (cols)
samsung2@samsung2:~$ julia
_
_ _ _(_)_ | A fresh approach to technical computing
(_) | (_) (_)| Documentation: http://docs.julialang.org
_ _ _| |_ __ _ | Type "help()" for help.
| | | | | | |/ _` | |
| | |
| | |_| | | | (_| | | Version 0.3.3 (2014-10-21 20:18 UTC)
_/ |\__'_|_|_|\__'_| | Official http://julialang.org release
|__/ | i686-linux-gnu
julia> using HDF5
julia> hfi=h5open("F_big.h5","w");close(hfi)
julia> k,l=8,8;
julia> fid = h5open("F_big.h5","r+")
HDF5 data file: F_big.h5
julia> fid["mygroup/A"]=rand(2)#niepotrzebny
2-element Array{Float64,1}:
0.318459
0.258055
julia> g = fid["mygroup"]
HDF5 group: /mygroup (file: F_big.h5)
julia> dset = d_create(g, "F", datatype(Float64), dataspace(10^k,1*10^l))
HDF5 dataset: /mygroup/F (file: F_big.h5)
julia> h5read("F_big.h5","mygroup/F",(:,1))
1x1 Array{Float64,2}:
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
⋮
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
julia> close(fid)
julia> hfi=h5open("F_big.h5","w");close(hfi)
julia> k,l=9,8;
julia> fid = h5open("F_big.h5","r+")
HDF5 data file: F_big.h5
julia> fid["mygroup/A"]=rand(2)#niepotrzebny
2-element Array{Float64,1}:
0.129214
0.4785
julia> g = fid["mygroup"]
HDF5 group: /mygroup (file: F_big.h5)
julia> dset = d_create(g, "F", datatype(Float64), dataspace(10^k,1*10^l))
HDF5 dataset: /mygroup/F (file: F_big.h5)
julia> h5read("F_big.h5","mygroup/F",(:,1))
ERROR: invalid Array size
in Array at base.jl:223
in _getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1557
in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1550
in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1620
in h5read at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:602
julia> close(fid)
julia> h5read("F_big.h5","mygroup/F",(1:2,:))
2x1 Array{Float64,2}:
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 … 0.0 0.0 0.0 0.0 0.0 0.0
0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0
julia> h5read("F_big.h5","mygroup/F",(:,1))
ERROR: invalid Array size
in Array at base.jl:223
in _getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1557
in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1550
in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1620
in h5read at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:602
julia>
Paul
Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
Did you try calculating how much memory the right hand side of that assignment
requires? (Remember that sizeof(Float64) == 8).
--Tim
On Wednesday, January 07, 2015 03:25:54 AM paul analyst wrote:
>_ _ _(_)_ | A fresh approach to technical computing
> (_) | (_) (_)| Documentation: http://docs.julialang.org
>_ _ _| |_ __ _ | Type "help()" for help.
>
> | | | | | | |/ _` | |
> | | |
> | | |_| | | | (_| | | Version 0.3.3 (2014-10-21 20:18 UTC)
>
> _/ |\__'_|_|_|\__'_| | Official http://julialang.org release
>
> |__/ | i686-linux-gnu
>
>
>
> dset[1:10^8,1]=rand(10^8);
>
> julia> size(dset)
> (10,1000)
>
> julia> dset[1:10^9,1]=rand(10^9);
> ERROR: invalid Array size
> in rand at random.jl:123
>
> julia>
>
> Julia now is using 32% of ram, 2.5 GB of 7.8 GB aviable..
> Ubuntu 14.10
> Why ? free size of HD is now 30GB
> Paul
>
>
> W dniu wtorek, 6 stycznia 2015 21:04:53 UTC+1 użytkownik paul analyst
>
> napisał:
> > > 2.if sum k and l > 9 Julia cant work. Is it to big size for hdf5 or for
> > >
> >> > system (Win7 64 Home Premium) ?
> >>
> >> Not sure. It works for me (Kubuntu Linux 14.04).
> >
> >> I checked on Ubuntu :
> > If sum of "k" or/and "l" is more then 7 I have problem with reading
> > vectors (cols)
> >
> >> samsung2@samsung2:~$ julia
> >>
> >>_
> >>
> >>_ _ _(_)_ | A fresh approach to technical computing
> >>
> >> (_) | (_) (_)| Documentation: http://docs.julialang.org
> >>
> >>_ _ _| |_ __ _ | Type "help()" for help.
> >>
> >> | | | | | | |/ _` | |
> >> | | |
> >> | | |_| | | | (_| | | Version 0.3.3 (2014-10-21 20:18 UTC)
> >>
> >> _/ |\__'_|_|_|\__'_| | Official http://julialang.org release
> >>
> >> |__/ | i686-linux-gnu
> >>
> >> julia> using HDF5
> >>
> >> julia> hfi=h5open("F_big.h5","w");close(hfi)
> >>
> >> julia> k,l=8,8;
> >>
> >> julia> fid = h5open("F_big.h5","r+")
> >> HDF5 data file: F_big.h5
> >>
> >> julia> fid["mygroup/A"]=rand(2)#niepotrzebny
> >>
> >> 2-element Array{Float64,1}:
> >> 0.318459
> >> 0.258055
> >>
> >> julia> g = fid["mygroup"]
> >> HDF5 group: /mygroup (file: F_big.h5)
> >>
> >> julia> dset = d_create(g, "F", datatype(Float64), dataspace(10^k,1*10^l))
> >> HDF5 dataset: /mygroup/F (file: F_big.h5)
> >>
> >> julia> h5read("F_big.h5","mygroup/F",(:,1))
> >>
> >> 1x1 Array{Float64,2}:
> >> 0.0
> >> 0.0
> >> 0.0
> >> 0.0
> >> 0.0
> >> 0.0
> >> 0.0
> >> 0.0
> >> 0.0
> >> 0.0
> >> ⋮
> >> 0.0
> >> 0.0
> >> 0.0
> >> 0.0
> >> 0.0
> >> 0.0
> >> 0.0
> >> 0.0
> >> 0.0
> >>
> >> julia> close(fid)
> >>
> >> julia> hfi=h5open("F_big.h5","w");close(hfi)
> >>
> >> julia> k,l=9,8;
> >>
> >> julia> fid = h5open("F_big.h5","r+")
> >> HDF5 data file: F_big.h5
> >>
> >> julia> fid["mygroup/A"]=rand(2)#niepotrzebny
> >>
> >> 2-element Array{Float64,1}:
> >> 0.129214
> >> 0.4785
> >>
> >> julia> g = fid["mygroup"]
> >> HDF5 group: /mygroup (file: F_big.h5)
> >>
> >> julia> dset = d_create(g, "F", datatype(Float64), dataspace(10^k,1*10^l))
> >> HDF5 dataset: /mygroup/F (file: F_big.h5)
> >>
> >> julia> h5read("F_big.h5","mygroup/F",(:,1))
> >> ERROR: invalid Array size
> >>
> >> in Array at base.jl:223
> >> in _getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1557
> >> in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1550
> >> in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1620
> >> in h5read at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:602
> >>
> >> julia> close(fid)
> >>
> >> julia> h5read("F_big.h5","mygroup/F",(1:2,:))
> >>
> >> 2x1 Array{Float64,2}:
> >> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 … 0.0 0.0 0.0 0.0 0.0 0.0
> >>
> >> 0.0
> >>
> >> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
> >>
> >> 0.0
> >>
> >> julia> h5read("F_big.h5","mygroup/F",(:,1))
> >> ERROR: invalid Array size
> >>
> >> in Array at base.jl:223
> >> in _getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1557
> >> in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1550
> >> in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1620
> >> in h5read at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:602
> >>
> >> julia>
> >>
> >> Paul
Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
_ _ _(_)_ | A fresh approach to technical computing
(_) | (_) (_)| Documentation: http://docs.julialang.org
_ _ _| |_ __ _ | Type "help()" for help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 0.3.3 (2014-10-21 20:18 UTC)
_/ |\__'_|_|_|\__'_| | Official http://julialang.org release
|__/ | i686-linux-gnu
dset[1:10^8,1]=rand(10^8);
julia> size(dset)
(10,1000)
julia> dset[1:10^9,1]=rand(10^9);
ERROR: invalid Array size
in rand at random.jl:123
julia>
Julia now is using 32% of ram, 2.5 GB of 7.8 GB aviable..
Ubuntu 14.10
Why ? free size of HD is now 30GB
Paul
W dniu wtorek, 6 stycznia 2015 21:04:53 UTC+1 użytkownik paul analyst
napisał:
>
> > 2.if sum k and l > 9 Julia cant work. Is it to big size for hdf5 or for
>
>> > system (Win7 64 Home Premium) ?
>>
>> Not sure. It works for me (Kubuntu Linux 14.04).
>>
>> I checked on Ubuntu :
>
> If sum of "k" or/and "l" is more then 7 I have problem with reading
> vectors (cols)
>
>
>
>> samsung2@samsung2:~$ julia
>>_
>>_ _ _(_)_ | A fresh approach to technical computing
>> (_) | (_) (_)| Documentation: http://docs.julialang.org
>>_ _ _| |_ __ _ | Type "help()" for help.
>> | | | | | | |/ _` | |
>> | | |_| | | | (_| | | Version 0.3.3 (2014-10-21 20:18 UTC)
>> _/ |\__'_|_|_|\__'_| | Official http://julialang.org release
>> |__/ | i686-linux-gnu
>>
>> julia> using HDF5
>>
>> julia> hfi=h5open("F_big.h5","w");close(hfi)
>>
>> julia> k,l=8,8;
>>
>> julia> fid = h5open("F_big.h5","r+")
>> HDF5 data file: F_big.h5
>>
>> julia> fid["mygroup/A"]=rand(2)#niepotrzebny
>> 2-element Array{Float64,1}:
>> 0.318459
>> 0.258055
>>
>> julia> g = fid["mygroup"]
>> HDF5 group: /mygroup (file: F_big.h5)
>>
>> julia> dset = d_create(g, "F", datatype(Float64), dataspace(10^k,1*10^l))
>> HDF5 dataset: /mygroup/F (file: F_big.h5)
>>
>> julia> h5read("F_big.h5","mygroup/F",(:,1))
>> 1x1 Array{Float64,2}:
>> 0.0
>> 0.0
>> 0.0
>> 0.0
>> 0.0
>> 0.0
>> 0.0
>> 0.0
>> 0.0
>> 0.0
>> ⋮
>> 0.0
>> 0.0
>> 0.0
>> 0.0
>> 0.0
>> 0.0
>> 0.0
>> 0.0
>> 0.0
>>
>> julia> close(fid)
>>
>> julia> hfi=h5open("F_big.h5","w");close(hfi)
>>
>> julia> k,l=9,8;
>>
>> julia> fid = h5open("F_big.h5","r+")
>> HDF5 data file: F_big.h5
>>
>> julia> fid["mygroup/A"]=rand(2)#niepotrzebny
>> 2-element Array{Float64,1}:
>> 0.129214
>> 0.4785
>>
>> julia> g = fid["mygroup"]
>> HDF5 group: /mygroup (file: F_big.h5)
>>
>> julia> dset = d_create(g, "F", datatype(Float64), dataspace(10^k,1*10^l))
>> HDF5 dataset: /mygroup/F (file: F_big.h5)
>>
>> julia> h5read("F_big.h5","mygroup/F",(:,1))
>> ERROR: invalid Array size
>> in Array at base.jl:223
>> in _getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1557
>> in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1550
>> in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1620
>> in h5read at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:602
>>
>> julia> close(fid)
>>
>> julia> h5read("F_big.h5","mygroup/F",(1:2,:))
>> 2x1 Array{Float64,2}:
>> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 … 0.0 0.0 0.0 0.0 0.0 0.0
>> 0.0
>> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
>> 0.0
>>
>> julia> h5read("F_big.h5","mygroup/F",(:,1))
>> ERROR: invalid Array size
>> in Array at base.jl:223
>> in _getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1557
>> in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1550
>> in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1620
>> in h5read at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:602
>>
>> julia>
>>
>> Paul
>
>
Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
> 2.if sum k and l > 9 Julia cant work. Is it to big size for hdf5 or for
> > system (Win7 64 Home Premium) ?
>
> Not sure. It works for me (Kubuntu Linux 14.04).
>
> I checked on Ubuntu :
If sum of "k" or/and "l" is more then 7 I have problem with reading
vectors (cols)
> samsung2@samsung2:~$ julia
>_
>_ _ _(_)_ | A fresh approach to technical computing
> (_) | (_) (_)| Documentation: http://docs.julialang.org
>_ _ _| |_ __ _ | Type "help()" for help.
> | | | | | | |/ _` | |
> | | |_| | | | (_| | | Version 0.3.3 (2014-10-21 20:18 UTC)
> _/ |\__'_|_|_|\__'_| | Official http://julialang.org release
> |__/ | i686-linux-gnu
>
> julia> using HDF5
>
> julia> hfi=h5open("F_big.h5","w");close(hfi)
>
> julia> k,l=8,8;
>
> julia> fid = h5open("F_big.h5","r+")
> HDF5 data file: F_big.h5
>
> julia> fid["mygroup/A"]=rand(2)#niepotrzebny
> 2-element Array{Float64,1}:
> 0.318459
> 0.258055
>
> julia> g = fid["mygroup"]
> HDF5 group: /mygroup (file: F_big.h5)
>
> julia> dset = d_create(g, "F", datatype(Float64), dataspace(10^k,1*10^l))
> HDF5 dataset: /mygroup/F (file: F_big.h5)
>
> julia> h5read("F_big.h5","mygroup/F",(:,1))
> 1x1 Array{Float64,2}:
> 0.0
> 0.0
> 0.0
> 0.0
> 0.0
> 0.0
> 0.0
> 0.0
> 0.0
> 0.0
> ⋮
> 0.0
> 0.0
> 0.0
> 0.0
> 0.0
> 0.0
> 0.0
> 0.0
> 0.0
>
> julia> close(fid)
>
> julia> hfi=h5open("F_big.h5","w");close(hfi)
>
> julia> k,l=9,8;
>
> julia> fid = h5open("F_big.h5","r+")
> HDF5 data file: F_big.h5
>
> julia> fid["mygroup/A"]=rand(2)#niepotrzebny
> 2-element Array{Float64,1}:
> 0.129214
> 0.4785
>
> julia> g = fid["mygroup"]
> HDF5 group: /mygroup (file: F_big.h5)
>
> julia> dset = d_create(g, "F", datatype(Float64), dataspace(10^k,1*10^l))
> HDF5 dataset: /mygroup/F (file: F_big.h5)
>
> julia> h5read("F_big.h5","mygroup/F",(:,1))
> ERROR: invalid Array size
> in Array at base.jl:223
> in _getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1557
> in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1550
> in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1620
> in h5read at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:602
>
> julia> close(fid)
>
> julia> h5read("F_big.h5","mygroup/F",(1:2,:))
> 2x1 Array{Float64,2}:
> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 … 0.0 0.0 0.0 0.0 0.0 0.0
> 0.0
> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
> 0.0
>
> julia> h5read("F_big.h5","mygroup/F",(:,1))
> ERROR: invalid Array size
> in Array at base.jl:223
> in _getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1557
> in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1550
> in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1620
> in h5read at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:602
>
> julia>
>
> Paul
Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
On Monday, January 05, 2015 09:52:28 PM Paul Analyst wrote: > dset = d_create( "F", datatype(Float64), dataspace(10,10))/ You're missing the `fid`. You have to tell it where (which file, or which group) you want to create that dataset. --Tim
Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
Tim , thx for hints, but do not work without this line
#fid["mygroup/A"]=rand(2)
becouse I vave not "g" , and nothing to declare in this line /dset =
d_create(g, "F", datatype(Float64), dataspace(10,10))/
/
dset = d_create( "F", datatype(Float64), dataspace(10,10))/
do not work ...
Paul
using HDF5
hfi=h5open("test.h5","w");close(hfi)
fid = h5open("test.h5","r+")
#fid["mygroup/A"]=rand(2)
g = fid["mygroup"]
dset = d_create(g, "F", datatype(Float64), dataspace(10,10))
dset[:,1] = rand(10)
h5read("test.h5","mygroup/F",(:,1))
close(fid)
W dniu 2015-01-05 o 15:39, Tim Holy pisze:
On Monday, January 05, 2015 03:17:12 PM Paul Analyst wrote:
Thx, Tim,
I have solution but is 2 questions:
1. Whay must be line: /fid["mygroup/A"]=rand(2)/ ?
That line just means 'create a variable called "A" inside a group called
"mygroup", and assign it a value of rand(2)'. If you don't need that variable,
you don't need that line. You also don't have to create a group called
"mygroup", if you prefer you can store everything in the top level of the file.
2.if sum k and l > 9 Julia cant work. Is it to big size for hdf5 or for
system (Win7 64 Home Premium) ?
Not sure. It works for me (Kubuntu Linux 14.04).
--Tim
using HDF5
hfi=h5open("bigfile.h5","w")
close(hfi)
k,l=6,3;
fid = h5open("bigfile.h5","r+")
fid["mygroup/A"]=rand(2)
g = fid["mygroup"]
dset = d_create(g, "F", datatype(Float64), dataspace(10^k,10^l))
dset[:,1] = rand(10^k)
h5read("bigfile.h5","mygroup/F",(:,1))
close(fid)
h5read("bigfile.h5","mygroup/F",(:,1:2))
Is OK
but if
k,l=6,4;
julia> close(fid)
HDF5-DIAG: Error detected in HDF5 (1.8.13) thread 0:
#000: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 2070 in
H5Fclose(): decrementing file ID failed
major: Object atom
minor: Unable to close file
#001: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5I.c line 1464 in
H5I_dec_app_ref(): can't decrement ID ref c
ount
major: Object atom
minor: Unable to decrement reference count
#002: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 1847 in
H5F_close(): can't close file
major: File accessibilty
minor: Unable to close file
#003: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 2009 in
H5F_try_close(): problems closing file
major: File accessibilty
minor: Unable to close file
#004: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 1161 in
H5F_dest(): low level truncate failed
major: File accessibilty
minor: Write failed
#005: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5FD.c line 1895 in
H5FD_truncate(): driver truncate request f
ailed
major: Virtual File Layer
minor: Can't update object
#006: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5FDsec2.c line 900
in H5FD_sec2_truncate(): unable to extend
file properly
major: Low-level I/O
minor: Seek failed
ERROR: Error closing file
in h5f_close at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:1924
julia> h5read("bigfile.h5","mygroup/F",(:,1:2))
HDF5-DIAG: Error detected in HDF5 (1.8.13) thread 0:
#000: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 1594 in
H5Fopen(): unable to open file
major: File accessibilty
minor: Unable to open file
#001: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 1385 in
H5F_open(): unable to read superblock
major: File accessibilty
minor: Read failed
#002: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5Fsuper.c line 353
in H5F_super_read(): unable to load superb
lock
major: Object cache
minor: Unable to protect metadata
#003: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5AC.c line 1323 in
H5AC_protect(): H5C_protect() failed.
major: Object cache
minor: Unable to protect metadata
#004: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5C.c line 3574 in
H5C_protect(): can't load entry
major: Object cache
minor: Unable to load metadata into cache
#005: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5C.c line 7954 in
H5C_load_entry(): unable to load entry
major: Object cache
minor: Unable to load metadata into cache
#006: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5Fsuper_cache.c
line 471 in H5F_sblock_load(): truncated file
: eof = 8002864, sblock->base_addr = 0, stored_eoa = 8002864
major: File accessibilty
minor: File has been truncated
ERROR: Error opening file bigfile.h5
in h5f_open at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:2023
in h5open at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:554
Paul
W dniu 2015-01-04 o 18:37, Tim Holy pisze:
Do note there are two additional pages of documentation in the doc/
folder.
--Tim
On Sunday, January 04, 2015 06:59:53 AM paul analyst wrote:
Of course, first I read :)
Is there about reading range array. I need to save a range of In analogy
to.
A = reshape (1: 120, 15, 8)
h5write ("/ tmp / test2.h5", "mygroup2 / A", A)
data = h5read ("/ tmp / test2.h5", "mygr
Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
On Monday, January 05, 2015 03:17:12 PM Paul Analyst wrote:
> Thx, Tim,
> I have solution but is 2 questions:
> 1. Whay must be line: /fid["mygroup/A"]=rand(2)/ ?
That line just means 'create a variable called "A" inside a group called
"mygroup", and assign it a value of rand(2)'. If you don't need that variable,
you don't need that line. You also don't have to create a group called
"mygroup", if you prefer you can store everything in the top level of the file.
> 2.if sum k and l > 9 Julia cant work. Is it to big size for hdf5 or for
> system (Win7 64 Home Premium) ?
Not sure. It works for me (Kubuntu Linux 14.04).
--Tim
>
>
> using HDF5
> hfi=h5open("bigfile.h5","w")
> close(hfi)
>
> k,l=6,3;
> fid = h5open("bigfile.h5","r+")
> fid["mygroup/A"]=rand(2)
> g = fid["mygroup"]
> dset = d_create(g, "F", datatype(Float64), dataspace(10^k,10^l))
> dset[:,1] = rand(10^k)
> h5read("bigfile.h5","mygroup/F",(:,1))
> close(fid)
> h5read("bigfile.h5","mygroup/F",(:,1:2))
>
> Is OK
> but if
> k,l=6,4;
>
> julia> close(fid)
> HDF5-DIAG: Error detected in HDF5 (1.8.13) thread 0:
>#000: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 2070 in
> H5Fclose(): decrementing file ID failed
> major: Object atom
> minor: Unable to close file
>#001: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5I.c line 1464 in
> H5I_dec_app_ref(): can't decrement ID ref c
> ount
> major: Object atom
> minor: Unable to decrement reference count
>#002: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 1847 in
> H5F_close(): can't close file
> major: File accessibilty
> minor: Unable to close file
>#003: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 2009 in
> H5F_try_close(): problems closing file
> major: File accessibilty
> minor: Unable to close file
>#004: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 1161 in
> H5F_dest(): low level truncate failed
> major: File accessibilty
> minor: Write failed
>#005: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5FD.c line 1895 in
> H5FD_truncate(): driver truncate request f
> ailed
> major: Virtual File Layer
> minor: Can't update object
>#006: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5FDsec2.c line 900
> in H5FD_sec2_truncate(): unable to extend
> file properly
> major: Low-level I/O
> minor: Seek failed
> ERROR: Error closing file
> in h5f_close at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:1924
>
>
> julia> h5read("bigfile.h5","mygroup/F",(:,1:2))
> HDF5-DIAG: Error detected in HDF5 (1.8.13) thread 0:
>#000: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 1594 in
> H5Fopen(): unable to open file
> major: File accessibilty
> minor: Unable to open file
>#001: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 1385 in
> H5F_open(): unable to read superblock
> major: File accessibilty
> minor: Read failed
>#002: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5Fsuper.c line 353
> in H5F_super_read(): unable to load superb
> lock
> major: Object cache
> minor: Unable to protect metadata
>#003: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5AC.c line 1323 in
> H5AC_protect(): H5C_protect() failed.
> major: Object cache
> minor: Unable to protect metadata
>#004: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5C.c line 3574 in
> H5C_protect(): can't load entry
> major: Object cache
> minor: Unable to load metadata into cache
>#005: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5C.c line 7954 in
> H5C_load_entry(): unable to load entry
> major: Object cache
> minor: Unable to load metadata into cache
>#006: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5Fsuper_cache.c
> line 471 in H5F_sblock_load(): truncated file
>
> : eof = 8002864, sblock->base_addr = 0, stored_eoa = 8002864
>
> major: File accessibilty
> minor: File has been truncated
> ERROR: Error opening file bigfile.h5
> in h5f_open at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:2023
> in h5open at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:554
>
>
>
> Paul
>
> W dniu 2015-01-04 o 18:37, Tim Holy pisze:
> > Do note there are two additional pages of documentation in the doc/
> > folder.
> >
> > --Tim
> >
> > On Sunday, January 04, 2015 06:59:53 AM paul analyst wrote:
> >> Of course, first I read :)
> >> Is there about reading range array. I need to save a range of In analogy
> >> to.
> >>
> >> A = reshape (1: 120, 15, 8)
> >> h5write ("/ tmp / test2.h5", "mygroup2 / A", A)
> >> data = h5read ("/ tmp / test2.h5", "mygroup2 / A" (2: 3: 15: 3: 5))
> >>
> >> Paul
> >>
> >>
> >> W dniu niedziela, 4 stycznia 2015 14:14:03 UTC+1 użytkownik Tim Holy
> >>
> >> napisał:
> >>> If I understand correctly, then yes, that's possible. See the HDF5 docs.
> >>>
> >>> --Tim
> >>>
> >>> On Sunday, January 04, 2015 04:25:13 AM paul analyst wrote:
> How to overwrite to an existing file, only
Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
Thx, Tim,
I have solution but is 2 questions:
1. Whay must be line: /fid["mygroup/A"]=rand(2)/ ?
2.if sum k and l > 9 Julia cant work. Is it to big size for hdf5 or for
system (Win7 64 Home Premium) ?
using HDF5
hfi=h5open("bigfile.h5","w")
close(hfi)
k,l=6,3;
fid = h5open("bigfile.h5","r+")
fid["mygroup/A"]=rand(2)
g = fid["mygroup"]
dset = d_create(g, "F", datatype(Float64), dataspace(10^k,10^l))
dset[:,1] = rand(10^k)
h5read("bigfile.h5","mygroup/F",(:,1))
close(fid)
h5read("bigfile.h5","mygroup/F",(:,1:2))
Is OK
but if
k,l=6,4;
julia> close(fid)
HDF5-DIAG: Error detected in HDF5 (1.8.13) thread 0:
#000: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 2070 in
H5Fclose(): decrementing file ID failed
major: Object atom
minor: Unable to close file
#001: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5I.c line 1464 in
H5I_dec_app_ref(): can't decrement ID ref c
ount
major: Object atom
minor: Unable to decrement reference count
#002: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 1847 in
H5F_close(): can't close file
major: File accessibilty
minor: Unable to close file
#003: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 2009 in
H5F_try_close(): problems closing file
major: File accessibilty
minor: Unable to close file
#004: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 1161 in
H5F_dest(): low level truncate failed
major: File accessibilty
minor: Write failed
#005: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5FD.c line 1895 in
H5FD_truncate(): driver truncate request f
ailed
major: Virtual File Layer
minor: Can't update object
#006: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5FDsec2.c line 900
in H5FD_sec2_truncate(): unable to extend
file properly
major: Low-level I/O
minor: Seek failed
ERROR: Error closing file
in h5f_close at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:1924
julia> h5read("bigfile.h5","mygroup/F",(:,1:2))
HDF5-DIAG: Error detected in HDF5 (1.8.13) thread 0:
#000: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 1594 in
H5Fopen(): unable to open file
major: File accessibilty
minor: Unable to open file
#001: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 1385 in
H5F_open(): unable to read superblock
major: File accessibilty
minor: Read failed
#002: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5Fsuper.c line 353
in H5F_super_read(): unable to load superb
lock
major: Object cache
minor: Unable to protect metadata
#003: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5AC.c line 1323 in
H5AC_protect(): H5C_protect() failed.
major: Object cache
minor: Unable to protect metadata
#004: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5C.c line 3574 in
H5C_protect(): can't load entry
major: Object cache
minor: Unable to load metadata into cache
#005: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5C.c line 7954 in
H5C_load_entry(): unable to load entry
major: Object cache
minor: Unable to load metadata into cache
#006: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5Fsuper_cache.c
line 471 in H5F_sblock_load(): truncated file
: eof = 8002864, sblock->base_addr = 0, stored_eoa = 8002864
major: File accessibilty
minor: File has been truncated
ERROR: Error opening file bigfile.h5
in h5f_open at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:2023
in h5open at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:554
Paul
W dniu 2015-01-04 o 18:37, Tim Holy pisze:
Do note there are two additional pages of documentation in the doc/ folder.
--Tim
On Sunday, January 04, 2015 06:59:53 AM paul analyst wrote:
Of course, first I read :)
Is there about reading range array. I need to save a range of In analogy to.
A = reshape (1: 120, 15, 8)
h5write ("/ tmp / test2.h5", "mygroup2 / A", A)
data = h5read ("/ tmp / test2.h5", "mygroup2 / A" (2: 3: 15: 3: 5))
Paul
W dniu niedziela, 4 stycznia 2015 14:14:03 UTC+1 użytkownik Tim Holy
napisał:
If I understand correctly, then yes, that's possible. See the HDF5 docs.
--Tim
On Sunday, January 04, 2015 04:25:13 AM paul analyst wrote:
How to overwrite to an existing file, only range of data?
In HDF5 can do this?
I have an array of zeros 10 x 10
I need an existing file owerwrite range rand (5x5), for example.
Existingfile [2: 7.3: 8]
Paul
Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
Big Thx for big lesson, is no too "dark" now
Paul
W dniu niedziela, 4 stycznia 2015 17:58:28 UTC+1 użytkownik Adrian
Cuthbertson napisał:
>
> You can also work directly with the HDF5 file as an array object...
>
> using HDF5
>
> hfi=h5open("myfile.h5","w"); # create the file
> close(hfi)
>
> A = reshape(1: 120, 15, 8);
>
> hfi = h5open("myfile.h5","r+") # read/write access
>
> hfi["mygroup/A"] = A
> 15x8 Array{Int64,2}:
> 1 16 31 46 61 76 91 106
> ...
> 14 29 44 59 74 89 104 119
> 15 30 45 60 75 90 105 120
>
> data = hfi["mygroup/A"][2:3:15,3:5]
> 5x3 Array{Int64,2}:
> 32 47 62
> 35 50 65
> 38 53 68
> 41 56 71
> 44 59 74
>
> hfi["mygroup/A"][2:3,3:5]
> 2x3 Array{Int64,2}:
> 32 47 62
> 33 48 63
>
> hfi["mygroup/A"][2:3,3:5]=[-1 -2 -3; -4 -5 -6]
> 2x3 Array{Int64,2}:
> -1 -2 -3
> -4 -5 -6
>
> hfi["mygroup/A"][1:5,:]
> 5x8 Array{Int64,2}:
> 1 16 31 46 61 76 91 106
> 2 17 -1 -2 -3 77 92 107
> 3 18 -4 -5 -6 78 93 108
> 4 19 34 49 64 79 94 109
> 5 20 35 50 65 80 95 110
>
> foo = hfi["mygroup/A"]
> HDF5 dataset: /mygroup/A (file: myfile.h5)
>
> foo[1:5,:]
> 5x8 Array{Int64,2}:
> 1 16 31 46 61 76 91 106
> 2 17 -1 -2 -3 77 92 107
> 3 18 -4 -5 -6 78 93 108
> 4 19 34 49 64 79 94 109
> 5 20 35 50 65 80 95 110
>
> close(hfi)
>
> HDF5 is awesome!
>
>
>
>
> On Sun, Jan 4, 2015 at 4:59 PM, paul analyst > wrote:
>
>> Of course, first I read :)
>> Is there about reading range array. I need to save a range of In analogy
>> to.
>>
>> A = reshape (1: 120, 15, 8)
>> h5write ("/ tmp / test2.h5", "mygroup2 / A", A)
>> data = h5read ("/ tmp / test2.h5", "mygroup2 / A" (2: 3: 15: 3: 5))
>>
>> Paul
>>
>>
>> W dniu niedziela, 4 stycznia 2015 14:14:03 UTC+1 użytkownik Tim Holy
>> napisał:
>>>
>>> If I understand correctly, then yes, that's possible. See the HDF5 docs.
>>>
>>> --Tim
>>>
>>> On Sunday, January 04, 2015 04:25:13 AM paul analyst wrote:
>>> > How to overwrite to an existing file, only range of data?
>>> > In HDF5 can do this?
>>> > I have an array of zeros 10 x 10
>>> > I need an existing file owerwrite range rand (5x5), for example.
>>> > Existingfile [2: 7.3: 8]
>>> > Paul
>>>
>>>
>
Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
Thx, form me this moment
*hfi["mygroup/A"] = A*
was not too clear
Paul
W dniu niedziela, 4 stycznia 2015 18:37:30 UTC+1 użytkownik Tim Holy
napisał:
>
> See the part about "...incrementally save to very large ..."
>
> --Tim
>
> On Sunday, January 04, 2015 06:59:53 AM paul analyst wrote:
> > Of course, first I read :)
> > Is there about reading range array. I need to save a range of In analogy
> to.
> >
> > A = reshape (1: 120, 15, 8)
> > h5write ("/ tmp / test2.h5", "mygroup2 / A", A)
> > data = h5read ("/ tmp / test2.h5", "mygroup2 / A" (2: 3: 15: 3: 5))
> >
> > Paul
> >
> >
> > W dniu niedziela, 4 stycznia 2015 14:14:03 UTC+1 użytkownik Tim Holy
> >
> > napisał:
> > > If I understand correctly, then yes, that's possible. See the HDF5
> docs.
> > >
> > > --Tim
> > >
> > > On Sunday, January 04, 2015 04:25:13 AM paul analyst wrote:
> > > > How to overwrite to an existing file, only range of data?
> > > > In HDF5 can do this?
> > > > I have an array of zeros 10 x 10
> > > > I need an existing file owerwrite range rand (5x5), for example.
> > > > Existingfile [2: 7.3: 8]
> > > > Paul
>
>
Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
Do note there are two additional pages of documentation in the doc/ folder.
--Tim
On Sunday, January 04, 2015 06:59:53 AM paul analyst wrote:
> Of course, first I read :)
> Is there about reading range array. I need to save a range of In analogy to.
>
> A = reshape (1: 120, 15, 8)
> h5write ("/ tmp / test2.h5", "mygroup2 / A", A)
> data = h5read ("/ tmp / test2.h5", "mygroup2 / A" (2: 3: 15: 3: 5))
>
> Paul
>
>
> W dniu niedziela, 4 stycznia 2015 14:14:03 UTC+1 użytkownik Tim Holy
>
> napisał:
> > If I understand correctly, then yes, that's possible. See the HDF5 docs.
> >
> > --Tim
> >
> > On Sunday, January 04, 2015 04:25:13 AM paul analyst wrote:
> > > How to overwrite to an existing file, only range of data?
> > > In HDF5 can do this?
> > > I have an array of zeros 10 x 10
> > > I need an existing file owerwrite range rand (5x5), for example.
> > > Existingfile [2: 7.3: 8]
> > > Paul
Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
See the part about "...incrementally save to very large ..."
--Tim
On Sunday, January 04, 2015 06:59:53 AM paul analyst wrote:
> Of course, first I read :)
> Is there about reading range array. I need to save a range of In analogy to.
>
> A = reshape (1: 120, 15, 8)
> h5write ("/ tmp / test2.h5", "mygroup2 / A", A)
> data = h5read ("/ tmp / test2.h5", "mygroup2 / A" (2: 3: 15: 3: 5))
>
> Paul
>
>
> W dniu niedziela, 4 stycznia 2015 14:14:03 UTC+1 użytkownik Tim Holy
>
> napisał:
> > If I understand correctly, then yes, that's possible. See the HDF5 docs.
> >
> > --Tim
> >
> > On Sunday, January 04, 2015 04:25:13 AM paul analyst wrote:
> > > How to overwrite to an existing file, only range of data?
> > > In HDF5 can do this?
> > > I have an array of zeros 10 x 10
> > > I need an existing file owerwrite range rand (5x5), for example.
> > > Existingfile [2: 7.3: 8]
> > > Paul
Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
You can also work directly with the HDF5 file as an array object...
using HDF5
hfi=h5open("myfile.h5","w"); # create the file
close(hfi)
A = reshape(1: 120, 15, 8);
hfi = h5open("myfile.h5","r+") # read/write access
hfi["mygroup/A"] = A
15x8 Array{Int64,2}:
1 16 31 46 61 76 91 106
...
14 29 44 59 74 89 104 119
15 30 45 60 75 90 105 120
data = hfi["mygroup/A"][2:3:15,3:5]
5x3 Array{Int64,2}:
32 47 62
35 50 65
38 53 68
41 56 71
44 59 74
hfi["mygroup/A"][2:3,3:5]
2x3 Array{Int64,2}:
32 47 62
33 48 63
hfi["mygroup/A"][2:3,3:5]=[-1 -2 -3; -4 -5 -6]
2x3 Array{Int64,2}:
-1 -2 -3
-4 -5 -6
hfi["mygroup/A"][1:5,:]
5x8 Array{Int64,2}:
1 16 31 46 61 76 91 106
2 17 -1 -2 -3 77 92 107
3 18 -4 -5 -6 78 93 108
4 19 34 49 64 79 94 109
5 20 35 50 65 80 95 110
foo = hfi["mygroup/A"]
HDF5 dataset: /mygroup/A (file: myfile.h5)
foo[1:5,:]
5x8 Array{Int64,2}:
1 16 31 46 61 76 91 106
2 17 -1 -2 -3 77 92 107
3 18 -4 -5 -6 78 93 108
4 19 34 49 64 79 94 109
5 20 35 50 65 80 95 110
close(hfi)
HDF5 is awesome!
On Sun, Jan 4, 2015 at 4:59 PM, paul analyst wrote:
> Of course, first I read :)
> Is there about reading range array. I need to save a range of In analogy
> to.
>
> A = reshape (1: 120, 15, 8)
> h5write ("/ tmp / test2.h5", "mygroup2 / A", A)
> data = h5read ("/ tmp / test2.h5", "mygroup2 / A" (2: 3: 15: 3: 5))
>
> Paul
>
>
> W dniu niedziela, 4 stycznia 2015 14:14:03 UTC+1 użytkownik Tim Holy
> napisał:
>>
>> If I understand correctly, then yes, that's possible. See the HDF5 docs.
>>
>> --Tim
>>
>> On Sunday, January 04, 2015 04:25:13 AM paul analyst wrote:
>> > How to overwrite to an existing file, only range of data?
>> > In HDF5 can do this?
>> > I have an array of zeros 10 x 10
>> > I need an existing file owerwrite range rand (5x5), for example.
>> > Existingfile [2: 7.3: 8]
>> > Paul
>>
>>
Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
Of course, first I read :)
Is there about reading range array. I need to save a range of In analogy to.
A = reshape (1: 120, 15, 8)
h5write ("/ tmp / test2.h5", "mygroup2 / A", A)
data = h5read ("/ tmp / test2.h5", "mygroup2 / A" (2: 3: 15: 3: 5))
Paul
W dniu niedziela, 4 stycznia 2015 14:14:03 UTC+1 użytkownik Tim Holy
napisał:
>
> If I understand correctly, then yes, that's possible. See the HDF5 docs.
>
> --Tim
>
> On Sunday, January 04, 2015 04:25:13 AM paul analyst wrote:
> > How to overwrite to an existing file, only range of data?
> > In HDF5 can do this?
> > I have an array of zeros 10 x 10
> > I need an existing file owerwrite range rand (5x5), for example.
> > Existingfile [2: 7.3: 8]
> > Paul
>
>
Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
If I understand correctly, then yes, that's possible. See the HDF5 docs. --Tim On Sunday, January 04, 2015 04:25:13 AM paul analyst wrote: > How to overwrite to an existing file, only range of data? > In HDF5 can do this? > I have an array of zeros 10 x 10 > I need an existing file owerwrite range rand (5x5), for example. > Existingfile [2: 7.3: 8] > Paul
[julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
How to overwrite to an existing file, only range of data? In HDF5 can do this? I have an array of zeros 10 x 10 I need an existing file owerwrite range rand (5x5), for example. Existingfile [2: 7.3: 8] Paul
