Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
_ _ _(_)_ | A fresh approach to technical computing
(_) | (_) (_)| Documentation: http://docs.julialang.org
_ _ _| |_ __ _ | Type help() for help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 0.3.3 (2014-10-21 20:18 UTC)
_/ |\__'_|_|_|\__'_| | Official http://julialang.org release
|__/ | i686-linux-gnu
dset[1:10^8,1]=rand(10^8);
julia size(dset)
(10,1000)
julia dset[1:10^9,1]=rand(10^9);
ERROR: invalid Array size
in rand at random.jl:123
julia
Julia now is using 32% of ram, 2.5 GB of 7.8 GB aviable..
Ubuntu 14.10
Why ? free size of HD is now 30GB
Paul
W dniu wtorek, 6 stycznia 2015 21:04:53 UTC+1 użytkownik paul analyst
napisał:
2.if sum k and l 9 Julia cant work. Is it to big size for hdf5 or for
system (Win7 64 Home Premium) ?
Not sure. It works for me (Kubuntu Linux 14.04).
I checked on Ubuntu :
If sum of k or/and l is more then 7 I have problem with reading
vectors (cols)
samsung2@samsung2:~$ julia
_
_ _ _(_)_ | A fresh approach to technical computing
(_) | (_) (_)| Documentation: http://docs.julialang.org
_ _ _| |_ __ _ | Type help() for help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 0.3.3 (2014-10-21 20:18 UTC)
_/ |\__'_|_|_|\__'_| | Official http://julialang.org release
|__/ | i686-linux-gnu
julia using HDF5
julia hfi=h5open(F_big.h5,w);close(hfi)
julia k,l=8,8;
julia fid = h5open(F_big.h5,r+)
HDF5 data file: F_big.h5
julia fid[mygroup/A]=rand(2)#niepotrzebny
2-element Array{Float64,1}:
0.318459
0.258055
julia g = fid[mygroup]
HDF5 group: /mygroup (file: F_big.h5)
julia dset = d_create(g, F, datatype(Float64), dataspace(10^k,1*10^l))
HDF5 dataset: /mygroup/F (file: F_big.h5)
julia h5read(F_big.h5,mygroup/F,(:,1))
1x1 Array{Float64,2}:
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
⋮
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
julia close(fid)
julia hfi=h5open(F_big.h5,w);close(hfi)
julia k,l=9,8;
julia fid = h5open(F_big.h5,r+)
HDF5 data file: F_big.h5
julia fid[mygroup/A]=rand(2)#niepotrzebny
2-element Array{Float64,1}:
0.129214
0.4785
julia g = fid[mygroup]
HDF5 group: /mygroup (file: F_big.h5)
julia dset = d_create(g, F, datatype(Float64), dataspace(10^k,1*10^l))
HDF5 dataset: /mygroup/F (file: F_big.h5)
julia h5read(F_big.h5,mygroup/F,(:,1))
ERROR: invalid Array size
in Array at base.jl:223
in _getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1557
in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1550
in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1620
in h5read at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:602
julia close(fid)
julia h5read(F_big.h5,mygroup/F,(1:2,:))
2x1 Array{Float64,2}:
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 … 0.0 0.0 0.0 0.0 0.0 0.0
0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0
julia h5read(F_big.h5,mygroup/F,(:,1))
ERROR: invalid Array size
in Array at base.jl:223
in _getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1557
in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1550
in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1620
in h5read at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:602
julia
Paul
Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
one vector 10^9 it is about 16 GB,
Paul
W dniu 2015-01-07 o 13:13, Tim Holy pisze:
Did you try calculating how much memory the right hand side of that assignment
requires? (Remember that sizeof(Float64) == 8).
--Tim
On Wednesday, January 07, 2015 03:25:54 AM paul analyst wrote:
_ _ _(_)_ | A fresh approach to technical computing
(_) | (_) (_)| Documentation: http://docs.julialang.org
_ _ _| |_ __ _ | Type help() for help.
| | | | | | |/ _` | |
| | |
| | |_| | | | (_| | | Version 0.3.3 (2014-10-21 20:18 UTC)
_/ |\__'_|_|_|\__'_| | Official http://julialang.org release
|__/ | i686-linux-gnu
dset[1:10^8,1]=rand(10^8);
julia size(dset)
(10,1000)
julia dset[1:10^9,1]=rand(10^9);
ERROR: invalid Array size
in rand at random.jl:123
julia
Julia now is using 32% of ram, 2.5 GB of 7.8 GB aviable..
Ubuntu 14.10
Why ? free size of HD is now 30GB
Paul
W dniu wtorek, 6 stycznia 2015 21:04:53 UTC+1 użytkownik paul analyst
napisał:
2.if sum k and l 9 Julia cant work. Is it to big size for hdf5 or for
system (Win7 64 Home Premium) ?
Not sure. It works for me (Kubuntu Linux 14.04).
I checked on Ubuntu :
If sum of k or/and l is more then 7 I have problem with reading
vectors (cols)
samsung2@samsung2:~$ julia
_
_ _ _(_)_ | A fresh approach to technical computing
(_) | (_) (_)| Documentation: http://docs.julialang.org
_ _ _| |_ __ _ | Type help() for help.
| | | | | | |/ _` | |
| | |
| | |_| | | | (_| | | Version 0.3.3 (2014-10-21 20:18 UTC)
_/ |\__'_|_|_|\__'_| | Official http://julialang.org release
|__/ | i686-linux-gnu
julia using HDF5
julia hfi=h5open(F_big.h5,w);close(hfi)
julia k,l=8,8;
julia fid = h5open(F_big.h5,r+)
HDF5 data file: F_big.h5
julia fid[mygroup/A]=rand(2)#niepotrzebny
2-element Array{Float64,1}:
0.318459
0.258055
julia g = fid[mygroup]
HDF5 group: /mygroup (file: F_big.h5)
julia dset = d_create(g, F, datatype(Float64), dataspace(10^k,1*10^l))
HDF5 dataset: /mygroup/F (file: F_big.h5)
julia h5read(F_big.h5,mygroup/F,(:,1))
1x1 Array{Float64,2}:
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
⋮
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
julia close(fid)
julia hfi=h5open(F_big.h5,w);close(hfi)
julia k,l=9,8;
julia fid = h5open(F_big.h5,r+)
HDF5 data file: F_big.h5
julia fid[mygroup/A]=rand(2)#niepotrzebny
2-element Array{Float64,1}:
0.129214
0.4785
julia g = fid[mygroup]
HDF5 group: /mygroup (file: F_big.h5)
julia dset = d_create(g, F, datatype(Float64), dataspace(10^k,1*10^l))
HDF5 dataset: /mygroup/F (file: F_big.h5)
julia h5read(F_big.h5,mygroup/F,(:,1))
ERROR: invalid Array size
in Array at base.jl:223
in _getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1557
in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1550
in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1620
in h5read at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:602
julia close(fid)
julia h5read(F_big.h5,mygroup/F,(1:2,:))
2x1 Array{Float64,2}:
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 … 0.0 0.0 0.0 0.0 0.0 0.0
0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0
julia h5read(F_big.h5,mygroup/F,(:,1))
ERROR: invalid Array size
in Array at base.jl:223
in _getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1557
in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1550
in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1620
in h5read at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:602
julia
Paul
Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
On Wednesday, January 07, 2015 01:42:24 PM Paul Analyst wrote:
one vector 10^9 it is about 16 GB,
...and that explains it. (Compare against your RAM size.)
--Tim
Paul
W dniu 2015-01-07 o 13:13, Tim Holy pisze:
Did you try calculating how much memory the right hand side of that
assignment requires? (Remember that sizeof(Float64) == 8).
--Tim
On Wednesday, January 07, 2015 03:25:54 AM paul analyst wrote:
_ _ _(_)_ | A fresh approach to technical computing
(_) | (_) (_)| Documentation: http://docs.julialang.org
_ _ _| |_ __ _ | Type help() for help.
| | | | | | |/ _` | |
| | |
| | |_| | | | (_| | | Version 0.3.3 (2014-10-21 20:18 UTC)
_/ |\__'_|_|_|\__'_| | Official http://julialang.org release
|
|__/ | i686-linux-gnu
dset[1:10^8,1]=rand(10^8);
julia size(dset)
(10,1000)
julia dset[1:10^9,1]=rand(10^9);
ERROR: invalid Array size
in rand at random.jl:123
julia
Julia now is using 32% of ram, 2.5 GB of 7.8 GB aviable..
Ubuntu 14.10
Why ? free size of HD is now 30GB
Paul
W dniu wtorek, 6 stycznia 2015 21:04:53 UTC+1 użytkownik paul analyst
napisał:
2.if sum k and l 9 Julia cant work. Is it to big size for hdf5 or for
system (Win7 64 Home Premium) ?
Not sure. It works for me (Kubuntu Linux 14.04).
I checked on Ubuntu :
If sum of k or/and l is more then 7 I have problem with reading
vectors (cols)
samsung2@samsung2:~$ julia
_
_ _ _(_)_ | A fresh approach to technical computing
(_) | (_) (_)| Documentation: http://docs.julialang.org
_ _ _| |_ __ _ | Type help() for help.
| | | | | | |/ _` | |
| | |
| | |_| | | | (_| | | Version 0.3.3 (2014-10-21 20:18 UTC)
_/ |\__'_|_|_|\__'_| | Official http://julialang.org release
|
|__/ | i686-linux-gnu
julia using HDF5
julia hfi=h5open(F_big.h5,w);close(hfi)
julia k,l=8,8;
julia fid = h5open(F_big.h5,r+)
HDF5 data file: F_big.h5
julia fid[mygroup/A]=rand(2)#niepotrzebny
2-element Array{Float64,1}:
0.318459
0.258055
julia g = fid[mygroup]
HDF5 group: /mygroup (file: F_big.h5)
julia dset = d_create(g, F, datatype(Float64),
dataspace(10^k,1*10^l))
HDF5 dataset: /mygroup/F (file: F_big.h5)
julia h5read(F_big.h5,mygroup/F,(:,1))
1x1 Array{Float64,2}:
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
⋮
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
julia close(fid)
julia hfi=h5open(F_big.h5,w);close(hfi)
julia k,l=9,8;
julia fid = h5open(F_big.h5,r+)
HDF5 data file: F_big.h5
julia fid[mygroup/A]=rand(2)#niepotrzebny
2-element Array{Float64,1}:
0.129214
0.4785
julia g = fid[mygroup]
HDF5 group: /mygroup (file: F_big.h5)
julia dset = d_create(g, F, datatype(Float64),
dataspace(10^k,1*10^l))
HDF5 dataset: /mygroup/F (file: F_big.h5)
julia h5read(F_big.h5,mygroup/F,(:,1))
ERROR: invalid Array size
in Array at base.jl:223
in _getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1557
in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1550
in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1620
in h5read at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:602
julia close(fid)
julia h5read(F_big.h5,mygroup/F,(1:2,:))
2x1 Array{Float64,2}:
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 … 0.0 0.0 0.0 0.0 0.0
0.0
0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0
0.0
julia h5read(F_big.h5,mygroup/F,(:,1))
ERROR: invalid Array size
in Array at base.jl:223
in _getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1557
in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1550
in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1620
in h5read at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:602
julia
Paul
Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
Did you try calculating how much memory the right hand side of that assignment
requires? (Remember that sizeof(Float64) == 8).
--Tim
On Wednesday, January 07, 2015 03:25:54 AM paul analyst wrote:
_ _ _(_)_ | A fresh approach to technical computing
(_) | (_) (_)| Documentation: http://docs.julialang.org
_ _ _| |_ __ _ | Type help() for help.
| | | | | | |/ _` | |
| | |
| | |_| | | | (_| | | Version 0.3.3 (2014-10-21 20:18 UTC)
_/ |\__'_|_|_|\__'_| | Official http://julialang.org release
|__/ | i686-linux-gnu
dset[1:10^8,1]=rand(10^8);
julia size(dset)
(10,1000)
julia dset[1:10^9,1]=rand(10^9);
ERROR: invalid Array size
in rand at random.jl:123
julia
Julia now is using 32% of ram, 2.5 GB of 7.8 GB aviable..
Ubuntu 14.10
Why ? free size of HD is now 30GB
Paul
W dniu wtorek, 6 stycznia 2015 21:04:53 UTC+1 użytkownik paul analyst
napisał:
2.if sum k and l 9 Julia cant work. Is it to big size for hdf5 or for
system (Win7 64 Home Premium) ?
Not sure. It works for me (Kubuntu Linux 14.04).
I checked on Ubuntu :
If sum of k or/and l is more then 7 I have problem with reading
vectors (cols)
samsung2@samsung2:~$ julia
_
_ _ _(_)_ | A fresh approach to technical computing
(_) | (_) (_)| Documentation: http://docs.julialang.org
_ _ _| |_ __ _ | Type help() for help.
| | | | | | |/ _` | |
| | |
| | |_| | | | (_| | | Version 0.3.3 (2014-10-21 20:18 UTC)
_/ |\__'_|_|_|\__'_| | Official http://julialang.org release
|__/ | i686-linux-gnu
julia using HDF5
julia hfi=h5open(F_big.h5,w);close(hfi)
julia k,l=8,8;
julia fid = h5open(F_big.h5,r+)
HDF5 data file: F_big.h5
julia fid[mygroup/A]=rand(2)#niepotrzebny
2-element Array{Float64,1}:
0.318459
0.258055
julia g = fid[mygroup]
HDF5 group: /mygroup (file: F_big.h5)
julia dset = d_create(g, F, datatype(Float64), dataspace(10^k,1*10^l))
HDF5 dataset: /mygroup/F (file: F_big.h5)
julia h5read(F_big.h5,mygroup/F,(:,1))
1x1 Array{Float64,2}:
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
⋮
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
julia close(fid)
julia hfi=h5open(F_big.h5,w);close(hfi)
julia k,l=9,8;
julia fid = h5open(F_big.h5,r+)
HDF5 data file: F_big.h5
julia fid[mygroup/A]=rand(2)#niepotrzebny
2-element Array{Float64,1}:
0.129214
0.4785
julia g = fid[mygroup]
HDF5 group: /mygroup (file: F_big.h5)
julia dset = d_create(g, F, datatype(Float64), dataspace(10^k,1*10^l))
HDF5 dataset: /mygroup/F (file: F_big.h5)
julia h5read(F_big.h5,mygroup/F,(:,1))
ERROR: invalid Array size
in Array at base.jl:223
in _getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1557
in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1550
in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1620
in h5read at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:602
julia close(fid)
julia h5read(F_big.h5,mygroup/F,(1:2,:))
2x1 Array{Float64,2}:
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 … 0.0 0.0 0.0 0.0 0.0 0.0
0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0
julia h5read(F_big.h5,mygroup/F,(:,1))
ERROR: invalid Array size
in Array at base.jl:223
in _getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1557
in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1550
in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1620
in h5read at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:602
julia
Paul
Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
2.if sum k and l 9 Julia cant work. Is it to big size for hdf5 or for
system (Win7 64 Home Premium) ?
Not sure. It works for me (Kubuntu Linux 14.04).
I checked on Ubuntu :
If sum of k or/and l is more then 7 I have problem with reading
vectors (cols)
samsung2@samsung2:~$ julia
_
_ _ _(_)_ | A fresh approach to technical computing
(_) | (_) (_)| Documentation: http://docs.julialang.org
_ _ _| |_ __ _ | Type help() for help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 0.3.3 (2014-10-21 20:18 UTC)
_/ |\__'_|_|_|\__'_| | Official http://julialang.org release
|__/ | i686-linux-gnu
julia using HDF5
julia hfi=h5open(F_big.h5,w);close(hfi)
julia k,l=8,8;
julia fid = h5open(F_big.h5,r+)
HDF5 data file: F_big.h5
julia fid[mygroup/A]=rand(2)#niepotrzebny
2-element Array{Float64,1}:
0.318459
0.258055
julia g = fid[mygroup]
HDF5 group: /mygroup (file: F_big.h5)
julia dset = d_create(g, F, datatype(Float64), dataspace(10^k,1*10^l))
HDF5 dataset: /mygroup/F (file: F_big.h5)
julia h5read(F_big.h5,mygroup/F,(:,1))
1x1 Array{Float64,2}:
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
⋮
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
julia close(fid)
julia hfi=h5open(F_big.h5,w);close(hfi)
julia k,l=9,8;
julia fid = h5open(F_big.h5,r+)
HDF5 data file: F_big.h5
julia fid[mygroup/A]=rand(2)#niepotrzebny
2-element Array{Float64,1}:
0.129214
0.4785
julia g = fid[mygroup]
HDF5 group: /mygroup (file: F_big.h5)
julia dset = d_create(g, F, datatype(Float64), dataspace(10^k,1*10^l))
HDF5 dataset: /mygroup/F (file: F_big.h5)
julia h5read(F_big.h5,mygroup/F,(:,1))
ERROR: invalid Array size
in Array at base.jl:223
in _getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1557
in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1550
in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1620
in h5read at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:602
julia close(fid)
julia h5read(F_big.h5,mygroup/F,(1:2,:))
2x1 Array{Float64,2}:
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 … 0.0 0.0 0.0 0.0 0.0 0.0
0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0
julia h5read(F_big.h5,mygroup/F,(:,1))
ERROR: invalid Array size
in Array at base.jl:223
in _getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1557
in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1550
in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1620
in h5read at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:602
julia
Paul
Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
On Monday, January 05, 2015 03:17:12 PM Paul Analyst wrote: Thx, Tim, I have solution but is 2 questions: 1. Whay must be line: /fid[mygroup/A]=rand(2)/ ? That line just means 'create a variable called A inside a group called mygroup, and assign it a value of rand(2)'. If you don't need that variable, you don't need that line. You also don't have to create a group called mygroup, if you prefer you can store everything in the top level of the file. 2.if sum k and l 9 Julia cant work. Is it to big size for hdf5 or for system (Win7 64 Home Premium) ? Not sure. It works for me (Kubuntu Linux 14.04). --Tim using HDF5 hfi=h5open(bigfile.h5,w) close(hfi) k,l=6,3; fid = h5open(bigfile.h5,r+) fid[mygroup/A]=rand(2) g = fid[mygroup] dset = d_create(g, F, datatype(Float64), dataspace(10^k,10^l)) dset[:,1] = rand(10^k) h5read(bigfile.h5,mygroup/F,(:,1)) close(fid) h5read(bigfile.h5,mygroup/F,(:,1:2)) Is OK but if k,l=6,4; julia close(fid) HDF5-DIAG: Error detected in HDF5 (1.8.13) thread 0: #000: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 2070 in H5Fclose(): decrementing file ID failed major: Object atom minor: Unable to close file #001: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5I.c line 1464 in H5I_dec_app_ref(): can't decrement ID ref c ount major: Object atom minor: Unable to decrement reference count #002: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 1847 in H5F_close(): can't close file major: File accessibilty minor: Unable to close file #003: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 2009 in H5F_try_close(): problems closing file major: File accessibilty minor: Unable to close file #004: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 1161 in H5F_dest(): low level truncate failed major: File accessibilty minor: Write failed #005: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5FD.c line 1895 in H5FD_truncate(): driver truncate request f ailed major: Virtual File Layer minor: Can't update object #006: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5FDsec2.c line 900 in H5FD_sec2_truncate(): unable to extend file properly major: Low-level I/O minor: Seek failed ERROR: Error closing file in h5f_close at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:1924 julia h5read(bigfile.h5,mygroup/F,(:,1:2)) HDF5-DIAG: Error detected in HDF5 (1.8.13) thread 0: #000: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 1594 in H5Fopen(): unable to open file major: File accessibilty minor: Unable to open file #001: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 1385 in H5F_open(): unable to read superblock major: File accessibilty minor: Read failed #002: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5Fsuper.c line 353 in H5F_super_read(): unable to load superb lock major: Object cache minor: Unable to protect metadata #003: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5AC.c line 1323 in H5AC_protect(): H5C_protect() failed. major: Object cache minor: Unable to protect metadata #004: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5C.c line 3574 in H5C_protect(): can't load entry major: Object cache minor: Unable to load metadata into cache #005: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5C.c line 7954 in H5C_load_entry(): unable to load entry major: Object cache minor: Unable to load metadata into cache #006: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5Fsuper_cache.c line 471 in H5F_sblock_load(): truncated file : eof = 8002864, sblock-base_addr = 0, stored_eoa = 8002864 major: File accessibilty minor: File has been truncated ERROR: Error opening file bigfile.h5 in h5f_open at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:2023 in h5open at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:554 Paul W dniu 2015-01-04 o 18:37, Tim Holy pisze: Do note there are two additional pages of documentation in the doc/ folder. --Tim On Sunday, January 04, 2015 06:59:53 AM paul analyst wrote: Of course, first I read :) Is there about reading range array. I need to save a range of In analogy to. A = reshape (1: 120, 15, 8) h5write (/ tmp / test2.h5, mygroup2 / A, A) data = h5read (/ tmp / test2.h5, mygroup2 / A (2: 3: 15: 3: 5)) Paul W dniu niedziela, 4 stycznia 2015 14:14:03 UTC+1 użytkownik Tim Holy napisał: If I understand correctly, then yes, that's possible. See the HDF5 docs. --Tim On Sunday, January 04, 2015 04:25:13 AM paul analyst wrote: How to overwrite to an existing file, only range of data? In HDF5 can do this? I have an array of zeros 10 x 10 I need an existing file owerwrite range rand (5x5), for example. Existingfile [2: 7.3: 8] Paul
Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
On Monday, January 05, 2015 09:52:28 PM Paul Analyst wrote: dset = d_create( F, datatype(Float64), dataspace(10,10))/ You're missing the `fid`. You have to tell it where (which file, or which group) you want to create that dataset. --Tim
Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
Tim , thx for hints, but do not work without this line #fid[mygroup/A]=rand(2) becouse I vave not g , and nothing to declare in this line /dset = d_create(g, F, datatype(Float64), dataspace(10,10))/ / dset = d_create( F, datatype(Float64), dataspace(10,10))/ do not work ... Paul using HDF5 hfi=h5open(test.h5,w);close(hfi) fid = h5open(test.h5,r+) #fid[mygroup/A]=rand(2) g = fid[mygroup] dset = d_create(g, F, datatype(Float64), dataspace(10,10)) dset[:,1] = rand(10) h5read(test.h5,mygroup/F,(:,1)) close(fid) W dniu 2015-01-05 o 15:39, Tim Holy pisze: On Monday, January 05, 2015 03:17:12 PM Paul Analyst wrote: Thx, Tim, I have solution but is 2 questions: 1. Whay must be line: /fid[mygroup/A]=rand(2)/ ? That line just means 'create a variable called A inside a group called mygroup, and assign it a value of rand(2)'. If you don't need that variable, you don't need that line. You also don't have to create a group called mygroup, if you prefer you can store everything in the top level of the file. 2.if sum k and l 9 Julia cant work. Is it to big size for hdf5 or for system (Win7 64 Home Premium) ? Not sure. It works for me (Kubuntu Linux 14.04). --Tim using HDF5 hfi=h5open(bigfile.h5,w) close(hfi) k,l=6,3; fid = h5open(bigfile.h5,r+) fid[mygroup/A]=rand(2) g = fid[mygroup] dset = d_create(g, F, datatype(Float64), dataspace(10^k,10^l)) dset[:,1] = rand(10^k) h5read(bigfile.h5,mygroup/F,(:,1)) close(fid) h5read(bigfile.h5,mygroup/F,(:,1:2)) Is OK but if k,l=6,4; julia close(fid) HDF5-DIAG: Error detected in HDF5 (1.8.13) thread 0: #000: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 2070 in H5Fclose(): decrementing file ID failed major: Object atom minor: Unable to close file #001: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5I.c line 1464 in H5I_dec_app_ref(): can't decrement ID ref c ount major: Object atom minor: Unable to decrement reference count #002: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 1847 in H5F_close(): can't close file major: File accessibilty minor: Unable to close file #003: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 2009 in H5F_try_close(): problems closing file major: File accessibilty minor: Unable to close file #004: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 1161 in H5F_dest(): low level truncate failed major: File accessibilty minor: Write failed #005: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5FD.c line 1895 in H5FD_truncate(): driver truncate request f ailed major: Virtual File Layer minor: Can't update object #006: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5FDsec2.c line 900 in H5FD_sec2_truncate(): unable to extend file properly major: Low-level I/O minor: Seek failed ERROR: Error closing file in h5f_close at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:1924 julia h5read(bigfile.h5,mygroup/F,(:,1:2)) HDF5-DIAG: Error detected in HDF5 (1.8.13) thread 0: #000: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 1594 in H5Fopen(): unable to open file major: File accessibilty minor: Unable to open file #001: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 1385 in H5F_open(): unable to read superblock major: File accessibilty minor: Read failed #002: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5Fsuper.c line 353 in H5F_super_read(): unable to load superb lock major: Object cache minor: Unable to protect metadata #003: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5AC.c line 1323 in H5AC_protect(): H5C_protect() failed. major: Object cache minor: Unable to protect metadata #004: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5C.c line 3574 in H5C_protect(): can't load entry major: Object cache minor: Unable to load metadata into cache #005: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5C.c line 7954 in H5C_load_entry(): unable to load entry major: Object cache minor: Unable to load metadata into cache #006: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5Fsuper_cache.c line 471 in H5F_sblock_load(): truncated file : eof = 8002864, sblock-base_addr = 0, stored_eoa = 8002864 major: File accessibilty minor: File has been truncated ERROR: Error opening file bigfile.h5 in h5f_open at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:2023 in h5open at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:554 Paul W dniu 2015-01-04 o 18:37, Tim Holy pisze: Do note there are two additional pages of documentation in the doc/ folder. --Tim On Sunday, January 04, 2015 06:59:53 AM paul analyst wrote: Of course, first I read :) Is there about reading range array. I need to save a range of In analogy to. A = reshape (1: 120, 15, 8) h5write (/ tmp / test2.h5, mygroup2 / A, A) data = h5read (/ tmp / test2.h5, mygroup2 / A (2: 3: 15: 3: 5)) Paul W dniu niedziela, 4 stycznia 2015
[julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
How to overwrite to an existing file, only range of data? In HDF5 can do this? I have an array of zeros 10 x 10 I need an existing file owerwrite range rand (5x5), for example. Existingfile [2: 7.3: 8] Paul
Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
If I understand correctly, then yes, that's possible. See the HDF5 docs. --Tim On Sunday, January 04, 2015 04:25:13 AM paul analyst wrote: How to overwrite to an existing file, only range of data? In HDF5 can do this? I have an array of zeros 10 x 10 I need an existing file owerwrite range rand (5x5), for example. Existingfile [2: 7.3: 8] Paul
Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
Of course, first I read :) Is there about reading range array. I need to save a range of In analogy to. A = reshape (1: 120, 15, 8) h5write (/ tmp / test2.h5, mygroup2 / A, A) data = h5read (/ tmp / test2.h5, mygroup2 / A (2: 3: 15: 3: 5)) Paul W dniu niedziela, 4 stycznia 2015 14:14:03 UTC+1 użytkownik Tim Holy napisał: If I understand correctly, then yes, that's possible. See the HDF5 docs. --Tim On Sunday, January 04, 2015 04:25:13 AM paul analyst wrote: How to overwrite to an existing file, only range of data? In HDF5 can do this? I have an array of zeros 10 x 10 I need an existing file owerwrite range rand (5x5), for example. Existingfile [2: 7.3: 8] Paul
Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
You can also work directly with the HDF5 file as an array object...
using HDF5
hfi=h5open(myfile.h5,w); # create the file
close(hfi)
A = reshape(1: 120, 15, 8);
hfi = h5open(myfile.h5,r+) # read/write access
hfi[mygroup/A] = A
15x8 Array{Int64,2}:
1 16 31 46 61 76 91 106
...
14 29 44 59 74 89 104 119
15 30 45 60 75 90 105 120
data = hfi[mygroup/A][2:3:15,3:5]
5x3 Array{Int64,2}:
32 47 62
35 50 65
38 53 68
41 56 71
44 59 74
hfi[mygroup/A][2:3,3:5]
2x3 Array{Int64,2}:
32 47 62
33 48 63
hfi[mygroup/A][2:3,3:5]=[-1 -2 -3; -4 -5 -6]
2x3 Array{Int64,2}:
-1 -2 -3
-4 -5 -6
hfi[mygroup/A][1:5,:]
5x8 Array{Int64,2}:
1 16 31 46 61 76 91 106
2 17 -1 -2 -3 77 92 107
3 18 -4 -5 -6 78 93 108
4 19 34 49 64 79 94 109
5 20 35 50 65 80 95 110
foo = hfi[mygroup/A]
HDF5 dataset: /mygroup/A (file: myfile.h5)
foo[1:5,:]
5x8 Array{Int64,2}:
1 16 31 46 61 76 91 106
2 17 -1 -2 -3 77 92 107
3 18 -4 -5 -6 78 93 108
4 19 34 49 64 79 94 109
5 20 35 50 65 80 95 110
close(hfi)
HDF5 is awesome!
On Sun, Jan 4, 2015 at 4:59 PM, paul analyst paul.anal...@mail.com wrote:
Of course, first I read :)
Is there about reading range array. I need to save a range of In analogy
to.
A = reshape (1: 120, 15, 8)
h5write (/ tmp / test2.h5, mygroup2 / A, A)
data = h5read (/ tmp / test2.h5, mygroup2 / A (2: 3: 15: 3: 5))
Paul
W dniu niedziela, 4 stycznia 2015 14:14:03 UTC+1 użytkownik Tim Holy
napisał:
If I understand correctly, then yes, that's possible. See the HDF5 docs.
--Tim
On Sunday, January 04, 2015 04:25:13 AM paul analyst wrote:
How to overwrite to an existing file, only range of data?
In HDF5 can do this?
I have an array of zeros 10 x 10
I need an existing file owerwrite range rand (5x5), for example.
Existingfile [2: 7.3: 8]
Paul
Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
See the part about ...incrementally save to very large ... --Tim On Sunday, January 04, 2015 06:59:53 AM paul analyst wrote: Of course, first I read :) Is there about reading range array. I need to save a range of In analogy to. A = reshape (1: 120, 15, 8) h5write (/ tmp / test2.h5, mygroup2 / A, A) data = h5read (/ tmp / test2.h5, mygroup2 / A (2: 3: 15: 3: 5)) Paul W dniu niedziela, 4 stycznia 2015 14:14:03 UTC+1 użytkownik Tim Holy napisał: If I understand correctly, then yes, that's possible. See the HDF5 docs. --Tim On Sunday, January 04, 2015 04:25:13 AM paul analyst wrote: How to overwrite to an existing file, only range of data? In HDF5 can do this? I have an array of zeros 10 x 10 I need an existing file owerwrite range rand (5x5), for example. Existingfile [2: 7.3: 8] Paul
Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
Do note there are two additional pages of documentation in the doc/ folder. --Tim On Sunday, January 04, 2015 06:59:53 AM paul analyst wrote: Of course, first I read :) Is there about reading range array. I need to save a range of In analogy to. A = reshape (1: 120, 15, 8) h5write (/ tmp / test2.h5, mygroup2 / A, A) data = h5read (/ tmp / test2.h5, mygroup2 / A (2: 3: 15: 3: 5)) Paul W dniu niedziela, 4 stycznia 2015 14:14:03 UTC+1 użytkownik Tim Holy napisał: If I understand correctly, then yes, that's possible. See the HDF5 docs. --Tim On Sunday, January 04, 2015 04:25:13 AM paul analyst wrote: How to overwrite to an existing file, only range of data? In HDF5 can do this? I have an array of zeros 10 x 10 I need an existing file owerwrite range rand (5x5), for example. Existingfile [2: 7.3: 8] Paul
Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
Thx, form me this moment *hfi[mygroup/A] = A* was not too clear Paul W dniu niedziela, 4 stycznia 2015 18:37:30 UTC+1 użytkownik Tim Holy napisał: See the part about ...incrementally save to very large ... --Tim On Sunday, January 04, 2015 06:59:53 AM paul analyst wrote: Of course, first I read :) Is there about reading range array. I need to save a range of In analogy to. A = reshape (1: 120, 15, 8) h5write (/ tmp / test2.h5, mygroup2 / A, A) data = h5read (/ tmp / test2.h5, mygroup2 / A (2: 3: 15: 3: 5)) Paul W dniu niedziela, 4 stycznia 2015 14:14:03 UTC+1 użytkownik Tim Holy napisał: If I understand correctly, then yes, that's possible. See the HDF5 docs. --Tim On Sunday, January 04, 2015 04:25:13 AM paul analyst wrote: How to overwrite to an existing file, only range of data? In HDF5 can do this? I have an array of zeros 10 x 10 I need an existing file owerwrite range rand (5x5), for example. Existingfile [2: 7.3: 8] Paul
Re: [julia-users] How to overwrite to an existing file, only range of data? HDF5 can do this ?
Big Thx for big lesson, is no too dark now
Paul
W dniu niedziela, 4 stycznia 2015 17:58:28 UTC+1 użytkownik Adrian
Cuthbertson napisał:
You can also work directly with the HDF5 file as an array object...
using HDF5
hfi=h5open(myfile.h5,w); # create the file
close(hfi)
A = reshape(1: 120, 15, 8);
hfi = h5open(myfile.h5,r+) # read/write access
hfi[mygroup/A] = A
15x8 Array{Int64,2}:
1 16 31 46 61 76 91 106
...
14 29 44 59 74 89 104 119
15 30 45 60 75 90 105 120
data = hfi[mygroup/A][2:3:15,3:5]
5x3 Array{Int64,2}:
32 47 62
35 50 65
38 53 68
41 56 71
44 59 74
hfi[mygroup/A][2:3,3:5]
2x3 Array{Int64,2}:
32 47 62
33 48 63
hfi[mygroup/A][2:3,3:5]=[-1 -2 -3; -4 -5 -6]
2x3 Array{Int64,2}:
-1 -2 -3
-4 -5 -6
hfi[mygroup/A][1:5,:]
5x8 Array{Int64,2}:
1 16 31 46 61 76 91 106
2 17 -1 -2 -3 77 92 107
3 18 -4 -5 -6 78 93 108
4 19 34 49 64 79 94 109
5 20 35 50 65 80 95 110
foo = hfi[mygroup/A]
HDF5 dataset: /mygroup/A (file: myfile.h5)
foo[1:5,:]
5x8 Array{Int64,2}:
1 16 31 46 61 76 91 106
2 17 -1 -2 -3 77 92 107
3 18 -4 -5 -6 78 93 108
4 19 34 49 64 79 94 109
5 20 35 50 65 80 95 110
close(hfi)
HDF5 is awesome!
On Sun, Jan 4, 2015 at 4:59 PM, paul analyst paul.a...@mail.com
javascript: wrote:
Of course, first I read :)
Is there about reading range array. I need to save a range of In analogy
to.
A = reshape (1: 120, 15, 8)
h5write (/ tmp / test2.h5, mygroup2 / A, A)
data = h5read (/ tmp / test2.h5, mygroup2 / A (2: 3: 15: 3: 5))
Paul
W dniu niedziela, 4 stycznia 2015 14:14:03 UTC+1 użytkownik Tim Holy
napisał:
If I understand correctly, then yes, that's possible. See the HDF5 docs.
--Tim
On Sunday, January 04, 2015 04:25:13 AM paul analyst wrote:
How to overwrite to an existing file, only range of data?
In HDF5 can do this?
I have an array of zeros 10 x 10
I need an existing file owerwrite range rand (5x5), for example.
Existingfile [2: 7.3: 8]
Paul
