from:"Chao YUE"

[Pytables-users] should I use pytables?

2013-08-07 Thread Chao YUE

Dear all,

I have a hierachical nested python dictionaries with the end of the branch
as either pandas dataframe, or np.ndarray or list or plain scalars.

let's say the different levels of keys are:

1st level: ['top1', 'top2', 'top3']
2nd level: ['mid1','mid2','mid3']
3rd level: ['bot1','bot2','bot3','bot4']

I think I am looking for some data strucuture that allow easy retrieving of
the data at different levels as dictionaries (I cannot think out something
better yet).

for example: data.ix['top1',:,'bot1'] will have keys only at the middle
levels.

I have a quick look of pytables document but not very sure, should I use
pytables for this purpose?

thanks a lot for any idea.

cheers,

Chao

-- 
***
Chao YUE
Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
UMR 1572 CEA-CNRS-UVSQ
Batiment 712 - Pe 119
91191 GIF Sur YVETTE Cedex
Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16

--
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with 2% overhead. 
Download for free and get started troubleshooting in minutes. 
http://pubads.g.doubleclick.net/gampad/clk?id=48897031iu=/4140/ostg.clktrk___
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users

Re: [Pytables-users] should I use pytables?

2013-08-07 Thread Chao YUE

Thanks Anthony, I think I will give a try, apprently at some stage I would
like to flush the data into disk :p

cheers,

Chao

On Wed, Aug 7, 2013 at 6:44 PM, Anthony Scopatz scop...@gmail.com wrote:

 On Wed, Aug 7, 2013 at 5:44 AM, Chao YUE chaoyue...@gmail.com wrote:

 Dear all,

 I have a hierachical nested python dictionaries with the end of the
 branch as either pandas dataframe, or np.ndarray or list or plain scalars.

 let's say the different levels of keys are:

 1st level: ['top1', 'top2', 'top3']
 2nd level: ['mid1','mid2','mid3']
 3rd level: ['bot1','bot2','bot3','bot4']

 I think I am looking for some data strucuture that allow easy retrieving
 of the data at different levels as dictionaries (I cannot think out
 something better yet).

 for example: data.ix['top1',:,'bot1'] will have keys only at the middle
 levels.

 I have a quick look of pytables document but not very sure, should I use
 pytables for this purpose?


 Hello Chao,

 If you are only ever going to use this data structure in memory, you
 shouldn't use pytables.  If you are going to persist this information to
 disk than pytables is a great choice!  Every dictionary will become a group
 and every leaf data structure will become an Array or a Table.

 Be Well
 Anthony



 thanks a lot for any idea.

 cheers,

 Chao

 --

 ***
 Chao YUE
 Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
 UMR 1572 CEA-CNRS-UVSQ
 Batiment 712 - Pe 119
 91191 GIF Sur YVETTE Cedex
 Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16

 


 --
 Get 100% visibility into Java/.NET code with AppDynamics Lite!
 It's a free troubleshooting tool designed for production.
 Get down to code-level detail for bottlenecks, with 2% overhead.
 Download for free and get started troubleshooting in minutes.

 http://pubads.g.doubleclick.net/gampad/clk?id=48897031iu=/4140/ostg.clktrk
 ___
 Pytables-users mailing list
 Pytables-users@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/pytables-users




 --
 Get 100% visibility into Java/.NET code with AppDynamics Lite!
 It's a free troubleshooting tool designed for production.
 Get down to code-level detail for bottlenecks, with 2% overhead.
 Download for free and get started troubleshooting in minutes.
 http://pubads.g.doubleclick.net/gampad/clk?id=48897031iu=/4140/ostg.clktrk
 ___
 Pytables-users mailing list
 Pytables-users@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/pytables-users




-- 
***
Chao YUE
Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
UMR 1572 CEA-CNRS-UVSQ
Batiment 712 - Pe 119
91191 GIF Sur YVETTE Cedex
Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16

--
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with 2% overhead. 
Download for free and get started troubleshooting in minutes. 
http://pubads.g.doubleclick.net/gampad/clk?id=48897031iu=/4140/ostg.clktrk___
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users

[Pytables-users] Can't open the group: '/'. 'File' object has no attribute 'root'

2012-06-02 Thread Chao YUE

Dear all,

I tried to use pytalbes to read a hdf file, but I got error:
I searched a little bit online, there might be cases you have more than 2
file handlers for the same file and they are opened for both read and
write, you'll probably have this error.
But it's not my case that I open it only for the first time.  From the
error message, it seems that there is no root group.

In [1]: h5file=tables.openFile('LISOTD_HRMC_V2.3.2011.hdf','r')
---
HDF5ExtError  Traceback (most recent call last)
/home/chaoyue/Downloads/LISOTD/ipython-input-1-b53d861308cf in module()
 1 h5file=tables.openFile('LISOTD_HRMC_V2.3.2011.hdf','r')

/usr/local/lib/python2.7/dist-packages/tables/file.pyc in
openFile(filename, mode, title, rootUEP, filters, **kwargs)
256 return filehandle
257 # Finally, create the File instance, and return it

-- 258 return File(filename, mode, title, rootUEP, filters, **kwargs)
259
260

/usr/local/lib/python2.7/dist-packages/tables/file.pyc in __init__(self,
filename, mode, title, rootUEP, filters, **kwargs)
565
566 # Get the root group from this file

-- 567 self.root = root = self.__getRootGroup(rootUEP, title,
filters)
568 # Complete the creation of the root node

569 # (see the explanation in ``RootGroup.__init__()``.


/usr/local/lib/python2.7/dist-packages/tables/file.pyc in
__getRootGroup(self, rootUEP, title, filters)
614 # Create new attributes for the root Group instance and

615 # create the object tree

-- 616 return RootGroup(self, rootUEP, title=title, new=new,
filters=filters)
617
618

/usr/local/lib/python2.7/dist-packages/tables/group.pyc in __init__(self,
ptFile, name, title, new, filters)
   1155 self._g_new(ptFile, name, init=True)
   1156 #   Open the node and get its object ID.

- 1157 self._v_objectID = self._g_open()
   1158
   1159 # Set disk attributes and read children names.


/usr/local/lib/python2.7/dist-packages/tables/hdf5Extension.so in
tables.hdf5Extension.Group._g_open (tables/hdf5Extension.c:5521)()

HDF5ExtError: Can't open the group: '/'.

If I open the file with this command, it gave no error:

In [2]: h5file=tables.openFile('LISOTD_HRMC_V2.3.2011.hdf',mode='r')

But when I try to print the file information, I get:

In [3]: print h5file
Exception tables.exceptions.HDF5ExtError: HDF5ExtError('Problems closing
the Group /',) in  ignored
---
AttributeErrorTraceback (most recent call last)
/home/chaoyue/Downloads/LISOTD/ipython-input-3-64c76de88957 in module()
 1 print h5file

/usr/local/lib/python2.7/dist-packages/tables/file.pyc in __str__(self)
   2197 # Print all the nodes (Group and Leaf objects) on object
tree

   2198 date =
time.asctime(time.localtime(os.stat(self.filename)[8]))
- 2199 astring =  self.filename + ' (File) ' + repr(self.title) +
'\n'
   2200 # astring += 'rootUEP :=' + repr(self.rootUEP) + '; '

   2201 # astring += 'format_version := ' + self.format_version +
'\n'


/usr/local/lib/python2.7/dist-packages/tables/file.pyc in _gettitle(self)
474
475 def _gettitle(self):
-- 476 return self.root._v_title
477 def _settitle(self, title):
478 self.root._v_title = title

AttributeError: 'File' object has no attribute 'root'

I have not too much experience handling HDF data but it's the second time I
have this problem.
In both cases the data are downloaded from official release of research
data so I think it's unlikely that the data itself are badly produced.
But if anyone has any interest trying to have a look of the issue, the data
is at:
ftp://ghrc.nsstc.nasa.gov/pub/lis/climatology/HRMC/data/

The ftp is anonymous and the data released by NASA.

thanks for any help in advance,

best regards,

Chao

-- 
***
Chao YUE
Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
UMR 1572 CEA-CNRS-UVSQ
Batiment 712 - Pe 119
91191 GIF Sur YVETTE Cedex
Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables

[Pytables-users] should I use pytables?

Re: [Pytables-users] should I use pytables?

[Pytables-users] Can't open the group: '/'. 'File' object has no attribute 'root'

3 matches

Site Navigation

Mail list logo

Footer information