Re: Running queries on large data structure

2006-08-04 Thread Christoph Haas
On Friday 04 August 2006 15:23, Amit Khemka wrote: > Though some sugggested maintaining data in some XML structures, I was > wondering that > if you considered using some native XML database like BDB XML. > > 1. It allows you to retain hierarchical structure of data. > 2. It also has support for pr

Re: Running queries on large data structure

2006-08-04 Thread Amit Khemka
On 8/3/06, Christoph Haas <[EMAIL PROTECTED]> wrote: > On Wednesday 02 August 2006 22:24, Christoph Haas wrote: > > I have written an application in Perl some time ago (I was young and > > needed the money) that parses multiple large text files containing > > nested data structures and allows the u

Re: Running queries on large data structure

2006-08-04 Thread Christoph Haas
On Friday 04 August 2006 01:39, John Machin wrote: > Christoph Haas wrote: > > I assume that XQuery can't to weird queries like IP ranges, or can it? > > That's twice now you've indicated that IP ranges are causing you some > problems. What's the big deal? > > (a) If you don't have a specialised da

Re: Running queries on large data structure

2006-08-04 Thread H J van Rooyen
"Christoph Haas" <[EMAIL PROTECTED]> wrote" | On Wednesday 02 August 2006 22:24, Christoph Haas wrote: | > I have written an application in Perl some time ago (I was young and | > needed the money) that parses multiple large text files containing | > nested data structures and allows the user to

Re: Running queries on large data structure

2006-08-03 Thread John Machin
Christoph Haas wrote: > I assume that XQuery can't to weird queries like IP ranges, or can it? That's twice now you've indicated that IP ranges are causing you some problems. What's the big deal? (a) If you don't have a specialised data type, just parse an IP address into an unsigned 32-bit int

Re: Running queries on large data structure

2006-08-03 Thread Paul Rubin
Christoph Haas <[EMAIL PROTECTED]> writes: > So the question is: would you rather force the data into a relational > database and write object-relational wrappers around it? Or would you > pickle it and load it later and work on the data? The latter application > is currently a CGI. I'm open to

Re: Running queries on large data structure

2006-08-03 Thread jay graves
Christoph Haas wrote: > On Thursday 03 August 2006 17:40, jay graves wrote: > > How hard would it be to create this nested structure? > Not hard. Instead of doing "INSERT INTO" I would add values to a dictionary > or list. That's even simpler. > > I've found > > pickling really large data structure

Re: Running queries on large data structure

2006-08-03 Thread Christoph Haas
On Thursday 03 August 2006 19:54, hiaips wrote: > Well, if you format the data as a Python dictionary and give the data > file a .py extension, it becomes a Python module that you can load and > reload dynamically. That's sort of what I was thinking. Ah, okay. You mean like just using expr() on th

Re: Running queries on large data structure

2006-08-03 Thread hiaips
Christoph, Well, if you format the data as a Python dictionary and give the data file a .py extension, it becomes a Python module that you can load and reload dynamically. That's sort of what I was thinking. --Dave -- http://mail.python.org/mailman/listinfo/python-list

Re: Running queries on large data structure

2006-08-03 Thread Christoph Haas
On Thursday 03 August 2006 17:45, hiaips wrote: > Christoph, > > Several possibilities come to mind... > > From your description, maybe something like Postgres, MySql, or sqlite > would not be the best option. (However, I'm wondering what your query > requirements are Imagine this example firewall

Re: Running queries on large data structure

2006-08-03 Thread Christoph Haas
On Thursday 03 August 2006 17:40, jay graves wrote: > Christoph Haas wrote: > > The situation is that I have input data that take ~1 minute to parse > > while the users need to run queries on that within seconds. I can > > think of two ways: > > What is the raw data size? The file containing the o

Re: Running queries on large data structure

2006-08-03 Thread hiaips
Christoph, Several possibilities come to mind... >From your description, maybe something like Postgres, MySql, or sqlite would not be the best option. (However, I'm wondering what your query requirements are -- for example, if you really need the power of SQL, maybe you should just bite the bulle

Re: Running queries on large data structure

2006-08-03 Thread jay graves
Christoph Haas wrote: > On Wednesday 02 August 2006 22:24, Christoph Haas wrote: > I suppose my former posting was too long and concrete. So allow me to try > it in a different way. :) OK. I'll bite. > The situation is that I have input data that take ~1 minute to parse while > the users need to

Re: Running queries on large data structure

2006-08-03 Thread Christoph Haas
On Wednesday 02 August 2006 22:24, Christoph Haas wrote: > I have written an application in Perl some time ago (I was young and > needed the money) that parses multiple large text files containing > nested data structures and allows the user to run quick queries on the > data. [...] I suppose my f

Running queries on large data structure

2006-08-02 Thread Christoph Haas
Hi, list... I have written an application in Perl some time ago (I was young and needed the money) that parses multiple large text files containing nested data structures and allows the user to run quick queries on the data. (For the firewall admins among you: it's a parser and web-based query t