Re: [W3af-develop] core/data/db/history.py and .trace files

2011-02-09 Thread Andres Riancho
Martin, On Wed, Feb 9, 2011 at 4:39 PM, Martin Holst Swende mar...@swende.se wrote: On 02/09/2011 02:31 AM, Steve Pinkham wrote: On 02/08/2011 08:08 PM, Andres Riancho wrote: Steve, noSQL servers are usually fast because they are in-memory systems. sqlite can be used in that mode also if

Re: [W3af-develop] core/data/db/history.py and .trace files

2011-02-09 Thread Martin Holst Swende
On 02/09/2011 02:31 AM, Steve Pinkham wrote: On 02/08/2011 08:08 PM, Andres Riancho wrote: Steve, noSQL servers are usually fast because they are in-memory systems. sqlite can be used in that mode also if you like. mongodb is not an in-memory db! In practice, it is. It stores all indexes in

Re: [W3af-develop] core/data/db/history.py and .trace files

2011-02-09 Thread Steve Pinkham
The only nosql databases I've used so far are key/value oriented(mostly riak). You've convinced me that there might be some benefits to documented oriented storage I haven't considered, so thank you for that. On 02/09/2011 02:39 PM, Martin Holst Swende wrote: I also disagree that they are

Re: [W3af-develop] core/data/db/history.py and .trace files

2011-02-08 Thread Leandro Reox
A very common no-sql document related database are IBM Lotus Notes db ( .nsf ) . These dbs are usually used for team room applications or for storing transaccional data between IBM mainframe and other platforms like HP-NON STOP most seen on Banking environments. Inside IBM doors these nsf

Re: [W3af-develop] core/data/db/history.py and .trace files

2011-02-08 Thread Andres Riancho
Lean, Do you know if the format is open? Do we have a Python binding to write to them? Any clue on how they scale in performance when saving thousands of registries? Regards, On Tue, Feb 8, 2011 at 4:42 PM, Leandro Reox leandro.r...@gmail.com wrote: A very common no-sql document related

Re: [W3af-develop] core/data/db/history.py and .trace files

2011-02-08 Thread Leandro Reox
Andres, Sadly the format is not open. Theres a few ways to write and retrieve data via Python to this kind of databases (like jython + and the notes.jar classes - notessql drivers on win, etc). Regarding performance a 100.000 records with attachments databases are very common on IBM

Re: [W3af-develop] core/data/db/history.py and .trace files

2011-02-08 Thread Andres Riancho
The only issue with mongodb is that its a daemon, I'm not sure if we want to have mongod as a w3af dependency. It could complicate packaging and install process. Regards, -- Andres Riancho El feb 8, 2011 6:39 p.m., Leandro Reox leandro.r...@gmail.com escribió: Here is a living proof of MongoDB

Re: [W3af-develop] core/data/db/history.py and .trace files

2011-02-08 Thread Andres Riancho
Steve, On Tue, Feb 8, 2011 at 8:45 PM, Steve Pinkham steve.pink...@gmail.com wrote: On 02/03/2011 12:04 PM, Andres Riancho wrote: Do we know about any noSQL database that's file based like sqlite? Maybe we could use this small rewrite to compare the performance of those backends. Regards,

Re: [W3af-develop] core/data/db/history.py and .trace files

2011-02-08 Thread Andres Riancho
Steve, On Tue, Feb 8, 2011 at 9:07 PM, Andres Riancho andres.rian...@gmail.com wrote: Berkeley DB is what was used as a file based key-value store before sqlite, but has no major benefits in most uses over sqlite which is why it didn't spring to mind. ;-) If you have many threads writing

Re: [W3af-develop] core/data/db/history.py and .trace files

2011-02-08 Thread Steve Pinkham
On 02/08/2011 07:07 PM, Andres Riancho wrote: Steve, On Tue, Feb 8, 2011 at 8:45 PM, Steve Pinkham steve.pink...@gmail.com wrote: On 02/03/2011 12:04 PM, Andres Riancho wrote: Do we know about any noSQL database that's file based like sqlite? Maybe we could use this small rewrite to compare

Re: [W3af-develop] core/data/db/history.py and .trace files

2011-02-08 Thread Steve Pinkham
On 02/08/2011 08:08 PM, Andres Riancho wrote: Steve, noSQL servers are usually fast because they are in-memory systems. sqlite can be used in that mode also if you like. mongodb is not an in-memory db! In practice, it is. It stores all indexes in memory and uses memory mapped files. It

Re: [W3af-develop] core/data/db/history.py and .trace files

2011-02-04 Thread Taras
Andres, I didn't use noSQL databases but it can be interesting research =) But for the first lets simply fix this bug with files. Do we know about any noSQL database that's file based like sqlite? Maybe we could use this small rewrite to compare the performance of those backends. Regards,

Re: [W3af-develop] core/data/db/history.py and .trace files

2011-02-04 Thread Andres Riancho
Agreed, we need fixed. Lots of bug reports about it in Trac. -- Andres Riancho El feb 4, 2011 6:51 p.m., Taras ox...@oxdef.info escribió: Andres, I didn't use noSQL databases but it can be interesting research =) But for the first lets simply fix this bug with files. Do we know about any

[W3af-develop] core/data/db/history.py and .trace files

2011-01-31 Thread Andres Riancho
Oxdef, We've been getting a lot [0] of automatic bug reports that look like this: w3afException: An internal error ocurred while searching for id 246. Original exception: [Errno 2] No such file or directory: '/root/.w3af/sessions/some-site.com-2011-Jan-31_12-56-05/246.trace' The only

Re: [W3af-develop] core/data/db/history.py and .trace files

2011-01-31 Thread Andres Riancho
Taras, On Mon, Jan 31, 2011 at 6:08 PM, Taras ox...@oxdef.info wrote: Andres, Oh, it is bad and good bug in same time =) Bad side is that bug is not trivial to reproduce and it occurs suddenly. But it looks like I found the problem. It is because of mistiming of db file and transactions