Re: [W3af-develop] 2 ideas
On 02/08/2011 05:15 AM, Taras wrote: Hi, all! There are 2 ideas: 1. What do you think about simple false-positive management in w3af? For example, we can add capability to read list of regex patterns from special file and test them against request before it will be reported. It can be useful in automated usage (scan+reporting) of w3af. This is a good idea, but I'm not sure blind regex is the way to do it. It might be better to give the user more info in the lists of issues and allow sorting by request time/delay/size/search term/HTTP status code/etc. Then it's easy to multi-select from that sorted list and right click or hit a menu entry to mark as false positive, confirmed, etc. Hmm.. I suppose the text interface could do the same by numbering sorted rows of findings and allow marking false positive, confirmed, etc by range. Those are just my ideas. BTW, most of my ideas are be more like burp ;-) Give the user as much feedback and tools as possible instead of trying to do magic in the background. There's too many corner cases in web app security to trust magic in the background yet ;-) 2. Last days sf.net works too slowly and had been attacked. Don't you think about migration to something like googlecode project hosting or even github? With git server speed doesn't matter near as much. The server is a sync point, and for most of your work you never touch the server. When I'm working on Web Security Dojo (which is in git on SF) I only really need to interact with the server once a day or so for a quick sync over SSH. Everything else happens client side, so who cares what the speed of the server is? It pains me to wait for svn Git does SHA-1 signing of each changeset as the identifier, so it would be hard for an attacker to change the code without git screaming bloody murder when you try to pull from the server next time. If svn on SF is causing problems, I recommend git. Git on SF works fine for my usage, but github does have better collaboration features that make it easier to get contributions from non-core developers. Last time I was singing the praises of git, the the discussion stopped with: On 01/20/2011 01:56 PM, Andres Riancho wrote: On Thu, Jan 20, 2011 at 1:03 PM, Steve Pinkham steve.pink...@gmail.com wrote: My main point is that if you're not branching for tool limitation reasons, perhaps it's time to re-evaluate your tools. ;-) Agreed! Git(or hg) makes branching and merging so easy you'll do it all the time, and discover when it's useful in your methodology instead of having the fear of merging syndrome that svn inspires. This tends to lead to both more stability and innovation since it's simple to try out a new idea in a branch and merge it if it works out well. Agreed! @Javier: you're going to be one of the most affected persons if we change to git, any comments on this thread? So Javier, what's your take? Some excellent free git resources for the uninitiated: http://progit.org/book/ http://library.edgecase.com/git_immersion/index.html http://help.github.com/ http://peepcode.com/products/git (I'll buy this video for any of the core contributors if it would help) Moving off trac would be... Annoying and time consuming. ;-) If SF isn't a good choice for hosting, you can host trac yourself(then have to do your own backups, patches, etc) or there are free services like http://www.assembla.com/ that allow importing databases. Personally I doubt moving trac off of sourceforge would be worth it in the long run. I'd wait for a while until the pain gets higher before moving trac if it was me. -- | Steven Pinkham, Security Consultant| | http://www.mavensecurity.com | | GPG public key ID CD31CAFB | smime.p7s Description: S/MIME Cryptographic Signature -- The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE: Pinpoint memory and threading errors before they happen. Find and fix more than 250 security defects in the development cycle. Locate bottlenecks in serial and parallel code that limit performance. http://p.sf.net/sfu/intel-dev2devfeb___ W3af-develop mailing list W3af-develop@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/w3af-develop
[W3af-develop] From OWASP summit 2011: Tools Interoperability (Data Instrumentation)
I'm planning on remotely attending the following OWASP Summit session (as well as others), and I think it is relavent both to the w3af project and open source web appsec improvement in general. http://www.owasp.org/index.php/Summit_2011_Working_Sessions/Session056 Unfortunately, it's not scheduled yet, so I'll try t update the list when it is scheduled. The times are GMT 0, so may or may not be convenient for those of us in the americas. ;-) I was up at 6:30am for the XSS session. Anyway, there is a discussion page from the wiki linked above if you have any comments before the session starts. -- | Steven Pinkham, Security Consultant| | http://www.mavensecurity.com | | GPG public key ID CD31CAFB | smime.p7s Description: S/MIME Cryptographic Signature -- The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE: Pinpoint memory and threading errors before they happen. Find and fix more than 250 security defects in the development cycle. Locate bottlenecks in serial and parallel code that limit performance. http://p.sf.net/sfu/intel-dev2devfeb___ W3af-develop mailing list W3af-develop@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/w3af-develop
Re: [W3af-develop] Sprint #13 - Javier's playground
Hello everyone, The sprint 13 has finally been filled up. I included some tasks [1] that (when done) will make our lives happier :-) If you think that something else related to them should be included please let us know. Thanks! Javier [1] https://sourceforge.net/apps/trac/w3af/milestone/owls-sprint-13 On Mon, Feb 7, 2011 at 9:36 AM, Andres Riancho andres.rian...@gmail.com wrote: Javier, On Wed, Feb 2, 2011 at 5:46 PM, Andres Riancho andres.rian...@gmail.com wrote: I would like to see this sprint planned by the end of this week if possible. Today is your last chance! If not, I'll enjoy setting up a whole sprint of documentation and grunt work ! :) Regards, -- Andrés Riancho Director of Web Security at Rapid7 LLC Founder at Bonsai Information Security Project Leader at w3af -- The modern datacenter depends on network connectivity to access resources and provide services. The best practices for maximizing a physical server's connectivity to a physical network are well understood - see how these rules translate into the virtual world? http://p.sf.net/sfu/oracle-sfdevnlfb ___ W3af-develop mailing list W3af-develop@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/w3af-develop -- The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE: Pinpoint memory and threading errors before they happen. Find and fix more than 250 security defects in the development cycle. Locate bottlenecks in serial and parallel code that limit performance. http://p.sf.net/sfu/intel-dev2devfeb ___ W3af-develop mailing list W3af-develop@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/w3af-develop
Re: [W3af-develop] core/data/db/history.py and .trace files
A very common no-sql document related database are IBM Lotus Notes db ( .nsf ) . These dbs are usually used for team room applications or for storing transaccional data between IBM mainframe and other platforms like HP-NON STOP most seen on Banking environments. Inside IBM doors these nsf databases are used for almost everything (team rooms, phone db, rrhh dbs, physical security dbs). Regards -- Leandro Reox On Fri, Feb 4, 2011 at 8:39 PM, Andres Riancho andres.rian...@gmail.comwrote: Agreed, we need fixed. Lots of bug reports about it in Trac. -- Andres Riancho El feb 4, 2011 6:51 p.m., Taras ox...@oxdef.info escribió: Andres, I didn't use noSQL databases but it can be interesting research =) But for the first lets simply fix this bug with files. Do we know about any noSQL database that's file based like sqlite? Maybe we could use this s... -- The modern datacenter depends on network connectivity to access resources and provide services. The best practices for maximizing a physical server's connectivity to a physical network are well understood - see how these rules translate into the virtual world? http://p.sf.net/sfu/oracle-sfdevnlfb ___ W3af-develop mailing list W3af-develop@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/w3af-develop -- The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE: Pinpoint memory and threading errors before they happen. Find and fix more than 250 security defects in the development cycle. Locate bottlenecks in serial and parallel code that limit performance. http://p.sf.net/sfu/intel-dev2devfeb___ W3af-develop mailing list W3af-develop@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/w3af-develop
Re: [W3af-develop] core/data/db/history.py and .trace files
Lean, Do you know if the format is open? Do we have a Python binding to write to them? Any clue on how they scale in performance when saving thousands of registries? Regards, On Tue, Feb 8, 2011 at 4:42 PM, Leandro Reox leandro.r...@gmail.com wrote: A very common no-sql document related database are IBM Lotus Notes db ( .nsf ) . These dbs are usually used for team room applications or for storing transaccional data between IBM mainframe and other platforms like HP-NON STOP most seen on Banking environments. Inside IBM doors these nsf databases are used for almost everything (team rooms, phone db, rrhh dbs, physical security dbs). Regards -- Leandro Reox On Fri, Feb 4, 2011 at 8:39 PM, Andres Riancho andres.rian...@gmail.com wrote: Agreed, we need fixed. Lots of bug reports about it in Trac. -- Andres Riancho El feb 4, 2011 6:51 p.m., Taras ox...@oxdef.info escribió: Andres, I didn't use noSQL databases but it can be interesting research =) But for the first lets simply fix this bug with files. Do we know about any noSQL database that's file based like sqlite? Maybe we could use this s... -- The modern datacenter depends on network connectivity to access resources and provide services. The best practices for maximizing a physical server's connectivity to a physical network are well understood - see how these rules translate into the virtual world? http://p.sf.net/sfu/oracle-sfdevnlfb ___ W3af-develop mailing list W3af-develop@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/w3af-develop -- Andrés Riancho Director of Web Security at Rapid7 LLC Founder at Bonsai Information Security Project Leader at w3af -- The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE: Pinpoint memory and threading errors before they happen. Find and fix more than 250 security defects in the development cycle. Locate bottlenecks in serial and parallel code that limit performance. http://p.sf.net/sfu/intel-dev2devfeb ___ W3af-develop mailing list W3af-develop@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/w3af-develop
Re: [W3af-develop] core/data/db/history.py and .trace files
Andres, Sadly the format is not open. Theres a few ways to write and retrieve data via Python to this kind of databases (like jython + and the notes.jar classes - notessql drivers on win, etc). Regarding performance a 100.000 records with attachments databases are very common on IBM infraestructure and the database itself performs like a charm. Have you consider using the open-source alternative to .nsf , mongodb ? Its an document oriented database type like nsf, open format, fully compatible with python and the performance is pretty awesome Regards On Tue, Feb 8, 2011 at 5:08 PM, Andres Riancho andres.rian...@gmail.comwrote: Lean, Do you know if the format is open? Do we have a Python binding to write to them? Any clue on how they scale in performance when saving thousands of registries? Regards, On Tue, Feb 8, 2011 at 4:42 PM, Leandro Reox leandro.r...@gmail.com wrote: A very common no-sql document related database are IBM Lotus Notes db ( .nsf ) . These dbs are usually used for team room applications or for storing transaccional data between IBM mainframe and other platforms like HP-NON STOP most seen on Banking environments. Inside IBM doors these nsf databases are used for almost everything (team rooms, phone db, rrhh dbs, physical security dbs). Regards -- Leandro Reox On Fri, Feb 4, 2011 at 8:39 PM, Andres Riancho andres.rian...@gmail.com wrote: Agreed, we need fixed. Lots of bug reports about it in Trac. -- Andres Riancho El feb 4, 2011 6:51 p.m., Taras ox...@oxdef.info escribió: Andres, I didn't use noSQL databases but it can be interesting research =) But for the first lets simply fix this bug with files. Do we know about any noSQL database that's file based like sqlite? Maybe we could use this s... -- The modern datacenter depends on network connectivity to access resources and provide services. The best practices for maximizing a physical server's connectivity to a physical network are well understood - see how these rules translate into the virtual world? http://p.sf.net/sfu/oracle-sfdevnlfb ___ W3af-develop mailing list W3af-develop@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/w3af-develop -- Andrés Riancho Director of Web Security at Rapid7 LLC Founder at Bonsai Information Security Project Leader at w3af -- The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE: Pinpoint memory and threading errors before they happen. Find and fix more than 250 security defects in the development cycle. Locate bottlenecks in serial and parallel code that limit performance. http://p.sf.net/sfu/intel-dev2devfeb___ W3af-develop mailing list W3af-develop@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/w3af-develop
Re: [W3af-develop] core/data/db/history.py and .trace files
The only issue with mongodb is that its a daemon, I'm not sure if we want to have mongod as a w3af dependency. It could complicate packaging and install process. Regards, -- Andres Riancho El feb 8, 2011 6:39 p.m., Leandro Reox leandro.r...@gmail.com escribió: Here is a living proof of MongoDB deployed on large scale scenarios : http://www.mongodb.org/display/DOCS/Production+Deployments Regards Lean On Tue, Feb 8, 2011 at 6:08 PM, Leandro Reox leandro.r...@gmail.com wrote: Andres, Sadl... -- The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE: Pinpoint memory and threading errors before they happen. Find and fix more than 250 security defects in the development cycle. Locate bottlenecks in serial and parallel code that limit performance. http://p.sf.net/sfu/intel-dev2devfeb___ W3af-develop mailing list W3af-develop@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/w3af-develop
Re: [W3af-develop] core/data/db/history.py and .trace files
Steve, On Tue, Feb 8, 2011 at 8:45 PM, Steve Pinkham steve.pink...@gmail.com wrote: On 02/03/2011 12:04 PM, Andres Riancho wrote: Do we know about any noSQL database that's file based like sqlite? Maybe we could use this small rewrite to compare the performance of those backends. Regards, I'm somewhat at a loss of what you think noSQL will buy you. It's useful in distributed, massively parallel systems, but offers no real benefit for single user databases. I disagree. I've seen how sqlite3's performance impacted in the framework's performance before mainly because of its slow access (SELECT). For what I can understand from the noSQL databases, the access to any row should be ultra fast, even if we save whole HTTP requests and responses to it. noSQL is just the new term for key-value stores. Yes. Berkeley DB is what was used as a file based key-value store before sqlite, but has no major benefits in most uses over sqlite which is why it didn't spring to mind. ;-) If you have many threads writing concurrently, BDB can be faster, but you have a great decrease in functionality as a cost. http://en.wikipedia.org/wiki/Berkeley_DB I already took a look into BDB and for some reason I discarded it, now I don't remember why :( Here's one set of benchmarks. For low number of records, BDB was faster, for number of high records sqlite was faster. Both should be fast enough. You shouldn't need transactional capabilities where sqlite was the slowest. http://www.sqlite.org/cvstrac/wiki?p=KeyValueDatabase What I read from this performance test is: BDB is faster in 90% of the cases. In the cases where BDB is faster, its ~50% faster in average. Regards, -- | Steven Pinkham, Security Consultant | | http://www.mavensecurity.com | | GPG public key ID CD31CAFB | -- Andrés Riancho Director of Web Security at Rapid7 LLC Founder at Bonsai Information Security Project Leader at w3af -- The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE: Pinpoint memory and threading errors before they happen. Find and fix more than 250 security defects in the development cycle. Locate bottlenecks in serial and parallel code that limit performance. http://p.sf.net/sfu/intel-dev2devfeb ___ W3af-develop mailing list W3af-develop@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/w3af-develop
Re: [W3af-develop] core/data/db/history.py and .trace files
Steve, On Tue, Feb 8, 2011 at 9:07 PM, Andres Riancho andres.rian...@gmail.com wrote: Berkeley DB is what was used as a file based key-value store before sqlite, but has no major benefits in most uses over sqlite which is why it didn't spring to mind. ;-) If you have many threads writing concurrently, BDB can be faster, but you have a great decrease in functionality as a cost. http://en.wikipedia.org/wiki/Berkeley_DB I already took a look into BDB and for some reason I discarded it, now I don't remember why :( Ahh, this is why! Deprecated since version 2.6: The bsddb module has been deprecated for removal in Python 3.0. [0] [0] http://docs.python.org/library/bsddb.html -- Andrés Riancho Director of Web Security at Rapid7 LLC Founder at Bonsai Information Security Project Leader at w3af -- The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE: Pinpoint memory and threading errors before they happen. Find and fix more than 250 security defects in the development cycle. Locate bottlenecks in serial and parallel code that limit performance. http://p.sf.net/sfu/intel-dev2devfeb ___ W3af-develop mailing list W3af-develop@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/w3af-develop
Re: [W3af-develop] core/data/db/history.py and .trace files
On 02/08/2011 07:07 PM, Andres Riancho wrote: Steve, On Tue, Feb 8, 2011 at 8:45 PM, Steve Pinkham steve.pink...@gmail.com wrote: On 02/03/2011 12:04 PM, Andres Riancho wrote: Do we know about any noSQL database that's file based like sqlite? Maybe we could use this small rewrite to compare the performance of those backends. Regards, I'm somewhat at a loss of what you think noSQL will buy you. It's useful in distributed, massively parallel systems, but offers no real benefit for single user databases. I disagree. I've seen how sqlite3's performance impacted in the framework's performance before mainly because of its slow access (SELECT). For what I can understand from the noSQL databases, the access to any row should be ultra fast, even if we save whole HTTP requests and responses to it. noSQL servers are usually fast because they are in-memory systems. sqlite can be used in that mode also if you like. If select is your problem, you're probably not indexing properly or your selects are waiting on writes. Both are fixable. That said, if all you ever want is a key value store and you never see yourself use any more complicated searches than that, maybe a key value store is for you. Otherwise writing better selects and tuning your indexing is probably a bigger win. I haven't looked at what you're using the database for or how you have it tuned yet, but I'll try to soon. Here's one set of benchmarks. For low number of records, BDB was faster, for number of high records sqlite was faster. Both should be fast enough. You shouldn't need transactional capabilities where sqlite was the slowest. http://www.sqlite.org/cvstrac/wiki?p=KeyValueDatabase What I read from this performance test is: BDB is faster in 90% of the cases. In the cases where BDB is faster, its ~50% faster in average. What I see is if you're making more than 100,000 selects/second in a web app scanner you seriously screwed up somewhere and need to be caching more. Being 2x or 10x faster will still lose to better design. -- | Steven Pinkham, Security Consultant| | http://www.mavensecurity.com | | GPG public key ID CD31CAFB | smime.p7s Description: S/MIME Cryptographic Signature -- The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE: Pinpoint memory and threading errors before they happen. Find and fix more than 250 security defects in the development cycle. Locate bottlenecks in serial and parallel code that limit performance. http://p.sf.net/sfu/intel-dev2devfeb___ W3af-develop mailing list W3af-develop@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/w3af-develop
Re: [W3af-develop] core/data/db/history.py and .trace files
On 02/08/2011 08:08 PM, Andres Riancho wrote: Steve, noSQL servers are usually fast because they are in-memory systems. sqlite can be used in that mode also if you like. mongodb is not an in-memory db! In practice, it is. It stores all indexes in memory and uses memory mapped files. It will automatically consume all available memory (which is a good thing or bad thing depending on what else you want to use the server for). http://www.mongodb.org/display/DOCS/Indexing+Advice+and+FAQ#IndexingAdviceandFAQ-MakesureyourindexescanfitinRAM. http://www.mongodb.org/display/DOCS/Caching -- | Steven Pinkham, Security Consultant| | http://www.mavensecurity.com | | GPG public key ID CD31CAFB | smime.p7s Description: S/MIME Cryptographic Signature -- The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE: Pinpoint memory and threading errors before they happen. Find and fix more than 250 security defects in the development cycle. Locate bottlenecks in serial and parallel code that limit performance. http://p.sf.net/sfu/intel-dev2devfeb___ W3af-develop mailing list W3af-develop@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/w3af-develop
[W3af-develop] sqlite3 weirdness (AKA I hate python ;-)
Meh. sqlite has been threadsafe since 2006, and the python adapter still won't let you use connections across multiple threads because you might have an old version. You're using an explicitly unsupported workaround (check_same_thread=False) that may cause dataloss due to optimisations in python 2.6 and later. See http://bugs.python.org/issue3846 I guess your workaround coupled with the mutexes should be a safe solution but may not be the most performant. I'm still researching and trying to wrap my head around the weirdness that is the python sqlite3 adapter though. It does some *strange* things. The ruby adapter is threadsafe. Just sayin.. ;-) -- | Steven Pinkham, Security Consultant| | http://www.mavensecurity.com | | GPG public key ID CD31CAFB | smime.p7s Description: S/MIME Cryptographic Signature -- The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE: Pinpoint memory and threading errors before they happen. Find and fix more than 250 security defects in the development cycle. Locate bottlenecks in serial and parallel code that limit performance. http://p.sf.net/sfu/intel-dev2devfeb___ W3af-develop mailing list W3af-develop@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/w3af-develop
