Re: [Windows] proposed binary build changes
On Wednesday, 25 January 2012, Filipe David Manana fdman...@apache.org wrote: On Wed, Jan 25, 2012 at 7:24 PM, Jeroen Janssen jeroen.jans...@gmail.com wrote: Hi, Is the issue Filipe mentioned (where a not compressed version 6 database doesn't compress on compaction) going to be fixed before the actual 1.2.0 release? I would say so. Here's a jira ticket I just created with a patch: https://issues.apache.org/jira/browse/COUCHDB-1394 this works fine on mac Jeroen. if you apply the patch attached you can check it directly on windows before i get around to rolling a new snapshot. I assume you're ok with erlang? +1 for having this committed for 1.2 please. Thanks Filipe! Dave
Re: [Windows] proposed binary build changes
On Thu, Jan 26, 2012 at 1:09 PM, Dave Cottlehuber d...@muse.net.nz wrote: On Wednesday, 25 January 2012, Filipe David Manana fdman...@apache.org wrote: On Wed, Jan 25, 2012 at 7:24 PM, Jeroen Janssen jeroen.jans...@gmail.com wrote: Hi, Is the issue Filipe mentioned (where a not compressed version 6 database doesn't compress on compaction) going to be fixed before the actual 1.2.0 release? I would say so. Here's a jira ticket I just created with a patch: https://issues.apache.org/jira/browse/COUCHDB-1394 this works fine on mac Jeroen. Thanks for testing Dave. It's now included in 1.2.x. if you apply the patch attached you can check it directly on windows before i get around to rolling a new snapshot. I assume you're ok with erlang? +1 for having this committed for 1.2 please. Thanks Filipe! Dave -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: [Windows] proposed binary build changes
On Wed, Jan 25, 2012 at 7:24 PM, Jeroen Janssen jeroen.jans...@gmail.com wrote: Hi, Is the issue Filipe mentioned (where a not compressed version 6 database doesn't compress on compaction) going to be fixed before the actual 1.2.0 release? I would say so. Here's a jira ticket I just created with a patch: https://issues.apache.org/jira/browse/COUCHDB-1394 this works fine on mac Jeroen. Thanks for testing Dave. It's now included in 1.2.x. Jeroen, Thanks for your patience. LGTME on windows too - fresh build at https://www.dropbox.com/s/jeifcxpbtpo78ak/snapshots/20120127?v=l A+ Dave
Re: [Windows] proposed binary build changes
On Wed, Jan 25, 2012 at 12:12 AM, Dave Cottlehuber d...@muse.net.nz wrote: On 23 January 2012 08:00, Jeroen Janssen jeroen.jans...@gmail.com wrote: Hi, It seems to be ok, since I get: All 2 tests passed. I also tried compaction again, but it didn't compress the 800Mb .couch file, so I guess something else might be needed aswell? Best regards, Jeroen Hi Jeroen, Thanks heaps, missing DLL is sorted - https://www.dropbox.com/s/jeifcxpbtpo78ak/snapshots/20120124/SDK7.1?v=l Either of these will work but in my testing R15B seems both faster and more memory efficient. NB I am testing compaction by going from 1.1.1 - 1.2.x here, as the on-disk format changes in between. I don't get compression either using your procedure above even with a POST /_restart in between. Hopefully Filipe can clarify what's expected here. If the disk version of a database is already at version 6 (version 1.2) and the database doesn't have compression, compaction will not compress it. I'll correct this soon. If you grab a database file from CouchDB 1.1 (or older), it will be compressed on compaction. 01/23/2012 03:25 PM 246,907,000 fatty_no_snappy.couch 01/23/2012 03:19 PM 38,830,200 fatty_snappy.couch 01/23/2012 01:57 PM 3,104,881 sluggy_no_snappy.couch 01/23/2012 01:58 PM 3,072,113 sluggy_snappy.couch - sluggy is a set of small docs, all around 4KiB size, generated via http://json-generator.appspot.com/ - fatty is a set of large docs, all around ~400KiB size, artificially full of http://baconipsum.com/ These json blobs are in https://www.dropbox.com/s/jeifcxpbtpo78ak/testing/ and are formatted for _bulk_docs. I'm testing your approach using http://friendpaste.com/6t4KwKYWKofHvVzF3kDMLQ A+ Dave -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: [Windows] proposed binary build changes
Hi, I can confirm that the new windows binaries work for the following: *) version 5 database, run compaction, reduced disksize (due to compression working) *) create new database, add 200k document, results in ~33k database (also due to compression) So that is good news! :) Is the issue Filipe mentioned (where a not compressed version 6 database doesn't compress on compaction) going to be fixed before the actual 1.2.0 release? Best regards, Jeroen On Wed, Jan 25, 2012 at 1:12 AM, Dave Cottlehuber d...@muse.net.nz wrote: On 23 January 2012 08:00, Jeroen Janssen jeroen.jans...@gmail.com wrote: Hi, It seems to be ok, since I get: All 2 tests passed. I also tried compaction again, but it didn't compress the 800Mb .couch file, so I guess something else might be needed aswell? Best regards, Jeroen Hi Jeroen, Thanks heaps, missing DLL is sorted - https://www.dropbox.com/s/jeifcxpbtpo78ak/snapshots/20120124/SDK7.1?v=l Either of these will work but in my testing R15B seems both faster and more memory efficient. NB I am testing compaction by going from 1.1.1 - 1.2.x here, as the on-disk format changes in between. I don't get compression either using your procedure above even with a POST /_restart in between. Hopefully Filipe can clarify what's expected here. 01/23/2012 03:25 PM 246,907,000 fatty_no_snappy.couch 01/23/2012 03:19 PM 38,830,200 fatty_snappy.couch 01/23/2012 01:57 PM 3,104,881 sluggy_no_snappy.couch 01/23/2012 01:58 PM 3,072,113 sluggy_snappy.couch - sluggy is a set of small docs, all around 4KiB size, generated via http://json-generator.appspot.com/ - fatty is a set of large docs, all around ~400KiB size, artificially full of http://baconipsum.com/ These json blobs are in https://www.dropbox.com/s/jeifcxpbtpo78ak/testing/ and are formatted for _bulk_docs. I'm testing your approach using http://friendpaste.com/6t4KwKYWKofHvVzF3kDMLQ A+ Dave
Re: [Windows] proposed binary build changes
On Wed, Jan 25, 2012 at 7:24 PM, Jeroen Janssen jeroen.jans...@gmail.com wrote: Hi, I can confirm that the new windows binaries work for the following: *) version 5 database, run compaction, reduced disksize (due to compression working) *) create new database, add 200k document, results in ~33k database (also due to compression) So that is good news! :) Is the issue Filipe mentioned (where a not compressed version 6 database doesn't compress on compaction) going to be fixed before the actual 1.2.0 release? I would say so. Here's a jira ticket I just created with a patch: https://issues.apache.org/jira/browse/COUCHDB-1394 Best regards, Jeroen On Wed, Jan 25, 2012 at 1:12 AM, Dave Cottlehuber d...@muse.net.nz wrote: On 23 January 2012 08:00, Jeroen Janssen jeroen.jans...@gmail.com wrote: Hi, It seems to be ok, since I get: All 2 tests passed. I also tried compaction again, but it didn't compress the 800Mb .couch file, so I guess something else might be needed aswell? Best regards, Jeroen Hi Jeroen, Thanks heaps, missing DLL is sorted - https://www.dropbox.com/s/jeifcxpbtpo78ak/snapshots/20120124/SDK7.1?v=l Either of these will work but in my testing R15B seems both faster and more memory efficient. NB I am testing compaction by going from 1.1.1 - 1.2.x here, as the on-disk format changes in between. I don't get compression either using your procedure above even with a POST /_restart in between. Hopefully Filipe can clarify what's expected here. 01/23/2012 03:25 PM 246,907,000 fatty_no_snappy.couch 01/23/2012 03:19 PM 38,830,200 fatty_snappy.couch 01/23/2012 01:57 PM 3,104,881 sluggy_no_snappy.couch 01/23/2012 01:58 PM 3,072,113 sluggy_snappy.couch - sluggy is a set of small docs, all around 4KiB size, generated via http://json-generator.appspot.com/ - fatty is a set of large docs, all around ~400KiB size, artificially full of http://baconipsum.com/ These json blobs are in https://www.dropbox.com/s/jeifcxpbtpo78ak/testing/ and are formatted for _bulk_docs. I'm testing your approach using http://friendpaste.com/6t4KwKYWKofHvVzF3kDMLQ A+ Dave -- Filipe David Manana, Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men.
Re: [Windows] proposed binary build changes
On 23 January 2012 08:00, Jeroen Janssen jeroen.jans...@gmail.com wrote: Hi, It seems to be ok, since I get: All 2 tests passed. I also tried compaction again, but it didn't compress the 800Mb .couch file, so I guess something else might be needed aswell? Best regards, Jeroen Hi Jeroen, Thanks heaps, missing DLL is sorted - https://www.dropbox.com/s/jeifcxpbtpo78ak/snapshots/20120124/SDK7.1?v=l Either of these will work but in my testing R15B seems both faster and more memory efficient. NB I am testing compaction by going from 1.1.1 - 1.2.x here, as the on-disk format changes in between. I don't get compression either using your procedure above even with a POST /_restart in between. Hopefully Filipe can clarify what's expected here. 01/23/2012 03:25 PM 246,907,000 fatty_no_snappy.couch 01/23/2012 03:19 PM38,830,200 fatty_snappy.couch 01/23/2012 01:57 PM 3,104,881 sluggy_no_snappy.couch 01/23/2012 01:58 PM 3,072,113 sluggy_snappy.couch - sluggy is a set of small docs, all around 4KiB size, generated via http://json-generator.appspot.com/ - fatty is a set of large docs, all around ~400KiB size, artificially full of http://baconipsum.com/ These json blobs are in https://www.dropbox.com/s/jeifcxpbtpo78ak/testing/ and are formatted for _bulk_docs. I'm testing your approach using http://friendpaste.com/6t4KwKYWKofHvVzF3kDMLQ A+ Dave
Re: [Windows] proposed binary build changes
Hi, Just a quick check since I am not able to get (snappy) compression working on windows. Steps I did: *) install couchdb 1.2.0a from [5] (disabled service/autostart) *) edit local.ini add file_compression = snappy to the [couchdb] section *) copied an old (800Mb) 1.0.1 database to var\lib\couchdb *) start couchdb\bin\couchdb.bat *) run compaction on the old database. still 800Mb size additional test I did was to: *) create a new database *) created a 200kb document and saved *) database file is ~200kb in size similar results with file_compression = deflate_9 Should I do additional steps to enable compression? Can anyone else confirm this behaviour on windows? Thanks in advance, Jeroen On Tue, Jan 17, 2012 at 8:16 PM, Dave Cottlehuber d...@muse.net.nz wrote: Hi folks, Quick questions followed by a long update. 1. Does anybody rely on the compiler that I use to build CouchDB? I'm considering dropping VS2008/SDK7.0 builds in favour of the newer VS2010/SDK7.1, mainly because I don't really have the space to build both. Although feel free to send me a 500GiB SSD… 2. Unless there are strenuous objections, I'm proposing to bump the Windows binaries and dependencies to enable that, details below. You can test a variety of 1.1.1 combos if this is important to you [1]. 3. Any further volunteers for testing voting on windows binary builds? - Kevin Coombes - Nick North - Ryan Ramage - pblakez - Dirkjan It should take ~ 1h per release I'm not expecting a commitment from you! 4. Feedback is hard to come by so, I'd love some use cases, success stories, and even the occasional abject failure for Couch on Windows. Some background details follow, mainly to ensure we have these archived on the list. HISTORY: Up until Nov 2011, the official binary release from Erlang/OTP team was done with VS2005 / VC++6. However Erlang CouchDB require ICU, OpenSSL, wxWidgets, Mozilla SpiderMonkey and optionally cURL, none of which compile cleanly under VS2005 these days. So I've been rolling my own using VS2008 / SDK 7.0, which is not so difficult now, and has also enabled me to provide fixes for uncommitted upstream patches where necessary. To work around some of the above, I've built binaries against a single, very specific release of the vcredist_x86.exe package. Thanks to some advice from Pierre at Microsoft, I can now remove this requirement. This should simplify requirements for anybody who needs to repackage. He's also given excellent suggestions on improving the performance of Apache CouchDB on Windows, and sharing building the dependencies with projects like PHP in future. Let's hope this progresses further. TODAY: Erlang/OTP have a VS2010 set of binaries available, with current components, and no outstanding critical bugs as yet. R15B has error messages with line numbers which is an immense help for troubleshooting. The build scripts I use are now largely compatible between SDKs, so it's possible to create SDK 7.0 and 7.1 versions when I merge document it properly. Thanks Wohali for the suggestion some help along the way! LIBRARIES INCLUDED IN THE BUILD: Unless there are strenuous objections, I'm proposing to move from/to: Compiler: VS2008/ SDK 7.0 - VS2010 / SDK 7.1 compiler Visual C++ runtime Erlang/OTP R15B - wxWidgets 2.8.11 - 2.8.12 - OpenSSL 1.0.0e - 0.9.8r to align with OTP team's release For CouchDB 1.2.0 and the future: - ICU 4.4.2 - 4.6.1 as it compiles cleanly under both SDKs. - cURL 7.23.1 - no change - SpiderMonkey 1.8.0-rc1 - 1.8.5 This sets us up nicely for providing a much simpler build process, perhaps without needing cygwin at all, just common Windows tools. I don't see any issues with this in the last month of testing, YMMV! The artefacts used to build these ( their debug symbols) are all available [4], and there's a build off the 1.2.x branch including these [5]. FUTURE: Feel free to add but this is where I'm thinking of going, in no particular order: - Makefile target for a trimmed OTP + Couch build upx compression - MSI support (thanks in advance to Wohali!!) - add a GeoCouchified version with erica + kanso support built in - performance improvements c/- Microsoft's PGO [2] - simpler build process perhaps using Mozilla tool chain only - stabilise my glazier build scripts [3] - use Windows-approved locations for datafiles - aim for Microsoft Certification - good general CouchDB publicity - enable JS and ETAP test framework on Windows - look at 64 bit builds in June or July once OTP end has settled down - move at least the .sha and .md5 files onto apache.org pages to ensure people have a trusted source of binaries. Feel free to prioritise this, or add to it. [1]: https://www.dropbox.com/s/jeifcxpbtpo78ak/snapshots/20120117?v=l [2]: http://msdn.microsoft.com/en-us/library/e7k32f4k.aspx [3]: http://github.com/dch/glazier [4]:
Re: [Windows] proposed binary build changes
On 22 January 2012 13:28, Jeroen Janssen jeroen.jans...@gmail.com wrote: Hi, Just a quick check since I am not able to get (snappy) compression working on windows. Hi Jeroen, Thanks for reporting this. AFAICT the snappy_nif.dll is not being copied during make dist or install. IIRC you ran into this last year on master and resulted in COUCHDB-1197 fixes. So I'm not sure what is missing on 1.2.x for this, or if its more likely an issue with my packaging steps. I'll keep you posted. BTW compression is on by default so no need to tweak ini files. A+ Dave
Re: [Windows] proposed binary build changes
This all sounds great, Dave. Once we get the source distribution working for 1.2.x, I'd like to co-ordinate with you to get the Windows build prepped at the same time. We should aim to have you uploading the .exe, .exe.md5, and .exe.sha files to your p.a.o space, and calling a vote on the artefacts like we do for the source artefacts. We can then announce both distributions at the same time. Thanks, On Sun, Jan 22, 2012 at 2:49 PM, Dave Cottlehuber d...@muse.net.nz wrote: On 22 January 2012 13:28, Jeroen Janssen jeroen.jans...@gmail.com wrote: Hi, Just a quick check since I am not able to get (snappy) compression working on windows. Hi Jeroen, Thanks for reporting this. AFAICT the snappy_nif.dll is not being copied during make dist or install. IIRC you ran into this last year on master and resulted in COUCHDB-1197 fixes. So I'm not sure what is missing on 1.2.x for this, or if its more likely an issue with my packaging steps. I'll keep you posted. BTW compression is on by default so no need to tweak ini files. A+ Dave
Re: [Windows] proposed binary build changes
On 22 January 2012 15:49, Dave Cottlehuber d...@muse.net.nz wrote:
On 22 January 2012 13:28, Jeroen Janssen jeroen.jans...@gmail.com wrote:
Hi,
Just a quick check since I am not able to get (snappy) compression
working on windows.
Hi Jeroen,
Thanks for reporting this. AFAICT the snappy_nif.dll is not being copied
during
make dist or install.
IIRC you ran into this last year on master and resulted in COUCHDB-1197 fixes.
So I'm not sure what is missing on 1.2.x for this, or if its more
likely an issue with
my packaging steps.
I'll keep you posted.
BTW compression is on by default so no need to tweak ini files.
A+
Dave
Hi Jeroen,
While I look for why this is not working in the build, can you copy
snappy_nif.dll [1] into $COUCH/lib/snappy-1.0.3/priv/ and then test
with snappy_test.erl courtesy of Filipe [2]?
3 pwd().
C:/couch/1.2.0a-8d83b39-git_otp_R15B_SDK7.1/bin
ok
4 c(/couch/snappy_tests).
{ok,snappy_tests}
5 snappy_tests:test().
All 2 tests passed.
ok
Your higher-level compression tests should be fine.
[1]: https://www.dropbox.com/s/jeifcxpbtpo78ak/snapshots/20120116?v=l
[2]
https://github.com/fdmanana/snappy-erlang-nif/blob/master/test/snappy_tests.erl
A+
Dave
Re: [Windows] proposed binary build changes
Hi,
It seems to be ok, since I get:
All 2 tests passed.
I also tried compaction again, but it didn't compress the 800Mb .couch
file, so I guess something else might be needed aswell?
Best regards,
Jeroen
On Sun, Jan 22, 2012 at 11:53 PM, Dave Cottlehuber d...@muse.net.nz wrote:
On 22 January 2012 15:49, Dave Cottlehuber d...@muse.net.nz wrote:
On 22 January 2012 13:28, Jeroen Janssen jeroen.jans...@gmail.com wrote:
Hi,
Just a quick check since I am not able to get (snappy) compression
working on windows.
Hi Jeroen,
Thanks for reporting this. AFAICT the snappy_nif.dll is not being copied
during
make dist or install.
IIRC you ran into this last year on master and resulted in COUCHDB-1197
fixes.
So I'm not sure what is missing on 1.2.x for this, or if its more
likely an issue with
my packaging steps.
I'll keep you posted.
BTW compression is on by default so no need to tweak ini files.
A+
Dave
Hi Jeroen,
While I look for why this is not working in the build, can you copy
snappy_nif.dll [1] into $COUCH/lib/snappy-1.0.3/priv/ and then test
with snappy_test.erl courtesy of Filipe [2]?
3 pwd().
C:/couch/1.2.0a-8d83b39-git_otp_R15B_SDK7.1/bin
ok
4 c(/couch/snappy_tests).
{ok,snappy_tests}
5 snappy_tests:test().
All 2 tests passed.
ok
Your higher-level compression tests should be fine.
[1]: https://www.dropbox.com/s/jeifcxpbtpo78ak/snapshots/20120116?v=l
[2]
https://github.com/fdmanana/snappy-erlang-nif/blob/master/test/snappy_tests.erl
A+
Dave
[Windows] proposed binary build changes
Hi folks, Quick questions followed by a long update. 1. Does anybody rely on the compiler that I use to build CouchDB? I'm considering dropping VS2008/SDK7.0 builds in favour of the newer VS2010/SDK7.1, mainly because I don't really have the space to build both. Although feel free to send me a 500GiB SSD… 2. Unless there are strenuous objections, I'm proposing to bump the Windows binaries and dependencies to enable that, details below. You can test a variety of 1.1.1 combos if this is important to you [1]. 3. Any further volunteers for testing voting on windows binary builds? - Kevin Coombes - Nick North - Ryan Ramage - pblakez - Dirkjan It should take ~ 1h per release I'm not expecting a commitment from you! 4. Feedback is hard to come by so, I'd love some use cases, success stories, and even the occasional abject failure for Couch on Windows. Some background details follow, mainly to ensure we have these archived on the list. HISTORY: Up until Nov 2011, the official binary release from Erlang/OTP team was done with VS2005 / VC++6. However Erlang CouchDB require ICU, OpenSSL, wxWidgets, Mozilla SpiderMonkey and optionally cURL, none of which compile cleanly under VS2005 these days. So I've been rolling my own using VS2008 / SDK 7.0, which is not so difficult now, and has also enabled me to provide fixes for uncommitted upstream patches where necessary. To work around some of the above, I've built binaries against a single, very specific release of the vcredist_x86.exe package. Thanks to some advice from Pierre at Microsoft, I can now remove this requirement. This should simplify requirements for anybody who needs to repackage. He's also given excellent suggestions on improving the performance of Apache CouchDB on Windows, and sharing building the dependencies with projects like PHP in future. Let's hope this progresses further. TODAY: Erlang/OTP have a VS2010 set of binaries available, with current components, and no outstanding critical bugs as yet. R15B has error messages with line numbers which is an immense help for troubleshooting. The build scripts I use are now largely compatible between SDKs, so it's possible to create SDK 7.0 and 7.1 versions when I merge document it properly. Thanks Wohali for the suggestion some help along the way! LIBRARIES INCLUDED IN THE BUILD: Unless there are strenuous objections, I'm proposing to move from/to: Compiler: VS2008/ SDK 7.0 - VS2010 / SDK 7.1 compiler Visual C++ runtime Erlang/OTP R15B - wxWidgets 2.8.11 - 2.8.12 - OpenSSL 1.0.0e - 0.9.8r to align with OTP team's release For CouchDB 1.2.0 and the future: - ICU 4.4.2 - 4.6.1 as it compiles cleanly under both SDKs. - cURL 7.23.1- no change - SpiderMonkey 1.8.0-rc1 - 1.8.5 This sets us up nicely for providing a much simpler build process, perhaps without needing cygwin at all, just common Windows tools. I don't see any issues with this in the last month of testing, YMMV! The artefacts used to build these ( their debug symbols) are all available [4], and there's a build off the 1.2.x branch including these [5]. FUTURE: Feel free to add but this is where I'm thinking of going, in no particular order: - Makefile target for a trimmed OTP + Couch build upx compression - MSI support (thanks in advance to Wohali!!) - add a GeoCouchified version with erica + kanso support built in - performance improvements c/- Microsoft's PGO [2] - simpler build process perhaps using Mozilla tool chain only - stabilise my glazier build scripts [3] - use Windows-approved locations for datafiles - aim for Microsoft Certification - good general CouchDB publicity - enable JS and ETAP test framework on Windows - look at 64 bit builds in June or July once OTP end has settled down - move at least the .sha and .md5 files onto apache.org pages to ensure people have a trusted source of binaries. Feel free to prioritise this, or add to it. [1]: https://www.dropbox.com/s/jeifcxpbtpo78ak/snapshots/20120117?v=l [2]: http://msdn.microsoft.com/en-us/library/e7k32f4k.aspx [3]: http://github.com/dch/glazier [4]: https://www.dropbox.com/s/jeifcxpbtpo78ak/Building_from_Source [5]: https://www.dropbox.com/s/jeifcxpbtpo78ak/snapshots/20120116?v=l A+ Dave
